https://pantsbuild.org/ logo
Join Slack
Powered by
Hi all, I am having some problems with wheels when...
# general
p
Hi all, I am having some problems with wheels when building PEX for a wheel I have built. I get the following exception:
Copy code
Exception message: Package SourcePackage('file:///Users/jay.lawrence/code/pantstheon/.pants.d/python-setup/resolved_requirements/CPython-3.7.7/PyYAML-5.3.1.tar.gz') is not translateable by ChainedTranslator(WheelTranslator, EggTranslator, SourceTranslator)
I will add the results of PEX_VERBOSE=9 as follows:
message has been deleted
Copy code
ex: R: tags for Platform(platform='macosx_10_15_x86_64', impl='cp', version='37', abi='cp37m') x PythonInterpreter('/usr/local/Cellar/python/3.7.7/Frameworks/Python.framework/Versions/3.7/bin/python3.7', PythonIdentity('cp', 'cp37m', '37', 3, 7, 7)) -> [('cp37', 'cp37m', 'macosx_10_15_x86_64'), ('cp37', 'cp37m', 'macosx_10_15_intel'), ('cp37', 'cp37m', 'macosx_10_15_fat64'), ('cp37', 'cp37m', 'macosx_10_15_fat32'), ('cp37', 'cp37m', 'macosx_10_15_universal'), ... ]
pex: Constructed RequestsContext context <pex.http.RequestsContext object at 0x10fcc5d90>
pex: crawling link i=0 link=Link('file:///Users/jay.lawrence/code/pantstheon/.pants.d/python-setup/resolved_requirements/CPython-3.7.7') follow_links=False
pex: crawling link i=0 link=Link('file:///Users/<user>/code/<repo>/3rdparty/python/wheelhouse') follow_links=False
pex: crawling link i=1 link=Link('<https://pypi.org/simple/pyyaml/>') follow_links=False
pex: Fetching file:///Users/<user>/code/<repo>/.pants.d/python-setup/resolved_requirements/CPython-3.7.7/PyYAML-5.3.1-cp37-cp37m-macosx_10_15_x86_64.whl :: Fetching file:///Users/jay.lawrence/codepex: Fetching file:///Users/<user>/code/<repo>/.pants.d/python-setup/resolved_requirements/CPython-3.7.7/PyYAML-5.3.1-cp37-cp37m-macosx_10_15_x86_64.whl: 1.5ms

pex:   Fetching file:///Users/<user>/code/<repo>/python-setup/resolved_requirements/CPython-3.7.7/PyYAML-5.3.1-cp37-cp37m-macosx_10_15_x86_64.whl: 0.6ms
pex: Translating /private/var/folders/pf/dcm97_h14qj0ylrsp91gx0k4ssnkj4/T/tmp5syvia8n/PyYAML-5.3.1-cp37-cp37m-macosx_10_15_x86_64.whl into distribution: 4.2ms
pex: R: tags for Platform(platform='linux_x86_64', impl=None, version=None, abi=None) x PythonInterpreter('/usr/local/Cellar/python/3.7.7/Frameworks/Python.framework/Versions/3.7/bin/python3.7', PythonIdentity('cp', 'cp37m', '37', 3, 7, 7)) -> [('cp37', 'abi3', 'manylinux1_x86_64'), ('cp33', 'abi3', 'manylinux1_x86_64'), ('cp37', 'cp37u', 'linux_x86_64'), ('cp37', 'cp37dm', 'manylinux1_x86_64'), ('py3', 'none', 'any'), ('cp32', 'abi3', 'manylinux1_x86_64'), ('cp37', 'cp37dm', 'linux_x86_64'), ('py36', 'none', 'any'), ('py3', 'none', 'manylinux1_x86_64'), ('cp37', 'abi3', 'linux_x86_64'), ('cp33', 'abi3', 'linux_x86_64'), ('cp3', 'none', 'any'), ('cp34', 'abi3', 'linux_x86_64'), ('cp37', 'cp37m', 'linux_x86_64'), ('py3', 'none', 'linux_x86_64'), ('cp35', 'abi3', 'linux_x86_64'), ('py30', 'none', 'any'), ('cp37', 'none', 'manylinux1_x86_64'), ('cp32', 'abi3', 'linux_x86_64'), ('py33', 'none', 'any'), ('cp37', 'cp37mu', 'manylinux1_x86_64'), ('py35', 'none', 'any'), ('cp37', 'cp37mu', 'linux_x86_64'), ('cp37', 'none', 'any'), ('cp37', 'cp37dmu', 'manylinux1_x86_64'), ('cp37', 'cp37u', 'manylinux1_x86_64'), ('py32', 'none', 'any'), ('py37', 'none', 'any'), ('cp36', 'abi3', 'manylinux1_x86_64'), ('py34', 'none', 'any'), ('cp37', 'none', 'linux_x86_64'), ('py31', 'none', 'any'), ('cp37', 'cp37d', 'manylinux1_x86_64'), ('cp34', 'abi3', 'manylinux1_x86_64'), ('cp35', 'abi3', 'manylinux1_x86_64'), ('cp37', 'cp37d', 'linux_x86_64'), ('cp37', 'cp37m', 'manylinux1_x86_64'), ('cp37', 'cp37dmu', 'linux_x86_64'), ('cp36', 'abi3', 'linux_x86_64')]
pex: Constructed RequestsContext context <pex.http.RequestsContext object at 0x1b912c85b90>

pex: Translating /private/var/folders/pf/dcm97_h14qj0ylrsp91gx0k4ssnkj4/T/tmpscet6umc/PyYAML-5.3.1.tar.gz into distribution :: Packaging PyYAML :: Installing /var/folders/pf/dcm97_h14qj0ylrsp91gx0k4ssnkj4/Tpex: Translating /private/var/folders/pf/dcm97_h14qj0ylrsp91gx0k4ssnkj4/T/tmpscet6umc/PyYAML-5.3.1.tar.gz into distribution :: Packaging PyYAML :: Installing /var/folders/pf/dcm97_h14qj0ylrsp91gx0k4ssnkj4/Tpex: Target package WheelPackage('file:///private/var/folders/pf/dcm97_h14qj0ylrsp91gx0k4ssnkj4/T/tmpugtxlkiv/PyYAML-5.3.1-cp37-cp37m-macosx_10_15_x86_64.whl') is not compatible with [('cp37', 'abi3', 'manylinux1_x86_64'), ('cp33', 'abi3', 'manylinux1_x86_64'), ('cp37', 'cp37u', 'linux_x86_64'), ('cp37', 'cp37dm', 'manylinux1_x86_64'), ('py3', 'none', 'any'), ('cp32', 'abi3', 'manylinux1_x86_64'), ('cp37', 'cp37dm', 'linux_x86_64'), ('py36', 'none', 'any'), ('py3', 'none', 'manylinux1_x86_64'), ('cp37', 'abi3', 'linux_x86_64'), ('cp33', 'abi3', 'linux_x86_64'), ('cp3', 'none', 'any'), ('cp34', 'abi3', 'linux_x86_64'), ('cp37', 'cp37m', 'linux_x86_64'), ('py3', 'none', 'linux_x86_64'), ('cp35', 'abi3', 'linux_x86_64'), ('py30', 'none', 'any'), ('cp37', 'none', 'manylinux1_x86_64'), ('cp32', 'abi3', 'linux_x86_64'), ('py33', 'none', 'any'), ('cp37', 'cp37mu', 'manylinux1_x86_64'), ('py35', 'none', 'any'), ('cp37', 'cp37mu', 'linux_x86_64'), ('cp37', 'none', 'any'), ('cp37', 'cp37dmu', 'manylinux1_x86_64'), ('cp37', 'cp37u', 'manylinux1_x86_64'), ('py32', 'none', 'any'), ('py37', 'none', 'any'), ('cp36', 'abi3', 'manylinux1_x86_64'), ('py34', 'none', 'any'), ('cp37', 'none', 'linux_x86_64'), ('py31', 'none', 'any'), ('cp37', 'cp37d', 'manylinux1_x86_64'), ('cp34', 'abi3', 'manylinux1_x86_64'), ('cp35', 'abi3', 'manylinux1_x86_64'), ('cp37', 'cp37d', 'linux_x86_64'), ('cp37', 'cp37m', 'manylinux1_x86_64'), ('cp37', 'cp37dmu', 'linux_x86_64'), ('cp36', 'abi3', 'linux_x86_64')]
pex: Translating /private/var/folders/pf/dcm97_h14qj0ylrsp91gx0k4ssnkj4/T/tmpscet6umc/PyYAML-5.3.1.tar.gz into distribution: 5164.6ms
pex:   Packaging PyYAML: 4785.1ms
pex:     Installing /var/folders/pf/dcm97_h14qj0ylrsp91gx0k4ssnkj4/T/tmpv1e9o0fp: 4783.7ms
pex:       Isolating pex in Chroot(/var/folders/pf/dcm97_h14qj0ylrsp91gx0k4ssnkj4/T/tmp6md8nxi3 {fs:}): 178.5ms
I have the following in my 
pants.ini
(at version 1.25 and will convert to toml soon)
Copy code
[python-setup]
interpreter_constraints: ['CPython==3.7.*']
interpreter_search_paths: ['/usr/local/bin']
platforms: ['current', 'linux-x86_64', 'macosx-10.9-x86_64']
I have built the linux wheel through the manylinux docker image, and used pip3 on the mac laptop to create the PyYAML mac wheel
I have the following wheels:
Copy code
ls -1 3rdparty/python/wheelhouse/
PyYAML-5.3.1-cp37-cp37m-linux_x86_64.whl
PyYAML-5.3.1-cp37-cp37m-macosx_10_15_x86_64.whl
PyYAML-5.3.1-cp37-cp37m-manylinux1_x86_64.whl
p
p
Thanks @polite-garden-50641 I will take a look
Ran the with 
~/bin/pex -v pyyaml==5.3.1 2
and seemed alright.
pex_output_for_PyYAML_5_3_1.txt
Copy code
./pants -q options --scope=python-setup --name=interpreter-constraints
python-setup.interpreter_constraints = ['CPython==3.7.*'] (from CONFIG in pants.ini)
I believe my interpreter is correct and it is using the Python 3.7 that I expect
If I remove all platforms but 
current
, then I will be able to create the pex, but I am not able to provide any other platforms, which is the problem I do not understand right now
p
@enough-analyst-54434
e
Thanks Asher. This looks like an issue with old (pre 2.0) Pex. I'm in transit at the moment but will take a look a little later this afternoon.
p
Thanks John 👍. I tried running on pants v1.25.0 and got the same result, however, I may have needed to remove caches. Would that still involve a pre 2.0 pex problem?
e
p
Thanks! I will see if that fixes the issue
I will check if that release also requires TOML
p
@hundreds-father-404 are ini file supported in recent pants versions ?
p
Maybe it is deprecated? I checked in v1.26.0 and ini is legacy. I was able to get past that issue! Thanks @enough-analyst-54434! I just have to figure out the proper platform strings for linux/manylinux since the following cause an error: 
['linux-x86_64', 'manylinux1-x86_64', 'macosx-10.9-x86_64']
Copy code
ERROR: Not a valid platform specifier: linux-x86_64

Platform strings must be in one of two forms:
1. Canonical: <platform>-<python impl abbr>-<python version>-<abi>
2. Abbreviated: <platform>-<python impl abbr>-<python version>-<abbr abi>

Given a canonical platform string for CPython 3.7.5 running on 64 bit linux of:
  linux-x86_64-cp-37-cp37m

Where the fields above are:
+ <platform>: linux-x86_64
+ <python impl abbr>: cp
+ <python version>: 37
+ <abi>: cp37m

The abbreviated platform string is:
  linux-x86_64-cp-37-m
h
Hi Jay, sorry you ran into this issue! I agree with John’s recommendation to try out Pants 1.26.x. Yes, @polite-garden-50641, we still support pants.ini. 1.26.x is the first to support pants.toml. We will remove support for pants.ini in 1.31, though (https://github.com/pantsbuild/pants/pull/9879) Jay, we recommend upgrading to pants.toml because it’s a much nicer experience, but you probably want to first get the Pex issue resolved. When upgrading, refer to https://groups.google.com/forum/#!searchin/pants-devel/pants.toml%7Csort:date/pants-devel/N1H03oJONco/fJplpGixAwAJ. It’s important to update the 
./pants
script by redownloading it (instructions in the email)
🎉 1
Awesome! So, the issue here is that the platform string isn’t specific enough. It’s ambiguous what Python version 
linux-x86_64
refers to. With Python native wheels, there is a difference between something built for Python 3.5 vs. 3.6 etc. It wasn’t safe that Pex 1.x allowed you to use this loose string. Instead of 
linux-x86_64
, try 
linux-x86_64-cp-37-m
p
Thanks @hundreds-father-404! I will give that a shot. I plan to upgrade to TOML after I get past this part, and I agree it looks much better 😄
SUCCESS! Thanks for the help all!
❤️ 1
h
Awesome! Also, have you seen the new docs about how to use the V2 implementation of Python? https://pants.readme.io/docs/welcome-to-pants Still a couple issues we’re prioritizing fixing, but overall V2 brings a couple big improvements, including: * Support for Python lockfiles * New linters: Black, Bandit, Flake8, Pylint, and Docformatter * Saner caching * Better concurrency. For example, you can run all of your tests in parallel (limited by your cores) * Less verbose/clunky UI thanks to a new dynamic UI * Precise file arguments, meaning that if you run 
./pants test project/demo.py
, Pants will only run tests for that specific file
p
I was generally aware that v2 was being worked on, but not on the specifics. Looks great!
I will need to look at the details. I see Python support is there now, and look forward to seeing Scala 😄 By the way, is Rust going to be supposed outside of the engine?
h
Let us know if you end up trying it out and any feedback or issues you run into! We recommend using 1.28.0 or 1.29.0.dev1 for the recent improvements. You can also run in “mixed mode” for now, where you still use V1 for most things, but V2 for some things. Add 
pants.backend.python
to the 
backend_packages2
option, along with any linters like 
pants.backend.python.lint.black
, then use 
--no-v1
or 
--no-v2
to turn off what you don’t want. In our internal repo, we have a 
./pants
script that uses V1, and a 
./v2
script that calls 
./pants
but with the option 
--v2
hardcoded.
By the way, is Rust going to be supposed outside of the engine?
Good question! @hundreds-breakfast-49010 and I think @aloof-angle-91616 have been looking at adding it and have some works in progress. By the way, if you have a moment, it would be super helpful to fill out this survey https://t.co/za6xeOxZ6S (if you haven’t yet) to help us plan the roadmap.
p
Sounds great! I will go ahead and do the roadmap survey 😄
❤️ 1
7 Views
Open in Slack
PreviousNext
Powered by