https://pantsbuild.org/ logo
Join Slack
Powered by
Hi all, I upgrading an existing pants repo that ha...
# general
h
Hi all, I upgrading an existing pants repo that has some python pex configuration that is dependent on binaries.path_by_id when creating cross-platform pex binaries. This section in the pants.ini config is removed post 1.6.x, so I was wondering if there is an alternative that I could use in order to build pex binaries on OS X to be run in Linux docker containers.
e
Can you be more explicit about your configuration? binaries.path_by_id has never been part of pants intrinsic handling of cross-platform pex building. It has been 
python_binary.platforms
at the target level and 
python-setup.platforms
at the repo level.
h
Hi John, We have the following for the python-setup section in pants.ini:
Copy code
[python-setup]
platforms: [ 'linux-x86_64', 'maxosx-10.6-intel']
And the following in pants.ini for path-by-id:
Copy code
[binaries]
path_by_id:   {
  ('darwin', '17'): ('mac', '10.12')}
The path_by_id was present since we get an untranslatable error while building the pex dist. For example:
Copy code
Exception message: Package SourcePackage(u'file:///Users/.../repo/.pants.d/python-setup/resolved_requirements/CPython-2.7.10/greenlet-0.4.12.tar.gz') is not translateable by ChainedTranslator(WheelTranslator, EggTranslator, SourceTranslator)
With PEX_VEBOSE on we get the following:
Copy code
ex: Target package WheelPackage('file:///private/var/folders/r0/1b1z1cwx7bg1gf9mt5m20mzhq807mv/T/tmp1PbRRO/greenlet-0.4.12-cp27-cp27m-macosx_10_12_intel.whl') is not compatible with [('cp27', 'cp27u', 'macosx_10_0_universal'), ('cp27', 'cp27dmu', 'macosx_10_2_universal'), ('cp27', 'cp27d', 'macosx_10_0_universal'), ('cp27', 'cp27d', 'macosx_10_6_intel'), ('cp27', 'cp27m', 'macosx_10_0_universal'), ('cp27', 'cp27dmu', 'macosx_10_6_universal'), ('cp27', 'none', 'macosx_10_5_universal'), ('cp27', 'cp27dm', 'macosx_10_2_universal'), ('cp27', 'cp27mu', 'macosx_10_0_universal'), ('cp27', 'cp27u', 'macosx_10_6_universal'), ('cp27', 'cp27dm', 'macosx_10_1_universal'), ('cp27', 'cp27dm', 'macosx_10_5_intel'), ('py2', 'none', 'any'), ('cp27', 'none', 'any'), ('cp27', 'cp27mu', 'macosx_10_5_universal'), ('cp27', 'cp27d', 'macosx_10_3_universal'), ('cp27', 'cp27dm', 'macosx_10_4_intel'), ('cp27', 'cp27d', 'macosx_10_1_universal'), ('cp27', 'cp27mu', 'macosx_10_4_intel'), ('cp27', 'cp27m', 'macosx_10_6_universal'), ('cp27', 'cp27u', 'macosx_10_5_universal'), ('cp27', 'none', 'macosx_10_4_universal'), ('cp27', 'cp27u', 'macosx_10_1_universal'), ('cp27', 'cp27mu', 'macosx_10_4_universal'), ('cp27', 'cp27dmu', 'macosx_10_4_intel'), ('cp27', 'cp27dm', 'macosx_10_3_universal'), ('cp27', 'none', 'macosx_10_4_intel'), ('cp27', 'cp27m', 'macosx_10_4_universal'), ('cp27', 'cp27m', 'macosx_10_2_universal'), ('cp27', 'cp27dm', 'macosx_10_0_universal'), ('py2', 'none', 'macosx_10_0_universal'), ('cp27', 'none', 'macosx_10_2_universal'), ('cp27', 'cp27m', 'macosx_10_5_intel'), ('cp27', 'cp27m', 'macosx_10_5_universal'), ('cp2', 'none', 'any'), ('cp27', 'cp27d', 'macosx_10_6_universal'), ('py2', 'none', 'macosx_10_3_universal'), ('cp27', 'cp27d', 'macosx_10_2_universal'), ('py2', 'none', 'macosx_10_2_universal'), ('py2', 'none', 'macosx_10_6_intel'), ('cp27', 'cp27dm', 'macosx_10_6_intel'), ('cp27', 'cp27m', 'macosx_10_4_intel'), ('py2', 'none', 'macosx_10_4_universal'), ('cp27', 'none', 'macosx_10_6_intel'), ('cp27', 'none', 'macosx_10_5_intel'), ('cp27', 'cp27dm', 'macosx_10_5_universal'), ('py2', 'none', 'macosx_10_5_intel'), ('py2', 'none', 'macosx_10_5_universal'), ('cp27', 'cp27d', 'macosx_10_4_intel'), ('cp27', 'cp27mu', 'macosx_10_6_universal'), ('cp27', 'cp27m', 'macosx_10_3_universal'), ('cp27', 'cp27dm', 'macosx_10_4_universal'), ('cp27', 'cp27u', 'macosx_10_4_universal'), ('cp27', 'cp27mu', 'macosx_10_3_universal'), ('py27', 'none', 'any'), ('py2', 'none', 'macosx_10_1_universal'), ('cp27', 'none', 'macosx_10_1_universal'), ('cp27', 'cp27d', 'macosx_10_5_universal'), ('cp27', 'cp27mu', 'macosx_10_6_intel'), ('cp27', 'none', 'macosx_10_6_universal'), ('cp27', 'cp27u', 'macosx_10_2_universal'), ('cp27', 'cp27d', 'macosx_10_4_universal'), ('cp27', 'cp27dmu', 'macosx_10_3_universal'), ('cp27', 'cp27d', 'macosx_10_5_intel'), ('cp27', 'cp27mu', 'macosx_10_2_universal'), ('cp27', 'cp27mu', 'macosx_10_5_intel'), ('cp27', 'cp27m', 'macosx_10_6_intel'), ('py2', 'none', 'macosx_10_4_intel'), ('cp27', 'cp27dmu', 'macosx_10_5_universal'), ('cp27', 'cp27dmu', 'macosx_10_5_intel'), ('cp27', 'cp27u', 'macosx_10_3_universal'), ('cp27', 'cp27dmu', 'macosx_10_1_universal'), ('cp27', 'cp27m', 'macosx_10_1_universal'), ('cp27', 'cp27u', 'macosx_10_6_intel'), ('cp27', 'cp27dmu', 'macosx_10_4_universal'), ('cp27', 'none', 'macosx_10_0_universal'), ('cp27', 'cp27mu', 'macosx_10_1_universal'), ('cp27', 'cp27dmu', 'macosx_10_6_intel'), ('cp27', 'none', 'macosx_10_3_universal'), ('cp27', 'cp27u', 'macosx_10_4_intel'), ('cp27', 'cp27u', 'macosx_10_5_intel'), ('cp27', 'cp27dm', 'macosx_10_6_universal'), ('py2', 'none', 'macosx_10_6_universal'), ('cp27', 'cp27dmu', 'macosx_10_0_universal')]
pex:   Packaging greenlet: 1584.5ms
e
Reading details but off the top looks like https://github.com/pantsbuild/pex/issues/523
In 
platforms: [ 'linux-x86_64', 'maxosx-10.6-intel']
why do you say 
-intel
? Have you tried 
platforms: [ 'linux-x86_64', 'maxosx-10.6-x86-64']
?
h
We haven’t tried that. I can’t say why we have that as the mac platform, but I can try with what you suggest
e
I'm at a loss for how path-by-id is having anything to do with this. Clearly you see a behavior change, but these things are not related at all in any way I know!
h
I don’t really understand it myself,but I am wondering if it somehow transforms the platform to target properly
I am targeting pantsbuild 1.9.0, in case that changes anythying
e
OK - ty. 
-intel
is a bogus platform only claimed by Apple-shipped python interpreters fwiw. If you build a Python interpreter yourself from python.org tarballs you'll get the binary identifying as 
-x86-64
(so ditto for pyenv, homebrew, etc - the only bad apple is apple)
h
now have the following in pants.ini:
Copy code
[python-setup]
platforms: [
    'linux-x86_64', 'macosx-10.6-x86-64'
  ]
I have removed the path_by_id section and I get the following error:
Copy code
No cached artifacts for 5 targets.
pex: Target package WheelPackage('file:///private/var/folders/r0/1b1z1cwx7bg1gf9mt5m20mzhq807mv/T/tmpP3Y8Az/MarkupSafe-1.0-cp27-cp27m-macosx_10_12_intel.whl') is not compatible with [('cp27', 'cp27u', 'linux_x86_64'), ('cp27', 'cp27m', 'manylinux1_x86_64'), ('py2', 'none', 'any'), ('cp27', 'cp27mu', 'manylinux1_x86_64'), ('cp27', 'none', 'any'), ('cp27', 'cp27dm', 'manylinux1_x86_64'), ('cp27', 'cp27u', 'manylinux1_x86_64'), ('cp27', 'cp27d', 'manylinux1_x86_64'), ('cp27', 'cp27mu', 'linux_x86_64'), ('cp27', 'none', 'linux_x86_64'), ('py2', 'none', 'linux_x86_64'), ('cp27', 'cp27dm', 'linux_x86_64'), ('cp27', 'cp27d', 'linux_x86_64'), ('cp27', 'cp27m', 'linux_x86_64'), ('py27', 'none', 'any'), ('cp27', 'cp27dmu', 'linux_x86_64'), ('cp27', 'none', 'manylinux1_x86_64'), ('cp2', 'none', 'any'), ('py2', 'none', 'manylinux1_x86_64'), ('cp27', 'cp27dmu', 'manylinux1_x86_64')]
pex:   Packaging MarkupSafe: 712.2ms

               Waiting for background workers to finish.
12:33:14 00:41   [complete]
               FAILURE
Exception caught: (<class 'pex.resolver.Untranslateable'>)

Exception message: Package SourcePackage(u'file:///Users/jay.lawrence/allclear/pantstheon/.pants.d/python-setup/resolved_requirements/CPython-2.7.10/MarkupSafe-1.0.tar.gz') is not translateable by ChainedTranslator(WheelTranslator, EggTranslator, SourceTranslator)
hmmm… intel is showing up, which may be related to the apple supplied python you mentioned?
I am currently using homebrew python fwiw at version 
2.7.15
e
And do you have a [python-repos] that adds in a pre-built-wheel repo for the OSX machine to fetch linux wheels and vice-versa?
h
Yes, that is
Copy code
[python-repos]
repos: +[
    '%(buildroot)s/3rdparty/python/wheelhouse/'
  ]
e
To be clear, when you say 2 platforms, pants will need to get the not-the-current-platform eggs/wheels from somewhere else - it has no way to build them.
And that wheelhouse has linux + mac?
IE: MarkupSafe is there? Can you list that dir?
h
Sure:
Copy code
$ ls -1 3rdparty/python/wheelhouse/*.whl
3rdparty/python/wheelhouse/MarkupSafe-1.0-cp27-cp27m-macosx_10_11_x86_64.whl
3rdparty/python/wheelhouse/MarkupSafe-1.0-cp27-cp27mu-linux_x86_64.whl
3rdparty/python/wheelhouse/PyYAML-3.12-cp27-cp27m-macosx_10_11_x86_64.whl
3rdparty/python/wheelhouse/PyYAML-3.12-cp27-cp27mu-linux_x86_64.whl
3rdparty/python/wheelhouse/gevent-1.2.2-cp27-cp27m-macosx_10_6_intel.whl
3rdparty/python/wheelhouse/gevent-1.2.2-cp27-cp27mu-linux_x86_64.whl
3rdparty/python/wheelhouse/greenlet-0.4.12-cp27-cp27m-macosx_10_11_x86_64.whl
3rdparty/python/wheelhouse/greenlet-0.4.12-cp27-cp27mu-linux_x86_64.whl
e
OK - can you please re-run with 
PEX_VERBOSE=9
? The likely candidates are: 1. The interpreter you think is being used is not - ie bad apple vs homebrew 2. cp27m vs cp27mu
The new more verbose output will make that clear, but not so much in slack, this might best be moved to a github issue soon.
h
Sounds good: the following is the result:
Copy code
2:39:30 00:02       [cache]
                   No cached artifacts for 5 targets.
                   Invalidated 5 targets.pex: Constructed RequestsContext context <pex.http.RequestsContext object at 0x117b09550>
pex: R: tags for Platform(platform='linux_x86_64', impl=None, version=None, abi=None) x PythonInterpreter('/System/Library/Frameworks/Python.framework/Versions/2.7/bin/python2.7', PythonIdentity('CPython', 2, 7, 10), {(u'wheel', u'0.29.0'): u'/Users/repo/.pants.d/python-setup/interpreters/CPython-2.7.10/wheel-0.29.0-py2.7.egg', (u'setuptools', u'30.0.0'): u'/Users/jay.lawrence/allclear/pantstheon/.pants.d/python-setup/interpreters/CPython-2.7.10/setuptools-30.0.0-py2.7.egg'}) -> [('cp27', 'cp27u', 'linux_x86_64'), ('cp27', 'cp27m', 'manylinux1_x86_64'), ('py2', 'none', 'any'), ('cp27', 'cp27mu', 'manylinux1_x86_64'), ('cp27', 'none', 'any'), ('cp27', 'cp27dm', 'manylinux1_x86_64'), ('cp27', 'cp27u', 'manylinux1_x86_64'), ('cp27', 'cp27d', 'manylinux1_x86_64'), ('cp27', 'cp27mu', 'linux_x86_64'), ('cp27', 'none', 'linux_x86_64'), ('py2', 'none', 'linux_x86_64'), ('cp27', 'cp27dm', 'linux_x86_64'), ('cp27', 'cp27d', 'linux_x86_64'), ('cp27', 'cp27m', 'linux_x86_64'), ('py27', 'none', 'any'), ('cp27', 'cp27dmu', 'linux_x86_64'), ('cp27', 'none', 'manylinux1_x86_64'), ('cp2', 'none', 'any'), ('py2', 'none', 'manylinux1_x86_64'), ('cp27', 'cp27dmu', 'manylinux1_x86_64')]
…
Copy code
ex: Constructed RequestsContext context <pex.http.RequestsContext object at 0x117b09910>
pex: crawling link i=0 link=Link(u'file:///Users/repo/.pants.d/python-setup/resolved_requirements/CPython-2.7.10') follow_links=False
pex: crawling link i=0 link=Link('<https://pypi.python.org/simple/requests/>') follow_links=False
pex: Constructed RequestsContext context <pex.http.RequestsContext object at 0x117b09950>
pex: crawling link i=0 link=Link('<https://pypi.python.org/simple/ansicolors/>') follow_links=False
pex: Constructed RequestsContext context <pex.http.RequestsContext object at 0x117b09dd0>
pex: crawling link i=0 link=Link('<https://pypi.python.org/simple/gevent/>') follow_links=False
pex: Constructed RequestsContext context <pex.http.RequestsContext object at 0x117b09e90>
pex: crawling link i=0 link=Link('<https://pypi.python.org/simple/greenlet/>') follow_links=False
pex: Constructed RequestsContext context <pex.http.RequestsContext object at 0x117b09950>
pex: crawling link i=0 link=Link('<https://pypi.python.org/simple/jinja2/>') follow_links=False
pex: Fetching file:///Users/repo/.pants.d/python-setup/resolved_requirements/CPython-2.7.10/Jinja2-2.9.6-py2.py3-none-any.whl: 4.2ms
pex:   Fetching file:///Users/repo/.pants.d/python-setup/resolved_requirements/CPython-2.7.10/Jinja2-2.9.6-py2.py3-none-any.whl: 3.1ms
pex: Translating /private/var/folders/r0/1b1z1cwx7bg1gf9mt5m20mzhq807mv/T/tmp6_R1He/Jinja2-2.9.6-py2.py3-none-any.whl into distribution: 2.8ms
pex: Fetching file:///Users/repo/.pants.d/python-setup/resolved_requirements/CPython-2.7.10/requests-2.13.0-py2.py3-none-any.whl: 4.1ms
pex:   Fetching file:///Users/repo/.pants.d/python-setup/resolved_requirements/CPython-2.7.10/requests-2.13.0-py2.py3-none-any.whl: 3.2ms
pex: Translating /private/var/folders/r0/1b1z1cwx7bg1gf9mt5m20mzhq807mv/T/tmpRpL4ab/requests-2.13.0-py2.py3-none-any.whl into distribution: 15.4ms
pex: Fetching file:///Users/repo/.pants.d/python-setup/resolved_requirements/CPython-2.7.10/gevent-1.2.2-cp27-cp27mu-manylinux1_x86_64.whl :: Fetching file:///Users/jay.lawrence/allclear/pantstheon/.pants.d/python-setup/resolved_requirements/CPython-2.7.10/gevent-1.2.2-cp27-cp27mu-manpex: Fetching file:///Users/jay.lawrence/allclear/pantstheon/.pants.d/python-setup/resolved_requirements/CPython-2.7.10/gevent-1.2.2-cp27-cp27mu-manylinux1_x86_64.whl: 45.4ms
pex:   Fetching file:///Users/repo/.pants.d/python-setup/resolved_requirements/CPython-2.7.10/gevent-1.2.2-cp27-cp27mu-manylinux1_x86_64.whl: 42.3ms
pex: Translating /private/var/folders/r0/1b1z1cwx7bg1gf9mt5m20mzhq807mv/T/tmpxPJIMA/gevent-1.2.2-cp27-cp27mu-manylinux1_x86_64.whl into distribution: 7.8ms
pex: Fetching file:///Users/repo/.pants.d/python-setup/resolved_requirements/CPython-2.7.10/greenlet-0.4.12-cp27-cp27m-manylinux1_x86_64.whl :: Fetching file:///Users/jay.lawrence/allclear/pantstheon/.pants.d/python-setup/resolved_requirements/CPython-2.7.10/greenlet-0.4.12-cp27-cp27mpex: Fetching file:///Users/jay.lawrence/allclear/pantstheon/.pants.d/python-setup/resolved_requirements/CPython-2.7.10/greenlet-0.4.12-cp27-cp27m-manylinux1_x86_64.whl: 3.3ms
pex:   Fetching file:///Users/repo/.pants.d/python-setup/resolved_requirements/CPython-2.7.10/greenlet-0.4.12-cp27-cp27m-manylinux1_x86_64.whl: 1.8ms
pex: Translating /private/var/folders/r0/1b1z1cwx7bg1gf9mt5m20mzhq807mv/T/tmpTPQ0ad/greenlet-0.4.12-cp27-cp27m-manylinux1_x86_64.whl into distribution: 2.1ms
pex: Fetching file:///Users/repo/.pants.d/python-setup/resolved_requirements/CPython-2.7.10/ansicolors-1.1.8-py2.py3-none-any.whl: 2.8ms
pex:   Fetching file:///Users/repo/.pants.d/python-setup/resolved_requirements/CPython-2.7.10/ansicolors-1.1.8-py2.py3-none-any.whl: 1.3ms
…
e
BAD APPLE: 
PythonInterpreter('/System/Library/Frameworks/Python.framework/Versions/2.7/bin/python2.7', PythonIdentity('CPython', 2, 7, 10)
h
Copy code
pex: Translating /private/var/folders/r0/1b1z1cwx7bg1gf9mt5m20mzhq807mv/T/tmp1bR2p1/ansicolors-1.1.8-py2.py3-none-any.whl into distribution: 4.2ms
pex: Constructed RequestsContext context <pex.http.RequestsContext object at 0x118796d10>
pex: crawling link i=0 link=Link('<https://pypi.python.org/simple/markupsafe/>') follow_links=False
pex: Constructed RequestsContext context <pex.http.RequestsContext object at 0x118796fd0>
pex: Fetching file:///Users/repo/.pants.d/python-setup/resolved_requirements/CPython-2.7.10/MarkupSafe-1.0.tar.gz: 7.7ms
pex:   Fetching file:///Users/repo/.pants.d/python-setup/resolved_requirements/CPython-2.7.10/MarkupSafe-1.0.tar.gz: 3.5ms
pex: Target package WheelPackage('file:///private/var/folders/r0/1b1z1cwx7bg1gf9mt5m20mzhq807mv/T/tmpCYFw21/MarkupSafe-1.0-cp27-cp27m-macosx_10_12_intel.whl') is not compatible with [('cp27', 'cp27u', 'linux_x86_64'), ('cp27', 'cp27m', 'manylinux1_x86_64'), ('py2', 'none', 'any'), ('cp27', 'cp27mu', 'manylinux1_x86_64'), ('cp27', 'none', 'any'), ('cp27', 'cp27dm', 'manylinux1_x86_64'), ('cp27', 'cp27u', 'manylinux1_x86_64'), ('cp27', 'cp27d', 'manylinux1_x86_64'), ('cp27', 'cp27mu', 'linux_x86_64'), ('cp27', 'none', 'linux_x86_64'), ('py2', 'none', 'linux_x86_64'), ('cp27', 'cp27dm', 'linux_x86_64'), ('cp27', 'cp27d', 'linux_x86_64'), ('cp27', 'cp27m', 'linux_x86_64'), ('py27', 'none', 'any'), ('cp27', 'cp27dmu', 'linux_x86_64'), ('cp27', 'none', 'manylinux1_x86_64'), ('cp2', 'none', 'any'), ('py2', 'none', 'manylinux1_x86_64'), ('cp27', 'cp27dmu', 'manylinux1_x86_64')]
pex: Translating /private/var/folders/r0/1b1z1cwx7bg1gf9mt5m20mzhq807mv/T/tmpD3oqmc/MarkupSafe-1.0.tar.gz into distribution: 1875.2ms
pex:   Packaging MarkupSafe: 1839.5ms
pex:     Installing /var/folders/r0/1b1z1cwx7bg1gf9mt5m20mzhq807mv/T/tmpZL0nbc: 1838.2ms

               Waiting for background workers to finish.
12:39:35 00:07   [complete]
               FAILURE
Exception caught: (<class 'pex.resolver.Untranslateable'>)

Exception message: Package SourcePackage(u'file:///Users/repo/.pants.d/python-setup/resolved_requirements/CPython-2.7.10/MarkupSafe-1.0.tar.gz') is not translateable by ChainedTranslator(WheelTranslator, EggTranslator, SourceTranslator)
e
h
Ah, sounds more like a problem on my end than a bug
e
Sortof. So - to be clear, conclusions: 1. Your fundamental issues is https://github.com/pantsbuild/pex/issues/523 2. path-by-id never should have worked - unrelated 3. You can only workaround by excluding Apple interpreters
h
ah, got it. Not sure how to do (3), but will read though the issue and work on resolving the apple interpreter. There seems to be no end to trouble from the Apple interpreter…
Thanks for all the help!
e
For 3, there are a few techniques that work with pants today: 1. use 
python-setup.interpreter-constraints
to version exclude known-bad OSX interpreters. This might not work for you though if you have a linux interpreter of the same version you do want to use (say 2.7.10 as in this case) 2. use 
python-setup.interpreter_search_paths
to whitelist known good paths on osx and linux. This might not work for you since the linux good interpreters tend to be in /usr/bin which is where bad interpreters shipped by Apple tend to be
h
Awesome! We should be fine with the 
interpreter-constraints
2 Views
Open in Slack
PreviousNext
Powered by