https://pantsbuild.org/ logo
Powered by
h

high-night-61226

09/25/2018, 4:53 PM
Hi all, I upgrading an existing pants repo that has some python pex configuration that is dependent on binaries.path_by_id when creating cross-platform pex binaries. This section in the pants.ini config is removed post 1.6.x, so I was wondering if there is an alternative that I could use in order to build pex binaries on OS X to be run in Linux docker containers.
e

enough-analyst-54434

09/25/2018, 5:11 PM
Can you be more explicit about your configuration? binaries.path_by_id has never been part of pants intrinsic handling of cross-platform pex building. It has been 
python_binary.platforms
at the target level and 
python-setup.platforms
at the repo level.
h

high-night-61226

09/25/2018, 5:19 PM
Hi John, We have the following for the python-setup section in pants.ini:
Copy code
[python-setup]
platforms: [ 'linux-x86_64', 'maxosx-10.6-intel']
And the following in pants.ini for path-by-id:
Copy code
[binaries]
path_by_id:   {
  ('darwin', '17'): ('mac', '10.12')}
The path_by_id was present since we get an untranslatable error while building the pex dist. For example:
Copy code
Exception message: Package SourcePackage(u'file:///Users/.../repo/.pants.d/python-setup/resolved_requirements/CPython-2.7.10/greenlet-0.4.12.tar.gz') is not translateable by ChainedTranslator(WheelTranslator, EggTranslator, SourceTranslator)
With PEX_VEBOSE on we get the following:
Copy code
ex: Target package WheelPackage('file:///private/var/folders/r0/1b1z1cwx7bg1gf9mt5m20mzhq807mv/T/tmp1PbRRO/greenlet-0.4.12-cp27-cp27m-macosx_10_12_intel.whl') is not compatible with [('cp27', 'cp27u', 'macosx_10_0_universal'), ('cp27', 'cp27dmu', 'macosx_10_2_universal'), ('cp27', 'cp27d', 'macosx_10_0_universal'), ('cp27', 'cp27d', 'macosx_10_6_intel'), ('cp27', 'cp27m', 'macosx_10_0_universal'), ('cp27', 'cp27dmu', 'macosx_10_6_universal'), ('cp27', 'none', 'macosx_10_5_universal'), ('cp27', 'cp27dm', 'macosx_10_2_universal'), ('cp27', 'cp27mu', 'macosx_10_0_universal'), ('cp27', 'cp27u', 'macosx_10_6_universal'), ('cp27', 'cp27dm', 'macosx_10_1_universal'), ('cp27', 'cp27dm', 'macosx_10_5_intel'), ('py2', 'none', 'any'), ('cp27', 'none', 'any'), ('cp27', 'cp27mu', 'macosx_10_5_universal'), ('cp27', 'cp27d', 'macosx_10_3_universal'), ('cp27', 'cp27dm', 'macosx_10_4_intel'), ('cp27', 'cp27d', 'macosx_10_1_universal'), ('cp27', 'cp27mu', 'macosx_10_4_intel'), ('cp27', 'cp27m', 'macosx_10_6_universal'), ('cp27', 'cp27u', 'macosx_10_5_universal'), ('cp27', 'none', 'macosx_10_4_universal'), ('cp27', 'cp27u', 'macosx_10_1_universal'), ('cp27', 'cp27mu', 'macosx_10_4_universal'), ('cp27', 'cp27dmu', 'macosx_10_4_intel'), ('cp27', 'cp27dm', 'macosx_10_3_universal'), ('cp27', 'none', 'macosx_10_4_intel'), ('cp27', 'cp27m', 'macosx_10_4_universal'), ('cp27', 'cp27m', 'macosx_10_2_universal'), ('cp27', 'cp27dm', 'macosx_10_0_universal'), ('py2', 'none', 'macosx_10_0_universal'), ('cp27', 'none', 'macosx_10_2_universal'), ('cp27', 'cp27m', 'macosx_10_5_intel'), ('cp27', 'cp27m', 'macosx_10_5_universal'), ('cp2', 'none', 'any'), ('cp27', 'cp27d', 'macosx_10_6_universal'), ('py2', 'none', 'macosx_10_3_universal'), ('cp27', 'cp27d', 'macosx_10_2_universal'), ('py2', 'none', 'macosx_10_2_universal'), ('py2', 'none', 'macosx_10_6_intel'), ('cp27', 'cp27dm', 'macosx_10_6_intel'), ('cp27', 'cp27m', 'macosx_10_4_intel'), ('py2', 'none', 'macosx_10_4_universal'), ('cp27', 'none', 'macosx_10_6_intel'), ('cp27', 'none', 'macosx_10_5_intel'), ('cp27', 'cp27dm', 'macosx_10_5_universal'), ('py2', 'none', 'macosx_10_5_intel'), ('py2', 'none', 'macosx_10_5_universal'), ('cp27', 'cp27d', 'macosx_10_4_intel'), ('cp27', 'cp27mu', 'macosx_10_6_universal'), ('cp27', 'cp27m', 'macosx_10_3_universal'), ('cp27', 'cp27dm', 'macosx_10_4_universal'), ('cp27', 'cp27u', 'macosx_10_4_universal'), ('cp27', 'cp27mu', 'macosx_10_3_universal'), ('py27', 'none', 'any'), ('py2', 'none', 'macosx_10_1_universal'), ('cp27', 'none', 'macosx_10_1_universal'), ('cp27', 'cp27d', 'macosx_10_5_universal'), ('cp27', 'cp27mu', 'macosx_10_6_intel'), ('cp27', 'none', 'macosx_10_6_universal'), ('cp27', 'cp27u', 'macosx_10_2_universal'), ('cp27', 'cp27d', 'macosx_10_4_universal'), ('cp27', 'cp27dmu', 'macosx_10_3_universal'), ('cp27', 'cp27d', 'macosx_10_5_intel'), ('cp27', 'cp27mu', 'macosx_10_2_universal'), ('cp27', 'cp27mu', 'macosx_10_5_intel'), ('cp27', 'cp27m', 'macosx_10_6_intel'), ('py2', 'none', 'macosx_10_4_intel'), ('cp27', 'cp27dmu', 'macosx_10_5_universal'), ('cp27', 'cp27dmu', 'macosx_10_5_intel'), ('cp27', 'cp27u', 'macosx_10_3_universal'), ('cp27', 'cp27dmu', 'macosx_10_1_universal'), ('cp27', 'cp27m', 'macosx_10_1_universal'), ('cp27', 'cp27u', 'macosx_10_6_intel'), ('cp27', 'cp27dmu', 'macosx_10_4_universal'), ('cp27', 'none', 'macosx_10_0_universal'), ('cp27', 'cp27mu', 'macosx_10_1_universal'), ('cp27', 'cp27dmu', 'macosx_10_6_intel'), ('cp27', 'none', 'macosx_10_3_universal'), ('cp27', 'cp27u', 'macosx_10_4_intel'), ('cp27', 'cp27u', 'macosx_10_5_intel'), ('cp27', 'cp27dm', 'macosx_10_6_universal'), ('py2', 'none', 'macosx_10_6_universal'), ('cp27', 'cp27dmu', 'macosx_10_0_universal')]
pex:   Packaging greenlet: 1584.5ms
e

enough-analyst-54434

09/25/2018, 5:20 PM
Reading details but off the top looks like https://github.com/pantsbuild/pex/issues/523
In 
platforms: [ 'linux-x86_64', 'maxosx-10.6-intel']
why do you say 
-intel
? Have you tried 
platforms: [ 'linux-x86_64', 'maxosx-10.6-x86-64']
?
h

high-night-61226

09/25/2018, 5:22 PM
We haven’t tried that. I can’t say why we have that as the mac platform, but I can try with what you suggest
e

enough-analyst-54434

09/25/2018, 5:22 PM
I'm at a loss for how path-by-id is having anything to do with this. Clearly you see a behavior change, but these things are not related at all in any way I know!
h

high-night-61226

09/25/2018, 5:23 PM
I don’t really understand it myself,but I am wondering if it somehow transforms the platform to target properly
I am targeting pantsbuild 1.9.0, in case that changes anythying
e

enough-analyst-54434

09/25/2018, 5:24 PM
OK - ty. 
-intel
is a bogus platform only claimed by Apple-shipped python interpreters fwiw. If you build a Python interpreter yourself from python.org tarballs you'll get the binary identifying as 
-x86-64
(so ditto for pyenv, homebrew, etc - the only bad apple is apple)
h

high-night-61226

09/25/2018, 5:33 PM
now have the following in pants.ini:
Copy code
[python-setup]
platforms: [
    'linux-x86_64', 'macosx-10.6-x86-64'
  ]
I have removed the path_by_id section and I get the following error:
Copy code
No cached artifacts for 5 targets.
pex: Target package WheelPackage('file:///private/var/folders/r0/1b1z1cwx7bg1gf9mt5m20mzhq807mv/T/tmpP3Y8Az/MarkupSafe-1.0-cp27-cp27m-macosx_10_12_intel.whl') is not compatible with [('cp27', 'cp27u', 'linux_x86_64'), ('cp27', 'cp27m', 'manylinux1_x86_64'), ('py2', 'none', 'any'), ('cp27', 'cp27mu', 'manylinux1_x86_64'), ('cp27', 'none', 'any'), ('cp27', 'cp27dm', 'manylinux1_x86_64'), ('cp27', 'cp27u', 'manylinux1_x86_64'), ('cp27', 'cp27d', 'manylinux1_x86_64'), ('cp27', 'cp27mu', 'linux_x86_64'), ('cp27', 'none', 'linux_x86_64'), ('py2', 'none', 'linux_x86_64'), ('cp27', 'cp27dm', 'linux_x86_64'), ('cp27', 'cp27d', 'linux_x86_64'), ('cp27', 'cp27m', 'linux_x86_64'), ('py27', 'none', 'any'), ('cp27', 'cp27dmu', 'linux_x86_64'), ('cp27', 'none', 'manylinux1_x86_64'), ('cp2', 'none', 'any'), ('py2', 'none', 'manylinux1_x86_64'), ('cp27', 'cp27dmu', 'manylinux1_x86_64')]
pex:   Packaging MarkupSafe: 712.2ms

               Waiting for background workers to finish.
12:33:14 00:41   [complete]
               FAILURE
Exception caught: (<class 'pex.resolver.Untranslateable'>)

Exception message: Package SourcePackage(u'file:///Users/jay.lawrence/allclear/pantstheon/.pants.d/python-setup/resolved_requirements/CPython-2.7.10/MarkupSafe-1.0.tar.gz') is not translateable by ChainedTranslator(WheelTranslator, EggTranslator, SourceTranslator)
hmmm… intel is showing up, which may be related to the apple supplied python you mentioned?
I am currently using homebrew python fwiw at version 
2.7.15
e

enough-analyst-54434

09/25/2018, 5:35 PM
And do you have a [python-repos] that adds in a pre-built-wheel repo for the OSX machine to fetch linux wheels and vice-versa?
h

high-night-61226

09/25/2018, 5:35 PM
Yes, that is
Copy code
[python-repos]
repos: +[
    '%(buildroot)s/3rdparty/python/wheelhouse/'
  ]
e

enough-analyst-54434

09/25/2018, 5:35 PM
To be clear, when you say 2 platforms, pants will need to get the not-the-current-platform eggs/wheels from somewhere else - it has no way to build them.
And that wheelhouse has linux + mac?
IE: MarkupSafe is there? Can you list that dir?
h

high-night-61226

09/25/2018, 5:37 PM
Sure:
Copy code
$ ls -1 3rdparty/python/wheelhouse/*.whl
3rdparty/python/wheelhouse/MarkupSafe-1.0-cp27-cp27m-macosx_10_11_x86_64.whl
3rdparty/python/wheelhouse/MarkupSafe-1.0-cp27-cp27mu-linux_x86_64.whl
3rdparty/python/wheelhouse/PyYAML-3.12-cp27-cp27m-macosx_10_11_x86_64.whl
3rdparty/python/wheelhouse/PyYAML-3.12-cp27-cp27mu-linux_x86_64.whl
3rdparty/python/wheelhouse/gevent-1.2.2-cp27-cp27m-macosx_10_6_intel.whl
3rdparty/python/wheelhouse/gevent-1.2.2-cp27-cp27mu-linux_x86_64.whl
3rdparty/python/wheelhouse/greenlet-0.4.12-cp27-cp27m-macosx_10_11_x86_64.whl
3rdparty/python/wheelhouse/greenlet-0.4.12-cp27-cp27mu-linux_x86_64.whl
e

enough-analyst-54434

09/25/2018, 5:38 PM
OK - can you please re-run with 
PEX_VERBOSE=9
? The likely candidates are: 1. The interpreter you think is being used is not - ie bad apple vs homebrew 2. cp27m vs cp27mu
The new more verbose output will make that clear, but not so much in slack, this might best be moved to a github issue soon.
h

high-night-61226

09/25/2018, 5:41 PM
Sounds good: the following is the result:
Copy code
2:39:30 00:02       [cache]
                   No cached artifacts for 5 targets.
                   Invalidated 5 targets.pex: Constructed RequestsContext context <pex.http.RequestsContext object at 0x117b09550>
pex: R: tags for Platform(platform='linux_x86_64', impl=None, version=None, abi=None) x PythonInterpreter('/System/Library/Frameworks/Python.framework/Versions/2.7/bin/python2.7', PythonIdentity('CPython', 2, 7, 10), {(u'wheel', u'0.29.0'): u'/Users/repo/.pants.d/python-setup/interpreters/CPython-2.7.10/wheel-0.29.0-py2.7.egg', (u'setuptools', u'30.0.0'): u'/Users/jay.lawrence/allclear/pantstheon/.pants.d/python-setup/interpreters/CPython-2.7.10/setuptools-30.0.0-py2.7.egg'}) -> [('cp27', 'cp27u', 'linux_x86_64'), ('cp27', 'cp27m', 'manylinux1_x86_64'), ('py2', 'none', 'any'), ('cp27', 'cp27mu', 'manylinux1_x86_64'), ('cp27', 'none', 'any'), ('cp27', 'cp27dm', 'manylinux1_x86_64'), ('cp27', 'cp27u', 'manylinux1_x86_64'), ('cp27', 'cp27d', 'manylinux1_x86_64'), ('cp27', 'cp27mu', 'linux_x86_64'), ('cp27', 'none', 'linux_x86_64'), ('py2', 'none', 'linux_x86_64'), ('cp27', 'cp27dm', 'linux_x86_64'), ('cp27', 'cp27d', 'linux_x86_64'), ('cp27', 'cp27m', 'linux_x86_64'), ('py27', 'none', 'any'), ('cp27', 'cp27dmu', 'linux_x86_64'), ('cp27', 'none', 'manylinux1_x86_64'), ('cp2', 'none', 'any'), ('py2', 'none', 'manylinux1_x86_64'), ('cp27', 'cp27dmu', 'manylinux1_x86_64')]
…
Copy code
ex: Constructed RequestsContext context <pex.http.RequestsContext object at 0x117b09910>
pex: crawling link i=0 link=Link(u'file:///Users/repo/.pants.d/python-setup/resolved_requirements/CPython-2.7.10') follow_links=False
pex: crawling link i=0 link=Link('<https://pypi.python.org/simple/requests/>') follow_links=False
pex: Constructed RequestsContext context <pex.http.RequestsContext object at 0x117b09950>
pex: crawling link i=0 link=Link('<https://pypi.python.org/simple/ansicolors/>') follow_links=False
pex: Constructed RequestsContext context <pex.http.RequestsContext object at 0x117b09dd0>
pex: crawling link i=0 link=Link('<https://pypi.python.org/simple/gevent/>') follow_links=False
pex: Constructed RequestsContext context <pex.http.RequestsContext object at 0x117b09e90>
pex: crawling link i=0 link=Link('<https://pypi.python.org/simple/greenlet/>') follow_links=False
pex: Constructed RequestsContext context <pex.http.RequestsContext object at 0x117b09950>
pex: crawling link i=0 link=Link('<https://pypi.python.org/simple/jinja2/>') follow_links=False
pex: Fetching file:///Users/repo/.pants.d/python-setup/resolved_requirements/CPython-2.7.10/Jinja2-2.9.6-py2.py3-none-any.whl: 4.2ms
pex:   Fetching file:///Users/repo/.pants.d/python-setup/resolved_requirements/CPython-2.7.10/Jinja2-2.9.6-py2.py3-none-any.whl: 3.1ms
pex: Translating /private/var/folders/r0/1b1z1cwx7bg1gf9mt5m20mzhq807mv/T/tmp6_R1He/Jinja2-2.9.6-py2.py3-none-any.whl into distribution: 2.8ms
pex: Fetching file:///Users/repo/.pants.d/python-setup/resolved_requirements/CPython-2.7.10/requests-2.13.0-py2.py3-none-any.whl: 4.1ms
pex:   Fetching file:///Users/repo/.pants.d/python-setup/resolved_requirements/CPython-2.7.10/requests-2.13.0-py2.py3-none-any.whl: 3.2ms
pex: Translating /private/var/folders/r0/1b1z1cwx7bg1gf9mt5m20mzhq807mv/T/tmpRpL4ab/requests-2.13.0-py2.py3-none-any.whl into distribution: 15.4ms
pex: Fetching file:///Users/repo/.pants.d/python-setup/resolved_requirements/CPython-2.7.10/gevent-1.2.2-cp27-cp27mu-manylinux1_x86_64.whl :: Fetching file:///Users/jay.lawrence/allclear/pantstheon/.pants.d/python-setup/resolved_requirements/CPython-2.7.10/gevent-1.2.2-cp27-cp27mu-manpex: Fetching file:///Users/jay.lawrence/allclear/pantstheon/.pants.d/python-setup/resolved_requirements/CPython-2.7.10/gevent-1.2.2-cp27-cp27mu-manylinux1_x86_64.whl: 45.4ms
pex:   Fetching file:///Users/repo/.pants.d/python-setup/resolved_requirements/CPython-2.7.10/gevent-1.2.2-cp27-cp27mu-manylinux1_x86_64.whl: 42.3ms
pex: Translating /private/var/folders/r0/1b1z1cwx7bg1gf9mt5m20mzhq807mv/T/tmpxPJIMA/gevent-1.2.2-cp27-cp27mu-manylinux1_x86_64.whl into distribution: 7.8ms
pex: Fetching file:///Users/repo/.pants.d/python-setup/resolved_requirements/CPython-2.7.10/greenlet-0.4.12-cp27-cp27m-manylinux1_x86_64.whl :: Fetching file:///Users/jay.lawrence/allclear/pantstheon/.pants.d/python-setup/resolved_requirements/CPython-2.7.10/greenlet-0.4.12-cp27-cp27mpex: Fetching file:///Users/jay.lawrence/allclear/pantstheon/.pants.d/python-setup/resolved_requirements/CPython-2.7.10/greenlet-0.4.12-cp27-cp27m-manylinux1_x86_64.whl: 3.3ms
pex:   Fetching file:///Users/repo/.pants.d/python-setup/resolved_requirements/CPython-2.7.10/greenlet-0.4.12-cp27-cp27m-manylinux1_x86_64.whl: 1.8ms
pex: Translating /private/var/folders/r0/1b1z1cwx7bg1gf9mt5m20mzhq807mv/T/tmpTPQ0ad/greenlet-0.4.12-cp27-cp27m-manylinux1_x86_64.whl into distribution: 2.1ms
pex: Fetching file:///Users/repo/.pants.d/python-setup/resolved_requirements/CPython-2.7.10/ansicolors-1.1.8-py2.py3-none-any.whl: 2.8ms
pex:   Fetching file:///Users/repo/.pants.d/python-setup/resolved_requirements/CPython-2.7.10/ansicolors-1.1.8-py2.py3-none-any.whl: 1.3ms
…
e

enough-analyst-54434

09/25/2018, 5:42 PM
BAD APPLE: 
PythonInterpreter('/System/Library/Frameworks/Python.framework/Versions/2.7/bin/python2.7', PythonIdentity('CPython', 2, 7, 10)
h

high-night-61226

09/25/2018, 5:42 PM
Copy code
pex: Translating /private/var/folders/r0/1b1z1cwx7bg1gf9mt5m20mzhq807mv/T/tmp1bR2p1/ansicolors-1.1.8-py2.py3-none-any.whl into distribution: 4.2ms
pex: Constructed RequestsContext context <pex.http.RequestsContext object at 0x118796d10>
pex: crawling link i=0 link=Link('<https://pypi.python.org/simple/markupsafe/>') follow_links=False
pex: Constructed RequestsContext context <pex.http.RequestsContext object at 0x118796fd0>
pex: Fetching file:///Users/repo/.pants.d/python-setup/resolved_requirements/CPython-2.7.10/MarkupSafe-1.0.tar.gz: 7.7ms
pex:   Fetching file:///Users/repo/.pants.d/python-setup/resolved_requirements/CPython-2.7.10/MarkupSafe-1.0.tar.gz: 3.5ms
pex: Target package WheelPackage('file:///private/var/folders/r0/1b1z1cwx7bg1gf9mt5m20mzhq807mv/T/tmpCYFw21/MarkupSafe-1.0-cp27-cp27m-macosx_10_12_intel.whl') is not compatible with [('cp27', 'cp27u', 'linux_x86_64'), ('cp27', 'cp27m', 'manylinux1_x86_64'), ('py2', 'none', 'any'), ('cp27', 'cp27mu', 'manylinux1_x86_64'), ('cp27', 'none', 'any'), ('cp27', 'cp27dm', 'manylinux1_x86_64'), ('cp27', 'cp27u', 'manylinux1_x86_64'), ('cp27', 'cp27d', 'manylinux1_x86_64'), ('cp27', 'cp27mu', 'linux_x86_64'), ('cp27', 'none', 'linux_x86_64'), ('py2', 'none', 'linux_x86_64'), ('cp27', 'cp27dm', 'linux_x86_64'), ('cp27', 'cp27d', 'linux_x86_64'), ('cp27', 'cp27m', 'linux_x86_64'), ('py27', 'none', 'any'), ('cp27', 'cp27dmu', 'linux_x86_64'), ('cp27', 'none', 'manylinux1_x86_64'), ('cp2', 'none', 'any'), ('py2', 'none', 'manylinux1_x86_64'), ('cp27', 'cp27dmu', 'manylinux1_x86_64')]
pex: Translating /private/var/folders/r0/1b1z1cwx7bg1gf9mt5m20mzhq807mv/T/tmpD3oqmc/MarkupSafe-1.0.tar.gz into distribution: 1875.2ms
pex:   Packaging MarkupSafe: 1839.5ms
pex:     Installing /var/folders/r0/1b1z1cwx7bg1gf9mt5m20mzhq807mv/T/tmpZL0nbc: 1838.2ms

               Waiting for background workers to finish.
12:39:35 00:07   [complete]
               FAILURE
Exception caught: (<class 'pex.resolver.Untranslateable'>)

Exception message: Package SourcePackage(u'file:///Users/repo/.pants.d/python-setup/resolved_requirements/CPython-2.7.10/MarkupSafe-1.0.tar.gz') is not translateable by ChainedTranslator(WheelTranslator, EggTranslator, SourceTranslator)
e

enough-analyst-54434

09/25/2018, 5:42 PM
h

high-night-61226

09/25/2018, 5:42 PM
Ah, sounds more like a problem on my end than a bug
e

enough-analyst-54434

09/25/2018, 5:43 PM
Sortof. So - to be clear, conclusions: 1. Your fundamental issues is https://github.com/pantsbuild/pex/issues/523 2. path-by-id never should have worked - unrelated 3. You can only workaround by excluding Apple interpreters
h

high-night-61226

09/25/2018, 5:44 PM
ah, got it. Not sure how to do (3), but will read though the issue and work on resolving the apple interpreter. There seems to be no end to trouble from the Apple interpreter…
Thanks for all the help!
e

enough-analyst-54434

09/25/2018, 5:45 PM
For 3, there are a few techniques that work with pants today: 1. use 
python-setup.interpreter-constraints
to version exclude known-bad OSX interpreters. This might not work for you though if you have a linux interpreter of the same version you do want to use (say 2.7.10 as in this case) 2. use 
python-setup.interpreter_search_paths
to whitelist known good paths on osx and linux. This might not work for you since the linux good interpreters tend to be in /usr/bin which is where bad interpreters shipped by Apple tend to be
h

high-night-61226

09/25/2018, 5:46 PM
Awesome! We should be fine with the 
interpreter-constraints
2 Views
Powered by