https://pantsbuild.org/ logo
#pex
Title
# pex
a

aloof-angle-91616

07/05/2018, 7:35 PM
@enough-analyst-54434 there's a CI failure in my PR https://github.com/pantsbuild/pants/pull/6022 in the osx shard where pex is failing to resolve locally-built dists in
PythonRun
only, with `Failed to execute PEX file, missing macosx_10_12_x86_64-cp-27-cp27m compatible dependencies for: my local dist`: see https://travis-ci.org/pantsbuild/pants/jobs/400569783. this does not repro on my osx laptop. in
SetupPyRunner
we're tagging the locally-built dist with
--plat-name
set to
get_local_platform()
if there are native sources.
get_local_platform()
has a TODO linking to this issue: https://github.com/pantsbuild/pex/issues/511, and it seems reasonable to think these are related, but it's not clear to me what would cause this resolution failure.
afaict
SetupPyRunner
is being invoked with the
interpreter
from
self.context.products.get_data(PythonInterpreter)
, and that is also used to create the pex that is consumed in
PythonRun
. i tried setting
--pre
in the
WrappedPex#cmdline()
method, which did not seem to resolve the issue (nor did blasting caches), but especially since this is only on the osx shard i would suspect the platform is different somehow.
any thoughts just on how the platforms might be out of sync is all i'm looking for when you have the time, thanks
also, I wrote the SetupPyRunner part about a week ago, so that may be suspect
the catch is that this failure doesn't occur when the
--tag-build
is not used for local dists though (as is done on master)
e

enough-analyst-54434

07/05/2018, 7:43 PM
I'll ignore platform differences for a moment. So the resolution for the PythonRun pex requirements happens way back in pyprep.requirements`. That is where allow_prereleases would be needed.
But just always
allow_prereleases
whenever there are
python_dists
in play is incorrect.
a

aloof-angle-91616

07/05/2018, 7:47 PM
ok, i'm looking through
ResolveRequirementsTaskBase
now. why would this resolution error only show up when invoking the pex (as opposed to when building it)?
and i understand the point about not having
allow_prereleases
on all the time, will figure out how to ensure it's only to resolve just the local dists
e

enough-analyst-54434

07/05/2018, 7:51 PM
why would this resolution error only show up when invoking the pex (as opposed to when building it)?
Agreed. Thinking more.
a

aloof-angle-91616

07/05/2018, 7:52 PM
locally on osx, we are detecting correctly whether the target set has any native sources in
resolve_requirements()
(and setting the wheel platform), and i don't think that would be different in travis
e

enough-analyst-54434

07/05/2018, 7:53 PM
I do not know off the top of my head but can check: are you sure this is a valid version?: 1.0.0+dcffd59dfb65eb2abd72f11322448606a7c76dc6.defaultfingerprintstrategy.da39a3ee5e6b.897870053801
In python setuptools eyes?
I know +[a-za-Z]+ is ok
Not sure about dots
a

aloof-angle-91616

07/05/2018, 7:55 PM
there is a page i have open explaining legal tagged builds -- that's what the raw fingerprint passed to
--tag-build
is converted to when making the dist by distutils. i will find that page
looks ok
a

aloof-angle-91616

07/05/2018, 7:56 PM
i doubt case sensitivity would affect this, not in this way
e

enough-analyst-54434

07/05/2018, 8:02 PM
The thing to do here then is to 1. edit .travis.yml on your branch down to just the 1 shard and 2. add PEX_VERBOSE=5 into the mix. Easiiest way would be to modify the .travis.yml script line and prepend that I think.
a

aloof-angle-91616

07/05/2018, 8:03 PM
got it
e

enough-analyst-54434

07/05/2018, 8:03 PM
The pex verbose will show us what is scanned (and rejected) for that dep
a

aloof-angle-91616

07/05/2018, 8:05 PM
doing that now, will post a link when ci is up. i'll ping you again when i've figured it out or figured out that i can't
grr, PEX_VERBOSE wasn't passed down to the subprocess. i think i know how to do that though dw
e

enough-analyst-54434

07/05/2018, 8:32 PM
OK, slowing down. Its an intg that is failing - some of those run
hermetic
which scrubs env
a

aloof-angle-91616

07/05/2018, 8:32 PM
yeah
e

enough-analyst-54434

07/05/2018, 8:32 PM
Perhaps target the failing test and edit it
a

aloof-angle-91616

07/05/2018, 8:32 PM
right there with you
also removed the rust testing so it's faster this time lol
so if you search
test_pants_run
in this log you'll see the pex verbose output: https://api.travis-ci.org/v3/job/400607859/log.txt. one thing i think is fine but want to verify is that the wheel filename ends in
-cp27-cp27m-macosx_10_12_x86_64.whl
, but pex is looking for the platform
macosx_10_12_x86_64-cp-27-cp27m
-- i'm assuming that's just a matter of display and not identity, but maybe not?
it does show that the dist we want is contained in the pex, but it doesn't satisfy the dep (for that exact dist, but without a version tag)
lots of output that i'm going through now so need to context switch yet
there is
pex: Please build pex with the subprocess32 module for more reliable requirement installation and interpreter execution.
-- i'm not seeing this locally
...i am, actually, but not on all locally-built dists with native code, just the one that's failing on travis
PEX_VERBOSE=5 ./pants -ldebug run examples/src/python/example/python_distribution/hello/fasthello:main
that's a lie, it's on all locally-built dists, this is not different from ci, sorry
the test in ci is using 2.7.10 instead of 2.7.14, otherwise the fingerprints/paths are the same
everything else is using 2.7.10 as well though it seems (declared as such in the pytest output). still looking
the timings are cut off for some weird reason at the "searching dependency cache" line in which it should have found the dist:
pex: Activating PEX virtual environment from /Users/travis/build/pantsbuild/pants/.pants.d/tmp/tmp75lcmR.pants.d/pyprep/requirements/CPython-2.7.10/24a44c0071b9e805fe7b2172be7b3e498b6f69e1-DefaultFingerprintStrategy_8c9b6e40089d :: Searching dependency cache: /Users/travis/build/pantsbuild/pants/.pants.d/tmp/tmp75lcmR.pants.d/pyprep/requirements/CPython-2.7.10/24a44c0071b9e805fe7b2172be7b3e498b6f69e1-DefaultFingerprintStrategy_8c9b6e40089d/.deps
and it's all on one line, it should be two and have
: 2.0ms
or so at the end of both
"should have found the dist" meaning "it does that on my laptop":
Copy code
pex: Activating PEX virtual environment from /Users/dmcclanahan/tools/pants/.pants.d/pyprep/requirements/CPython-2.7.14/f14bea5cea130765649dd5c2465d9be7af894484-DefaultFingerprintStrategy_8c9b6e40089d: 4.6ms                                                                                                                                       
pex:   Searching dependency cache: /Users/dmcclanahan/tools/pants/.pants.d/pyprep/requirements/CPython-2.7.14/f14bea5cea130765649dd5c2465d9be7af894484-DefaultFingerprintStrategy_8c9b6e40089d/.deps: 2.0ms
pex:     Adding fasthello 1.0.0+b2746123820ccd47afb2e520476673e3aea8a39a.defaultfingerprintstrategy.da39a3ee5e6b.897870053801: 0.2ms
pex:   Resolving fasthello==1.0.0+b2746123820ccd47afb2e520476673e3aea8a39a.defaultfingerprintstrategy.da39a3ee5e6b.897870053801: 0.7ms
pex:   Activating fasthello 1.0.0+b2746123820ccd47afb2e520476673e3aea8a39a.defaultfingerprintstrategy.da39a3ee5e6b.897870053801: 0.9ms
pex:     Adding sitedir: 0.5ms
pex: Activating PEX virtual environment from /Users/dmcclanahan/tools/pants/.pants.d/pyprep/sources/b416569e036d849d23748fcd2cd6aab6409ccc01-DefaultFingerprintStrategy_4645322e1ea7 pex: Activating PEX virtual environment from /Users/dmcclanahan/tools/pants/.pants.d/pyprep/sources/b416569e036d849d23748fcd2cd6aab6409ccc01-DefaultFingerprintStrategy_4645322e1ea7: 0.6ms
and the interpreter it's using at
/System/Library/Frameworks/Python.framework/Versions/2.7/bin/python2.7
is
2.7.10
on this machine, which lines up (everything else has 2.7.10 in the path names, so that's correct)
thinking that what
--plat-name
wants may not be the exact platform name (???), looking at https://stackoverflow.com/questions/35112511/pip-setup-py-bdist-wheel-no-longer-builds-forced-non-pure-wheels#36886459. looking for release notes between 2.7.10 vs 2.7.14 to see if there's a difference in behavior (or i could just try invoking 2.7.10 locally...)
looking in the
wheel
source, the
--plat-name
argument is definitely incorrect, so i'll try fixing that and seeing what happens
https://github.com/pypa/wheel/blob/master/wheel/bdist_wheel.py (it uses a
get_platform()
method defined in an adjacent module)
e

enough-analyst-54434

07/05/2018, 10:02 PM
Yeah pex_uitil.get_local_platform is definitely not what you want
The release.sh script does this right
a

aloof-angle-91616

07/05/2018, 10:03 PM
i'll check it
that's great to hear because i was just diving into cpython source again
I included the link in a comment when I figured out the right way.
This explains why osx and not linux
This should fail on all OSX
a

aloof-angle-91616

07/05/2018, 10:06 PM
that was my concern as well, i'm going to blast all of the caches i know of and try again (after making the platform change)
thanks a lot
pex's platform method describes a nuance with MACOSX_DEPLOYMENT_TARGET which i will address after we can figure out how to build anything at all
it's definitely different (it ends with
_intel.whl
now) so this may be the right track
tossing it up into ci in a sec
e

enough-analyst-54434

07/05/2018, 10:13 PM
From all the side reading on this I'm pretty positive local version identifiers don't require allow_prereleases
pre and local are different beasts
So that whole potential problem is not a problem
a

aloof-angle-91616

07/05/2018, 10:14 PM
that makes sense,
--pre
was 100% a shot in the dark
yes
come to think of it i should have realized
--plat-name
was almost definitely the first thing to look into, lesson learned. ci up at https://travis-ci.org/pantsbuild/pants/builds/400643859?utm_source=github_status&utm_medium=notification
i also used the name
Platform
for platform-specific native code stuff, was going to change that anyway and realizing the current naming ambiguity that i wasted time on gives me more impetus to change that name too
e

enough-analyst-54434

07/05/2018, 10:41 PM
pex.environment.Environment
- which is relevant erroring at runtime bit here, logs platform tags it thinks it supports at level 9
So maybe try againt with PEX_VERBOSE=9
a

aloof-angle-91616

07/05/2018, 10:45 PM
i will try that -- i was seeing a wheel ending in
macosx_10_13_intel.whl
locally, but the remote version was still showing the old
macosx_10_12_x86_64.whl
, so i thought blasting the cache might work (haven't rebuilt since then). pushing the verbosity increase now
e

enough-analyst-54434

07/05/2018, 10:46 PM
Remote fixes osx version down
Which should explain 10.12 vs 10.13
a

aloof-angle-91616

07/05/2018, 10:46 PM
yeah sorry that was not the concern
e

enough-analyst-54434

07/05/2018, 10:46 PM
oh, intel vs x86_64
gotcha
a

aloof-angle-91616

07/05/2018, 10:47 PM
but locally the wheel was named
_x86_64
before the platform change, now it's intel, yeah
e

enough-analyst-54434

07/05/2018, 10:50 PM
May be worth the next shot - using that util.
a

aloof-angle-91616

07/05/2018, 10:52 PM
hmmmmm thanks, i will try that
(still fails using the distutils platform, looking through the output now)
and haven't used the pex util you just linked yet
i actually just restarted that build and realized that would clear it for you too oops
travis for the PR on pantsbuild/pants isn't running yet, which is fine
speak of the devil
also going to add
-ldebug
to the pants invocation in the test, should have done that earlier
ok so it still fails, but i also noticed that there's this "Translating <the whl we care about> into dist" section in travis, but not locally (i grepped for
trans
in the stderr), so that seems highly relevant. going to push a commit with debug on to see what the produced wheel file's name is (because it's still looking for
_x86_64
in ci)
i built python 2.7.10 so i could see if that was involved with the whl name changing but i can't figure out how to make pants accept that
sorry, haven't looked, not can't figure out -- i meant that i am looking
yep found
using
pex.pep425tags.get_platform()
returns it to being
_x86_64.whl
, but the same error persists. i'm clearing the caches and rebuilding, i don't expect that to help.