https://pantsbuild.org/ logo
#general
Title
# general
n

numerous-fall-96475

05/06/2020, 4:06 PM
Got v2 working locally (hoooray), but for the life of me I can't figure out what's wrong on Jenkins. The Dockerfile handles v1. End of the stack dump is this, and I can't even really tell which file it's not finding. Happens during a ./pants binary call to build a single pex.
👀 1
🎉 1
h

hundreds-father-404

05/06/2020, 4:09 PM
Yay! We finished yesterday rewriting the docs at https://pants.readme.io/docs/welcome-to-pants. Hope those help, and please let us know if you have any suggestions! Hm, which Pants version is this? I’ve never seen this where it doesn’t say what file can’t be found..That error message is from the underlying OS
Ohhh, I suspect that we can’t find the “file”
python
a

aloof-angle-91616

05/06/2020, 4:11 PM
if we could repro this with process_executor, we could potentially get better info, but those tools aren’t embedded in the pants pex and it’s not easy to get to them. i think this error might happen if the output file wasn’t generated. process_executor would let us at least reproduce this error. i think we publish process_executor to s3. we could use that to repro the error here
i thought so too! but then i looked at the PATH!
h

hundreds-father-404

05/06/2020, 4:12 PM
@numerous-fall-96475 where is your Python 3.7 interpreter located, or any Python interpreter in general?
☝️ 1
a

aloof-angle-91616

05/06/2020, 4:13 PM
do we need to use the python exe in the command line here? or could we execute the pex directly?
h

hundreds-father-404

05/06/2020, 4:14 PM
do we need to use the python exe in the command line here?
Yes, I believe this is necessary. See a comment in
hermetic_pex.py
lines 43-37 for why we use the generic
python
name rather than something more specific like
python3.7
a

aloof-angle-91616

05/06/2020, 4:16 PM
that comment mentions remote execution, which isn't relevant here i thought?
i'm just trying to figure out how to bisect the issue
h

hundreds-father-404

05/06/2020, 4:24 PM
Even in a world where remote execution didn’t exist, we would still use the generic
python
name because Pants does not know which single Python interpreter to use, nor does it know the absolute path to that interpreter. We leave it up to the OS + Pex to handle all binary selection. We might be able to get away with
/usr/bin/python
, but I think we had issues with each OS handling that differently. For example, macOS will in its next release not even ship with Python. We won’t know if users installed their
python
bin in
/usr/bin/python
or somewhere else.
n

numerous-fall-96475

05/06/2020, 4:25 PM
It's running on a modified nvidia/cuda image build ontop of 18.04, and it installs python3 with apt. So I imagine python would be in /usr/bin/ unless the upstream cuda image did somethink really funky.
👍 1
a

aloof-angle-91616

05/06/2020, 4:26 PM
i was under the impression that executing a pex file without the
python
prefix would select the correct interpreter automatically. i actually know that’s not true, because i recently had to make the change to several ci jobs to add it. but i don’t know why there’s a difference in behavior
n

numerous-fall-96475

05/06/2020, 4:26 PM
So I should be running python3 ./pants binary blah instead?
a

aloof-angle-91616

05/06/2020, 4:26 PM
no no not for pants
this is purely relevant for pants calling pex internally
👍 1
sorry for the noise
👍 1
breaking out the pex discussion into #pex, sorry for being more confusing
i think process_executor can be used here to repro the issue. if you make a new issue with the info from this thread and ping this thread when you’ve made it we should be able to investigate it todayish
just the info that seems necessary. the log + your second most recent comment would be the relevant info i think
n

numerous-fall-96475

05/06/2020, 4:30 PM
Forgive my ignorance here but is process_executor a pants specific flag? A different binary? Something else?
a

aloof-angle-91616

05/06/2020, 4:31 PM
it is a different binary that allows debugging process executions like this. the v2 engine relies on a serializable remote process execution API and process_executor is the top-level binary to test that. we also have fs_util which allows modifying and inspecting the pants virtual file store. basically if you make a ticket i’m thinking that i could provide a set of commands for you to run after downloading the process_executor binary to reproduce the issue
oh wait!!!! try with
--no-process-execution-cleanup-local-dirs
, look at the output for a directory path “preserved local process execution dir at”
and then let us know the contents of that directory
i can be more clear but it would be helpful at this point to make an issue. sorry to keep spamming the thread :) i know it’s confusing
n

numerous-fall-96475

05/06/2020, 4:34 PM
I'll get this image build local and shell into it to poke around. Will have something for you later tonight although I have meetings until about 5 pm EST.
a

aloof-angle-91616

05/06/2020, 4:35 PM
no problem!!! we totally understand busy schedules. i’ll see if i can jump in earlier than that given that you’ve already provided some quite nice info
w

witty-crayon-22786

05/06/2020, 4:37 PM
hey @numerous-fall-96475: sorry for the trouble. based on the command and the failure, this seems like the biggest open question: https://pantsbuild.slack.com/archives/C046T6T9U/p1588781541409300?thread_ts=1588781192.403400&cid=C046T6T9U
because the error is indicating that there is no
python
in that PATH, afaict.
👍 1
h

hundreds-father-404

05/06/2020, 4:38 PM
In particular, running
which -a python
in the CI job would help
w

witty-crayon-22786

05/06/2020, 4:39 PM
and note that it doesn’t need to be “the” python you’re expecting the run to actually use: once the
pex
has started, it will locate an appropriate one in the PATH to match the constraint.
👍 1
n

numerous-fall-96475

05/06/2020, 10:41 PM
Haven't forgotten about you guys. Meetings are done. Although having some trouble gaining access to the environments needed to provide a real apples to apples comparison on the which call.
no alias for python / python2 is not on the docker image, only python3
w

witty-crayon-22786

05/06/2020, 11:03 PM
yea, that should be fine. pex will bootstrap with either of them, as long as “something” is at
python
but you’re saying that your python3 install is not aliased to
python
?
n

numerous-fall-96475

05/06/2020, 11:03 PM
that's right
w

witty-crayon-22786

05/06/2020, 11:04 PM
ah. well… that would do it, heh. cc @hundreds-father-404
h

hundreds-father-404

05/06/2020, 11:04 PM
Hm, that is indeed the issue
n

numerous-fall-96475

05/06/2020, 11:04 PM
Which is very surprising, given that it's ubuntu 18.04 under the hood
👍 1
w

witty-crayon-22786

05/06/2020, 11:04 PM
well, that’s your workaround i think.
n

numerous-fall-96475

05/06/2020, 11:04 PM
But thank you for the helm
help*
a

aloof-angle-91616

05/06/2020, 11:05 PM
please feel free to ping us in the future about this sort of thing!!! glad we could help
h

hundreds-father-404

05/06/2020, 11:05 PM
You’re welcome! Two solutions here, as far as I can tell: 1) Add a symlink such that
python
shows up on your PATH 2) Modify the
interpreter_search_paths
option to point to a binary named
python
, if you have one. See https://pants.readme.io/v1.27/docs/python-interpreter-compatibility#changing-the-interpreter-search-path.
👖 1
a

aloof-angle-91616

05/06/2020, 11:06 PM
thank you for closing the loop @hundreds-father-404
❤️ 1
w

witty-crayon-22786

05/06/2020, 11:07 PM
@hundreds-father-404: the interpreter search paths should probably already find things with alternative names?
but i wonder if it’s possible to give pex an insanely permissive shebang that would bootstrap under ~anything it can find, so that we can avoid the
python ./pex
in favor of
#!/env/please/just/any python*
…i’ll take that question elsewhere.
👍 2
a

aloof-angle-91616

05/06/2020, 11:08 PM
#!/usr/bin/env python would allow us to revert several command lines i had to edit for MLWF
w

witty-crayon-22786

05/06/2020, 11:09 PM
won’t match
python3
, which was the bug here.
👍 1