https://pantsbuild.org/ logo
#pex
Title
w

witty-crayon-22786

05/06/2020, 11:08 PM
re: shebangs: i wonder if it’s possible to give pex an insanely permissive shebang that would bootstrap under ~anything it can find, so that we can avoid the 
python ./pex
 in favor of 
#!/env/please/just/any python*
a

aloof-angle-91616

05/06/2020, 11:09 PM
i don’t think shebang lines work with globs :)
i think we could work around that
h

hundreds-father-404

05/06/2020, 11:10 PM
I don’t think they do either. It’s a real problem when you want to say in a shebang
python3.7+
. You can either gamble on
python3
or settle for
python3.7
. I don’t think you can say
python3.7
or
python3.8
in a shebang
w

wide-energy-11069

05/06/2020, 11:10 PM
also the version python used for bootstrap matters. e.g. python3.6 some.pex works, but python3.7 some.pex will give
Copy code
TypeError: unhashable type: 'list'
👍 1
(depending on the pex version too i guess)
w

witty-crayon-22786

05/06/2020, 11:11 PM
which version of pex?
a

aloof-angle-91616

05/06/2020, 11:11 PM
right now i’m thinking this could be done with an extension to the bootstrap script. or we could extract the pex interpreter selection into a standalone script or something
w

witty-crayon-22786

05/06/2020, 11:11 PM
yea.
a

aloof-angle-91616

05/06/2020, 11:12 PM
@wide-energy-11069 the ipex launcher is written to be compatible with py2 for this reason — so it can be launched from an arbitrary interpreter. i agree that the pex version may be to blame for that since pex code shouldn’t be causing issues with diff interpreter versions. but not sure
👍 1
@witty-crayon-22786 i don’t know if you saw john’s recent issue to “run pex files as normal applications” but i just remembered you’re not in the build-internal chat anymore :’( miss you. i’ll link it in a new thread
❤️ 1
😢 1
e

enough-analyst-54434

05/06/2020, 11:50 PM
Folks - the shebang can be controlled today and made to just be
/usr/bin/env python
- no prefix and interpreter-constraints will force pex to re-exec itself after finding an appropriate interpreter no matter how its run.
👍 1
h

hundreds-father-404

05/06/2020, 11:51 PM
What prompted this is needing wanting to work with the bin name
python3
, not
python
. A user doesn’t have
python
on their ubuntu machine
e

enough-analyst-54434

05/06/2020, 11:51 PM
all
python file.pex
does is override the shebang.
👍 1
Wanting a pex file to work with python3?
IE: needing the shebang to say
/usr/bin/env python3
?
Did the user have control over the building of the pex to add the custom shebang?
a

aloof-angle-91616

05/06/2020, 11:53 PM
the only
python
available on the user's PATH in this case was named
python3
. we believe this is the case in ubuntu docker images
the issue was not the custom shebang, but that pants hardcodes the
python
executable and looks it up in the PATH when creating a pex file in v2
👍 1
being able to change that with a pants option might do it
e

enough-analyst-54434

05/06/2020, 11:54 PM
OK - so this has 0 to do with pex then. Its a v2 rule problem.
h

hundreds-father-404

05/06/2020, 11:54 PM
I’m not sure if the conversation has evolved to something else. Originally, it was that it would be nice for Pants in
hermetic_pex.py
to work with either
python
or
python3
in the line
argv=[python, ./example.pex]
, so that Pants can support that user today who only has
python3
on his machine and not
python
. I don’t think that’s very feasible, though, without gymnastics, and I don’t think it’s a problem worth prioritizing. For now, we can improve documentation
e

enough-analyst-54434

05/06/2020, 11:55 PM
Right - thats purely a v2 rule problem.
👍 2
a

aloof-angle-91616

05/06/2020, 11:55 PM
i think we agree on that now
or rather, if the pex file we create has the
/usr/bin/env
shebang as discussed, is it possible we could remove the
python
entirely? cc @hundreds-father-404
although, i think the issue here was that the downloaded
pex
executable is not user-controlled
e

enough-analyst-54434

05/06/2020, 11:56 PM
All that does is move the same problem into the shebang.
👍 1
a

aloof-angle-91616

05/06/2020, 11:56 PM
will investigate
All that does is move the same problem into the shebang.
yes, you're right. thanks
@hundreds-father-404 would you accept a PR to add a
--python-setup-python-interpreter-name
option?
e

enough-analyst-54434

05/06/2020, 11:57 PM
we believe this is the case in ubuntu docker images
Did anybody check this?
a

aloof-angle-91616

05/06/2020, 11:58 PM
the user reported that that was the case, in an ubuntu docker image. i'll see if that's the case on every ubuntu now
h

hundreds-father-404

05/06/2020, 11:58 PM
Did anybody check this?
I don’t think it’s true in general. For example, we use Ubuntu in our own CI. @aloof-angle-91616 the user said he has a very custom docker image
👍 1
a

aloof-angle-91616

05/06/2020, 11:58 PM
ah ok thank you
w

witty-crayon-22786

05/06/2020, 11:59 PM
the rule cannot know precisely what environment it will be running in
👍 2
a

aloof-angle-91616

05/06/2020, 11:59 PM
yes, the default
python
package installs a py2 interpreter with the name
python
w

witty-crayon-22786

05/06/2020, 11:59 PM
once the process is actually started, it can scan the PATH to see what exists… hence the shebang comment
👍 1
a

aloof-angle-91616

05/06/2020, 11:59 PM
so maybe an interpreter name option for local and remote?
w

witty-crayon-22786

05/07/2020, 12:00 AM
could do it with a bash wrapper in the EPR around the
pex
, but that’s sortof what a shebang is
👍 1
h

hundreds-father-404

05/07/2020, 12:00 AM
would you accept a PR to add a --python-setup-python-interpreter-name option?
I think this is good? Better than our current remedy we gave the user of needing to coerce their system to have
python
be discoverable. I don’t see any issues with this.
a

aloof-angle-91616

05/07/2020, 12:00 AM
in case you haven't seen it, i made a PR that makes bash invocations more streamlined: https://github.com/pantsbuild/pants/pull/9101 and it might make solving this sort of thing easier
e

enough-analyst-54434

05/07/2020, 12:01 AM
My question about image was just to prompt backoff if this is a truly weird case. I don't think its unreasonable for pants to say your environment must have at least 1 python interpreter named python.
👍 2
a

aloof-angle-91616

05/07/2020, 12:01 AM
that in itself is a configuration option
e

enough-analyst-54434

05/07/2020, 12:01 AM
But should not be if this is a truly wierd case.
a

aloof-angle-91616

05/07/2020, 12:01 AM
super easy to
ln -s
if the user doesn't have a
python
-named binary too
e

enough-analyst-54434

05/07/2020, 12:02 AM
Did the use say they didn't like the idea of fixing their image to have this link?
👀 1
a

aloof-angle-91616

05/07/2020, 12:02 AM
But should not be if this is a truly wierd case.
sorry, i meant that using the PATH to locate the first
python
is itself a form of configuration, which removes the need for a separate option
reading the thread, the user said "thanks for the help" immediately after we arrived at the solution of "add a
python
interpreter or symlink"
so i think the user is fine with this
👍 1
e

enough-analyst-54434

05/07/2020, 12:04 AM
OK - so documenting maybe that Pants requirement is not just python3.6+ but a sane python which installs a
python
binary in addition to
python3.6
etc is also a requirement.
👍 2
I have never seen a python package that does not do this.
👍 1
a

aloof-angle-91616

05/07/2020, 12:05 AM
that is absolutely good enough for me
h

hundreds-father-404

05/07/2020, 12:05 AM
Cool. Adding to the docs now
Thank you both
e

enough-analyst-54434

05/07/2020, 12:05 AM
I'm not hard opposed to a knob, but you can go too far.
💯 1
a

aloof-angle-91616

05/07/2020, 12:06 AM
i normally like more knobs to a fault but it feels like we're only adding complexity without any real need for a specific option for the python interpreter name
👍 1
i can easily see how different settings of that option in different pants.toml files for example would cause silent breakages
💯 1
@hundreds-father-404 thank you for volunteering to do the docs change, i was going to offer to switch my previous offer from options to docs. let me know if there's anything i can help clean up here
❤️ 1
h

hundreds-father-404

05/07/2020, 12:09 AM
let me know if there’s anything i can help clean up here
A review would be appreciated! Everything is done other than: - aws lambda - coverage - “project introspection” to describe things like
dependees
See https://pants.readme.io/docs/welcome-to-pants for 1.28.x, including a new page on codegen I wrote today :)
👍 1
a

aloof-angle-91616

05/07/2020, 12:10 AM
^_^_^_^_^_^_^_^
h

hundreds-father-404

05/07/2020, 12:12 AM
e

enough-analyst-54434

05/07/2020, 12:34 AM
Looks good. The wrinkle that may not be clear is that the requirement is that some
python
is on the
PATH
and you also have
python3.[6-9]
on the
PATH
(setup script needs this). So we actually have a push-pull. If the user's image would have had just a
python
binary for python3.6 then the setup script would fail.
👨‍🎓 2
The some
python
on the path can be 2.7+
👍 2
h

hundreds-father-404

05/07/2020, 1:22 AM
Okay updated again https://pants.readme.io/docs/installing-pants. I left off
python3.9
because we will soon stop using
abi3
(to unblock Stu’s migration from
cffi -> cpython
), and we won’t initially build py39
e

enough-analyst-54434

05/07/2020, 1:35 AM
Looks more correct now. Technically "You must also have python on your 
PATH
. This can be Python 2 or Python 3." should be "You must also have python on your 
PATH
. This can be Python 2.7+ or Python 3.5+." That said, the only important thing is Python 3.6+ on the
PATH
and all the rest of the lanuage applies to weird situations. How about de-escalating the weird situation from the main flow of the section. Something like: -- To run Pants, you need: * Linux or macOS. * Python 3.6+ discoverable on your
PATH
Note that your own code can use Python 2; this is only what Pants uses to run itself. If you run into problems but have Python 3.6+ installed you may have a non-standard python installation. Pants expects to find both
python
and one of
python3.6
,
python3.7
, or
python3.8
on your PATH.
👍 1
h

hundreds-father-404

05/07/2020, 1:36 AM
I like it. Thanks! I’ll try out using a “warning” block - those are nice because you can quickly decide it’s irrelevant to you
e

enough-analyst-54434

05/07/2020, 1:45 AM
Agreed.