https://pantsbuild.org/ logo
Join Slack
Powered by
re: shebangs: i wonder if it’s possible to give pe...
# pex
w
re: shebangs: i wonder if it’s possible to give pex an insanely permissive shebang that would bootstrap under ~anything it can find, so that we can avoid the 
python ./pex
 in favor of 
#!/env/please/just/any python*
a
i don’t think shebang lines work with globs :)
i think we could work around that
h
I don’t think they do either. It’s a real problem when you want to say in a shebang 
python3.7+
. You can either gamble on 
python3
or settle for 
python3.7
. I don’t think you can say 
python3.7
or 
python3.8
in a shebang
w
also the version python used for bootstrap matters. e.g. python3.6 some.pex works, but python3.7 some.pex will give
Copy code
TypeError: unhashable type: 'list'
👍 1
(depending on the pex version too i guess)
w
which version of pex?
a
right now i’m thinking this could be done with an extension to the bootstrap script. or we could extract the pex interpreter selection into a standalone script or something
w
yea.
a
@wide-energy-11069 the ipex launcher is written to be compatible with py2 for this reason — so it can be launched from an arbitrary interpreter. i agree that the pex version may be to blame for that since pex code shouldn’t be causing issues with diff interpreter versions. but not sure
👍 1
@witty-crayon-22786 i don’t know if you saw john’s recent issue to “run pex files as normal applications” but i just remembered you’re not in the build-internal chat anymore :’( miss you. i’ll link it in a new thread
❤️ 1
😢 1
e
Folks - the shebang can be controlled today and made to just be 
/usr/bin/env python
- no prefix and interpreter-constraints will force pex to re-exec itself after finding an appropriate interpreter no matter how its run.
👍 1
h
What prompted this is needing wanting to work with the bin name 
python3
, not 
python
. A user doesn’t have 
python
on their ubuntu machine
e
all 
python file.pex
does is override the shebang.
👍 1
Wanting a pex file to work with python3?
IE: needing the shebang to say 
/usr/bin/env python3
?
Did the user have control over the building of the pex to add the custom shebang?
a
the only 
python
available on the user's PATH in this case was named 
python3
. we believe this is the case in ubuntu docker images
the issue was not the custom shebang, but that pants hardcodes the 
python
executable and looks it up in the PATH when creating a pex file in v2
👍 1
being able to change that with a pants option might do it
e
OK - so this has 0 to do with pex then. Its a v2 rule problem.
h
I’m not sure if the conversation has evolved to something else. Originally, it was that it would be nice for Pants in 
hermetic_pex.py
to work with either 
python
or 
python3
in the line 
argv=[python, ./example.pex]
, so that Pants can support that user today who only has 
python3
on his machine and not 
python
. I don’t think that’s very feasible, though, without gymnastics, and I don’t think it’s a problem worth prioritizing. For now, we can improve documentation
e
Right - thats purely a v2 rule problem.
👍 2
a
i think we agree on that now
or rather, if the pex file we create has the 
/usr/bin/env
shebang as discussed, is it possible we could remove the 
python
entirely? cc @hundreds-father-404
although, i think the issue here was that the downloaded 
pex
executable is not user-controlled
e
All that does is move the same problem into the shebang.
👍 1
a
will investigate
All that does is move the same problem into the shebang.
yes, you're right. thanks
@hundreds-father-404 would you accept a PR to add a 
--python-setup-python-interpreter-name
option?
e
we believe this is the case in ubuntu docker images
Did anybody check this?
a
the user reported that that was the case, in an ubuntu docker image. i'll see if that's the case on every ubuntu now
h
Did anybody check this?
I don’t think it’s true in general. For example, we use Ubuntu in our own CI. @aloof-angle-91616 the user said he has a very custom docker image
👍 1
a
ah ok thank you
w
the rule cannot know precisely what environment it will be running in
👍 2
a
yes, the default 
python
package installs a py2 interpreter with the name 
python
w
once the process is actually started, it can scan the PATH to see what exists… hence the shebang comment
👍 1
a
so maybe an interpreter name option for local and remote?
w
could do it with a bash wrapper in the EPR around the 
pex
, but that’s sortof what a shebang is
👍 1
h
would you accept a PR to add a --python-setup-python-interpreter-name option?
I think this is good? Better than our current remedy we gave the user of needing to coerce their system to have 
python
be discoverable. I don’t see any issues with this.
a
in case you haven't seen it, i made a PR that makes bash invocations more streamlined: https://github.com/pantsbuild/pants/pull/9101 and it might make solving this sort of thing easier
e
My question about image was just to prompt backoff if this is a truly weird case. I don't think its unreasonable for pants to say your environment must have at least 1 python interpreter named python.
👍 2
a
that in itself is a configuration option
e
But should not be if this is a truly wierd case.
a
super easy to 
ln -s
if the user doesn't have a 
python
-named binary too
e
Did the use say they didn't like the idea of fixing their image to have this link?
👀 1
a
But should not be if this is a truly wierd case.
sorry, i meant that using the PATH to locate the first 
python
is itself a form of configuration, which removes the need for a separate option
reading the thread, the user said "thanks for the help" immediately after we arrived at the solution of "add a 
python
interpreter or symlink"
so i think the user is fine with this
👍 1
e
OK - so documenting maybe that Pants requirement is not just python3.6+ but a sane python which installs a 
python
binary in addition to 
python3.6
etc is also a requirement.
👍 2
I have never seen a python package that does not do this.
👍 1
a
that is absolutely good enough for me
h
Cool. Adding to the docs now
Thank you both
e
I'm not hard opposed to a knob, but you can go too far.
💯 1
a
i normally like more knobs to a fault but it feels like we're only adding complexity without any real need for a specific option for the python interpreter name
👍 1
i can easily see how different settings of that option in different pants.toml files for example would cause silent breakages
💯 1
@hundreds-father-404 thank you for volunteering to do the docs change, i was going to offer to switch my previous offer from options to docs. let me know if there's anything i can help clean up here
❤️ 1
h
let me know if there’s anything i can help clean up here
A review would be appreciated! Everything is done other than: - aws lambda - coverage - “project introspection” to describe things like 
dependees
See https://pants.readme.io/docs/welcome-to-pants for 1.28.x, including a new page on codegen I wrote today :)
👍 1
a
^_^_^_^_^_^_^_^
h
e
Looks good. The wrinkle that may not be clear is that the requirement is that some 
python
is on the 
PATH
and you also have 
python3.[6-9]
on the 
PATH
(setup script needs this). So we actually have a push-pull. If the user's image would have had just a 
python
binary for python3.6 then the setup script would fail.
👨‍🎓 2
The some 
python
on the path can be 2.7+
👍 2
h
Okay updated again https://pants.readme.io/docs/installing-pants. I left off 
python3.9
because we will soon stop using 
abi3
(to unblock Stu’s migration from 
cffi -> cpython
), and we won’t initially build py39
e
Looks more correct now. Technically "You must also have python on your 
PATH
. This can be Python 2 or Python 3." should be "You must also have python on your 
PATH
. This can be Python 2.7+ or Python 3.5+." That said, the only important thing is Python 3.6+ on the 
PATH
and all the rest of the lanuage applies to weird situations. How about de-escalating the weird situation from the main flow of the section. Something like: -- To run Pants, you need: * Linux or macOS. * Python 3.6+ discoverable on your 
PATH
Note that your own code can use Python 2; this is only what Pants uses to run itself. If you run into problems but have Python 3.6+ installed you may have a non-standard python installation. Pants expects to find both 
python
and one of 
python3.6
, 
python3.7
, or 
python3.8
on your PATH.
👍 1
h
I like it. Thanks! I’ll try out using a “warning” block - those are nice because you can quickly decide it’s irrelevant to you
e
Agreed.
Open in Slack
PreviousNext
Powered by