re shebangs i wonder if it s possible to give pex an insanel

Question

re shebangs i wonder if it s possible to give pex an insanely permissive shebang that would bootstrap under ~anything it can find so that we can avoid the `python pex` in favor of ` env please just any python `

aloof-angle-91616 · Answer

i don t think shebang lines work with globs

aloof-angle-91616 · Answer

i think we could work around that

hundreds-father-404 · Answer

I don t think they do either It s a real problem when you want to say in a shebang `python3 7+` You can either gamble on `python3` or settle for `python3 7` I don t think you can say `python3 7` or `python3 8` in a shebang

wide-energy-11069 · Answer

also the version python used for bootstrap matters e g python3 6 some pex works but python3 7 some pex will give ```TypeError unhashable type list ```

wide-energy-11069 · Answer

depending on the pex version too i guess

witty-crayon-22786 · Answer

which version of pex

aloof-angle-91616 · Answer

right now i m thinking this could be done with an extension to the bootstrap script or we could extract the pex interpreter selection into a standalone script or something

witty-crayon-22786 · Answer

yea

aloof-angle-91616 · Answer

< wide energy 11069> the ipex launcher is written to be compatible with py2 for this reason so it can be launched from an arbitrary interpreter i agree that the pex version may be to blame for that since pex code shouldn t be causing issues with diff interpreter versions but not sure

aloof-angle-91616 · Answer

< witty crayon 22786> i don t know if you saw john s recent issue to run pex files as normal applications but i just remembered you re not in the build internal chat anymore miss you i ll link it in a new thread

enough-analyst-54434 · Answer

Folks the shebang can be controlled today and made to just be ` usr bin env python` no prefix and interpreter constraints will force pex to re exec itself after finding an appropriate interpreter no matter how its run

hundreds-father-404 · Answer

What prompted this is ~needing~ wanting to work with the bin name `python3` not `python` A user doesn t have `python` on their ubuntu machine

enough-analyst-54434 · Answer

all `python file pex` does is override the shebang

enough-analyst-54434 · Answer

Wanting a pex file to work with python3

enough-analyst-54434 · Answer

IE needing the shebang to say ` usr bin env python3`

enough-analyst-54434 · Answer

Did the user have control over the building of the pex to add the custom shebang

aloof-angle-91616 · Answer

the only `python` available on the user s PATH in this case was named `python3` we believe this is the case in ubuntu docker images

aloof-angle-91616 · Answer

the issue was not the custom shebang but that pants hardcodes the `python` executable and looks it up in the PATH when creating a pex file in v2

aloof-angle-91616 · Answer

being able to change that with a pants option might do it

enough-analyst-54434 · Answer

OK so this has 0 to do with pex then Its a v2 rule problem

hundreds-father-404 · Answer

I m not sure if the conversation has evolved to something else Originally it was that it would be nice for Pants in `hermetic pex py` to work with either `python` or `python3` in the line `argv= python example pex ` so that Pants can support that user today who only has `python3` on his machine and not `python` I don t think that s very feasible though without gymnastics and I don t think it s a problem worth prioritizing For now we can improve documentation

enough-analyst-54434 · Answer

Right thats purely a v2 rule problem

aloof-angle-91616 · Answer

i think we agree on that now

aloof-angle-91616 · Answer

or rather if the pex file we create has the ` usr bin env` shebang as discussed is it possible we could remove the `python` entirely cc < hundreds father 404>

aloof-angle-91616 · Answer

although i think the issue here was that the downloaded `pex` executable is not user controlled

enough-analyst-54434 · Answer

All that does is move the same problem into the shebang

aloof-angle-91616 · Answer

will investigate

aloof-angle-91616 · Answer

gt All that does is move the same problem into the shebang yes you re right thanks

aloof-angle-91616 · Answer

< hundreds father 404> would you accept a PR to add a ` python setup python interpreter name` option

enough-analyst-54434 · Answer

gt we believe this is the case in ubuntu docker images Did anybody check this

aloof-angle-91616 · Answer

the user reported that that was the case in an ubuntu docker image i ll see if that s the case on every ubuntu now

hundreds-father-404 · Answer

gt Did anybody check this I don t think it s true in general For example we use Ubuntu in our own CI < aloof angle 91616> the user said he has a very custom docker image

aloof-angle-91616 · Answer

ah ok thank you

witty-crayon-22786 · Answer

the rule cannot know precisely what environment it will be running in

aloof-angle-91616 · Answer

yes the default `python` package installs a py2 interpreter with the name `python`

witty-crayon-22786 · Answer

once the process is actually started it can scan the PATH to see what exists hence the shebang comment

aloof-angle-91616 · Answer

~so maybe an interpreter name option for local and remote ~

witty-crayon-22786 · Answer

could do it with a bash wrapper in the EPR around the `pex` but that s sortof what a shebang is

hundreds-father-404 · Answer

gt would you accept a PR to add a python setup python interpreter name option I think this is good Better than our current remedy we gave the user of needing to coerce their system to have `python` be discoverable I don t see any issues with this

aloof-angle-91616 · Answer

in case you haven t seen it i made a PR that makes bash invocations more streamlined <https github com pantsbuild pants pull 9101> and it might make solving this sort of thing easier

enough-analyst-54434 · Answer

My question about image was just to prompt backoff if this is a truly weird case I don t think its unreasonable for pants to say your environment must have at least 1 python interpreter named python

aloof-angle-91616 · Answer

that in itself is a configuration option

enough-analyst-54434 · Answer

But should not be if this is a truly wierd case

aloof-angle-91616 · Answer

super easy to `ln s` if the user doesn t have a `python` named binary too

enough-analyst-54434 · Answer

Did the use say they didn t like the idea of fixing their image to have this link

aloof-angle-91616 · Answer

gt But should not be if this is a truly wierd case sorry i meant that using the PATH to locate the first `python` is itself a form of configuration which removes the need for a separate option

aloof-angle-91616 · Answer

reading the thread the user said thanks for the help immediately after we arrived at the solution of add a `python` interpreter or symlink

aloof-angle-91616 · Answer

so i think the user is fine with this

enough-analyst-54434 · Answer

OK so documenting maybe that Pants requirement is not just python3 6+ but a sane python which installs a `python` binary in addition to `python3 6` etc is also a requirement

enough-analyst-54434 · Answer

I have never seen a python package that does not do this

aloof-angle-91616 · Answer

that is absolutely good enough for me

hundreds-father-404 · Answer

Cool Adding to the docs now

hundreds-father-404 · Answer

Thank you both

enough-analyst-54434 · Answer

I m not hard opposed to a knob but you can go too far

aloof-angle-91616 · Answer

i normally like more knobs to a fault but it feels like we re only adding complexity without any real need for a specific option for the python interpreter name

aloof-angle-91616 · Answer

i can easily see how different settings of that option in different pants toml files for example would cause silent breakages

aloof-angle-91616 · Answer

< hundreds father 404> thank you for volunteering to do the docs change i was going to offer to switch my previous offer from options to docs let me know if there s anything i can help clean up here

hundreds-father-404 · Answer

gt let me know if there s anything i can help clean up here A review would be appreciated Everything is done other than aws lambda coverage project introspection to describe things like `dependees` See <https pants readme io docs welcome to pants> for 1 28 x including a new page on codegen I wrote today

aloof-angle-91616 · Answer

^ ^ ^ ^ ^ ^ ^ ^

hundreds-father-404 · Answer

Okay how is <https pants readme io docs installing pants>

enough-analyst-54434 · Answer

Looks good The wrinkle that may not be clear is that the requirement is that some `python` is on the `PATH` and you also have `python3 6 9 ` on the `PATH` setup script needs this So we actually have a push pull If the user s image would have had just a `python` binary for python3 6 then the setup script would fail

enough-analyst-54434 · Answer

The some `python` on the path can be 2 7+

hundreds-father-404 · Answer

Okay updated again <https pants readme io docs installing pants> I left off `python3 9` because we will soon stop using `abi3` to unblock Stu s migration from `cffi gt cpython` and we won t initially build py39

enough-analyst-54434 · Answer

Looks more correct now Technically You must also have python on your `PATH` This can be Python 2 or Python 3 should be You must also have python on your `PATH` This can be Python 2 7+ or Python 3 5+ That said the only important thing is Python 3 6+ on the `PATH` and all the rest of the lanuage applies to weird situations How about de escalating the weird situation from the main flow of the section Something like To run Pants you need Linux or macOS Python 3 6+ discoverable on your `PATH` Note that your own code can use Python 2 this is only what Pants uses to run itself If you run into problems but have Python 3 6+ installed you may have a non standard python installation Pants expects to find both `python` and one of `python3 6` `python3 7` or `python3 8` on your PATH

hundreds-father-404 · Answer

I like it Thanks I ll try out using a warning block those are nice because you can quickly decide it s irrelevant to you

enough-analyst-54434 · Answer

Agreed