https://pantsbuild.org/ logo
#development
Title
# development
a

ancient-vegetable-10556

08/26/2022, 5:43 PM
Oook, I think this is enough PyOxidizer functionality to enable alpha-quality distribution. Feel free to take a look at the PR https://github.com/pantsbuild/pants/pull/16484 (but I suspect most of the review will need to be from @enough-analyst-54434 because
ox.py
is utter utter madness)
e

enough-analyst-54434

08/26/2022, 5:45 PM
What problem is that actually trying to solve? Is it that the Pants binary is now dual purpose and sometimes needs to be Pants and sometimes Python?
If so, could that just be inverted and run as Python by default but the pants wrapper script says PANTS_BE_PANTS=PLEASE ?
h

hundreds-father-404

08/26/2022, 5:46 PM
Is it that the Pants binary is now dual purpose and sometimes needs to be Pants and sometimes Python?
I know we said to do this project incrementally, but do we need to? We're on 2.15.0.dev0 today -- that's several releases to iron out kinks Imo, the important thing is that the
./pants
script can continue to support pre-2.15
a

ancient-vegetable-10556

08/26/2022, 5:46 PM
I’m answering the question
@enough-analyst-54434 the point here is that we can distribute the frozen binary without a Python interpreter, but we still need to be able to pull down plugin packages from PyPI without an actual Python interpreter hanging around.
The plugin bootstrapper in Pants uses
sys.executable
to find the python interpreter that Pants is running under, and it invokes
pex
with that interpreter. In a pyoxed binary, the way the Python interpreter is through the binary itself
and then, pex itself needs to be able to run pip, venv, and
python -c
and the only Python interpreter you can guarantee is the Pants binary
e

enough-analyst-54434

08/26/2022, 5:49 PM
I completely understand that the python is baked in the binary,
I'm asking if the main of the binary can be default just be python
a

ancient-vegetable-10556

08/26/2022, 5:49 PM
we do not have the Python command line parser available to us
I spoke with Nick Coghlan about that, and he’d be willing to help me deliver that for a future CPython release
e

enough-analyst-54434

08/26/2022, 5:50 PM
Cotcha.
Ok, so its a crippled Python.
Makes sense.
a

ancient-vegetable-10556

08/26/2022, 5:50 PM
(really productive discussion over the weekend. If the work comes out of it, it completely justifies the weekend away)
it’s a crippled Python that can support Pex downloading packages from Pip into a venv
e

enough-analyst-54434

08/26/2022, 5:51 PM
Yeah. From here as an outsider to this whle dev process, I assumed pyoxidizer could deliver a working python (in addition to the fact your rust code was in there and potentially owned main). That assumption was wrong.
a

ancient-vegetable-10556

08/26/2022, 5:52 PM
Yeah, the Python is fully workable except for not having access to the command line parser
e

enough-analyst-54434

08/26/2022, 5:53 PM
That sure is too bad, but thank you. Now oxy makes sense.
a

ancient-vegetable-10556

08/26/2022, 5:53 PM
Right. When I was speaking with Nick, who wrote
runpy
, he says that access to the raw
argv
would make a lot of sense, at a minimum
e

enough-analyst-54434

08/26/2022, 5:54 PM
This is great for not-python backends but disappointing for python since this means Pex still has the find 1 valid python interpreter to get bootstrapped with as a problem users will hit.
I was thinking we were guaranteed and we could always use the pyoxided binary as a valid python, guaranteed.
a

ancient-vegetable-10556

08/26/2022, 5:54 PM
It’s not so much of an issue, because Python development environments should have a Python available
e

enough-analyst-54434

08/26/2022, 5:55 PM
Well, history would disagree. Mac and Debian have been issues
Oh well, this is progress!
Ok. hopefully my 1 comment is easily swatted down or else easily leads to less insanity.
a

ancient-vegetable-10556

08/26/2022, 6:08 PM
it’s no worse than Pants is currently, if you have a working Python
Our JVM support is miles ahead in that regard
e

enough-analyst-54434

08/26/2022, 6:09 PM
Yeah, someone was nice enough to make downlaoding pre-packaged jvms easy IIUC!
Ok, afaict LGTM. I'm not sure how you figured it all out - that had to be gnarly! But it would be nice to drop the insane stuff / using the pyoxidized binary as a sys.executable for other Python processes on purpose at some point.
❤️ 1
a

ancient-vegetable-10556

08/26/2022, 6:30 PM
I’ll explain the necessity in the PR for the paper trail, but basically, I was logging the argv for every invocation of the pants binary to a file with a hardcoded path, so I could see all of the executions that Pex made
and then figuring out what each invocation actually did
e

enough-analyst-54434

08/26/2022, 6:37 PM
Ok. It would be good to also document why you don't just ditch that hardcoded use of sys.executable we specify elsewhere or else why ditching that didn't work, etc. Its basically alot of torture whose justification is not currently clear.
a

ancient-vegetable-10556

08/26/2022, 6:38 PM
Yup.
I’ve explained over here: https://github.com/pantsbuild/pants/pull/16484#discussion_r956334210 — if you think this belongs in-code too, I can add that.