https://pantsbuild.org/ logo
#development
Title
# development
b

bitter-ability-32190

02/21/2023, 4:41 PM
Why does the
find_python
rule in
system_binaries
not just use
sys.executable
? For
pants
it'd be the standlone, for
./pants
it'd be whatever the script found.
I'm very confused by all the Pythons floating around and how they are resolved. This Python I don't quite understand 😕
e

enough-analyst-54434

02/21/2023, 4:58 PM
Well, the rule takes a PythonBootstrap - that defines legal places to look for Python. The sys.executable may not be legal. So that's the obvious thing. As to whether that restriction is important, I'm not sure. But it could be that the
./pants
bootstrap script finds a bogus Mac system interpreter that happens to work to bootstrap Pants but then would fail for running user code through Pex. That's my guess.
The graveyard of Mac system interpreter issues is full but always expanding.
😂 1
The
pants
sys.executable is certainly bogus for building sdists! So it helps save PBS bootstraps for getting used for things they should not be used for.
It really truly causes unending confusion that Pants rules are written in Python. I think its best to always pretend they're written in COBOL.
You cannot cheat and use the runtime.
b

bitter-ability-32190

02/21/2023, 5:04 PM
When is this used to run user code though? Wouldn't we still need to filter the possible Pythons down to the relevant ICs?
I'm guessing this is only used to run the pex command that runs to find ICs, which isn't user code
e

enough-analyst-54434

02/21/2023, 5:04 PM
Follow the breadcrumb, grep says 2 places ...
How do you know though that Pex interpreter discovery doesn;t require a fully functioning interpreter?
b

bitter-ability-32190

02/21/2023, 5:06 PM
Does it?
e

enough-analyst-54434

02/21/2023, 5:06 PM
b

bitter-ability-32190

02/21/2023, 5:06 PM
PBS is "fully-functioning" is it not? Sans sdists which I know PEX will never depend on
e

enough-analyst-54434

02/21/2023, 5:07 PM
It is not fully funtioning! Parts of distutils do not work in PBS. Basically, you can take it from here I think, just dig and make sure - but truly the COBOL thing is really the way to think about things. The runtime really should not be used - period.
b

bitter-ability-32190

02/21/2023, 5:07 PM
(The
./pants
issue remains however)
Ok I think I get it. The Mac example was helpful
e

enough-analyst-54434

02/21/2023, 5:28 PM
Ok - but the COBOL point should be even more helpful. Any use of the current rule runtime to do anything other than run rule code is not a good idea. It should never be used for a Process is, I think, always safe to say.
p

proud-dentist-22844

02/21/2023, 8:31 PM
Now I’m imagining BUILD files w/ COBOL divisions, sections, and paragraphs… oh boilerplate hell! (yeah, I know it’s an illustration to help in not thinking of BUILD as python but as a DSL)
e

enough-analyst-54434

02/21/2023, 8:37 PM
It's not actually. It's to help with not thinking about rule code as Python. It is Python, but it's not real Python, or should not be treated that way. More to the point, whatever the rule code language was, say COBOL, we should never try to run a COBOL Process using the Pants COBOL runtime. The Pants configuration language is another matter entirely. For that, pretend BUILD is toml.
p

proud-dentist-22844

02/21/2023, 8:40 PM
ah yes. The rules are more of a DSL as well.
I ran up against that quite a bit when first learning to write rules… And I purposely broke some of the pants conventions in a couple of places, using some pythonic code with comments explaining why it was ok to not use a Get at that point… 😛
b

bitter-ability-32190

02/21/2023, 8:41 PM
So in thinking about de-coupling Pants' reliance on system Python, theoretically would
pyoxy
work as a Python exe for pex's sake?
Aww hell, let's find out 😛
e

enough-analyst-54434

02/21/2023, 8:42 PM
PyOxy == PBS
b

bitter-ability-32190

02/21/2023, 8:43 PM
Oh I was gonna go even harder and use the one-file
e

enough-analyst-54434

02/21/2023, 8:44 PM
Pex does not currently guard its sys.executable - unlike how Pants must. IOW, Pex will use sys.executable to do building if sys.executable meets ICs, etc. And so that is trouble.
b

bitter-ability-32190

02/21/2023, 8:45 PM
Hmm not following. Put another way? This is just for the case that we're running a PEX and no ICs were baked in
e

enough-analyst-54434

02/21/2023, 8:46 PM
Running a PEX or the Pex PEX?
b

bitter-ability-32190

02/21/2023, 8:46 PM
Certainly the latter. I think sometimes the former. Oh damn I see
e

enough-analyst-54434

02/21/2023, 8:46 PM
I'd really slow way the heck down and try to understand things.
You can run an experiment, succeed, and in the end be wrong here.
b

bitter-ability-32190

02/21/2023, 8:51 PM
OK I'm relatively certain it is only the latter.
e

enough-analyst-54434

02/21/2023, 8:54 PM
Pex works with any Python; so you can just pyenv ~random, then run the Pex PEX with that; then find out the real Pythons you need, then pyenv those. You are fully correct and divorced from the runtime. You have potentially wasted the initial rando pyenv bootstrap if that interpreter is not used by user code. That, at least, gets you fully correct. Later you can figure out how to optimize the case where rando gets never used again.
b

bitter-ability-32190

02/21/2023, 8:55 PM
That's.... extremely fair in terms of piecemeal development
e

enough-analyst-54434

02/21/2023, 8:56 PM
But I would expend some effort towards thinking about if the optimization is even structurally possible. I myself am not sure.
b

bitter-ability-32190

02/21/2023, 8:56 PM
I suspect it is, albeit using some other python. I don't think it's possible solely using pyenv
PBS/`pyoxy` seem like prime candidates for the decoupling
e

enough-analyst-54434

02/21/2023, 8:57 PM
Doesn't Pants have modeling for ICs? So rule code could know, for example, Python 3.7 is in-range for user code?
b

bitter-ability-32190

02/21/2023, 8:58 PM
Yes it does, however not all rule code is running in the context of ICs.
e

enough-analyst-54434

02/21/2023, 8:58 PM
If not, that sounds like a bug.
b

bitter-ability-32190

02/21/2023, 9:00 PM
Not all rule code is user code
I did find an example where it isn't PEX-related though.
GunzipBinary
e

enough-analyst-54434

02/21/2023, 9:02 PM
So, there is either a default IC or one or more explicit ones, right?
b

bitter-ability-32190

02/21/2023, 9:02 PM
Not for
GunzipBinary
. That should be usable without enabling the Python backend 😐
e

enough-analyst-54434

02/21/2023, 9:02 PM
At any rate, I work poorly in your mode. I think you have enough nudges to spin on this.
b

bitter-ability-32190

02/21/2023, 9:03 PM
Yeah certainly. I think best path forward is user-code-via-pyenv. Then untangling the Pants internals
3 Views