straight-action-8031803/29/2023, 6:01 PM
In Pants is there a way to inject a Python environment into a command like this? For example, perhaps I create a shell script that runs this command then Pants creates the Pex environment somehow?
broad-processor-9240003/29/2023, 7:14 PM
) or automatically as part of the build process?
pants run something
straight-action-8031803/29/2023, 7:15 PM
broad-processor-9240003/29/2023, 7:22 PM
supports a few different ways to specify what gets run, and that’ll construct the Python environment (I’m not in a position to dig up the relevant docs page); does that help?
In future (pants 2.16), the new
might be just the thing to run it automatically, even without a
Either way, you’ll need to add it as a dependency (maybe in its own resolve, if it’s just to use as a tool).
straight-action-8031803/29/2023, 8:21 PM
which seems to allow me to call the script entry for an arbitrary package
broad-processor-9240003/29/2023, 9:26 PM
is effectively acting as a wrapper for
in the sense that it needs to pull in the environment for
in addition to whatever
• I'd guess one can probably package the whole (composer and the script) into a single PEX binary, and then somehow convince it to run the appropriate file...
◦ maybe: https://www.pantsbuild.org/v2.16/docs/reference-pex_binary#codeargscode (in 2.16+), but I don't know how that would refer to something within the pex? can it be specified as an importable module rather than a file path?
• alternatively, can composer run a PEX directly? As in, package
into a PEX with its deps, and then let composer execute that (again might require composer to support executing importable things)
• building explicit PEXes just to execute them locally isn't great, though, so maybe there's an alternative ways
I'm at the limit of my pants knowledge, though, so someone else might have to jump in.
enough-analyst-5443403/29/2023, 9:37 PM
straight-action-8031803/29/2023, 9:58 PM
script is mainly to help with multiprocessing, so it spins up multiple processes for your ML training script to allow multigpu. I was able to create a Pex binary that targets the composer script, and run it successfully. However, trying to actually pass the training script it doesn’t seem to propagate the Python environment:
for example tells me that it’s missing
./pants run src:run_train -- src/train.py
even though I explicitly include that dep in the binary definition.
Is the issue that composer is creating another process which is then outside of the Pex environment when it forks?
enough-analyst-5443403/29/2023, 10:53 PM
target will net you the most compatible runtime environment to a vanilla venv:
You might try that.
straight-action-8031803/30/2023, 12:20 AM