Is it possible to get a pip friendly version of Pa...
# general
Is it possible to get a pip friendly version of Pant's lockfiles? When the lockfile is regenerated after a new input or package upgrade, it would be easier to do something along the lines of pants export_constraints && pip upgrade rather than pants export venv && copy/replace.
You can ask
to convert the lockfile to a
file. I forget the exact command, so you'll wanna poke around `pex`'s CLI. I also have a separate Pants resolve for my tools, and then declare a
for each tool, so I can run them in Pants. So for me the command would look like
./pants run 3rdparty/python/tools:pex -- <pex_args>
. Would recommend
... it would be easier to ...
@nice-florist-55958 is the difficulty slowness of the export afresh?
The Pex command is
pex3 lock export --python path/to/venvs/python my.lock
. Unfortunately, Pants sticks invalid json on the top of the Pex lock file (comments); so you need to script in removal of those comments 1st before passing the lock file to Pex.
Perhaps the easiest setup: 1. Run
./pants export
an initial time, then, in the exported venv
pip install pex
2. When updating the venv, from within the venv (which has pex3 installed)
pex3 lock export my.lock
- the right interpreter will be used automatically.
It seems to me Pants really should have a
./pants export --lockfiles
where it can export lockfiles with the headers it adds having been stripped so that you can then use the unadulterated lockfiles as you please without having to worry about Pantisms.
👍 3
Pants already has the code to do the stipping so that it can interact with the underlying lock generating tool! In other words Pants itself runs into this same problem internally.
@nice-florist-55958 if you're willing to file an issue; it would be great to get this right. One way we fall down is when we add features we don't really use. Pants doesn't really dogfood actually using exported venvs day to day, much less very large venvs. As such we can err shipping a feature that's not actually all that useful in practice. Your actual real experience / feedback using the feature helps correct us.
Yah exactly. The first time someone's env is setup, use pants export. But then need a fast way to automate updating that exported venv whenever the lock is regenerated, especially for very large ones. Will try out the above suggestions. Maybe whatever turns out to be the best option could be baked into options for generate-lockfiles or export goals?
There are old issues filed against Pants / Pex to support mutating a venv. That's really what's needed here. Those issues stem from venv mutation being much faster for Pip in practice let alone Pants and Pex.
@polite-garden-50641 FWIW you can make the PEX call using pants so it's reproducible and doesn't dirty system deps if you do the tools-resolve mentioned above
so I do this in CI, in a dedicated GH job to run pip audit, in my scenario, for this I want to avoid bootstrapping pants in that job just to run export, doing a pip install for pex,pip-audit and then running them is much faster and more light weight.
Screen Shot 2022-07-13 at 11.32.46 AM.png
In my case, I want other devs to run the command, so the portability of a single command is invaluable. It all depends on the use-case
agreed. but this is a CI job that devs never run locally...
Sorry haven't had time to get back on this, just chiming in w/ three more points that came to mind: 1. May be good to have a pip-sync (see pip-tools) like feature, in case people mutate the venv directly such that its resolve would diverges from Pants' 2. Can the lockfile for a
target be exported (for metadata compliance purposes) 3. Can the list of requirements for a
target be exported (same reason)? This list should include first-party distributions (that are now third party from the package user's perspective) and third-party distributions, but not transitive (see this for some discussion). (As an aside, I'll point out something I learned from testing: Pants will not complain about incompatible resolves if a
has dependencies w/ different resolves, so you can easily end up with a that has incompatible requirements -- understandably Pants does not attempt to build a lock for a
as that would probably be overkill, but it might want to check that targets all have a compatible resolve, or provide options for all behaviors?)