https://pantsbuild.org/ logo
p

plain-night-51324

01/04/2023, 6:55 PM
we have a python monorepo, and each repo has a requirements.txt. We hacked together a script to combine them into a single all-requirements.txt to serve as a pseudo lockfile and use pip to install that. with potential introduction of
./pants generate-lockfile
how do we improve the workflow? Convert it to poetry? (it doesn’t seem to support monorepo) Do I still need to go through this process? it seems pants does not support a --check option currently for lockfiles so the lockfile and my requirements could go out of date.
p

polite-garden-50641

01/04/2023, 7:06 PM
We use something simliar to:
Copy code
sed '/^\/\//d' <PATH-TO-PEX-LOCKFILE> > dist/pex_lock.json  # remove pants comments from lock file
python3 -m venv .venv && source .venv/bin/activate && pip install pex==2.1.114
PEX_TOOLS=1 .venv/bin/pex --lock dist/pex_lock.json --include-tools -- venv --pip --bin-path prepend --collisions-ok .venv
you can also run
pex3 lock export dist/pex_lock.json > dist/pip-lockfile.txt
to generate a pip compatible lockfile with versions and hashes. (based on the requirements.txt format, I forget the pep #)
p

plain-night-51324

01/04/2023, 9:24 PM
Thanks, would you mind sharing what happens when a dependency is added/changed?
p

polite-garden-50641

01/04/2023, 9:28 PM
In our workflow we re-generate the lockfile(s) when that happens. to keep the virutalenv up to date we have a script we use to run a python interpreter. this script basically checks if the venv needs to be rebuilt and does it if that is the case. It determines that by hashing the pex lock file and checking if it is the same as a hash stored in a file located in virtualenv directory (this script will generate that hash file when building the venv)
p

plain-night-51324

01/04/2023, 9:48 PM
ok, final question, what is the source for generating lockfiles? is that some combination of requirements.txt? or a poetry lock file?
h

happy-kitchen-89482

01/04/2023, 10:13 PM
Your
all-requirements.txt
could be the source for generating the lockfile.
Generally the input is a requirements.txt, which represents "the universe of allowed direct 3rdparty deps in this repo" (or at least "in this resolve").
Pants then takes the relevant subset in each scenario, so you won't actually use the entire lockfile each time
c

chilly-holiday-77415

01/05/2023, 9:52 AM
is the pants lockfile functionally similar to poetry’s? I’m trying to figure out when I’d choose to follow the suggestion of
poetry export
over
./generate-lockfile
- my assumption is that you retain poetry’s lockfile behaviour being manipulated by
poetry add/remove package
that Pants doesn’t care about - are there other differences?
e

enough-analyst-54434

01/05/2023, 3:37 PM
Pants does not have add/remove, it expects you can add and remove on your own. Otherwise Pants lockfiles can be treated as functionally identical to Poetry lock files.
c

chilly-holiday-77415

01/06/2023, 12:46 PM
so the Pants workflow for adding/removing is just adding a line to requirements (let’s say unpinned), Pants will generate a lockfile using that point in time latest version and use that to cut into pexes, and going forward will use the locked version - changing what gets locked afterwards would be a case of pinning in requirements to a different version? Apologies for rubber ducking, I’m trying to figure out whether to drop Poetry entirely and just making sure I grok the situation correctly without any prod pants experience - my feeling is that Poetry really only adds minor value in it’s helper behaviour on 3rd party management locking/adding/removing/upgrading, but it’s not something I want to make harder for devs 🙂 Likely a decision to defer!
h

happy-kitchen-89482

01/06/2023, 2:14 PM
Yep, everything you stated in your first paragraph is correct.
I think using pex lockfiles makes more sense than using poetry, at this point
It's better integrated, the lockfiles are (I think) more robust
But YMMV of course
The inputs to lockfile generation are requirements, which you can constrain however you like, and
generate-lockfiles
will turn those into a full transitive lockfile
m

modern-monkey-78364

01/10/2023, 7:18 AM
@happy-kitchen-89482 I have a doubt on this conversation that has been shared above. I am also planning to completely transition from poetry to only use pants. So that means no need of pyproject.toml and
Copy code
Does pants handle dependency validation automatically (i.e. making sure that dep_a is compatible with dep_b) or will we need to use pip explicitly for it to add to requirements.txt?
p

plain-night-51324

01/10/2023, 8:34 AM
right now i feel like generate-lockfiles target is very slow. Takes 2 minutes to tell me whats wrong, then takes another 2 minutes after i make some small chanrge. not sure how it is for everyone else
h

happy-kitchen-89482

01/10/2023, 10:48 AM
Which version of Pants are you on?
generate-lockfiles
delegates to pex, which delegates to pip, and by default to an older pip that may be slower. If you're on a Pants 2.16 dev release you can set
pip_version=22.3
under
[python]
and get newer pip, which may have relevant perf improvements
@modern-monkey-78364 I'm not sure I understand what you mean by "dependency validation"
Generating the lockfile will fail if your requirements can't be resolved
So you don't need to use pip explicitly
e

enough-analyst-54434

01/10/2023, 11:59 AM
Although Benjy is right and the Pip version can be used to speed up some resolves, the general speed issue is well known. It was probably incorrect to close https://github.com/pantsbuild/pants/issues/14127 without forking an issue for incremental resolves. The basic issue is Pants / Pex / Pip re-resolve from scratch every time you lock. Unlike with Poetry or pretty much any other Python tool using lock files, an existing venv is not mutated, which is what generally speeds this up.
h

happy-kitchen-89482

01/10/2023, 12:39 PM
True, we might need a new ticket for incremental lockfile generation. The need is starting to pile up.
c

chilly-holiday-77415

01/10/2023, 12:45 PM
potentially a bit controversial, given that it’s quite Python specific and Pants is a broad church, but I’d love to see a helper function that’s something along the lines of
./pants add-my-dep
./pants remove-my-dep
./pants bump-my-dep
, as it’s not difficult to add a helper script to do that once you grok that it’s nearly the same as what a Poetry user is used to, but the small gap feels much larger in terms of ergonomics than it really is: just to support your earlier thought Benjy: > I think using pex lockfiles makes more sense than using poetry, at this point
m

modern-monkey-78364

01/10/2023, 6:12 PM
@modern-monkey-78364 I’m not sure I understand what you mean by “dependency validation”
@happy-kitchen-89482 I mean sometimes in requirements.txt we have a dependency prometheus-client=0.14.1 and we want to add another dependency ray=2.2.0 to the project. Now ray 2.2.0 internally dependency upon prometheus-client < 0.14. So, if I currently add ray dependency right now with
poetry add ray
, it will fail and mention that ray version is not compatible with existing prometheus-client version. Wanted to understand how it is taken care by pants.
h

happy-kitchen-89482

01/10/2023, 7:00 PM
Yes, lockfile generation will fail on that kind of thing
p

plain-night-51324

01/10/2023, 7:01 PM
I think if it takes a few seconds to tell me it fail to generate that would be awesome. I literally spent multiple days on playing nice with lockfile generation.
Screenshot 2023-01-10 at 11.02.20 AM.png
m

modern-monkey-78364

01/10/2023, 7:14 PM
Yes, lockfile generation will fail on that kind of thing
Would it also tell why it failed? E.g. prometheus-client is incompatible with ray?
p

plain-night-51324

01/10/2023, 7:14 PM
Yes it would
the error is very clear
m

modern-monkey-78364

01/10/2023, 7:15 PM
Ah got it, thanks 🙂
h

happy-kitchen-89482

01/10/2023, 10:49 PM
It delegates to pip in the end, so the error messages are pip's
9 Views