we have a python monorepo, and each repo has a requirements.txt. We hacked together a script to combine them into a single all-requirements.txt to serve as a pseudo lockfile and use pip to install that. with potential introduction of

./pants generate-lockfile

how do we improve the workflow? Convert it to poetry? (it doesn’t seem to support monorepo) Do I still need to go through this process? it seems pants does not support a --check option currently for lockfiles so the lockfile and my requirements could go out of date.

We use something simliar to:

sed '/^\/\//d' <PATH-TO-PEX-LOCKFILE> > dist/pex_lock.json  # remove pants comments from lock file
python3 -m venv .venv && source .venv/bin/activate && pip install pex==2.1.114
PEX_TOOLS=1 .venv/bin/pex --lock dist/pex_lock.json --include-tools -- venv --pip --bin-path prepend --collisions-ok .venv

Thanks, would you mind sharing what happens when a dependency is added/changed?

In our workflow we re-generate the lockfile(s) when that happens. to keep the virutalenv up to date we have a script we use to run a python interpreter. this script basically checks if the venv needs to be rebuilt and does it if that is the case. It determines that by hashing the pex lock file and checking if it is the same as a hash stored in a file located in virtualenv directory (this script will generate that hash file when building the venv)

ok, final question, what is the source for generating lockfiles? is that some combination of requirements.txt? or a poetry lock file?

Your

all-requirements.txt

could be the source for generating the lockfile.

is the pants lockfile functionally similar to poetry’s? I’m trying to figure out when I’d choose to follow the suggestion of

poetry export

over

./generate-lockfile

- my assumption is that you retain poetry’s lockfile behaviour being manipulated by

poetry add/remove package

that Pants doesn’t care about - are there other differences?

Pants does not have add/remove, it expects you can add and remove on your own. Otherwise Pants lockfiles can be treated as functionally identical to Poetry lock files.

so the Pants workflow for adding/removing is just adding a line to requirements (let’s say unpinned), Pants will generate a lockfile using that point in time latest version and use that to cut into pexes, and going forward will use the locked version - changing what gets locked afterwards would be a case of pinning in requirements to a different version? Apologies for rubber ducking, I’m trying to figure out whether to drop Poetry entirely and just making sure I grok the situation correctly without any prod pants experience - my feeling is that Poetry really only adds minor value in it’s helper behaviour on 3rd party management locking/adding/removing/upgrading, but it’s not something I want to make harder for devs 🙂 Likely a decision to defer!

Yep, everything you stated in your first paragraph is correct.

@happy-kitchen-89482 I have a doubt on this conversation that has been shared above. I am also planning to completely transition from poetry to only use pants. So that means no need of pyproject.toml and

Does pants handle dependency validation automatically (i.e. making sure that dep_a is compatible with dep_b) or will we need to use pip explicitly for it to add to requirements.txt?

right now i feel like generate-lockfiles target is very slow. Takes 2 minutes to tell me whats wrong, then takes another 2 minutes after i make some small chanrge. not sure how it is for everyone else

Which version of Pants are you on?

Although Benjy is right and the Pip version can be used to speed up some resolves, the general speed issue is well known. It was probably incorrect to close https://github.com/pantsbuild/pants/issues/14127 without forking an issue for incremental resolves. The basic issue is Pants / Pex / Pip re-resolve from scratch every time you lock. Unlike with Poetry or pretty much any other Python tool using lock files, an existing venv is not mutated, which is what generally speeds this up.

True, we might need a new ticket for incremental lockfile generation. The need is starting to pile up.

potentially a bit controversial, given that it’s quite Python specific and Pants is a broad church, but I’d love to see a helper function that’s something along the lines of

./pants add-my-dep

./pants remove-my-dep

./pants bump-my-dep

, as it’s not difficult to add a helper script to do that once you grok that it’s nearly the same as what a Poetry user is used to, but the small gap feels much larger in terms of ergonomics than it really is: just to support your earlier thought Benjy: > I think using pex lockfiles makes more sense than using poetry, at this point

@modern-monkey-78364 I’m not sure I understand what you mean by “dependency validation”

@happy-kitchen-89482 I mean sometimes in requirements.txt we have a dependency prometheus-client=0.14.1 and we want to add another dependency ray=2.2.0 to the project. Now ray 2.2.0 internally dependency upon prometheus-client < 0.14. So, if I currently add ray dependency right now with

poetry add ray

, it will fail and mention that ray version is not compatible with existing prometheus-client version. Wanted to understand how it is taken care by pants.

Yes, lockfile generation will fail on that kind of thing

I think if it takes a few seconds to tell me it fail to generate that would be awesome. I literally spent multiple days on playing nice with lockfile generation.

Yes, lockfile generation will fail on that kind of thing

Would it also tell why it failed? E.g. prometheus-client is incompatible with ray?

Yes it would

Ah got it, thanks 🙂

It delegates to pip in the end, so the error messages are pip's