Hello, a proposal to support per-tool lockfiles and a first draft of multiple lockfiles for user code. Unlike the current constraints.txt approach, this uses actual lockfiles with hashes and that must be fully comprehensive. Pants will generate them for you Feedback welcomed: https://docs.google.com/document/d/1bCYb0UQZx9a-9tAagydCN_z3826QRvz_3aVnXKSNTJw/edit#

I’m trying to digest that document. What’s the difference between a tool lockfile and a user lockfile? user lockfile seems to be what I’m familiar with - like

poetry export

of the constraints for the repository’s code. But a tool lockfile? Is that just the tool’s requirment? Would that include dependencies of the user’s code? Would that overlap with user lockfiles such that some tools include the user dependencies and others don’t (thinking of at least

pylint

)?

Thanks for taking a look! Exactly. I'm using user lockfile to refer to your code, whereas tool lockfile is specifically for the tools that Pants is running like black and Pytest. Some tools like Black and Flake8 don't care at all what your 3rd party dependencies are, whereas you're right that some like Pylint do need your third party dependencies ("user lockfile"). It's not totally clear to me yet what should happen in that situation - ideally we could use Pex so that the requirements for the tool don't need to be the same as the requirements for your own code, which I think is called "shadowing". I left that as a TODO, this is definitely only a first draft to check if I'm on the right track

Are you referring to "shading"?

Yeah I think I am referring to shading. Don't we need that? We merge tool pexes with user requirements and we don't shade. I think that makes us susceptible to conflicting requirements

It does, but shading is non-trivial

Hmm so then that's really tricky, the tool lockfiles cannot be completely self contained from user lockfiles...hm

Most of them can

MyPy, Pytest, and Pylint to start. The issue is that now the loose requirement will be solely used to generate the lockfile for the tool constraints, which means that even if the requirements is loose, the tool always end up with an exact pin. That exact pin must be compatible with the exact pin for the user lockfile

Well, for the most part this is only a problem for the transitive deps, as code tends not to depend on Pylint and MyPy.

Right. But the loose requirements aren't helpful because again, the tool lockfile will get pinned and always use an exact version Note that Poetry solve this with the idea of dev requirements. They use a single lockfile and include tools like Pytest and Flake8 in the dev section so that you get a single resolve but you don't bundle dev deps in your binary

Bundling dev deps is not a problem for us, so i don't see how that's relevant.

You think shading is a nonstarter?

No, I think it's a starter 🙂

We have experience with shading in Java, right?

Shading involves renaming all the symbols in a dist, and then renaming all references to those symbols.

So, the idea is that Pex would rewrite those dists when creating the pex, right? By parsing and rewriting all their code? Yeah, the only way I see tool lockfiles working is shading Without that, we have the status quo of confusing error messages where the user lockfile must be compatible with tool requirements - except even worse because tool uses pinned versions always now Mixing in tool locks into user locks sounds also confusing, including how the tool would know which of N user lockfiles to consume

Or, the tool lockfiles are only implemented for tools that can be cleanly isolated from user deps.

+1, but they need to be included into user lockfiles then if they can't be isolated, which also has some complications. We need to pin the tool's deps somehow for reproducible builds (This is a very real problem, several times in the past, Pants' users builds have broke because a transitive dep of a tool like Pytest made a breaking change overnight)

On my machine, I generally install black and flake8 with pipx so that they're isolated from everything else. pytest and pylint, however, need to be installed in the project virtualenv, so they're always dev dependencies (depending on the project, that means poetry dev deps, a

dev

extra, or something like

requirements-dev.txt