https://pantsbuild.org/ logo
Join Slack
Powered by
Hello, a proposal to support per-tool lockfiles an...
# general
h
Hello, a proposal to support per-tool lockfiles and a first draft of multiple lockfiles for user code. Unlike the current constraints.txt approach, this uses actual lockfiles with hashes and that must be fully comprehensive. Pants will generate them for you Feedback welcomed: https://docs.google.com/document/d/1bCYb0UQZx9a-9tAagydCN_z3826QRvz_3aVnXKSNTJw/edit#
The main part to focus on is "Modeling: per-tool lockfiles". "Modeling multiple user lockfiles" still needs more thought on how to specify what uses which target "Lockfile format" is sort of complete but missing how to actually generate it
cc @happy-kitchen-89482 @enough-analyst-54434
p
I’m trying to digest that document. What’s the difference between a tool lockfile and a user lockfile? user lockfile seems to be what I’m familiar with - like 
poetry export
of the constraints for the repository’s code. But a tool lockfile? Is that just the tool’s requirment? Would that include dependencies of the user’s code? Would that overlap with user lockfiles such that some tools include the user dependencies and others don’t (thinking of at least 
pylint
)?
h
Thanks for taking a look! Exactly. I'm using user lockfile to refer to your code, whereas tool lockfile is specifically for the tools that Pants is running like black and Pytest. Some tools like Black and Flake8 don't care at all what your 3rd party dependencies are, whereas you're right that some like Pylint do need your third party dependencies ("user lockfile"). It's not totally clear to me yet what should happen in that situation - ideally we could use Pex so that the requirements for the tool don't need to be the same as the requirements for your own code, which I think is called "shadowing". I left that as a TODO, this is definitely only a first draft to check if I'm on the right track
h
Are you referring to "shading"?
That's something pex does for its own dependencies (e.g., setuptools), so that they don't collide with whatever the user may want to bundle in the pex. I don't think that's directly applicable here.
h
Yeah I think I am referring to shading. Don't we need that? We merge tool pexes with user requirements and we don't shade. I think that makes us susceptible to conflicting requirements
h
It does, but shading is non-trivial
and we don't have general mechanism for it
pex can shade its own requirements
mostly because we happen to know that it works
but shading things generally is hard
because of references by name at runtime
for example
h
Hmm so then that's really tricky, the tool lockfiles cannot be completely self contained from user lockfiles...hm
h
Most of them can
I think
which tools need to run in the same venv as user deps?
mypy, I guess
We can try and run pex's shading code on those tools and see if they still work...
I would hope/expect that such tools are good about not having requirements, for exactly this reason
Other than maybe "setuptools at any recent version", stuff like that
h
MyPy, Pytest, and Pylint to start. The issue is that now the loose requirement will be solely used to generate the lockfile for the tool constraints, which means that even if the requirements is loose, the tool always end up with an exact pin. That exact pin must be compatible with the exact pin for the user lockfile
h
Well, for the most part this is only a problem for the transitive deps, as code tends not to depend on Pylint and MyPy.
Pytest is tricky though
since code does depend on it
tests import from it
it is the runner as well as the API
But yeah, tricky tricky
MyPy for example does have some requirements that your code might also depend on
such as typed-ast
and toml (albeit a loose req at any version)
maybe separate per-tool lockfiles are simply not feasible
in which case we have to take tools into account in whatever method we provide to users to generate their lockfiles
gah
h
Right. But the loose requirements aren't helpful because again, the tool lockfile will get pinned and always use an exact version Note that Poetry solve this with the idea of dev requirements. They use a single lockfile and include tools like Pytest and Flake8 in the dev section so that you get a single resolve but you don't bundle dev deps in your binary
h
Bundling dev deps is not a problem for us, so i don't see how that's relevant.
And my point is that tool lockfiles may not be feasible
at least for tools that expect to run in a venv with the code's own requirements
and for those tools we may not be able to do better than ensuring that they are included when resolving the user lockfiles
Right now, as far as I can tell, if your code depends on typed-ast>=1.5.0 for example, then you simply cannot use mypy...
h
You think shading is a nonstarter?
h
No, I think it's a starter 🙂
but it's far from obvious that it's a finisher
we can try and see what we get
it would be pretty epic if it works in practice
but, be clear about the complications
h
We have experience with shading in Java, right?
h
Shading involves renaming all the symbols in a dist, and then renaming all references to those symbols.
❗ 1
There are many ways this can go wrong
especially if there is any runtime loading by name, which for example mypy does a lot of
we have experience with shading in Python, but specifically for Pex's requirements
it's not a general purpose mechanism, although it could be made into one
For example, mypy depends on 
typing_extensions
, and looks up type info from that by name
So...
you can see the challenges
If a tool was designed to be run in the user's venv, we may have no choice but to do that
Ugh
h
So, the idea is that Pex would rewrite those dists when creating the pex, right? By parsing and rewriting all their code? Yeah, the only way I see tool lockfiles working is shading Without that, we have the status quo of confusing error messages where the user lockfile must be compatible with tool requirements - except even worse because tool uses pinned versions always now Mixing in tool locks into user locks sounds also confusing, including how the tool would know which of N user lockfiles to consume
p
Or, the tool lockfiles are only implemented for tools that can be cleanly isolated from user deps.
h
+1, but they need to be included into user lockfiles then if they can't be isolated, which also has some complications. We need to pin the tool's deps somehow for reproducible builds (This is a very real problem, several times in the past, Pants' users builds have broke because a transitive dep of a tool like Pytest made a breaking change overnight)
p
On my machine, I generally install black and flake8 with pipx so that they're isolated from everything else. pytest and pylint, however, need to be installed in the project virtualenv, so they're always dev dependencies (depending on the project, that means poetry dev deps, a 
dev
extra, or something like 
requirements-dev.txt
Yes. The lockfiles need to include all the transitive deps, including of the dev deps.
4 Views
Open in Slack
PreviousNext
Powered by