https://pantsbuild.org/ logo
h

high-yak-85899

09/28/2022, 5:27 PM
I think I've asked this before, but when (if at all) does pants determine a lockfile needs to be updated?
h

happy-kitchen-89482

09/28/2022, 5:28 PM
If any of the inputs to the lockfile generator have changed. Typically this means either your requirements.txt (or equivalent), or your interpreter constraints
h

high-yak-85899

09/28/2022, 5:31 PM
hmm, that's what I thought it was. I have a scenario right now where I generated my lockfile with
pyscard<2.0.4
, then changed my requirements file to
pyscard>2.0.4
but I'm not getting any notice about my lockfile needing an update.
Is there some behavior I need to turn on to have it validate the lockfile?
I do see
pyscard<2.0.4
in the lockfile metadata
h

happy-kitchen-89482

09/28/2022, 5:33 PM
Hmm, which version of Pants?
h

high-yak-85899

09/28/2022, 5:34 PM
2.13.0
The only command I've run after changing
requirements.txt
is
./pants test
in case that's relevant.
Did a
./pants package
on something else and did not see different behavior just to make sure.
If I add a completely new entry to
requirements.txt
, I still don't see any invalidation happen (double checked I didn't change
[python].invalid_lockfile_behavior
)
Is it possible that the lockfile validation is only working with the subset of the lockfile/requirements needed for the goals I am running?
I think that's what's happening
h

hundreds-father-404

09/28/2022, 5:50 PM
Is it possible that the lockfile validation is only working with the subset of the lockfile/requirements needed for the goals I am running?
Ahhhh, yeah that is it!
h

high-yak-85899

09/28/2022, 5:51 PM
I'm kind of torn on what I'd like to happen. I guess at some point, you should be running something (in CI or otherwise) that teases out that behavior.
h

happy-kitchen-89482

09/28/2022, 5:53 PM
Oh, yeah, makes sense.
Interesting question what the right thing to do is
h

hundreds-father-404

09/28/2022, 5:55 PM
I think we wanted to avoid the performance implications of needing to consult
AllTargets
to discover what goes into the lockfile. But honestly, that concern may be overblown: we already use
AllTargets
for dep inference
h

high-yak-85899

09/28/2022, 5:55 PM
It also seems like this validation only worries about direct dependencies potentially? In this scenario, we're using
yubikey-manager
.
yubikey-manager
needs
pyscard>=1.9,<3
.
pyscard==2.0.4
has installation issues so we add an additional requirement of
pyscard>2.0.4
. Running my test that uses
yubikey-manager
does not trigger an invalid lockfile error
h

hundreds-father-404

09/28/2022, 5:56 PM
yep, only tracks direct requirements. we want to avoid making network calls so don't know what the transitive deps of your third-party requirements are
h

high-yak-85899

09/28/2022, 5:57 PM
Don't you have to know at some point if I am to build/run/test things?
h

hundreds-father-404

09/28/2022, 5:57 PM
those third-party deps get resolved when building
requirements.pex
, but that gets cached. and they're agnostic to Pants the whole time
h

high-yak-85899

09/28/2022, 6:01 PM
What would you recommend for maintaining visibility on something like that? We have lots of cases where packages we directly depend on specify something incorrectly for their transitive dependencies and we have to fix that by adding more entries to our
requirements.txt
.
My only thought is to supplement with a unit test that makes sure the entries in
requirements.txt
exactly match the metadata of my
constraints.txt
h

happy-kitchen-89482

09/28/2022, 7:00 PM
The transitive third-party deps are the output of the lockfile generation, and we should invalidate based on inputs, so I’m not sure I understand the question?
h

high-yak-85899

09/28/2022, 7:03 PM
So scenario is • I use package A • package A depends on some range of package B • that range of B actually has issues • I add a constraint in requirements.txt for package B (even though I don't import it directly)
This is, without a doubt, unavoidable.
There are multiple dependencies in our system that leave their transitive dependencies either too flexible or just verifiably wrong that I am left to either 1) find a different package to use or 2) add my own constraints in requirements.txt to force package resolution to choose a valid transitive dependency.
I think my scenario is a little easier than the general behavior because I keep all dependency needs in
requirements.txt
. There's no searching through all targets to find places where other
python_requirement
targets have been called out. So I can ask my system the simple question like "does the metadata in the lockfile exactly match requirements.txt"
h

happy-kitchen-89482

09/28/2022, 10:11 PM
So this is what pip’s constraints.txt files are for. @hundreds-father-404, do we plan to continue to support those as inputs to a lockfile (separate from our former janky use of them as sorta-kinda lockfiles, before we had proper lockfiles)?
👍 1
h

hundreds-father-404

09/28/2022, 10:53 PM
6 Views