https://pantsbuild.org/ logo
#development
Title
# development
h

hundreds-father-404

05/05/2021, 2:58 PM
Hi, thoughts on how to handle this invalidation logic with the engine? For Pants to generate lockfiles, a requirement is that that users can manually change the file. This implies that Pants should not automatically rewrite the file ever because it would break that, and instead you need to manually run something like
./pants pin
However, we want to eagerly warn if we detect the input requirements have changed and the lockfile has not been changed, to suggest running again. How could we know when this is the case? Especially in a way that works without pantsd? And even immune to wiping lmdb store? My thinking is to hash all the input requirements and write the hash to the lockfile. Every run, we can compare the calculated hash with the hash saved in the lockfile
w

witty-crayon-22786

05/05/2021, 10:09 PM
That's one option... another might be to actually try to apply the requirements to the contents of the lockfile, and see whether they all match.
Which would have the advantage of "working" (not perfectly: wouldn't allow for detecting orphan lockfile entries) with a subset of requirements.
h

hundreds-father-404

05/05/2021, 10:11 PM
By doing something like running a subprocess, or static analysis?
w

witty-crayon-22786

05/05/2021, 10:13 PM
I just mean "does
requests>=0.0.7
match
requests==0.0.8
?"
Assuming that there is library code to do that.
h

hundreds-father-404

05/05/2021, 10:16 PM
Ah, so there we would only regenerate on certain changes, rather than always on changes. Imo, we don't need that: it's simpler to always attempt a regen if any of the inputs have changed, and let the regen possibly rewrite the file or not
I think this
hash
proposal is roughly what Poetry, Cargo, and friends are doing when they put a hash at the top of their lockfiles
w

witty-crayon-22786

05/05/2021, 10:16 PM
Yea, but that requires you to find all inputs, which you might not have in your fast path
Say you are building just one binary in the repo: finding all inputs/requirements means going and finding all other binaries.
h

hundreds-father-404

05/05/2021, 10:18 PM
It's easy/quick to have all inputs for tool lockfiles: read a subsystem option. You're right that that is trickier to do performantly for user lockfiles Although, that idea of
"does requests>=0.0.7 match requests==0.0.8?"
has the same problem, right? You still either way need the original inputs
w

witty-crayon-22786

05/05/2021, 10:19 PM
Only the ones relevant to a particular target/binary
Which are a subset of the whole resolve
h

hundreds-father-404

05/05/2021, 10:49 PM
Hm, tricky. In the revised proposal, I now suggest an option
[python-setup].lockfile_regeneration = {"generate", "error"}
. We could add an
ignore
? Purely intended for desktop builds for better performance, then use "error" for CI Also, pantsd would reduce the perf hit for this check