Posting this here because slack feels more ephemeral and I could be missing something obvious. In the new python backend wishlist it was mentioned that https://github.com/pantsbuild/pants/pull/20531 could help avoid invalidating all consumers of a resolve. Reading through the issue it looks like this would be done by utilizing line numbers in a requirements.txt. This almost feels like a hack. Keeping your requirements sorted alphabetically* for example would potentially invalidate everything again. Instead, why not model third party dependencies directly? To be more concrete, suppose I have a repo with a resolve in the usual location, a

requirements.txt

with

scikit-learn~=1.4.2

and a module

forecast.py

that depends on scikit learn. For example,

$ pants dependencies --transitive src/python/forecast.py
3rdparty/python/default.lock:_python-default_lockfile
3rdparty/python/requirements.txt:reqs
3rdparty/python:reqs#scikit-learn

In a new python backend, could we remove the dependency on

requirements.txt:reqs

? Could we just depend on

3rdparty/python:reqs#scikit-learn

? If Pants modeled 3rd Party Requirements, we'd see that

scikit-learn

itself depends on

joblib

,

numpy

,

scipy

, and

threadpoolctl

(pip show scikit-learn) but Pants doesn't model that. Are there fundamental reasons this wouldn't work? Edit - Formatting - Also sorry if this is the wrong channel. I can move it over to general if that's more appropriate

(just a quick note on ephemeralism; we have full history here, including web-searchable) I think the way forward is to lean more into the lockfile data (using the locked version as input to detect change, for instance). Currently, the granularity is rooted in changes to files on disk, and for lockfiles and requirements.txt files, these clearly lumps much more together than what is ideal, but this is due to a lack of data on the targets (e.g. the missing 3rdparty version info) which could otherwise negate the need for the file level invalidation. The reason I think is mostly historical, as initially there where no lockfiles, only requirements, and these could be without constraints, and in order to err on the side of doing too much invalidates all targets from the same source file lacking the data to be able to pin point each (generated) target accurately. When you suggest to only depend on

3rdparty/python:reqs#scikit-learn

for instance, we would also need to know the version of that requirement, along with all its transitive dependencies (from the lockfile) in order to not have to also include the lockfile itself as a dependency. Agree that using line numbers is a bit of a hack in this case, but it takes us a little closer reducing some superfluous work, until we have proper modelling for this, perhaps something along the lines outlined above.

@broad-processor-92400 recently wrote up some thoughts at https://github.com/pex-tool/pex/issues/2411