Is it possible to have a Python "resolve" that you...
# general
f
Is it possible to have a Python "resolve" that you never lock? We have a bunch of faked dependencies because they are provided by RPM actually and we just lie to Pants to map e.g. the
python3-requests
RPM package to the
request
module in module mapping, and deal with this by never running commands that Pants wants to resolve. This has worked "fine" until I've started trying to add another resolve with a subset of our deps that points at actual PyPI packages. I had to add a dummy lockfile, but obviously Pants blows up if it tries to read that. Is there a way to configure a resolve to just not try to read a lockfile?
b
I don't think it's really possible today without using dependency excludes. I expect this relates heavily to having Pants work well with
conda
... which I can't seem to find the ticket for
I added the ability for plugins to eject dependencies during inference. You might be able to get it working in a plugin...
f
What do you mean eject?
b
https://github.com/pantsbuild/pants/blob/31dc8684acdb330dbf831609499d82e75b6d6d6e/src/python/pants/engine/target.py#L2633 The return of the
InferDependenciesRequest
union implementation has an
exclude
field. So, after all dependencies (explicit and inferred) are gathered, then Pants processes the excludes (explicit and inferred). So in a plugin you can say "Toss these addresses out if you ever find them". It's a hammer, but it might be useful for your case.
e
I think the issue here is that editing is done above the layer of Pex. Pex requires a complete resolve, and so if Pants reads the lockfile, Pex will choke today. I think this feature would need to be added to keep having lockfiles read: https://github.com/pantsbuild/pex/issues/2097. Otherwise, I think @flat-zoo-31952 is right, you'd need to not have Pex involved at all on the resolve read side, or else tell it to
--ignore-errors
.
f
I mean the use case for this I think implies un-Pex-able dependencies, since afaik pex can't install things from distro-provided libraries or conda environments. So we're talking about worlds where we're not putting everything in Pex and running things from the environment. When you say "Pex requires a complete resolve" you mean you need to give it exact versions, but you don't actually have to build a complete pex do you? I mean, if you give it fewer packages it won't complain. I think this is more what I mean here. Within a resolve, I'd need to say "some of this stuff doesn't actually come from standard python packaging." I don't think that dependency exclusion is what I'd want either.
e
No, so Pex has the metadata of each requirement. That metadata says what each requires in turn and so on. Pex requires that graph is complete. The feature I linked would allow you to pick nodes in the graph by project name, say
pandas
and pick zap or zap-transitive and that would do 2 things: 1. Ensure those deps don't go into built PEXes and 2 they aren't required at boot time; i.e. the current Pex demand for closure is lifted in this case. Does that make sense @flat-zoo-31952?
IOW 1 of 2 reasons Pex even exists is the desire for hermeticity. That requires closure since that means you won't be getting deps from the ambient environment. This new feature would allow you to break that fundamental assumption / requirement of Pex since ~13 years by picking nodes to zap.
f
So yeah I think I may have been looking at this the wrong way. I think we should stop lying to Pants about our faked
python_requirement
targets and come up with a way to infer deps from our environmentally provided deps.
I get that Pex exists for hermeticity but we use the RPM ecosystem extensively and that decision was made far before my time, and even if I don't like how hard it is to make that work deterministically, it's what I have to work with. So my default resolve is necessarily "take what the OS has to provide you", and we're continually trying to figure out how to keep that working with the Pants python backend without rewriting it.
I've avoided this problem up until by just not using the bits of Pants that ever need to build a PEX, and I think I can still do that as long as I don't actually try to resolve this faked default resolve I have