@enough-analyst-54434: can

pex-tools … extract

be convinced to extract for multiple interpreters at once?

Ok. The answer, though, was no with the current code. You'd have to add a flag to the tool and do some code.

for posterity, my realization was that the

repository.pex

is not the issue (which should have been obvious as of Monday, but it took some time to sink in): rather, the fact that we computed so many overlapping subsets of it.

Yes, ~all pex tools build a PEX and, instead of running it, as it to resolve(). That means those tools all act on the interpreter you're running the tool with.

--pex-repository=.. --no-transitive $requirement

to build single-entry PEXes for all requirements in the closure of your root requirements is what i had been doing. have it mostly working, but am trying to see whether i need to special case

local_only

.

Oh right, forgot about the PEX_PATH hack. Yeah - that's a good way to experiment on this.

@enough-analyst-54434: if a particular requirement has no distribution, does that also mean it will never have dependencies rendered via

graph

? for example, for the

typing

module

but yea: 

graph

 being interpreter-specific makes sense at this juncture… because creating an interpreter/platform-oblivious graph is essentially what you have needed to patch 

pip

 to do, right?

No, these are at different levels. Here a PEX file contains a concrete set of dists. Those may satisfy one or more interpreters if any of the flags used to build the PEX admitted multiple pythons (

--python X --pythonY

,

--interpreter-constraint ...

,

--platform Z --platform W

, or any combination of those), then it can contain more than one graph. So the graph tool could be expanded to walk the root requirements using all the embdded dists and ignoring tags and env markers. There is no Pip involved in this process today as-is, nor in this feature expansion. Fundamentally, all pex tools are runtime tools. They work with a concrete PEX file. The pip thing you're referring to is a buildtime feature that requires pip and does not download all the dists, it just downloads 1 dist for each dist family (i.e.: 1 pytest dist, 1 requests dist, etc.), but it ignores tags and env markers at resolve time, which is different and can include more things than in a PEX. You don't care about those things though for your work. Your work works on realized PEX files and only wants to deal with the real dists contained therin. Concreteley, Given

pex --interpreter-constraint ">=3.7,<3.9" foo

ANd foo's dependencies are: "bar==1; python_version == '3.7'" "bar==2; python_version == '3.8'" "baz; sys_platform == 'darwin'" The PEX is built on Linux and that Linux machine has both a Python 3.7 and Python 3.8 on the PATH visible to it. The PEX file will contain one foo dist, and two bar dists. The graph tool is run against the PEX file also on Linux. The current graph tool will output "foo X -> bar 1" when run with Python 3.7 and "foo X -> bar 2" when run with Python 3.8. The graph tool could be expanded to output ~ "foo X -> ["bar 1; python_version == '3.7'", "bar 2; python_version == '3.8'"]". The pip feature will also include baz and its transitive dependencies.

got it: this all makes sense: thanks!

Correct. It will use env markers so it will prune to those.

yep, got it.

You'll know baz exists, since you ignore env markers, you just miss its dep graph.

yep. and

local_only

is only for a single interpreter anyway, and that’s where we need this most.