witty-crayon-2278608/25/2021, 9:32 PM
be convinced to extract for multiple interpreters at once?
pex-tools … extract
enough-analyst-5443408/26/2021, 4:34 PM
witty-crayon-2278608/26/2021, 5:20 PM
is not the issue (which should have been obvious as of Monday, but it took some time to sink in): rather, the fact that we computed so many overlapping subsets of it.
is ~50MB, but 94 overlapping subsets of that added up to 2.8GB =P
impacted by the interpreter used to extract? i.e., might different interpreters result in different graphs?
pex-tools _…_ graph
enough-analyst-5443408/26/2021, 6:01 PM
directory structure in the PEX file?
witty-crayon-2278608/26/2021, 6:08 PM
to build single-entry PEXes for all requirements in the closure of your root requirements is what i had been doing. have it mostly working, but am trying to see whether i need to special case
--pex-repository=.. --no-transitive $requirement
being interpreter-specific makes sense at this juncture… because creating an interpreter/platform-oblivious graph is essentially what you have needed to patch
to do, right?
Are you just manually extracting subsets of the zip by knowing abouti thought about doing this, but i figured that I’d also need to manipulate the PEX-INFO.directory structure in the PEX file?
enough-analyst-5443408/26/2021, 6:46 PM
witty-crayon-2278608/26/2021, 6:57 PM
? for example, for the
enough-analyst-5443408/26/2021, 7:10 PM
but yea:No, these are at different levels. Here a PEX file contains a concrete set of dists. Those may satisfy one or more interpreters if any of the flags used to build the PEX admitted multiple pythons (being interpreter-specific makes sense at this juncture… because creating an interpreter/platform-oblivious graph is essentially what you have needed to patch
graphto do, right?
--python X --pythonY
, or any combination of those), then it can contain more than one graph. So the graph tool could be expanded to walk the root requirements using all the embdded dists and ignoring tags and env markers. There is no Pip involved in this process today as-is, nor in this feature expansion. Fundamentally, all pex tools are runtime tools. They work with a concrete PEX file. The pip thing you're referring to is a buildtime feature that requires pip and does not download all the dists, it just downloads 1 dist for each dist family (i.e.: 1 pytest dist, 1 requests dist, etc.), but it ignores tags and env markers at resolve time, which is different and can include more things than in a PEX. You don't care about those things though for your work. Your work works on realized PEX files and only wants to deal with the real dists contained therin. Concreteley, Given
--platform Z --platform W
ANd foo's dependencies are: "bar==1; python_version == '3.7'" "bar==2; python_version == '3.8'" "baz; sys_platform == 'darwin'" The PEX is built on Linux and that Linux machine has both a Python 3.7 and Python 3.8 on the PATH visible to it. The PEX file will contain one foo dist, and two bar dists. The graph tool is run against the PEX file also on Linux. The current graph tool will output "foo X -> bar 1" when run with Python 3.7 and "foo X -> bar 2" when run with Python 3.8. The graph tool could be expanded to output ~ "foo X -> ["bar 1; python_version == '3.7'", "bar 2; python_version == '3.8'"]". The pip feature will also include baz and its transitive dependencies.
pex --interpreter-constraint ">=3.7,<3.9" foo
witty-crayon-2278608/26/2021, 7:12 PM
So the graph tool could be expanded to walk the root requirements using all the embdded dists and ignoring tags and env markers.this seems like it would be equivalent to walking the graph of https://github.com/pantsbuild/pants/blob/3319779e702f5ed616ec7f54d73159932f18fc4f/src/python/pants/backend/python/util_rules/pex.py#L909-L917 …?
deal with only one interpreter, and so that would mean that the
will only end up rendering info about distributions relevant to that interpreter? (i’ll try this out to confirm)
enough-analyst-5443408/26/2021, 7:16 PM
witty-crayon-2278608/26/2021, 7:18 PM
enough-analyst-5443408/26/2021, 7:19 PM
witty-crayon-2278608/26/2021, 7:20 PM
is only for a single interpreter anyway, and that’s where we need this most.