https://pantsbuild.org/ logo
Join Slack
Powered by
TIL: python requirements are very cyclic. unconfir...
# development
w
TIL: python requirements are very cyclic. unconfirmed!
đź‘€ 1
h
what do you mean?
w
pytest
depends on 
importlib-metadata
which depends on 
pytest
âž• 1
etc, etc, etc
(although i think that some of these are what other ecosystems would call “dev”/“test” dependencies…? not sure why they’re included)
h
What's the context? Ah, you're pex-per-wheel change?
w
yea.
e
They should be delineated by env markers. For example, run this:
Copy code
$ pex pytest -opytest.pex --include-tools
$ PEX_TOOLS=1 ./pytest.pex graph --open
And you'll see alot of gray rendered inactive nodes. lots of those have env marker 
extra = 'tests'
.
So, the delineation is there for good citizens.
So your graph code just needs to take that into account I think like the graph tool renderer does.
w
…aha. it does.
ok. good to know.
e
The pip hack diables all wheel tags and all env markers except the special extra marker and the python_version and python_full_version markers.
w
@enough-analyst-54434: in general though, are cycles allowed? or would you only encounter them if you are ignoring env markers like i apparently have here, and you are with the pip hack?
e
The extra marker is special in that its only ever injected - its not an intrinsic of the Python in play. Injection is pytest[tests] in a req.
The pip hack does not ignore extras for this reason!
w
sorry, for which reason? to avoid introducing cycles?
or just because it’s an edge* property that must be applied for the graph to be valid?
…the latter thing, i think.
so that brings me back to the question: should i be expecting cycles in this graph once extras are properly applied?
e
Yeah, ok - that was a lie - not cycles. I think Pip fails on cycles, but there is no spec for this, that's up to the resolver. The reason you don't include extras carte blanche though is if they were not actually requested, then you pull in resolved deps the user didn't ask for on any platform,
w
yea, that makes sense.
e
I honestly have no clue if you can expect cycles. You might make two local dists that depend on each other and throw them at pip to find out a non-hand wave answer.
đź‘Ť 1
âž• 1
w
thanks.
e
(don't throw them both at Pip though - throw one.
w
yea.
e
If you throw both you might trigger an edge where it allows it when it otherwise wouldn't.
w
i’m going to stash the cycle handling stuff i have and see how much more progress i can make with properly filtered extras before worrying about it again. thanks!
🤞 1
…hm. so, while 
pex-tools … graph
does grey out nodes that are completely eliminated, it doesn’t grey-out/disable edges, afaict: it labels them with the 
extras=
information though. so i think that i would have to actually implement filtering/application of extras here to consume this graph (potentially) without cycles.
e.g. 
pytest
and 
attrs
are both active, but `attrs`’ edge to 
pytest
would be disabled by 
extra=="tests_no_zope"
:
e
How is disabling a node different from disabling an edge? Is this just hard to get info since you're trying to parse .dot format? IOW: All the information you need is in the picture IIUC, therefore its in the .dot.
At any rate, the PEX_PATH thing turned out hard and the hacks are building fast. I think the 
--spread
format I proposed in the PEX_PATH bug is the way to go.
w
How is disabling a node different from disabling an edge?
because if both the source and the dest are still active, even though one of the edges was disabled, you still have a cycle (as in the `attrs`/`pytest` example above)
e
Ok.
w
but yea, mostly agreed: re PEX_PATH. i think that it is stuck somewhere between acting like the Manifest ClassPath embedded in a runnable JAR, and acting like a runtime declared classpath for a series of loose JARs.
e
It's like neither afaict since a jar is passive and a PEX is active. A PEX always resolves its platform specific deps internally 1st. Jars have no code to do this and the jvm does not do this.
A PEX is nothing like a jar except its zipped. Its much more like a custom classloader.
w
it’s original usecase was to run directly from the ZIP though, which is supported without the custom classloader… in that regard it is similar. although it’s definitely diverged from that
e
No!
Even directly from zip it has to custom classload.
w
i thought that you said it acted like an egg?
e
It has a ton of machinery to point sys.path at the internal .deps/<wheel>.whl
w
mm.
e
Yes, eggs themselves were embedded, but the python zipimport facility is not recursive.
đź‘Ť 1
So, on --spread. If you don't think that's crazy you can take a crack at a pex tool if you want. If not I can get to it later today / this weekend after I clear the Pex 2.1.46 issue for Pants to be able to use Pex next.
w
not crazy: thanks for sketching it. there is one other detail i think, commenting.
but i also think that i am probably unblocked by what is already posted on https://github.com/pantsbuild/pants/pull/12675, so i’m not sure what the priority of the followup will be
e
Ok, if you're unblocked I'll leave the issue as a feature and unwind my stack through to the Pex lock file work instead.
w
thank you. i do think that https://github.com/pantsbuild/pex/issues/1423 might represent an opportunity to either clarify the semantics of the PEX_PATH (i.e., “each PEX on the PEX_PATH must be self contained, with all of its transitive deps embedded”) or to adjust resolve to walk the entire path to find transitive deps, simiilar to a classpath. but also not blocking
e
K.
Open in Slack
PreviousNext
Powered by