https://pantsbuild.org/ logo
Powered by
w

witty-crayon-22786

08/26/2021, 11:38 PM
TIL: python requirements are very cyclic. unconfirmed!
đź‘€ 1
h

hundreds-father-404

08/26/2021, 11:38 PM
what do you mean?
w

witty-crayon-22786

08/26/2021, 11:38 PM
pytest
depends on 
importlib-metadata
which depends on 
pytest
âž• 1
etc, etc, etc
(although i think that some of these are what other ecosystems would call “dev”/“test” dependencies…? not sure why they’re included)
h

hundreds-father-404

08/26/2021, 11:44 PM
What's the context? Ah, you're pex-per-wheel change?
w

witty-crayon-22786

08/26/2021, 11:49 PM
yea.
e

enough-analyst-54434

08/26/2021, 11:57 PM
They should be delineated by env markers. For example, run this:
Copy code
$ pex pytest -opytest.pex --include-tools
$ PEX_TOOLS=1 ./pytest.pex graph --open
And you'll see alot of gray rendered inactive nodes. lots of those have env marker 
extra = 'tests'
.
So, the delineation is there for good citizens.
So your graph code just needs to take that into account I think like the graph tool renderer does.
w

witty-crayon-22786

08/26/2021, 11:59 PM
…aha. it does.
ok. good to know.
e

enough-analyst-54434

08/27/2021, 12:01 AM
The pip hack diables all wheel tags and all env markers except the special extra marker and the python_version and python_full_version markers.
w

witty-crayon-22786

08/27/2021, 12:02 AM
@enough-analyst-54434: in general though, are cycles allowed? or would you only encounter them if you are ignoring env markers like i apparently have here, and you are with the pip hack?
e

enough-analyst-54434

08/27/2021, 12:02 AM
The extra marker is special in that its only ever injected - its not an intrinsic of the Python in play. Injection is pytest[tests] in a req.
The pip hack does not ignore extras for this reason!
w

witty-crayon-22786

08/27/2021, 12:03 AM
sorry, for which reason? to avoid introducing cycles?
or just because it’s an edge* property that must be applied for the graph to be valid?
…the latter thing, i think.
so that brings me back to the question: should i be expecting cycles in this graph once extras are properly applied?
e

enough-analyst-54434

08/27/2021, 12:06 AM
Yeah, ok - that was a lie - not cycles. I think Pip fails on cycles, but there is no spec for this, that's up to the resolver. The reason you don't include extras carte blanche though is if they were not actually requested, then you pull in resolved deps the user didn't ask for on any platform,
w

witty-crayon-22786

08/27/2021, 12:07 AM
yea, that makes sense.
e

enough-analyst-54434

08/27/2021, 12:07 AM
I honestly have no clue if you can expect cycles. You might make two local dists that depend on each other and throw them at pip to find out a non-hand wave answer.
đź‘Ť 1
âž• 1
w

witty-crayon-22786

08/27/2021, 12:08 AM
thanks.
e

enough-analyst-54434

08/27/2021, 12:08 AM
(don't throw them both at Pip though - throw one.
w

witty-crayon-22786

08/27/2021, 12:08 AM
yea.
e

enough-analyst-54434

08/27/2021, 12:09 AM
If you throw both you might trigger an edge where it allows it when it otherwise wouldn't.
w

witty-crayon-22786

08/27/2021, 12:09 AM
i’m going to stash the cycle handling stuff i have and see how much more progress i can make with properly filtered extras before worrying about it again. thanks!
🤞 1
…hm. so, while 
pex-tools … graph
does grey out nodes that are completely eliminated, it doesn’t grey-out/disable edges, afaict: it labels them with the 
extras=
information though. so i think that i would have to actually implement filtering/application of extras here to consume this graph (potentially) without cycles.
e.g. 
pytest
and 
attrs
are both active, but `attrs`’ edge to 
pytest
would be disabled by 
extra=="tests_no_zope"
:
e

enough-analyst-54434

08/27/2021, 4:04 PM
How is disabling a node different from disabling an edge? Is this just hard to get info since you're trying to parse .dot format? IOW: All the information you need is in the picture IIUC, therefore its in the .dot.
At any rate, the PEX_PATH thing turned out hard and the hacks are building fast. I think the 
--spread
format I proposed in the PEX_PATH bug is the way to go.
w

witty-crayon-22786

08/27/2021, 4:06 PM
How is disabling a node different from disabling an edge?
because if both the source and the dest are still active, even though one of the edges was disabled, you still have a cycle (as in the `attrs`/`pytest` example above)
e

enough-analyst-54434

08/27/2021, 4:09 PM
Ok.
w

witty-crayon-22786

08/27/2021, 4:13 PM
but yea, mostly agreed: re PEX_PATH. i think that it is stuck somewhere between acting like the Manifest ClassPath embedded in a runnable JAR, and acting like a runtime declared classpath for a series of loose JARs.
e

enough-analyst-54434

08/27/2021, 4:17 PM
It's like neither afaict since a jar is passive and a PEX is active. A PEX always resolves its platform specific deps internally 1st. Jars have no code to do this and the jvm does not do this.
A PEX is nothing like a jar except its zipped. Its much more like a custom classloader.
w

witty-crayon-22786

08/27/2021, 4:19 PM
it’s original usecase was to run directly from the ZIP though, which is supported without the custom classloader… in that regard it is similar. although it’s definitely diverged from that
e

enough-analyst-54434

08/27/2021, 4:19 PM
No!
Even directly from zip it has to custom classload.
w

witty-crayon-22786

08/27/2021, 4:19 PM
i thought that you said it acted like an egg?
e

enough-analyst-54434

08/27/2021, 4:19 PM
It has a ton of machinery to point sys.path at the internal .deps/<wheel>.whl
w

witty-crayon-22786

08/27/2021, 4:19 PM
mm.
e

enough-analyst-54434

08/27/2021, 4:20 PM
Yes, eggs themselves were embedded, but the python zipimport facility is not recursive.
đź‘Ť 1
So, on --spread. If you don't think that's crazy you can take a crack at a pex tool if you want. If not I can get to it later today / this weekend after I clear the Pex 2.1.46 issue for Pants to be able to use Pex next.
w

witty-crayon-22786

08/27/2021, 4:22 PM
not crazy: thanks for sketching it. there is one other detail i think, commenting.
but i also think that i am probably unblocked by what is already posted on https://github.com/pantsbuild/pants/pull/12675, so i’m not sure what the priority of the followup will be
e

enough-analyst-54434

08/27/2021, 4:31 PM
Ok, if you're unblocked I'll leave the issue as a feature and unwind my stack through to the Pex lock file work instead.
w

witty-crayon-22786

08/27/2021, 4:34 PM
thank you. i do think that https://github.com/pantsbuild/pex/issues/1423 might represent an opportunity to either clarify the semantics of the PEX_PATH (i.e., “each PEX on the PEX_PATH must be self contained, with all of its transitive deps embedded”) or to adjust resolve to walk the entire path to find transitive deps, simiilar to a classpath. but also not blocking
e

enough-analyst-54434

08/27/2021, 5:01 PM
K.
Powered by