gt we should probably just re enable the pex cache and trust

Question

gt we should probably just re enable the pex cache and trust it to be concurrency safe We should probably take this to the new channel < C0105PY6BM5|performance> but we ve been discussing this proposal at Toolchain and think it s likely a good change to re enable At a minimum re enable when we know it s local execution Pex has sane caching afaict John put a lot of sweat into ensuring that

witty-crayon-22786 · Answer

yea

witty-crayon-22786 · Answer

ditto `coursier` for example

witty-crayon-22786 · Answer

it s not a universal truth but

hundreds-father-404 · Answer

MyPy is another example MyPy has very sane caching that the Dropbox engineers spent 2 years working on We ve found that ignoring that MyPy cache results in roughly 5 10x worse performance not formally benchmarked

witty-crayon-22786 · Answer

gt re enable when we know it s local execution i m not sure this differentiation is worth it in the remote case the cost of writing to the cache should be minimal and depending on the remoting implementation it might not be in docker which would allow it to actually share caches across invokes

hundreds-father-404 · Answer

The only gotcha is that Google RBE is read only That s why John originally turned off the Pex cache

hundreds-father-404 · Answer

But maybe we could use a simple heuristic or linux command to see if the FS is read only Only use the cache iff it s possible

witty-crayon-22786 · Answer

hm read only really

hundreds-father-404 · Answer

let me retrace that PR to confirm I remember seeing it last week when I checked this out

witty-crayon-22786 · Answer

funky

hundreds-father-404 · Answer

witty-crayon-22786 · Answer

Ah so not RBE My understanding is that in docker by default the homedir would be writable

witty-crayon-22786 · Answer

But what does default really mean

hundreds-father-404 · Answer

Do you think it would be better to implement In the Python rule allow passing extra args to the EPR iff running locally or Add some mechanism in the Python rule to inspect if the FS is readable then use that to conditionally create the EPR args

hundreds-father-404 · Answer

Neither sound great but re enabling the Pex cache would be an enormous win for V2 performance Resolving requirements is by far the biggest slowdown

enough-analyst-54434 · Answer

Google RBE is or was not read only which is bad Toolchain s remote exec env is though mod the chroot dir

enough-analyst-54434 · Answer

If we re going to support deemed safe caches we really need to plumb a hole into EPRs to allow for input digests demmed safe and not part of fingerprint

witty-crayon-22786 · Answer

well this gets to the point i initially raised then

witty-crayon-22786 · Answer

we have that actually had forgotten

enough-analyst-54434 · Answer

Ok

witty-crayon-22786 · Answer

with the downside that it actually computes the whole input digest which might not be the best for a cache

hundreds-father-404 · Answer

`unsafe local only files because we favor speed over correctness for this rule`

enough-analyst-54434 · Answer

This will be a problem for all resolvers afaict so a standard solution will be great

witty-crayon-22786 · Answer

and which gets to the point i was originally making about resolve per library

hundreds-breakfast-49010 · Answer

running locally seems like more of a fundamental operation as far as pants is concerned than checking if a file system is rw or not

happy-kitchen-89482 · Answer

Not sure I understand resolve per library Today e g when you run tests we resolve once for the dep closure What are you proposing

witty-crayon-22786 · Answer

resolving once per library literally

witty-crayon-22786 · Answer

that would have the effect of populating a cache of artifacts at each step

witty-crayon-22786 · Answer

which would grow as you moved up the graph

witty-crayon-22786 · Answer

such that when you were building a root most things would be in the cache

witty-crayon-22786 · Answer

since hitting the resolve cache is not massively faster than missing it due to overheads in pip mostly it s unlikely to be a win without refactorings

hundreds-father-404 · Answer

By library do you mean 3rd party distribution Then using PEX PATH to merge them all

witty-crayon-22786 · Answer

python library

hundreds-father-404 · Answer

Hm I don t follow What precisely do you mean by resolve here

witty-crayon-22786 · Answer

you would not be able to merge with pex path because the resolves would be different at each stage

witty-crayon-22786 · Answer

a pex pip resolve for the transitive dependencies of the target