I think i'm running into more issues with pex http cache behavior. It seems the first time I work on a target in a day or two, some kind of cache miss is triggered, and I'm back to 5-10 min of downloading again (which I can see via the network indicators). I have

[python-setup].resolver_http_cache_ttl = 0

set. This is hard to reproduce because it only happens once or so in a given day, usually when i'm not expecting it. From previous logs, I suspect this is the remote TTL from the server. I'm not sure whether this is a vendored-pip thing, or a pex thing, but I really really don't think the HTTP cache TTL should apply to immutable resources like this. This is important to me because I'm often working with a slowish connection and dealing with

opencv

and

torch

which are just enormous packages that i really want to avoid downloading as much as possible.

As a work around, you could run a local pypi cache using

devpi

, https://www.devpi.net/. I run a local server with an index that passes through to our pypi server. This caches locally all the packages and only downloads to my mac when a new version is added to the server.

i've tried this but it doesn't seem to work for me

Later today, I'll get you more details on my setup. I basically followed the Quickstart tutorial. In addition to the default root index, I set up a "dev" index with our pypi server as the base. I then pointed my pip to the my local dev index. The whole server was pointed to a local directory on my laptop's hard drive.

Cool, this seems like a good workaround!

is this in a CI environment?

not in a CI environment, on my workstation

hm

i think so...it's hard to say because my requirements change fairly frequently

ok, got it.

i suspect pants is getting it right based on what i've seen, except that sometimes it seems to resolve

constraints.txt

differently for different goals, if that makes sense?

@flat-zoo-31952: it’s not tremendously practical, but you can set

--pex-verbosity

to enable logging in general

yeah i can do that until i learn more

@flat-zoo-31952: re: different goals: it needs to do an additional resolve to get the “tool” for the goal, but that shouldn’t invalidate the underlying resolve

i can look into this, i can also look back through some of the logs i gathered when digging through this before

resolver_http_cache_ttl

was added

the presence of the option changes the cache key unfortunately.

it appears that verbosity 5 was too much though 😄

but assuming that

~/.cache/pants/named_caches

contains relevant content, it should re-run quickly

yeah it took ~5 min

yea.

trying verbosity 3

i’m wrapping up https://github.com/pantsbuild/pants/issues/10847 and can then take a look at this.

yeah i'm looking for that, but i'm not seeing that in these logs...it seems to be using the cache...but still taking 4 min

there are at least two caches within pex: one is for downloaded artifacts (source distributions); and another is for built wheels

i understand, no pressure on you guys....this is hard to reproduce and analyze

totally.

however it's still better then when i cache miss on the

RUN pip install -r requirements.txt

in dockerfiles, so i'll try to keep that in perspective 😁

😅

This took about 4 min...seems like it is using the cache for most items, but i guess i don't really understand why it takes so long

@flat-zoo-31952: i don’t really have a great understanding of pex’s logging, but i’ll spend some time today to get that, and then see what sense i can make of that log.

That makes sense...and the instability of my requirements is likely to remain perpetually. This is why I think having some kind of intermediate form of resolution (like lockfiles) makes sense. But I'm probably going to need to do a better job of elaborating my use cases

Just rechecked my configs. I had set up my

dev

index with an import of all the wheels and other packages from our company pypi. The root one had

"mirror_url": "<https://pypi.org/simple/>",
        "mirror_web_url_fmt": "<https://pypi.org/project/{name}/>",

set up. It was the one that caches everything locally.

i spent some more time looking at this, and it really seems like there is inherent overhead in the resolve re-running, even with 100% http cache hits for wheels.

./pants devpi start

?

@jolly-midnight-72759: my hope is that we can fix any glaring performance issues here at a fundamental level such that people mostly don’t need to run proxies

I thought you had identified a fundamental step that could not be avoided or worked on.

haven’t given up yet, heh

Oh... I thought putting 0 meant to never expire the cache

A "very large number", afaik. Would check back in on the TTL github issue, but if you look at pip's behavior when max_age==0, it just disables the cache.

no need, i'll take your word for it

for posterity: https://github.com/pantsbuild/pex/blob/7509a895ac5f5f0f67daff31f3d8d3a6e75c9f30/pex/vendor/_vendored/pip/pip/_vendor/cachecontrol/controller.py#L134-L136

Both, I think...

ok, thanks