One more question I ve got a CI going and I m caching the `~

Question

One more question I ve got a CI going and I m caching the `~ cache python setup` directory but on each build I see the ` pants` script re downloading pants and re installing the associated pip packages Am I doing something wrong

hundreds-father-404 · Answer

Not sure if that was a typo should be `~ cache pants setup` but otherwise that should work If not the typo I m wondering if the cache is being purged

plain-carpet-73994 · Answer

The BitBucket docs do say We do not cache symlinks so if your dependencies rely heavily on them caching may not be efficient and looking in the `setup` directory there are a bunch of symlinks like ```$ ll ~ cache pants setup bootstrap Linux x86 64 total 24K lrwxrwxrwx 1 oliver oliver 75 Apr 28 15 56 2 3 1rc0 py38 gt home oliver cache pants setup bootstrap Linux x86 64 pants NDy4Tu install lrwxrwxrwx 1 oliver oliver 75 Apr 27 16 47 2 4 0 py38 gt home oliver cache pants setup bootstrap Linux x86 64 pants Ctsgqr install drwx 3 oliver oliver 4 0K Apr 27 16 47 pants 3HQg6s drwx 3 oliver oliver 4 0K Apr 27 16 47 pants Ctsgqr drwx 3 oliver oliver 4 0K Apr 28 15 56 pants NDy4Tu lrwxrwxrwx 1 oliver oliver 85 Apr 27 16 47 virtualenv 16 4 3 gt home oliver cache pants setup bootstrap Linux x86 64 pants 3HQg6s virtualenv 16 4 3``` so maybe that s the issue

plain-carpet-73994 · Answer

oh yeah that was a typo The real cache value is correct

plain-carpet-73994 · Answer

I did a copy paste from where pants reports it s instlaling things and then `grep` d my pipeline config to make sure what I m caching is correct

happy-kitchen-89482 · Answer

Hmmm that symlinks issue could be the thing

hundreds-father-404 · Answer

Ah yeah You can alternatively install Pants as a Pex file <https www pantsbuild org v2 5 docs proxies installing pants> There used to be some perf hit before 2 4 x iirc because it was running a Pex as a zipapp but now it s less noticeable because it uses the unzip mode

happy-kitchen-89482 · Answer

But I wonder do they mean we don t follow symlinks or we don t cache symlinks themselves

plain-carpet-73994 · Answer

yeah I can read that either way too not sure which they mean

happy-kitchen-89482 · Answer

Because those symlinks appear to all be within the same top level directory that you re caching

happy-kitchen-89482 · Answer

Do you have a way to poke around in the cached dir E g sshing into the CI container

happy-kitchen-89482 · Answer

I m not too familiar with bitbucket

plain-carpet-73994 · Answer

I m not 100 sure but I don t think I can get into the running CI build Did a little looking through their docs and didn t see that but it might be possible

plain-carpet-73994 · Answer

The pex file idea is good though I could actually just add a build step where on CI I replace the ` pants` script with the ` pex` file and then cache that for subsequent runs

plain-carpet-73994 · Answer

oh I can add a step to my build to look inside those caches and see what s there That s the easy answer Will do that

plain-carpet-73994 · Answer

It appears they do cache the symlinks themselves just not what they point at ```ls lh ~ cache pants setup bootstrap Linux x86 64 total 16K lrwxrwxrwx 1 companion companion 78 May 1 01 22 2 4 0 py38 gt home companion cache pants setup bootstrap Linux x86 64 pants ZqaOql install drwx 2 companion companion 4 0K May 1 01 22 pants 7MCJXg drwx 2 companion companion 4 0K May 1 01 22 pants ZqaOql lrwxrwxrwx 1 companion companion 88 May 1 01 22 virtualenv 16 4 3 gt home companion cache pants setup bootstrap Linux x86 64 pants 7MCJXg virtualenv 16 4 3```

plain-carpet-73994 · Answer

That was run immediately after my cache was pulled and before any other build steps

plain-carpet-73994 · Answer

But then the 1st build step seems not to have used what s in the cache ``` pants test Total Received Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 0 100 122 100 122 0 0 1525 0 1525 0 0 0 0 0 0 0 0 0 100 281 100 281 0 0 1510 0 6108 0 0 0 0 0 0 0 0 0 100 3626k 100 3626k 0 0 14 2M 0 14 2M Using base prefix home companion pyenv versions 3 8 9 New python executable in home companion cache pants setup bootstrap Linux x86 64 pants mwDM6G install bin python3 8 Also creating executable in home companion cache pants setup bootstrap Linux x86 64 pants mwDM6G install bin python Installing setuptools pip wheel done Collecting pip Downloading <https files pythonhosted org packages cd 6f 43037c7bcc8bd8ba7c9074256b1a11596daa15555808ec748048c1507f08 pip 21 1 1 py3 none any whl> 1 5MB Installing collected packages pip Found existing installation pip 19 0 3 Uninstalling pip 19 0 3 Successfully uninstalled pip 19 0 3 ```

plain-carpet-73994 · Answer

I m done for the day but will check back tomorrow in hopes that one of you have a brilliant idea for what s wrong and how to fix it thanks again for all your help

hundreds-father-404 · Answer

I think most of us will be logged off during the weekend but happy to help on Monday trying to enjoy AZ outdoors before we get 100+ degree weather every day smiling face with tear

happy-kitchen-89482 · Answer

That s odd to me because isn t the symlink target in the thing you re caching anyway ` home companion cache`

plain-carpet-73994 · Answer

yeah I d think so But I didn t dig all the way in to see what all the other symlinks might be