https://pantsbuild.org/ logo
Join Slack
Powered by
One more question. I've got a CI going and I'm cac...
# general
p
One more question. I've got a CI going and I'm caching the 
~/.cache/python/setup
directory but on each build I see the 
./pants
script re-downloading pants and re-installing the associated pip packages. Am I doing something wrong?
h
Not sure if that was a typo? should be 
~/.cache/pants/setup
but otherwise that should work. If not the typo, I'm wondering if the cache is being purged?
p
The BitBucket docs do say "We do not cache symlinks, so if your dependencies rely heavily on them caching may not be efficient." and looking in the 
setup
directory there are a bunch of symlinks like:
Copy code
$ ll ~/.cache/pants/setup/bootstrap-Linux-x86_64/
total 24K
lrwxrwxrwx 1 oliver oliver   75 Apr 28 15:56 2.3.1rc0_py38 -> /home/oliver/.cache/pants/setup/bootstrap-Linux-x86_64/pants.NDy4Tu/install
lrwxrwxrwx 1 oliver oliver   75 Apr 27 16:47 2.4.0_py38 -> /home/oliver/.cache/pants/setup/bootstrap-Linux-x86_64/pants.Ctsgqr/install
drwx------ 3 oliver oliver 4.0K Apr 27 16:47 pants.3HQg6s
drwx------ 3 oliver oliver 4.0K Apr 27 16:47 pants.Ctsgqr
drwx------ 3 oliver oliver 4.0K Apr 28 15:56 pants.NDy4Tu
lrwxrwxrwx 1 oliver oliver   85 Apr 27 16:47 virtualenv-16.4.3 -> /home/oliver/.cache/pants/setup/bootstrap-Linux-x86_64/pants.3HQg6s/virtualenv-16.4.3
so maybe that's the issue
oh yeah - that was a typo. The real cache value is correct.
I did a copy-paste from where pants reports it's instlaling things and then `grep`'d my pipeline config to make sure what I'm caching is correct.
h
Hmmm, that symlinks issue could be the thing.
h
Ah, yeah. You can alternatively install Pants as a Pex file https://www.pantsbuild.org/v2.5/docs/proxies#installing-pants There used to be some perf hit before 2.4.x iirc because it was running a Pex as a "zipapp", but now it's less noticeable because it uses the unzip mode
h
But I wonder, do they mean "we don't follow symlinks" or "we don't cache symlinks themselves"?
p
yeah - I can read that either way too. not sure which they mean.
h
Because those symlinks appear to all be within the same top-level directory that you're caching
Do you have a way to poke around in the cached dir? E.g., sshing into the CI container?
I'm not too familiar with bitbucket
p
I'm not 100% sure but I don't think I can get into the running CI build. Did a little looking through their docs and didn't see that but it might be possible.
The .pex file idea is good though. I could actually just add a build step where, on CI, I replace the 
./pants
script with the 
.pex
file and then cache that for subsequent runs.
oh! I can add a step to my build to look inside those caches and see what's there. That's the easy answer. Will do that.
➕ 1
It appears they do cache the symlinks themselves, just not what they point at:
Copy code
ls -lh ~/.cache/pants/setup/bootstrap-Linux-x86_64/
total 16K
lrwxrwxrwx. 1 companion companion   78 May  1 01:22 2.4.0_py38 -> /home/companion/.cache/pants/setup/bootstrap-Linux-x86_64/pants.ZqaOql/install
drwx------. 2 companion companion 4.0K May  1 01:22 pants.7MCJXg
drwx------. 2 companion companion 4.0K May  1 01:22 pants.ZqaOql
lrwxrwxrwx. 1 companion companion   88 May  1 01:22 virtualenv-16.4.3 -> /home/companion/.cache/pants/setup/bootstrap-Linux-x86_64/pants.7MCJXg/virtualenv-16.4.3
That was run immediately after my cache was pulled and before any other build steps.
But then the 1st build step seems not to have used what's in the cache:
Copy code
./pants test ::
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
100   122  100   122    0     0   1525      0 --:--:-- --:--:-- --:--:--  1525
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
100   281  100   281    0     0   1510      0 --:--:-- --:--:-- --:--:--  6108
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
100 3626k  100 3626k    0     0  14.2M      0 --:--:-- --:--:-- --:--:-- 14.2M
Using base prefix '/home/companion/.pyenv/versions/3.8.9'
New python executable in /home/companion/.cache/pants/setup/bootstrap-Linux-x86_64/pants.mwDM6G/install/bin/python3.8
Also creating executable in /home/companion/.cache/pants/setup/bootstrap-Linux-x86_64/pants.mwDM6G/install/bin/python
Installing setuptools, pip, wheel...
done.
Collecting pip
  Downloading <https://files.pythonhosted.org/packages/cd/6f/43037c7bcc8bd8ba7c9074256b1a11596daa15555808ec748048c1507f08/pip-21.1.1-py3-none-any.whl> (1.5MB)
Installing collected packages: pip
  Found existing installation: pip 19.0.3
    Uninstalling pip-19.0.3:
      Successfully uninstalled pip-19.0.3
...
I'm done for the day but will check back tomorrow in hopes that one of you have a brilliant idea for what's wrong and how to fix it. thanks again for all your help!
h
I think most of us will be logged off during the weekend, but happy to help on Monday! (trying to enjoy AZ outdoors before we get 100+ degree weather every day 🥲)
h
That's odd to me because isn't the symlink target in the thing you're caching anyway (
/home/companion/.cache
)?
p
yeah, I'd think so. But I didn't dig all the way in to see what all the other symlinks might be.
2 Views
Open in Slack
PreviousNext
Powered by