https://pantsbuild.org/ logo
#general
Title
# general
c

calm-artist-46894

04/21/2020, 9:12 PM
Cache (build everything but share cache cross build) vs Incremental Build (only build based on git change), which one is more favorable in case of monorepo? Related question, many of the intermediate result (e.g. pyprep, source, requirements) are not in the
.cache
folder (they can be found in
.pants.d
folder), hence even with remote cache setup, those step would still be re-evaluated. Is this by design?
a

aloof-angle-91616

04/21/2020, 9:14 PM
pants will mirror things from the remote cache into
.pants.d
.cache
is for things that apply across multiple pants projects on the same machine (e.g. downloaded tools like compilers)
.pants.d
contains a configurable number of most recent versions of each target, and rebuilds incrementally
c

calm-artist-46894

04/21/2020, 9:16 PM
I found 3 folders, without modifying any cache related config:
Copy code
~/.cache
~/myproject/.pants.d
~/myproject/.cache
I understand
~/.cache
is global, but not sure about
~/myproject/.cache
a

aloof-angle-91616

04/21/2020, 9:17 PM
i don't remember what
~/myproject/.cache
is for
looking for it in the code now
🙏 1
c

calm-artist-46894

04/21/2020, 9:18 PM
Also by remote cache, I am referring to this config https://www.pantsbuild.org/setup_repo.html#build-cache
Copy code
[cache]
read_from = ['<https://myserver.co/pantscache>']
write_to = ['<https://myserver.co/pantscache>']
I assume this determine where to write the content which by default goes to
~/myproject/.cache
a

aloof-angle-91616

04/21/2020, 9:20 PM
i believe it should actually pull things to versioned directories in
.pants.d
what are you seeing within
~/myproject/.cache
?
oh!
you might be right, sorry
yes, that's totally right, it looks like pants mirrors things from the remote cache to
~/myproject/.cache
thank you for being patient :)
c

calm-artist-46894

04/21/2020, 9:22 PM
at least for v1 (sorry I am trying to pull my repo and see what I got in the cache folder, did pants clean-all recently)
a

aloof-angle-91616

04/21/2020, 9:22 PM
yes yes
v2 uses a different mechanism that is not stored directly on the filesystem
no worries! i found it in the code you do not have to check your .cache dir anymore
c

calm-artist-46894

04/21/2020, 9:23 PM
.pants.d (I think I might have some v2 content there as well)
Copy code
zxiong@zxiong-dt-02:~/cruise/ark$ ll .pants.d/
total 128
drwx------  21 zxiong zxiong  4096 Apr 18 22:56 ./
drwxr-xr-x  23 zxiong zxiong  4096 Apr 21 14:22 ../
drwxr-xr-x   3 zxiong zxiong  4096 Apr 18 14:50 binary/
drwxr-xr-x   3 zxiong zxiong  4096 Apr 18 14:50 build_invalidator/
drwxr-xr-x   3 zxiong zxiong  4096 Apr 18 15:02 bundle/
drwxr-xr-x   3 zxiong zxiong  4096 Apr 18 14:50 compile/
drwxr-xr-x   3 zxiong zxiong  4096 Apr 18 15:02 docker/
drwxr-xr-x   3 zxiong zxiong  4096 Apr 18 15:27 gen/
drwxr-xr-x   3 zxiong zxiong  4096 Apr 18 15:02 go/
drwxr-xr-x   3 zxiong zxiong  4096 Apr 18 14:50 link/
drwxr-xr-x   4 zxiong zxiong  4096 Apr 18 14:50 native-compile/
drwxr-xr-x   2 zxiong zxiong 12288 Apr 21 14:21 .pids/
drwxr-xr-x   5 zxiong zxiong  4096 Apr 18 14:50 pyprep/
drwxr-xr-x   4 zxiong zxiong  4096 Apr 18 14:50 python-setup/
drwxrwxr-x   3 zxiong zxiong  4096 Apr 18 22:56 repl/
drwxr-xr-x 147 zxiong zxiong 20480 Apr 21 14:21 reports/
drwxr-xr-x   3 zxiong zxiong  4096 Apr 18 15:01 resolve/
drwxr-xr-x 147 zxiong zxiong 20480 Apr 21 14:21 run-tracker/
drwxr-xr-x   4 zxiong zxiong  4096 Apr 18 14:54 test/
drwx------   2 zxiong zxiong  4096 Apr 18 22:50 tmp13co_3j9/
drwxr-xr-x   3 zxiong zxiong  4096 Apr 18 14:50 unpack-wheels/
👍 1
.cache
Copy code
zxiong@zxiong-dt-02:~/cruise/ark$ ll .cache/
total 0
drwxr-xr-x 1 zxiong zxiong 0 Apr 18 13:54 0e740d14080eb85699af81fd93f5ee8c33b0f109/
drwxr-xr-x 1 zxiong zxiong 0 Apr 18 11:04 2168b65f4e09795c8f0b91092957e3ea7458b120/
drwxr-xr-x 1 zxiong zxiong 0 Apr 18 11:04 297e7e6534394d4006588918197a050607f3ce4c/
drwxr-xr-x 1 zxiong zxiong 0 Apr 18 11:04 362a4df2b153a25df10921e21cd809739d4486e0/
drwxr-xr-x 1 zxiong zxiong 0 Apr 18 11:19 3bd7ab7f4d03d6820806fe206f4459a9d74fb4e0/
drwxr-xr-x 1 zxiong zxiong 0 Apr 18 11:04 474944122ed9a458d19352e72ac38f4491a36454/
drwxr-xr-x 1 zxiong zxiong 0 Apr 18 11:04 4a6b32ec8ea45ea0ee797ba9add41a8bd46374f0/
drwxr-xr-x 1 zxiong zxiong 0 Apr 18 11:04 648e011928491dad6ce41468d77369faeff09222/
drwxr-xr-x 1 zxiong zxiong 0 Apr 18 11:04 7a66eae588f8a6e3807b313651454d6b2250701d/
drwxr-xr-x 1 zxiong zxiong 0 Apr 18 22:56 a87bf29b7d0d6cae531099b2d263ab76bd9e703f/
drwxr-xr-x 1 zxiong zxiong 0 Apr 18 14:19 baec2a8cd2a2002bd1881305348fe47cb9318904/
drwxr-xr-x 1 zxiong zxiong 0 Apr 18 15:41 ce26c36c6162c5ac34c9c3a962cde48fc4d067e5/
drwxr-xr-x 1 zxiong zxiong 0 Apr 18 15:15 d503aaff9f6e85f4c2d2ebaa059687e9f3d84bf0/
drwxr-xr-x 1 zxiong zxiong 0 Apr 18 13:52 e608031cd5a231c1e2958ca930898478c52bfc92/
drwxr-xr-x 1 zxiong zxiong 0 Apr 18 11:04 e719225457f5118d9dc7171e65c1930cf4f2f450/
drwxr-xr-x 1 zxiong zxiong 0 Apr 18 11:04 e80ec75db251a3dc367c3fd4d8aa29e452e7e056/
drwxr-xr-x 1 zxiong zxiong 0 Apr 18 16:43 f0f6b7aadc113555d88665139a9670ab42485e37/
a

aloof-angle-91616

04/21/2020, 9:24 PM
ok yes perfect,
.cache/
is fingerprinted
c

calm-artist-46894

04/21/2020, 9:24 PM
yes
a

aloof-angle-91616

04/21/2020, 9:24 PM
we have it configured a little differently at twitter because we have it used a shared buildcache to support multiple checkouts of our monorepo
c

calm-artist-46894

04/21/2020, 9:25 PM
.pants.d
too fingerprinted but it is deeper in the folder structure (I think)
a

aloof-angle-91616

04/21/2020, 9:25 PM
yes that's current
it also has "current" symlinks which point to the most recent version
note that the structure of
.pants.d
is not guaranteed to be stable
c

calm-artist-46894

04/21/2020, 9:26 PM
yes, read the fine print
✍️ 1
👖 1
a

aloof-angle-91616

04/21/2020, 9:27 PM
please let me know if i can clear anything else up
c

calm-artist-46894

04/21/2020, 9:28 PM
I was trying to setup a simple remote cache by pointing my
cache.write_to
and
cache.read_from
to a remote web storage. Given that the
.pants.d
content are not persisted and pulled cross build, now I wonder if worth spending time on this (another reason is we are trying pants v2 as well, which may not be done over night neither…)
a

aloof-angle-91616

04/21/2020, 9:28 PM
.cache
is persisted across builds, though!
or, it should be
c

calm-artist-46894

04/21/2020, 9:28 PM
.cache
 is persisted across builds, though!
it is
a

aloof-angle-91616

04/21/2020, 9:29 PM
i would also love to align with any timelines you may have regarding pants v2 support for any partixular workflow
c

calm-artist-46894

04/21/2020, 9:29 PM
There is discussion in #gettingstartedv2 and #cruise-folks
🔥 1
a

aloof-angle-91616

04/21/2020, 9:30 PM
ok wonderful
c

calm-artist-46894

04/21/2020, 9:32 PM
Generally, I am working on it but at meantime keep getting distracted (a different discussion between me an my manager I guess 😉 ) I identified several gaps and @hundreds-father-404 and @happy-kitchen-89482 had been helping me a lot. That’s said, the timeline depends on how fast we can get all features (we currently use in v1) to v2, and how I can align my time
a

aloof-angle-91616

04/21/2020, 9:33 PM
yes absolutely
if it revolves around support for jvm languages, c++, rust, or go, please loop me in because i have plans for those that i'd want an excuse to start
👍 1
h

happy-kitchen-89482

04/21/2020, 9:59 PM
To clarify, those caching instructions and
.pants.d
are v1 things. Local caching is baked deeply into the design of v2. You don't have to set anything up to get it.
Remote, shared caching is another matter.
We will have more info on that soon.
c

calm-artist-46894

04/21/2020, 10:00 PM
got it
4 Views