Hey all 👋 Are the files (

*.zip

,

*.tar.xz

, etc) downloaded during the

TemplatedExternalTool

setup cached somehow? If yes, where/how are those files cached?

They're in the lmdb store, under the hash name of the file

I see. Is the digest of the file also used during the cache lookup? In the case where I have the same file name but I've changed the version of the external tool, is pants gonna download the new file and invalidate the old one in the cache?

It'll probably keep both, as the download is an append-only cache

Okay, but I shouldn't have any problems having the same .zip file name for different versions of my external tool, right?

I'm not sure I understand what you mean, but it's based on content when it comes to the cache part.

I'm currently storing my binaries on S3, the final zip file have the same name as I have the version as folder, e.g.

<s3://my_bucket/my_external_tool/1.0.0/my_tool.zip>

, so in this case every time I download my external tool, it will download

my_tool.zip

, but the content can be different according to the version.

Ahh, interesting... It's probabyl worth looking at this to see if there is any pre-download optimization it tries to go through (e.g. if it conditionally tries not to download). Also running with

-ldebug

might provide some insight https://github.com/pantsbuild/pants/blob/50a4e75b69321f3a2d3cb110433144c7f586ae38/src/python/pants/core/util_rules/external_tool.py#L353

Got it, makes sense. Thank you very much for all the explanations.

👍 I'd be curious what you find out. My gut feel is that, if you're already providing a digest value as part of the external tool downloads, then Pants wouldn't try to re-download if that's already in cache - as in, that's the cache key of interest

Yeah, in that case I would have problems when eventually specifying a different external tool version, as the new file wouldn't be downloaded and the old one would be always used.

Well, we're generally expecting a known size and known hash for supply chain reasons, but it sounds like you want to bypass that?

Hmm, my case is exactly like this

shunit2

, so I guess my question is: if I specify a different shunit2 version, pants would download the newer version or use an already downloaded

shunit2

if present on the cache? I mean, I could change my tool publishing to append the version to the zip file just to be sure it will not cause any problems, but I was just wondering if pants is already handling that.

I haven't tested it out, but it should pick the version you specify, however you're specifying versions. For example, in your pants.toml, you can update the digest, or known_versions, or whatever your mechanism is

Got it. Thanks a lot for all the explanations, it really helped!

(Just confirming the earliest question: the cache is based on the content of the files, not their filenames. This means both: files with the same name but different content are correctly cached separately, and, files with different names and the same content are deduplicated in the cache (saving disk space).)