https://pantsbuild.org/ logo
b

bored-art-40741

03/21/2021, 2:39 AM
OK, I think I've finally gotten stuck on how to do something in the new engine. I'm running
coursier fetch
, which returns me a report on which jars were fetched and the absolute paths where they were fetched. They're fetched into a subdir of
global_options.options.named_caches_dir
. But now I want to gather up only the jars that were relevant to this resolve into a
Digest
where the jars have been flattened out (or at least have the named cache prefix trimmed off). AFAICT the only way to do this would be with
PathGlobs
, but it only works on paths relative to the build root. Is there some other way to capture arbitrary files from known locations on disk into a
Digest
?
w

witty-crayon-22786

03/21/2021, 4:01 AM
Is there some other way to capture arbitrary files from known locations on disk into a 
Digest
?
the
named_caches
are only intended to be accessible from inside of
Process
runs: so running a process that copies the files out of the named_caches and into its own directory, where they can be captured using the
output_directories
or
output_files
of the
Process
would be the right way to go about it probably
the safest way to do that would be in a bash script wrapping your coursier invoke (to make it atomic with the fetching itself)… you could usually get away with doing it in a separate
Process
, but there is no guarantee that the first and second processes will run on the same machine
👍 1
@bored-art-40741: feel free to ping me if you have questions about this api!
b

bored-art-40741

03/21/2021, 5:23 PM
Yeah Tom (also, hey!), I was trying to get to the point of using
RemovePrefix
, but you need to get to a
Digest
first, which is where I was stuck.
Thanks Stu, that makes sense. I hadn't yet considered how this would play nicely with remoting, and clearly it wouldn't
I suppose I just trust that the CAKVS is smart enough to deduplicate individual jars that get slurped in from the named cache
w

witty-crayon-22786

03/21/2021, 6:03 PM
I suppose I just trust that the CAKVS is smart enough to deduplicate individual jars that get slurped in from the named cache
exactly: it ignores timestamps and filenames
so jars fetched off the network should very reliably be content addressed. repeatedly capturing out of the workdir of a
Process
is not a probelm.
Also: @bored-art-40741: heads up that John fixed a bug in named cache usage that was breaking in remote execution (in case you've reused any of the pex code): https://github.com/pantsbuild/pants/pull/11760
b

bored-art-40741

03/21/2021, 7:57 PM
Ah thanks, I did indeed base my code off of the changed PEX code