https://pantsbuild.org/ logo
Powered by
#development
Title
# development
a

ancient-vegetable-10556

12/08/2021, 9:47 PM
@witty-crayon-22786 At the moment it looks like we do all of our coursier fetches on the consumption side entirely from data in the serialized lockfile; I’m trying to get an idea of the extent to which I’m going to need to remodel things for local files
👍 1
@witty-crayon-22786 @fast-nail-55400 In 
fetch_with_coursier
, it’s clear that the fetch request is for a 
jvm_artifact
, but we call cs itself with details from the lockfile itself. I’m considering whether I should swap out the lockfile entries with declared artifacts at this point (if they have a URL or JAR declared). Is there any reason why I shouldn’t be doing that?
(the problem here is that fetching based on the lockfile doesn’t play well with using local files)
w

witty-crayon-22786

12/08/2021, 10:58 PM
i think that that should always be safe to do, because the request has a 
CoursierResolveKey
(iirc?) and we don’t allow duplicates of a coordinate as input?
…would want to confirm the latter thing, but i think that coursier blows up if you give it the same input twice
a

ancient-vegetable-10556

12/08/2021, 10:59 PM
I wouldn’t be giving it the same input twice
w

witty-crayon-22786

12/08/2021, 10:59 PM
i know: sorry, i meant that within a resolve, a coordinate is unique
so yes, i think that it is safe to do that
a

ancient-vegetable-10556

12/08/2021, 11:00 PM
so the idea would be to compare the 
CoursierLockfileEntries
with URL- or JAR- specifying artifacts, check if their resolves are compatible, and use the artifact info instead of the lockfile info in that case?
w

witty-crayon-22786

12/08/2021, 11:01 PM
their resolves should be compatible already i think:
because the request has a 
CoursierResolveKey
 (iirc?)
a

ancient-vegetable-10556

12/08/2021, 11:01 PM
Can we specify multiple artifacts with the same coordinates under different resolves?
w

witty-crayon-22786

12/08/2021, 11:02 PM
we can.
because the request has a 
CoursierResolveKey
 (iirc?)
but yes: this is true. 
CoursierFetchRequest
extends 
ClasspathEntryRequest
which has a resolve field
a

ancient-vegetable-10556

12/08/2021, 11:03 PM
Right, so I’ve got the resolve field, but I’d need to pull in the artifacts that are associated with that resolve key, yeah?
w

witty-crayon-22786

12/08/2021, 11:03 PM
need to load the lockfile, yea
a

ancient-vegetable-10556

12/08/2021, 11:03 PM
The lockfile is already loaded
I specifically need the 
jvm_artifacts
, because the lockfile is not Pants-aware
w

witty-crayon-22786

12/08/2021, 11:04 PM
that’s also part of the 
request
though, right?
(it is)
a

ancient-vegetable-10556

12/08/2021, 11:04 PM
The root artifact is, yes, but not its dependencies
w

witty-crayon-22786

12/08/2021, 11:04 PM
ahhh.
a

ancient-vegetable-10556

12/08/2021, 11:04 PM
and if the dependency is a 
jvm_artifact
, I need the 
jvm_artifact
so that I can yoink the file appropriately
w

witty-crayon-22786

12/08/2021, 11:05 PM
hm. but this is an intransitive resolve
…oh. shit. it’s not, is it.
a

ancient-vegetable-10556

12/08/2021, 11:05 PM
Nope
w

witty-crayon-22786

12/08/2021, 11:05 PM
argh.
oh, wait though.
…hm. dang.
a

ancient-vegetable-10556

12/08/2021, 11:06 PM
there is nothing good here 🙂
w

witty-crayon-22786

12/08/2021, 11:06 PM
yea.
a

ancient-vegetable-10556

12/08/2021, 11:07 PM
So I need to be able to request all of the artifacts that match a given resolve, and sub those in if their coordinates are a dependency
w

witty-crayon-22786

12/08/2021, 11:07 PM
so either the lockfile bakes that information in, or https://pantsbuild.slack.com/archives/C0D7TNJHL/p1639004580226000?thread_ts=1639000059.221600&cid=C0D7TNJHL … which is a bummer
so
one thing to consider here:
a

ancient-vegetable-10556

12/08/2021, 11:09 PM
Storing it in the lockfile is awkward, because it would need to be able to be handled by a 
SourceFilesRequest
w

witty-crayon-22786

12/08/2021, 11:09 PM
i’m not certain that “fetch individual artifact via a new process” is strictly necessary
a

ancient-vegetable-10556

12/08/2021, 11:09 PM
No, but I’d still need to assemble a coursier command line of some sort
which would need to know whether to fetch by URL or local file
the 
fetch_one_coord
thing makes for very fine-grained caching with minimal fuss, but you’re right, it’s not strictly necessary
w

witty-crayon-22786

12/08/2021, 11:11 PM
if the resolve that generates the report also split an output Snapshot into new Digests per file, then 
fetch_one_coord
could go away, and we could look up the 
Digest
to use in the report dataclass
i’m not sure what the original idea behind 
coursier_resolve_lockfile
running a 
fetch
but then not actually capturing the outputs was
… because at the point when it does that, it already has everything it needs to load it later
a

ancient-vegetable-10556

12/08/2021, 11:13 PM
coursier_resolve_lockfile
writes the lockfile to disk, and that’s infrequent
most people shouldn’t be resolving their lockfile
so even if we optimised that, it wouldn’t fix the general use case, which is just fetching
w

witty-crayon-22786

12/08/2021, 11:14 PM
hm: if we’re talking about 
coursier_fetch.coursier_resolve_lockfile
, it doesn’t write it to disk: just loads it into memory after running the fetch
a

ancient-vegetable-10556

12/08/2021, 11:15 PM
Right, that’s the task that produces the actual file file
I should know, I just refactored it significantly
w

witty-crayon-22786

12/08/2021, 11:15 PM
k
a

ancient-vegetable-10556

12/08/2021, 11:15 PM
coursier_resolve_lockfile
is not run when fetching
that produces the contents of the file that is loaded when fetching
w

witty-crayon-22786

12/08/2021, 11:15 PM
Copy code
description=(
                "Running `coursier fetch` against "
                f"{pluralize(len(artifact_requirements), 'requirement')}: "
                f"{', '.join(req.to_coord_str() for req in artifact_requirements)}"
            ),
a

ancient-vegetable-10556

12/08/2021, 11:16 PM
@witty-crayon-22786 Yeah, it uses 
coursier fetch
to resolve the lockfile. Apparently it’s a necessity?
The only thing retained is the report, which is used to verify file hashes when fetching the artifacts at consumption time
w

witty-crayon-22786

12/08/2021, 11:17 PM
it’s creating the lockfile in memory
a

ancient-vegetable-10556

12/08/2021, 11:17 PM
Yes, and it gets written to disk
it’s one rule in the process that writes the lockfile to disk, that’s the only time it gets called
w

witty-crayon-22786

12/08/2021, 11:18 PM
ok, i see what you are saying.
and yea, although we have everything we need at the end of that method, the trouble is that the digests can/will get GC’d out of the store eventually.
a

ancient-vegetable-10556

12/08/2021, 11:19 PM
Right, so 
coursier_resolve_lockfile
has access to the 
jvm_artifacts
, it dumps a serialized lockfile (by way of its calling context), and that serialized lockfile is loaded in 
fetch_with_coursier
So the question is — can I make the correct 
jvm_artifacts
available in 
fetch_with_coursier
?
w

witty-crayon-22786

12/08/2021, 11:20 PM
yea, you could. although it might be cleaner to change the lockfile format to actually include a local file and/or URL
…at which point, the Coordinate output would be very similar to the Coordinate input =x
a

ancient-vegetable-10556

12/08/2021, 11:22 PM
The problem with including a local file is that I need to resolve it with a 
SourceFilesRequest
, and the lockfile is not likely to be in a similar location to the BUILD file that brought it into being
w

witty-crayon-22786

12/08/2021, 11:22 PM
can include it in the lockfile as relative to the build root
a

ancient-vegetable-10556

12/08/2021, 11:23 PM
hrrrrrrrrrrrrrrrm ok
w

witty-crayon-22786

12/08/2021, 11:24 PM
sorry, this is all pretty mind bending… took me a while to catch up.
a

ancient-vegetable-10556

12/08/2021, 11:24 PM
Welcome 🙂
I think it took me a good 3 weeks before I understood lockfiles last time
if your head’s not in the space, it’s confusing
It works better than expected!
w

witty-crayon-22786

12/09/2021, 10:04 PM
🙌
a

ancient-vegetable-10556

12/09/2021, 10:11 PM
Powered by