https://pantsbuild.org/ logo
Join Slack
Powered by
<@U06A03HV1> At the moment it looks like we do all...
# development
a
@witty-crayon-22786 At the moment it looks like we do all of our coursier fetches on the consumption side entirely from data in the serialized lockfile; I’m trying to get an idea of the extent to which I’m going to need to remodel things for local files
👍 1
@witty-crayon-22786 @fast-nail-55400 In 
fetch_with_coursier
, it’s clear that the fetch request is for a 
jvm_artifact
, but we call cs itself with details from the lockfile itself. I’m considering whether I should swap out the lockfile entries with declared artifacts at this point (if they have a URL or JAR declared). Is there any reason why I shouldn’t be doing that?
(the problem here is that fetching based on the lockfile doesn’t play well with using local files)
w
i think that that should always be safe to do, because the request has a 
CoursierResolveKey
(iirc?) and we don’t allow duplicates of a coordinate as input?
…would want to confirm the latter thing, but i think that coursier blows up if you give it the same input twice
a
I wouldn’t be giving it the same input twice
w
i know: sorry, i meant that within a resolve, a coordinate is unique
so yes, i think that it is safe to do that
a
so the idea would be to compare the 
CoursierLockfileEntries
with URL- or JAR- specifying artifacts, check if their resolves are compatible, and use the artifact info instead of the lockfile info in that case?
w
their resolves should be compatible already i think:
because the request has a 
CoursierResolveKey
 (iirc?)
a
Can we specify multiple artifacts with the same coordinates under different resolves?
w
we can.
because the request has a 
CoursierResolveKey
 (iirc?)
but yes: this is true. 
CoursierFetchRequest
extends 
ClasspathEntryRequest
which has a resolve field
a
Right, so I’ve got the resolve field, but I’d need to pull in the artifacts that are associated with that resolve key, yeah?
w
need to load the lockfile, yea
a
The lockfile is already loaded
I specifically need the 
jvm_artifacts
, because the lockfile is not Pants-aware
w
that’s also part of the 
request
though, right?
(it is)
a
The root artifact is, yes, but not its dependencies
w
ahhh.
a
and if the dependency is a 
jvm_artifact
, I need the 
jvm_artifact
so that I can yoink the file appropriately
w
hm. but this is an intransitive resolve
…oh. shit. it’s not, is it.
a
Nope
w
argh.
oh, wait though.
…hm. dang.
a
there is nothing good here 🙂
w
yea.
a
So I need to be able to request all of the artifacts that match a given resolve, and sub those in if their coordinates are a dependency
w
so either the lockfile bakes that information in, or https://pantsbuild.slack.com/archives/C0D7TNJHL/p1639004580226000?thread_ts=1639000059.221600&amp;cid=C0D7TNJHL … which is a bummer
so
one thing to consider here:
a
Storing it in the lockfile is awkward, because it would need to be able to be handled by a 
SourceFilesRequest
w
i’m not certain that “fetch individual artifact via a new process” is strictly necessary
a
No, but I’d still need to assemble a coursier command line of some sort
which would need to know whether to fetch by URL or local file
the 
fetch_one_coord
thing makes for very fine-grained caching with minimal fuss, but you’re right, it’s not strictly necessary
w
if the resolve that generates the report also split an output Snapshot into new Digests per file, then 
fetch_one_coord
could go away, and we could look up the 
Digest
to use in the report dataclass
i’m not sure what the original idea behind 
coursier_resolve_lockfile
running a 
fetch
but then not actually capturing the outputs was
… because at the point when it does that, it already has everything it needs to load it later
a
coursier_resolve_lockfile
writes the lockfile to disk, and that’s infrequent
most people shouldn’t be resolving their lockfile
so even if we optimised that, it wouldn’t fix the general use case, which is just fetching
w
hm: if we’re talking about 
coursier_fetch.coursier_resolve_lockfile
, it doesn’t write it to disk: just loads it into memory after running the fetch
a
Right, that’s the task that produces the actual file file
I should know, I just refactored it significantly
w
k
a
coursier_resolve_lockfile
is not run when fetching
that produces the contents of the file that is loaded when fetching
w
Copy code
description=(
                "Running `coursier fetch` against "
                f"{pluralize(len(artifact_requirements), 'requirement')}: "
                f"{', '.join(req.to_coord_str() for req in artifact_requirements)}"
            ),
a
@witty-crayon-22786 Yeah, it uses 
coursier fetch
to resolve the lockfile. Apparently it’s a necessity?
The only thing retained is the report, which is used to verify file hashes when fetching the artifacts at consumption time
w
it’s creating the lockfile in memory
a
Yes, and it gets written to disk
it’s one rule in the process that writes the lockfile to disk, that’s the only time it gets called
w
ok, i see what you are saying.
and yea, although we have everything we need at the end of that method, the trouble is that the digests can/will get GC’d out of the store eventually.
a
Right, so 
coursier_resolve_lockfile
has access to the 
jvm_artifacts
, it dumps a serialized lockfile (by way of its calling context), and that serialized lockfile is loaded in 
fetch_with_coursier
So the question is — can I make the correct 
jvm_artifacts
available in 
fetch_with_coursier
?
w
yea, you could. although it might be cleaner to change the lockfile format to actually include a local file and/or URL
…at which point, the Coordinate output would be very similar to the Coordinate input =x
a
The problem with including a local file is that I need to resolve it with a 
SourceFilesRequest
, and the lockfile is not likely to be in a similar location to the BUILD file that brought it into being
w
can include it in the lockfile as relative to the build root
a
hrrrrrrrrrrrrrrrm ok
w
sorry, this is all pretty mind bending… took me a while to catch up.
a
Welcome 🙂
I think it took me a good 3 weeks before I understood lockfiles last time
if your head’s not in the space, it’s confusing
It works better than expected!
w
🙌
a
Open in Slack
PreviousNext
Powered by