Does the engine provide a mechanism to get just the

FileDigest

from a file within a

Digest

or

Snapshot

? I see I can get

FileContent

, but I don't actually need the bytes of the file, just the fingerprint

No.

The context here is that I already know the expected digest

Could you use

await Get(Digest, CreateDigest([file_content])

to get the expected

Digest

, then compare that way?

Interestingly it kind of does...

Yeah, I can. I just figured that for performance reasons I'd rather avoid slurping all of the bytes into memory within the rule, and I couldn't think of any reason why the engine wouldn't expose something like

Get(FileDigest, ExtractFileDigest(digest, "some/file"))

If you make a

DownloadFile

which has a bogus URL, if the digest of the

DownloadFile

happens to be in the local cache it'll return you the

Digest

of the

Directory

containing exactly that file...

Ah but that's not what I need. I have the

Digest

, I need the

FileDigest

use Snaphot

Ooh, yes! I forgot

Snapshot

existed!

How does

Snapshot

help?

it contains a list of files/dirs in a Digest and their individual digests

OK, that's what I want, but that's not the API I see on

Snapshot

Wait, I don't think it does on the

Python

side?

I see a

digest

of the entire snapshot, and a list of files and dirs, not their individual digests

it may have another name, let me find an example

That said, exposing either the sha256 of files in

Snapshot

, or an intrinsic to go from a

Digest

or

Snapshot

+ path to its sha256 doesn't seem unreasonable. (Right now we haven't because "we happen to know the sha256 of these files" is an implementation detail rather than part of the API, but it could reasonably be part of the API, even if a future change to the internals means it's less efficient to compute than it currently is, which is unlikely in the short/medium term)

Yeah, I've been idly noting in my head that there are good reasons to keep

sha256

an implementation detail, but with a little design that can be maintained. It basically boils down to making

FileDigest

serializable, and for it to self-describe the underlying hash in a way that doesn't preclude adding or removing hash functions in the future

FileDigest

is available in Python: https://github.com/pantsbuild/pants/blob/64a69b8e1f596308895153222c2b5e1377ccd5d3/src/python/pants/engine/fs.py#L47

Yes, but there's no way to produce one without slurping in all of the bytes and ctor'ing it in memory

yeah 😞

Snapshot

only produces the file and directory names

But would be trivial (both in code and runtime overhead) to extend to also include `FileDigest`s, I think?

so maybe we need to a new primitive to just not fully hydrate a

Digest

Yeah, I can easily go

Digest -> FileContent -> FileDigest

, but the

FileContent.content

is unnecessary for my use-case, and likely has sufficient network overhead to make a meaningful performance difference

do you know the file name you want?

Snapshot.files

(or some new field for backcompat) could be a

Map[str, FileDigest]

I suspect we may want it to be a method rather than an eagerly hydrated

Map

, to allow for memory optimisations, but they're morally equivalent 🙂

Yeah. I'd also be fine with my original suggestion:

Get(FileDigest, ExtractFileDigest(some_digest, "file/path"))

open an issue?

Sure

Thanks! How blocking is this? Sounds fun for me to do as a background task this week, but trying to prioritize

Not blocking at all. The in-memory workaround is fine to keep iterating; the improved API would just be a drop-in optimization