Hi. I’m trying to package a new app as a pex_binar...
# general
a
Hi. I’m trying to package a new app as a pex_binary and for the first time I’ve run into a problem that I can’t solve using just the docs. The app itself is in python, and has 32 dependencies, some of substantial size (e.g PyTorch). When I try to run
pants package
on the target, after around 10 minutes it crashes with “Failed to digest inputs”. I know this isn’t the most extensive description, but I’m not sure what and where I’d have to share for someone to help me. I’d be very grateful for any kind of help.
Copy code
level: Info,
    append_only_caches: {
        CacheName(
            "pex_root",
        ): RelativePath(
            ".cache/pex_root",
        ),
    },
    jdk_home: None,
    platform_constraint: Some(
        Linux_x86_64,
    ),
    cache_scope: Successful,
}

Failed to digest inputs: "Error storing Digest { hash: Fingerprint<3057fd045aa4878c7e691f962d4f56de483a2c461b2c6066fa94a67e0a32bd78>, size_bytes: 2663814611 }: Input/output error"
e
2GB certainly is a big input. You may need to adjust: https://www.pantsbuild.org/docs/reference-global#section-local-store-files-max-size-bytes Read that carefully though, there is another option to consider when coming up with a value to give you enough headroom to store a 2GB file.
Even if you get this working, the resulting PEX zip time and size will be huge. You may want to consider the packed layout for your PEX: https://www.pantsbuild.org/docs/reference-pex_binary#codelayoutcode It's a directory tree of zips instead of 1 zip, so less convenient to copy around, but may be the trick you need.
If this sort of thing turns out to be a common problem it may make sense to invest in 2 storage tiers for our local cache (LMDB for small to medium and read-only file blobs on disk for large) or else invent a new type for LMDB storage that transparently shards huge blobs into medium ones.
👍 1
1
h
opened https://github.com/pantsbuild/pants/issues/16697 so this doesn't get lost
a
Thanks for your replies. I'll give that a read and try out the solutions you mentioned. I have one more question. Is there a way to tell what's in this file being stored? If I understand everything correctly, this has to be the file containing the 3rdparty packages, but what (if anything) do I do to check?
e
Its a bit non-trivial since we don't publish the `fsutil`binaries. 1st stop, get set up to hack on Pants following this guide: https://www.pantsbuild.org/docs/contributor-setup and then build fsutil as per this guide: https://www.pantsbuild.org/docs/contributions-rust#the-fs_util-tool. I can provide more help using fsutil once you get that far. The key inputs to the tool, though, will be the fingerprint (
3057fd045aa4878c7e691f962d4f56de483a2c461b2c6066fa94a67e0a32bd78
) and the size (
2663814611
).
Actually, since the error is storing the digest, fsutil may not help at all. That only helps if the thing has been stored. You have a failure to store.