https://pantsbuild.org/ logo
Join Slack
Powered by
Within a rule, how can I extract not just a file, ...
# plugins
w
Within a rule, how can I extract not just a file, but a folder? When using certain types of libraries, PyOx needs to create a shared library folder next to the main binary - so instead of the output being a single file, it's a binary file + folder of libs. I can successfully extract my binary to the dist, but I'm not sure what the API to extract a folder would be. Using 
output_directories
doesn't appear to visibly do anything https://github.com/sureshjoshi/pants-pyoxidizer-plugin/blob/33be0f2e2d03b0a487d9e0[…]1062b3bb28afa85e/pants-plugins/experimental/pyoxidizer/rules.py
And to be more specific, I can SEE the files when I do this: 
snapshot = await Get(Snapshot, Digest, result.output_digest)
But I'm not sure of the mechanism which actually pulls them into the dist folder - that part is mildly magical. I see the BuildPackage takes in the output_digest, which contains everything - so does this mean each file I want to extract needs to be it's own BuiltPackageArtifact? So like, I'd pull 20-30 artifacts? Or have I missed an API that handles that?
Copy code
return BuiltPackage(
        result.output_digest,
        artifacts=(
            BuiltPackageArtifact(
                f"./build/x86_64-apple-darwin/debug/install/{output_filename}"
            ),
        ),
    )
Something like this appears to work, but I don't know that it's canonical:
Copy code
snapshot = await Get(Snapshot, Digest, result.output_digest)
artifacts = [BuiltPackageArtifact(file) for file in snapshot.files]
return BuiltPackage(
    result.output_digest,
    artifacts=tuple(artifacts),
)
h
I think that having multiple `BuiltPackageArtifact`s vs one solely impacts the logging of the 
./pants package
goal, per my read of 
core/goals/package.py
. What matters is the 
BuiltPackage
having the correct 
output_digest
, which it sounds like is working how you want? For better logging, perhaps you want two 
BuiltPackageArtifacts
? One for the binary file, and another where 
relpath=
the directory?
w
Well, the output digest is there and correct, but the files weren't being copied to dist until I mapped out all the artifacts I required - so it does work, but I wasn't sure if it was the correct route I was going
h
Hmmmm I'm not sure how that would be happening. This is the code in `package.py`:
Copy code
packages = await MultiGet(
        Get(BuiltPackage, PackageFieldSet, field_set)
        for field_set in target_roots_to_field_sets.field_sets
    )
    merged_snapshot = await Get(Snapshot, MergeDigests(pkg.digest for pkg in packages))
    workspace.write_digest(merged_snapshot.digest, path_prefix=str(dist_dir.relpath))
    for pkg in packages:
        for artifact in pkg.artifacts:
            msg = []
            if artifact.relpath:
                msg.append(f"Wrote {dist_dir.relpath / artifact.relpath}")
            msg.extend(str(line) for line in artifact.extra_log_lines)
            if msg:
                <http://logger.info|logger.info>("\n".join(msg))
It's only those first few lines before the 
for
loop that matter for writing things to disk. The for loop is purely for logging
w
🤯
I'll re-review - maybe I'm conflating the artifacts vs the PexProcess output?
h
Possibly, it is definitely necessary for the 
PexProcess
to have 
output_dirs
and/or 
output_files
set appropriately
w
Yep, you're right! I passed in an empty tuple for artifacts, and everything works as expected. So strange, I guess I had misconfigured PexProcess, and it was a red herring when I was looking at BuiltPackage
Thanks!
h
Awesome to hear! That being said, might be worth playing with how many `BuiltPackageArtifact`s you use for more useful logging. I suspect 1-2 is best
3 Views
Open in Slack
PreviousNext
Powered by