https://pantsbuild.org/ logo
Join Slack
Powered by
hi, im writing a couple of custom goals, one to ve...
# general
a
hi, im writing a couple of custom goals, one to verify built python packages, and the other to publish them to pypi i have got as far as target selection, but im not clear how/if i can invoke another goal from my goal (ie to build the package and i guess provide path to artefact) is it possible to call another goal from a goal in this way, or is there some build dependency magic im missing ?
c
Hi @adorable-engine-71736 Normally, you provide the goals you want to run, in the order you want them, on the command line, as:
Copy code
./pants lint custom-verify publish ::
Any reason why you want your custom goal to also run another publish goal, rather than be explicit and provide those goals when invoking pants?
a
i guess, because ive been working with bazel recently, im looking for an analog of using the product of the build rule
i guess my question then would be - is there a way of getting the path to artefacts built in a previous goal ?
c
Also related to your question (wrgt publishing to pypi), that there is a draft for a publish goal in a PR here: https://github.com/pantsbuild/pants/pull/12805
a
v useful, thanks
c
Actually, the publish goal also builds the packages to be published, so it is a one step operation for build&publish in that sense.
N.b. that if it was already built, that it will reuse that and pick it up, so can in practice be two separate steps, but the paths to what was built will then come from the cached results of that previous build.
h
H! I'd need to look at the specific example, but basically you'd just have to request the product of the goal programmatically.
I'll take a look at the code in a few minutes and post an example snippet
a
thanks, ill check out the publish pr, and step through to get a better idea of how its working
h
Just to verify, it's specifically python distributions (wheels or sdists) that you want to publish?
a
i can post the wip code im working on also...
yep, python packages
h
That would be very helpful!
a
removing breakpoints and other garbage...
h
My phone wants to autocorrect sdists to "sadists", what could go wrong?
🤪 2
a
h
Will look once I'm in front of a laptop 😃
a
c
Not sure if this is easier to follow, but perhaps gives a slightly different perspective. It does the same, given a set of targets, requests any artifacts to be built for those targets. The paths to which will be in the 
BuiltPackage
data. https://github.com/pantsbuild/pants/blob/main/src/python/pants/backend/docker/docker_build_context.py#L69-L87
h
So as @curved-television-6568 mentioned, the key data structure is 
BuiltPackage
. The 
package
@goal_rule requests it here: https://github.com/pantsbuild/pants/blob/145c5f9bec5dab8cad69e4c7a51435bc027c313f/src/python/pants/core/goals/package.py#L96 and then dumps it into the 
dist/
dir
You'd want to request it and then perform the publishing step instead
How are you publishing? Shelling out to Twine?
a
i didnt get that far, previously i was using a github action
ive managed to get a verification system bootstrapped based on the publishing code
v messy/wip atm and not 100% confirmed working, but the logs are looking pretty promising
Copy code
12:25:47.79 [INFO] stdout: "pytooling_distribution(address=\"envoy.gpg.sign:package\", dependencies=('!//:envoy.gpg.sign', 'envoy.gpg.sign/envoy/gpg/sign'), description=None, provides=envoy.gpg.sign, setup_py_commands=('bdist_wheel', 'sdist'), tags=None, verify_targets=('//:pytooling',))"
12:25:47.79 [INFO] stdout: "\n"
12:25:47.79 [INFO] stdout: "Targets([<class 'pytooling_verify.register.PytoolingVerifyTarget'>(address=//:pytooling, alias='verify_distribution', description=None, tags=None, verifier=verifier)])"
12:25:47.79 [INFO] stdout: "\n"
12:25:47.79 [INFO] stdout: "BuiltPackage(digest=Digest('c77df2f798ccb602899ca15f9ba9d3dfd0bd9270a388384db4d136b982ab8fb9', 224), artifacts=(BuiltPackageArtifact(relpath='envoy.gpg.sign-0.0.4.dev0-py3-none-any.whl', extra_log_lines=()), BuiltPackageArtifact(relpath='envoy.gpg.sign-0.0.4.dev0.tar.gz', extra_log_lines=())))"
h
Great!
a
if i can get the code to a reasonably clean/generic state (may take some time) im happy to pr if its helpful
h
I think you're on the right track.
Yeah, that could be great, if the functionality is sufficiently general-purpose it sounds like it could be useful!
Pardon my ignorance, but what is the nature of the verification?
a
making sure the metadata, etc is correct in the final package
h
Got it. And do you think you'll be able to use @ambitious-actor-36781’s nascent publish goal when it's ready, or will you roll your own?
a
obv correct is subjective, so i guess i need to make that as generic as possible
h
I'm wondering how verification and publishing interact
a
re publish goal
ill probs use it before its published - even if it just means copying out the code locally
no desire to reinvent wheels
h
For example, maybe there needs to be a 
VerifiedPackage
type, and that is the only thing that can be published
Which would guarantee via the type system that published artifacts have been verified
which is pretty neat
a
sgtm, but i guess you might want no validation if the validation is not v generic
h
And folks can opt-in to verifications (and maybe plug in their own)
a
yep, sounds really helpful
h
I'm just spitballing, but really happy to see folks thinking about these issues
Is your verification logic in python code, or does it involve calling a process?
a
i havent got that far in terms of implementation, but atm i am building the packages from a setup.cfg so it will probs check the fields against that
also
my first attempt missed all the packages
and had local deps, which i dont want
so its these things i want to add in ci
c
I like the verification idea. Guess it could be a separate target/config that defines what rules apply as criteria for a certain verification, and that when publishing, you can declare what verification is required to proceed. Something along those lines ought to be generic enough :)
h
Got it, well I don't want to confuse things with more verbiage, as it seems like you're on the right path. Don't hesitate to ask more questions here as needed!
a
cool thanks for help
happy to assist if i can re publishing <> verification
im @adorable-engine-71736 on github so feel free to ping me there if i can
h
Will do!
4 Views
Open in Slack
PreviousNext
Powered by