https://pantsbuild.org/ logo
Powered by
a

adorable-engine-71736

09/13/2021, 9:27 AM
hi, im writing a couple of custom goals, one to verify built python packages, and the other to publish them to pypi i have got as far as target selection, but im not clear how/if i can invoke another goal from my goal (ie to build the package and i guess provide path to artefact) is it possible to call another goal from a goal in this way, or is there some build dependency magic im missing ?
c

curved-television-6568

09/13/2021, 9:30 AM
Hi @adorable-engine-71736 Normally, you provide the goals you want to run, in the order you want them, on the command line, as:
Copy code
./pants lint custom-verify publish ::
Any reason why you want your custom goal to also run another publish goal, rather than be explicit and provide those goals when invoking pants?
a

adorable-engine-71736

09/13/2021, 9:31 AM
i guess, because ive been working with bazel recently, im looking for an analog of using the product of the build rule
i guess my question then would be - is there a way of getting the path to artefacts built in a previous goal ?
c

curved-television-6568

09/13/2021, 9:32 AM
Also related to your question (wrgt publishing to pypi), that there is a draft for a publish goal in a PR here: https://github.com/pantsbuild/pants/pull/12805
a

adorable-engine-71736

09/13/2021, 9:32 AM
v useful, thanks
c

curved-television-6568

09/13/2021, 9:33 AM
Actually, the publish goal also builds the packages to be published, so it is a one step operation for build&publish in that sense.
N.b. that if it was already built, that it will reuse that and pick it up, so can in practice be two separate steps, but the paths to what was built will then come from the cached results of that previous build.
h

happy-kitchen-89482

09/13/2021, 9:36 AM
H! I'd need to look at the specific example, but basically you'd just have to request the product of the goal programmatically.
I'll take a look at the code in a few minutes and post an example snippet
a

adorable-engine-71736

09/13/2021, 9:37 AM
thanks, ill check out the publish pr, and step through to get a better idea of how its working
h

happy-kitchen-89482

09/13/2021, 9:38 AM
Just to verify, it's specifically python distributions (wheels or sdists) that you want to publish?
a

adorable-engine-71736

09/13/2021, 9:38 AM
i can post the wip code im working on also...
yep, python packages
h

happy-kitchen-89482

09/13/2021, 9:38 AM
That would be very helpful!
a

adorable-engine-71736

09/13/2021, 9:38 AM
removing breakpoints and other garbage...
h

happy-kitchen-89482

09/13/2021, 9:39 AM
My phone wants to autocorrect sdists to "sadists", what could go wrong?
🤪 2
a

adorable-engine-71736

09/13/2021, 9:40 AM
h

happy-kitchen-89482

09/13/2021, 9:41 AM
Will look once I'm in front of a laptop 😃
a

adorable-engine-71736

09/13/2021, 9:43 AM
c

curved-television-6568

09/13/2021, 9:53 AM
Not sure if this is easier to follow, but perhaps gives a slightly different perspective. It does the same, given a set of targets, requests any artifacts to be built for those targets. The paths to which will be in the 
BuiltPackage
data. https://github.com/pantsbuild/pants/blob/main/src/python/pants/backend/docker/docker_build_context.py#L69-L87
h

happy-kitchen-89482

09/13/2021, 11:10 AM
So as @curved-television-6568 mentioned, the key data structure is 
BuiltPackage
. The 
package
@goal_rule requests it here: https://github.com/pantsbuild/pants/blob/145c5f9bec5dab8cad69e4c7a51435bc027c313f/src/python/pants/core/goals/package.py#L96 and then dumps it into the 
dist/
dir
You'd want to request it and then perform the publishing step instead
How are you publishing? Shelling out to Twine?
a

adorable-engine-71736

09/13/2021, 11:27 AM
i didnt get that far, previously i was using a github action
ive managed to get a verification system bootstrapped based on the publishing code
v messy/wip atm and not 100% confirmed working, but the logs are looking pretty promising
Copy code
12:25:47.79 [INFO] stdout: "pytooling_distribution(address=\"envoy.gpg.sign:package\", dependencies=('!//:envoy.gpg.sign', 'envoy.gpg.sign/envoy/gpg/sign'), description=None, provides=envoy.gpg.sign, setup_py_commands=('bdist_wheel', 'sdist'), tags=None, verify_targets=('//:pytooling',))"
12:25:47.79 [INFO] stdout: "\n"
12:25:47.79 [INFO] stdout: "Targets([<class 'pytooling_verify.register.PytoolingVerifyTarget'>(address=//:pytooling, alias='verify_distribution', description=None, tags=None, verifier=verifier)])"
12:25:47.79 [INFO] stdout: "\n"
12:25:47.79 [INFO] stdout: "BuiltPackage(digest=Digest('c77df2f798ccb602899ca15f9ba9d3dfd0bd9270a388384db4d136b982ab8fb9', 224), artifacts=(BuiltPackageArtifact(relpath='envoy.gpg.sign-0.0.4.dev0-py3-none-any.whl', extra_log_lines=()), BuiltPackageArtifact(relpath='envoy.gpg.sign-0.0.4.dev0.tar.gz', extra_log_lines=())))"
h

happy-kitchen-89482

09/13/2021, 11:29 AM
Great!
a

adorable-engine-71736

09/13/2021, 11:29 AM
if i can get the code to a reasonably clean/generic state (may take some time) im happy to pr if its helpful
h

happy-kitchen-89482

09/13/2021, 11:30 AM
I think you're on the right track.
Yeah, that could be great, if the functionality is sufficiently general-purpose it sounds like it could be useful!
Pardon my ignorance, but what is the nature of the verification?
a

adorable-engine-71736

09/13/2021, 11:33 AM
making sure the metadata, etc is correct in the final package
h

happy-kitchen-89482

09/13/2021, 11:33 AM
Got it. And do you think you'll be able to use @ambitious-actor-36781’s nascent publish goal when it's ready, or will you roll your own?
a

adorable-engine-71736

09/13/2021, 11:33 AM
obv correct is subjective, so i guess i need to make that as generic as possible
h

happy-kitchen-89482

09/13/2021, 11:33 AM
I'm wondering how verification and publishing interact
a

adorable-engine-71736

09/13/2021, 11:33 AM
re publish goal
ill probs use it before its published - even if it just means copying out the code locally
no desire to reinvent wheels
h

happy-kitchen-89482

09/13/2021, 11:34 AM
For example, maybe there needs to be a 
VerifiedPackage
type, and that is the only thing that can be published
Which would guarantee via the type system that published artifacts have been verified
which is pretty neat
a

adorable-engine-71736

09/13/2021, 11:35 AM
sgtm, but i guess you might want no validation if the validation is not v generic
h

happy-kitchen-89482

09/13/2021, 11:35 AM
And folks can opt-in to verifications (and maybe plug in their own)
a

adorable-engine-71736

09/13/2021, 11:35 AM
yep, sounds really helpful
h

happy-kitchen-89482

09/13/2021, 11:35 AM
I'm just spitballing, but really happy to see folks thinking about these issues
Is your verification logic in python code, or does it involve calling a process?
a

adorable-engine-71736

09/13/2021, 11:37 AM
i havent got that far in terms of implementation, but atm i am building the packages from a setup.cfg so it will probs check the fields against that
also
my first attempt missed all the packages
and had local deps, which i dont want
so its these things i want to add in ci
c

curved-television-6568

09/13/2021, 11:38 AM
I like the verification idea. Guess it could be a separate target/config that defines what rules apply as criteria for a certain verification, and that when publishing, you can declare what verification is required to proceed. Something along those lines ought to be generic enough :)
h

happy-kitchen-89482

09/13/2021, 11:39 AM
Got it, well I don't want to confuse things with more verbiage, as it seems like you're on the right path. Don't hesitate to ask more questions here as needed!
a

adorable-engine-71736

09/13/2021, 11:39 AM
cool thanks for help
happy to assist if i can re publishing <> verification
im @adorable-engine-71736 on github so feel free to ping me there if i can
h

happy-kitchen-89482

09/13/2021, 11:43 AM
Will do!
4 Views
Powered by