pantsbuild #plugins

Channels

welcome

github-notifications

development

polite-garden-50641

04/08/2021, 7:37 PM

I am having issues adding options w/ type dict and specifying the values via toml. I managed to repo in the pants, this is the diff:

fast-nail-55400

04/28/2021, 5:54 PM

what do I do about this mypy error? it seems like the error is coming from something that my plugin imports but is not from my plugin code directly:

Copy code

src/python/pants/backend/terraform/tffmt.py:21: note: In module imported here:
src/python/pants/engine/process.py: note: In class "InteractiveRunner":
src/python/pants/engine/process.py:334:2: error: Cannot determine type of
'side_effecting'  [has-type]
    @side_effecting
     ^

fast-nail-55400

05/04/2021, 7:48 PM

Is there a mechanism for inferring targets from sources? Or is it just dependency inference on existing targets?

plain-sundown-25537

05/05/2021, 9:12 PM

Do we have document about pulling deps different python index?

big-xylophone-43403

05/13/2021, 5:08 AM

Hello, I’m trying to write a plugin to provide custom setup_py() Kwargs based on this doc. The custom rule should run a git

Process

to read tags and populate the

version

kwarg. However, I’m not sure what

input_digest

and/or

working_dir

should be provided to the Git process 🧵

proud-dentist-22844

05/13/2021, 6:22 PM

ok, I'm designing a plugin that should run before pytest runs (

./pants test

goal) to ensure that services are running. If they are not, then it should bail with helpful instructions on setting up the developer environment. new target:

service

depend on it from python_tests target:

Copy code

service(name="service-mongo", ... other metadata describing the service here ...)
python_tests(
    name="tests",
    dependencies=["//:service-mongo"],
)

Then, we add an

@rule

with

targets: Targets

in the input that filters down to the

service

targets. That rule can use

Process

or similar to check for that the service is running. then

rules()

would return

[*collect_rules(), UnionRule(TestFieldSet, AssertServiceRequest)]

I think? But is there a way to insert my rule so that it has to run first, before pytest? Am I in the ballpark?

proud-dentist-22844

05/13/2021, 6:24 PM

Or, I suppose I could add a new field to

python_tests

called

service_dependencies

, but however it is defined, I'm still not sure how to add this pre-pytest step.

proud-dentist-22844

05/13/2021, 11:52 PM

I put

pants_requirement(name="pants", dist="pantsbuild.pants")

pylint-plugins/BUILD

as recommended in the docs. But now, I'm getting an ugly error when I try to regen constraints:

Copy code

ERROR: Could not find a version that satisfies the requirement pantsbuild.pants==2.5.0rc2 (from versions: 0.0.17, 0.0.18, 0.0.20, 0.0.21, 0.0.22, 0.0.23, 0.0.24, 0.0.25, 0.0.26, 0.0.27, 0.0.28, 0.0.29, 0.0.30, 0.0.31, 0.0.32, 0.0.33, 0.0.34, 0.0.35, 0.0.36, 0.0.37, 0.0.38, 0.0.39, 0.0.40, 0.0.41, 0.0.42, 0.0.43, 0.0.44, 0.0.45, 0.0.46, 0.0.47, 0.0.48, 0.0.49, 0.0.50, 0.0.51, 0.0.52, 0.0.53, 0.0.54, 0.0.55, 0.0.56, 0.0.57, 0.0.58, 0.0.59, 0.0.60, 0.0.61, 0.0.62, 0.0.63, 0.0.64, 0.0.65, 0.0.66, 0.0.67, 0.0.68, 0.0.69, 0.0.70, 0.0.71, 0.0.72, 0.0.73, 0.0.74, 0.0.75, 0.0.76, 0.0.77, 0.0.79, 0.0.80, 0.0.81, 0.0.82, 1.0.0, 1.0.1, 1.1.0, 1.2.0, 1.2.1, 1.3.0, 1.4.0, 1.5.0, 1.6.0, 1.7.0, 1.15.0, 1.16.0, 1.17.0, 1.18.0, 1.19.0, 1.20.0, 1.21.0, 1.22.0, 1.23.0, 1.24.0, 1.25.0, 1.26.0, 1.27.0, 1.28.0, 1.29.0, 1.30.0, 1.30.1, 1.30.2, 1.30.3, 1.30.4rc0, 1.30.4, 1.30.5rc0, 2.0.0a0, 2.0.0a1, 2.0.0a2, 2.0.0a3, 2.0.0b0, 2.0.0b1, 2.0.0b2, 2.0.0b3, 2.0.0rc0, 2.0.0rc1, 2.0.0rc2, 2.0.0rc3, 2.0.0, 2.0.1rc0, 2.0.1rc1, 2.0.1rc2, 2.0.1rc3, 2.0.1rc4, 2.0.1, 2.0.2rc0, 2.0.2, 2.1.0.dev0, 2.1.0rc0, 2.1.0rc1, 2.1.0rc2, 2.1.0rc3, 2.1.0, 2.1.1rc0, 2.1.1rc1, 2.1.1rc2, 2.1.1, 2.1.2rc0, 2.1.2)
ERROR: No matching distribution found for pantsbuild.pants==2.5.0rc2

Any ideas on what to do for that?

👀 1

proud-dentist-22844

05/14/2021, 8:25 PM

There is a warning not to use subprocess in pants plugins. What if I use a library that uses subprocess? 😉 Would you cringe at using

distro

in a plugin (in an

@_uncacheable_rule

)? https://pypi.org/project/distro/ https://github.com/nir0s/distro/blob/master/distro.py Looks like it runs these basic commands:

uname -rs

lsb_release -a

It also accesses a variety of files like

/etc/os-release

. Would I shoot myself in the foot by using this?

proud-dentist-22844

05/14/2021, 8:29 PM

It is not safe to use
subprocess.run()
like you normally would because this can break caching and will not leverage Pants's parallelism.

https://www.pantsbuild.org/v2.5/docs/rules-api-process

enough-analyst-54434

05/14/2021, 8:29 PM

If the input can change behind your back yes. For /etc/os-release - an upgrade changes the cntents of that file and Pants won't know. For uname ditto. Strongly consider running those libraries as tools in a Process cache_scope=NEVER. It may help to ignore Python and treat it as if it does not exist. Put another way - pretend you were writing this and all your support libraries were written in a jvm language. Then you'd need to shell out with Process. Do that here too.

enough-analyst-54434

05/14/2021, 8:30 PM

For example, Pants uses Pex heavily. We shell out to it and do not use its API.

👍 1

fast-nail-55400

05/14/2021, 8:31 PM

and using subprocess like that would probably also not work with remote execution where the process invocation is delegated to the remote server

fast-nail-55400

05/14/2021, 8:31 PM

and the Pants engine will (transparently to the rule) choose whether to execute locally or remotely based on configuration

enough-analyst-54434

05/14/2021, 8:32 PM

We do have a way to mark a Process platform. I'm not sure that was ever fully fleshed out.

enough-analyst-54434

05/14/2021, 8:32 PM

But you should be able to say this is a Process that must run local. Big handwave.

proud-dentist-22844

05/14/2021, 8:33 PM

But if I run the commands, then I have to parse the output - I'd rather leave that to a well tested library

enough-analyst-54434

05/14/2021, 8:33 PM

You can package that library in a PEX and run it

proud-dentist-22844

05/14/2021, 8:34 PM

You're saying to write a "script" (a pex) and shell out to it from within the plugin?

enough-analyst-54434

05/14/2021, 8:34 PM

That's my point above a jvm tool, How would you leverage jvm libs to say parse jvm imports? Package that up into a tool to run via a Process.

enough-analyst-54434

05/14/2021, 8:34 PM

Yes!

proud-dentist-22844

05/14/2021, 8:34 PM

Well see, I avoid the JVM 😉

enough-analyst-54434

05/14/2021, 8:35 PM

Ans shell out with

await Get(ProcessResult, Process(...))

- not subprocess.

enough-analyst-54434

05/14/2021, 8:36 PM

One dangerous thing about Pants v2 1st supporting Python is the rule API also happens - only happens - to be in Python. So that can leave a confusing mental model. You should view the rule language as not Python for the purposes of figuring out how to do things.

➕ 1

proud-dentist-22844

05/14/2021, 8:37 PM

https://github.com/st2sandbox/st2/blob/pants/pants-plugins/uses_services/platform.py

proud-dentist-22844

05/14/2021, 8:37 PM

This is what I was thinking of doing.

enough-analyst-54434

05/14/2021, 8:37 PM

If we had used COBOL like I wanted - this wouldn't ever be an issue for anyone 😉

fast-nail-55400

05/14/2021, 8:37 PM

You’re saying to write a “script” (a pex) and shell out to it from within the plugin?

there are some existing examples in the pants source code of a plugin writing a script to a digest and then invoking it with the Process-realted rules.

fast-nail-55400

05/14/2021, 8:37 PM

let me find the examples

fast-nail-55400

05/14/2021, 8:38 PM

https://github.com/pantsbuild/pants/blob/4f39186f780310e73a708b3b589635dfa2e09696/src/python/pants/engine/process.py#L497