pantsbuild #plugins

$ pex requests -orequests-lib.pex
$ cat <<EOF > exe.py
import requests

print(requests.get("<https://example.org>").content)
EOF
$ ./requests-lib.pex exe.py 
b'<!doctype html>\n<html>\n<head>\n    <title>Example Domain</title>\n\n    <meta charset="utf-8" />\n    <meta http-equiv="Content-type" content="text/html; charset=utf-8" />\n    <meta name="viewport" content="width=device-width, initial-scale=1" />\n    <style type="text/css">\n    body {\n        background-color: #f0f0f2;\n        margin: 0;\n        padding: 0;\n        font-family: -apple-system, system-ui, BlinkMacSystemFont, "Segoe UI", "Open Sans", "Helvetica Neue", Helvetica, Arial, sans-serif;\n        \n    }\n    div {\n        width: 600px;\n        margin: 5em auto;\n        padding: 2em;\n        background-color: #fdfdff;\n        border-radius: 0.5em;\n        box-shadow: 2px 3px 7px 2px rgba(0,0,0,0.02);\n    }\n    a:link, a:visited {\n        color: #38488f;\n        text-decoration: none;\n    }\n    @media (max-width: 700px) {\n        div {\n            margin: 0 auto;\n            width: auto;\n        }\n    }\n    </style>    \n</head>\n\n<body>\n<div>\n    <h1>Example Domain</h1>\n    <p>This domain is for use in illustrative examples in documents. You may use this\n    domain in literature without prior coordination or asking for permission.</p>\n    <p><a href="<https://www.iana.org/domains/example>">More information...</a></p>\n</div>\n</body>\n</html>\n'

In that mode - with no entry point defined - the PEX file acts like a python interpreter with dependencies; sort of like a portable venv.

👍 1

enough-analyst-54434

05/14/2021, 10:02 PM

But the basic idea is: get it working on the CLI 1st. Then translate what you did on the CLI into Pants `await Get`s with Pex and Process.

👍 2

proud-dentist-22844

05/15/2021, 7:12 AM

So, the script that I need to run with pex works when I manually build the pex. Based on those examples, I've fleshed out how to build the pex. Next I need to include the script itself in the input_digest for the VenvPexProcess. Normally, I would reach for

open()

but I suspect that is not very pantsy. The pylint plugin (which you pointed me to for how to make a pex) gets files based on a plugins listed in pants.toml, and the source files its checking, so all of the files are defined somehow outside of that plugin. But my file is right next to the plugin source file (in the same directory), so what's the best way to go about putting that in a digest?

proud-dentist-22844

05/15/2021, 7:15 AM

Repasting the links to my files: Here's the input_digest in the plugin: https://github.com/st2sandbox/st2/blob/pants/pants-plugins/uses_services/platform.py#L30-L37 Here's the script that it needs to run: https://github.com/st2sandbox/st2/blob/pants/pants-plugins/uses_services/inspect_platform.py

proud-dentist-22844

05/15/2021, 3:06 PM

That last thread is quite lengthy now so, I'll start a new one. How do I include a specific file in a

Digest

? I can do

CreateDigest([FileContent(script_path, <string>)])

but how do I grab the contents of a separate file to add to

FileContent

? Can I just

open()

it within the

@rule

? Here's the input_digest in the plugin: https://github.com/st2sandbox/st2/blob/pants/pants-plugins/uses_services/platform.py#L30-L37 Here's the script that it needs to run: https://github.com/st2sandbox/st2/blob/pants/pants-plugins/uses_services/inspect_platform.py Note that the file to include is in the same directory as the plugin python file.

enough-analyst-54434

05/15/2021, 3:47 PM

In this case,

open()

or https://docs.python.org/3/library/importlib.html#importlib.resources.read_binary or just embedding the script code in platform.py as a string are your likeliest options and they're all fine. The code Tom pointed to earlier has an example of creating a Digest given bytes (a string literal is the source in this case, but you can read in the bytes with

open()

importlib.resources

or ... however): https://github.com/pantsbuild/pants/blob/4f39186f780310e73a708b3b589635dfa2e09696/src/python/pants/engine/process.py#L511-L530 The reason

open()

and

importlib.resources

are OK here is they are reading content whose content changes Pants will be aware of. This is for two reasons: 1. Any path you add to

[GLOBAL] pythonpath

is watched by pantsd for changes and, upon change, pantsd willl restart itself to pick up the change. 2. Rule code is always re-run when Pants starts afresh. To explain 2 a bit more, the only long term caching Pants does is of

Process

executions done in rule code. Once those are executed successfully for a given set of inputs, they are never executed again - ever... unless you wipe out the pants database stored under in ~/.cache/pants (if you use remote caching you'd also have to invalidate that cache).

👍 1

proud-dentist-22844

05/15/2021, 3:56 PM

Oh Cool

proud-dentist-22844

05/19/2021, 3:34 PM

Given a target, if I get a list of Targets with SiblingAddresses, will the list of siblings include the original target?

Copy code

sibling_targets = await Get(
        Targets, AddressSpecs([SiblingAddresses(request.protocol_target.address)]),
    )

proud-dentist-22844

05/19/2021, 4:35 PM

How do I enable the

export-codegen

goal in my codegen plugin?

proud-dentist-22844

05/19/2021, 5:11 PM

I getting an error when I follow this example: https://www.pantsbuild.org/docs/rules-api-file-system#createdigest

Copy code

digest = await Get(
        Digest,
        CreateDigest([FileContent(entry_points_txt_path, entry_points_txt_contents)]),
    )

which gives me this error:

Copy code

TypeError: Invalid Get. The third argument `CreateDigest([FileContent(path=contrib/runners/noop_runner.egg-info/entry_points.txt, content=(len:59), is_executable=False)])` must have the exact same type as the second argument, <class 'pants.engine.internals.native_engine.PyDigest'>, but had the type <class 'pants.engine.fs.CreateDigest'>.

Am I doing something wrong with CreateDigest?

proud-dentist-22844

05/19/2021, 9:20 PM

I'm struggling to get my dependency injection & inference working in my

stevedore_extensions

plugin. https://github.com/st2sandbox/st2/tree/pants/pants-plugins/stevedore_extensions There are two pieces. 1. I have a

stevedore_extension

target with an

entry_points

field. That target should have python deps based on those entry_points using the

inject_stevedore_entry_points_dependencies

rule. But, when I do

./pants dependencies contrib/runners/noop_runner:runner

(where runner is the name of my stevedore_extension target), I get nothing. 2. I added a

stevedore_namespaces

field to the

python_tests

target. Each

stevedore_extension

target declares that it implements one stevedore

namespace

and this

stevedore_namespaces

field should infer additional dependencies for that test by going from namespace to the extension targets that implement the namespace, and by depending on those stevedore_extension targets, they should also get transitive dependencies on the modules that implement the relevant entry points. That should happen via the

infer_stevedore_dependencies

rule. But if I do

./pants dependees contrib//runners/noop_runner:runner

it does not list the python_tests target where I added my

stevedore_namespaces

field. So, can anyone help me debug where I'm going wrong wiring all of this up?

proud-dentist-22844

05/20/2021, 5:35 AM

Apparently, if I pass an int into argv:

VenvPexProcess(... argv=(..., 42, ...), ...)

I get a big ugly error:

Copy code

00:31:59.12 [ERROR] panic at 'called `Result::unwrap()` on an `Err` value: "Field `argv` was not convertible to type alloc::vec::Vec<alloc::string::String>: PyErr { ptype: <class \'TypeError\'>, pvalue: Some(\'Expected type that converts to PyString but received int\'), ptraceback: None }"', src/nodes.rs:308
00:31:59.12 [ERROR] Please set RUST_BACKTRACE=1, re-run, and then file a bug at <https://github.com/pantsbuild/pants/issues>.

I ended up grepping to find all the

argv

instances in my code to see what was up.

proud-dentist-22844

05/20/2021, 5:48 AM

I think I could use a new `ProcessCacheScope`:

PER_RESTART_IF_SUCCESSFUL

ie cache/memoize in memory, but only when it exits successfully. wdyt?

proud-dentist-22844

05/20/2021, 12:45 PM

What's the pants-plugin equivalent for

os.path.exists

? I have a list of candidate paths, and I need to choose the first directory that exists in the sources.

proud-dentist-22844

05/20/2021, 1:41 PM

I would love a code review on two plugins I wrote: https://github.com/st2sandbox/st2/tree/pants/pants-plugins stevedore_extensions -> enables defining entry_point data (typical of a stevedore extension) as targets and then depending on them via stevedore_namespaces field on python_tests targets. It uses the PytestSetupPlugin hook to generate egg-info data based on the target metadata, making it possible for stevedore to find the extensions while testing. uses_services -> a contributor UX improvement - Uses the PytestSetupPlugin hook to check for services like mongo before running tests that "use" those services (uses is a field on python_tests targets). This allows us to bail out as early as possible with an error message explaining how to install / start the required service. They are both essentially functional 🙂. Please let me know how they could be better. 🙂

👀 1

✔️ 1

gentle-guitar-84783

05/24/2021, 6:58 PM

When pulling in a 3rd party requirement in a rule as a dependency how do I hydrate the full file sources? It seems if I pull in Sources for a 3rd party requirement I simply get back

requirements.txt

big-fall-51153

05/25/2021, 12:00 PM

How do I debug a digest

Get

not seeming to match on anything? When I try to get

DigestContents

, there are no files coming back

proud-dentist-22844

05/30/2021, 4:39 AM

How can I extend

./pants tailor

? I'd like to have it add

skip_pylint=True

whenever it adds a

python_tests()

target. I would also like it to add another target (possibly a macro for a files target) in any directory with

pack.yaml

(which is a market file for source roots).

average-australia-85137

06/06/2021, 8:52 PM

Hello! I'm trying to get a digest from a relocated files target and for some reason all of the SourceFileRequests I'm sending aren't able to be evaluated:

Copy code

Exception: Could not find a rule to satisfy Get(SourceFiles, SourceFilesRequest, SourceFilesRequest(sources_fields=(<class 'pants.core.target_types.RelocatedFilesSources'>(alias='_sources', address=test_docker:moved_static, value={repr(self.value)}, default={repr(self.default)}),), for_sources_types=(<class 'pants.engine.target.Sources'>,), enable_codegen=False)).

Copy code

Exception: Could not find a rule to satisfy Get(SourceFiles, SourceFilesRequest, SourceFilesRequest(sources_fields=(<class 'pants.core.target_types.RelocatedFilesSources'>(alias='_sources', address=test_docker:moved_static, value={repr(self.value)}, default={repr(self.default)}),), for_sources_types=(<class 'pants.core.target_types.FilesSources'>,), enable_codegen=True)).

Looking at the code it looks like i should be able to use the codegen toggle with the HydrateSourcesRequest to create a

RelocateFilesViaCodegenRequest

(by specifying

FilesSources

as the requested generation type) however it doesn't seem like it ever gets that far?

full-policeman-85126

06/09/2021, 9:27 AM

Hi! I'm trying to write a plugin, which runs a Docker container. The container is supposed to build Python Wheels and I would like to mount a host directory into the container. Docker requires the mounts to be specified as absolute paths, though. Is there a way to retrieve the absolute Path of the current working directory (i.e. /tmp/processExecution…)?

plain-sundown-25537

06/20/2021, 8:42 PM

sorry for my ignorant, does pants v2 have a doc or example on how to write plugins?

ambitious-actor-36781

06/21/2021, 7:34 AM

The inconsistent use/lack-of of filenames in: https://www.pantsbuild.org/docs/plugins-setup-py makes it really hard to follow

ambitious-actor-36781

06/22/2021, 7:06 AM

so, I've written a plugin to inject a

version

on setup.py for when you run

pants package

But it's hardcoded and not entirely helpful. Is there an easy way to add a

--version

flag to the

package

goal? oh ding, this is ez. Create a new "subsystem" give it a "--version" option. Then in your rule that modifys your setup.py options use request your subsystem type (via typed dependency-injection) and 💥 ezpz

💯 1