In furthering the development of the prettier plugin, how can I get the root path of the targets? I ran

./pants -l=debug lint pants-plugins::

and it passes all the python files to prettier but not the markdown file. I want the prettier plugin to operate on all files in a project (BUILD context).

Some thoughts on the documentation for adding a linter/formatter while trying to add a linter/formatter. 1. The linked repository (https://github.com/pantsbuild/example-plugin/tree/main/pants-plugins/examples/bash) is seemingly out-of-date with the current version of Pants (E.g. some types no longer exist). 2. The linked repository also adds full support for a new language (Bash), which comes with additional bells and whistles simply adding a linter/formatter doesn't need. It's hard to see what I'd need for linting/formatting vs. what I need for making binaries and testing (which I don't need) 3. The documentation has code snippets for several files, but seems to straddle between "files needed to add new language support" and "files needed for your linter/formatter support". I'm very confused what goes where 😅 4. The last file snippet in each section has a filepath header. I think that should be added to all snippets, otherwise I'm not sure which snippet belongs in which file, or what the conventional name of each file should be. a. This would help with #3 as well I think 5. The guides also add a linter/formatter for a language already recognized in Pants (shell), so it uses Pants-defined types. I'm kind of lost at sea for defining the relevant types for a language Pants doesn't come with support for. a. Perhaps the guide can fork and provide info on these tasks for a new language or an already-existing one? b. More examples in the example repo could help as well

Tangentially related, I'm wondering if a

cookiecutter

script for common tasks would really speed up stamping out new linter/formatters: https://github.com/cookiecutter/cookiecutter (The result doesn't have to be perfect, but any templating goes a long way)

If I'm creating a plugin that takes in the output of a python_distribution (i.e. wheel), are there examples of where that's currently done in any existing plugins? I can see how I can grab source files just fine - but I'm less sure about outputs of other targets. I've pulled in and taken a dependency on the python_dist target, but I can't figure out which API will let me grab that wheel

Within a rule, how can I extract not just a file, but a folder? When using certain types of libraries, PyOx needs to create a shared library folder next to the main binary - so instead of the output being a single file, it's a binary file + folder of libs. I can successfully extract my binary to the dist, but I'm not sure what the API to extract a folder would be. Using

output_directories

doesn't appear to visibly do anything https://github.com/sureshjoshi/pants-pyoxidizer-plugin/blob/33be0f2e2d03b0a487d9e0[…]1062b3bb28afa85e/pants-plugins/experimental/pyoxidizer/rules.py

Could anyone recommend some next steps for me for making this PyOx plugin more usable for the community? Once I can import a PyOx configuration file, instead of using a coded template, it'll be more or less usable for my purposes. I haven't designed a target api or subsystem for it, as that would end up highly specific for my use cases and not much value to everyone else. I've got some tickets open in this repo that I'll be closing out as well, but in general - this has reached proof of concept level for me and I'll just be running it over some of my non-trivial projects for a while. https://github.com/sureshjoshi/pants-pyoxidizer-plugin

Do Pants plugin tests assume a specific python version on run? I just got hit with

E           Engine traceback:
E             in select
E             in experimental.pyoxidizer.rules.package_pyoxidizer_binary (src:test-bin)
E             in pants.backend.python.util_rules.pex.create_pex (pyoxidizer.pex)
E             in pants.backend.python.util_rules.pex.build_pex (pyoxidizer.pex)
E             in pants.backend.python.util_rules.pex.find_interpreter (CPython>=3.8)
E             in pants.engine.process.fallible_to_exec_result_or_raise
E             in pants.backend.python.util_rules.pex_cli.setup_pex_cli_process
E             in pants.backend.python.util_rules.pex_environment.find_pex_python
E             in pants.python._binaries_rules.find_python
E           Traceback (most recent call last):
E             File "/Users/sj/.cache/pants/named_caches/pex_root/venvs/de04010553f1713b72512ad07d1b20b66bf8f840/f5cef8ebdbaf03ed774d467ae52a457e8fe3774a/lib/python3.9/site-packages/pants/engine/internals/selectors.py", line 705, in native_engine_generator_send
E               res = func.send(arg)
E             File "/Users/sj/.cache/pants/named_caches/pex_root/venvs/de04010553f1713b72512ad07d1b20b66bf8f840/f5cef8ebdbaf03ed774d467ae52a457e8fe3774a/lib/python3.9/site-packages/pants/python/_binaries_rules.py", line 82, in find_python
E               raise BinaryNotFoundError(
E           pants.engine.process.BinaryNotFoundError: Was not able to locate a Python interpreter to execute rule code.
E           Please ensure that Python is available in one of the locations identified by `[python-bootstrap] search_path`, which currently expands to:
E             []

I think I might be mis-understanding how RuleRunner's work. If I create this fake BUILD file, and then run my pyoxidizer rule, should that automatically run the sources and distribution? Or do I need to explicitly add those rules as well to the fixture?

python_sources(
                name="libtest"
            )
            python_distribution(
                name="test-dist",
                dependencies=[":libtest"],
                wheel=True,
                sdist=False,
                provides=python_artifact(
                    name="test-dist",
                    version="0.1.0",
                ),
            )
            pyoxidizer_binary(
                name="test-bin",
                entry_point="hellotest.main",
                dependencies=[":test-dist"],
            )

My first plugin and my first Pants PR - please be gentle 🙂 https://github.com/pantsbuild/pants/pull/14183

fyi, feedback welcomed on what would help you with writing plugins

Is there an easy way to use my PR'd version of Pants in my other repos? There is a PANTS_SHA env variable, but I think that's only for mainline Pantses?

Is there an existing mechanism in

python_distribution

to add imports to the setup.py? in

setup_py.py

we have the following:

SETUP_BOILERPLATE = """
# DO NOT EDIT THIS FILE -- AUTOGENERATED BY PANTS
# Target: {target_address_spec}

from setuptools import setup

setup(**{setup_kwargs_str})
"""

but if I need another import in there, can that be added via the BUILD file? Or do I just need to use a custom setup.py/auto-generate my own

Hi, is there some list of (actively maintained) pants plugins or a naming convention in pypi?

Hi Simon, No, there isn’t (to my knowledge). The Plugin API is not yet marked stable, and as such, there has not been any widespread adoption/promotion of Pants plugins. @busy-vase-39202 is investigating various ways to list/index plugins going forward, such as tagging your GitHub repo as a Pants plugin or using classifiers on your Python distribution etc.

FWIW I'll add too that adding a plugin inside of Pants is pretty easy, and you get a review by the core Pants maintainers. So if your plugin idea is a good fit, I would recommend adding it to the pants repo 🙂

Yes, we've generally been encouraging people to develop their plugins directly in Pants so that you don't have to worry about API changes + it's easier for others to benefit We are planning/hoping to stabilize the Plugin API likely in the latter half of 2022 What plugins are you thinking of writing?

Actually I was looking for plugins rather than thinking of writing them.

Ah got it. In general, or for specific functionality?

Both, I was interested how the plugin landscape looks like in general and also looking for: • plugins related to building containers (see the other discussion) • plugins to build rust crates • plugins to integrate with https://www.pulumi.com

These are the public plugins that I'm aware of. https://docs.google.com/spreadsheets/d/1F3k4T8OkQpZ1pGA6YK-kh5AkbP3ma-C-QDljFi33ZTk/edit?usp=sharing Additions always welcome.

@busy-vase-39202 I've updated mine with the more generic pants-plugins github repo, where I'm prototyping a few different plugins

Does

DistBuildRequest

or

DistBuildChrootRequest

support adding arbitrary dependencies? https://github.com/pantsbuild/pants/blob/0ce449f9181c690dbf60b2bc51b0e97f285eb9ae/src/python/pants/backend/python/goals/setup_py.py#L385-L417 I'm working on the

mypyc

plugin, and it works relatively well, but when I add a 3rd party dep, it cannot be found - I assume because it can't find the dep within the build context. I manually hackjobbed this by creating a custom pyproject.toml and then adding my dep to the [build-system].requires field, just to see if it would work. Seemingly it does, so now I'm trying to figure out how to correctly get it into the pipeline. I can access the field in the transitive dependencies, so that's not a problem. The end goal is a wheel with the compiled objects inside.

I have a python_awslambda question: I have a large number of functions with the same third-party dependencies, and prior to using pants was using Lambda Layers to avoid reuploading 100MB of deps for every function. In python_awslambda, the deps get built into the pex so I'm back to repeating lots of large uploads. Is there a way to filter a third-party dep out of the pex being sent to lambda?

hello! I've inherited a codebase that uses Pants 1.26.0, Python 3.6.12, and Cython 0.28.2 (yeah, I know...) I'm trying to chip away at getting all of this upgraded, but my first priority is getting to Python 3.7 because 3.6 is EOL and no longer receiving security updates the Cython part of the build happens using a plugin that a former coworker wrote, apparently with help from someone in the Pants community. this was originally in May 2018 and the mailing list archives don't seem to stretch back that far. if I just upgrade to 3.7 (changing interpreter_constraints in pants.toml, and running Pants inside a Py3.7-based Docker container instead of the 3.6-based one we use now), with no other changes, the build fails with:

Traceback (most recent call last):
  File "/srv/submodules/obscura-plugins/plugins/setup.py", line 4, in <module>
    from Cython.Build import cythonize
ModuleNotFoundError: No module named 'Cython'

Exception message: Command '['/usr/bin/python3.7', '/srv/submodules/obscura-plugins/plugins/setup.py', 'build_ext', '--inplace', '--verbose']' returned non-zero exit status 1.

is there a "simple" fix to get this working under 3.7? if I have to upgrade Pants and/or Cython to get it to work I will, but my goal in the short term is to do the minimum necessary to get upgraded to 3.7, and then tackle the rest of the tech debt separately pants.toml: http://ix.io/3PIe register.py of the plugin: http://ix.io/3PIi compile_cython.py: http://ix.io/3PIj BUILD containing the Cython target: http://ix.io/3PIk the setup.py living next to that BUILD file, where the exception above is being thrown from: http://ix.io/3PIm

@witty-family-13337 has posted what looks like an awesome change to add

helm

support: https://github.com/pantsbuild/pants/pull/14487 … i’ve been busy in the last few days, but will take a look soon. in the meantime though, if anyone who is familiar with

helm

has time to take a look, that would be really helpful!

Hi all, drafted a PR for the

WSDL -> Java

code generation backend in case any early feedback can be received. I’m having issues now to run the tests in pants (some import error related to the

.so

binary of the engine) so I leave it as draft as I couldn’t run all the tests: https://github.com/pantsbuild/pants/pull/14545

I'm trying to write a formatter that does some very simple addition (copyright header) to the file contents. With the following code, Pants always says my subsystem made changes (likely my output digest isn't right)

source_files = await Get(
        SourceFiles,
        SourceFilesRequest(field_set.source for field_set in request.field_sets),
    )
    source_files_snapshot = (
        source_files.snapshot
        if request.prior_formatter_result is None
        else request.prior_formatter_result
    )
    input_digest = source_files_snapshot.digest
    digest_contents = await Get(DigestContents, Digest, input_digest)
    output_digest = await Get(
        Digest,
        CreateDigest(
            FileContent(path=file_content.path, content=maybe_add_copyright(file_content.content))
            for file_content in digest_contents
        ),
    )
    return FmtResult(
        input=input_digest,
        output=output_digest,
        stdout="",
        stderr="",
        formatter_name=request.name,
    )

Hi folks, I'm trying to make a plugin for

ansible-lint

, which requires the

ansible

script to be there too. I tried subclassing PythonToolBase and adding 'ansible==5.3.0' to the extra requirements, but it looks like it's not included in the PEX as I get "No such file or directory: 'ansible'" when running .

ansible-lint

. I'm not sure what question to ask. Is it possible to have a PEX with multiple binaries? Can I have a PythonToolBase depend/include another PythonToolBase? Is there a better way to do this? Any help is appreciated :)

Hey Team, looking to write an external tool to support twirp, specifically using the twirpy protoc plugin. One requirement to use the tool is to install it via

go

with

go install <http://github.com/verloop/twirpy/protoc-gen-twirpy@latest|github.com/verloop/twirpy/protoc-gen-twirpy@latest>

Is it possible to use

go

like this in pants via an external tool? Or would we need to install

go

as an external tool first and then we can install plugins with it?

Hey all, just wondering if I could get a sanity check on an idea for the ansible-playbook

deploy

target I'm working on. I was planning to do a similar dependencies inference as what Docker does re: copying PEXes and being able to pull in files/resource/etc (https://www.pantsbuild.org/docs/docker#adding-dependencies-to-your-docker_image-targets) In my current state of mind, this makes sense to me, given how I've been using my internal ansible deployment target. However, from a wider perspective, I don't know if this could cause more trouble, since there isn't a singular Dockerfile to look through, but rather a set of roles and a larger number of potential inference locations