Hi everyone, looking for advice in porting projects under separate repos with setup.py files into a monorepo. First question: Do I need to port the setup.py files into pyproject.toml files? Does Pants require pyproject.toml?

A little weird request here, but is it possible to defer a dependency that a pex needs and instead get it from the system install?

When building macros, is there an idiomatic way of listing all the target names that the macro created? I’m trying to create a macro that will define a bunch of

system-binary

dependencies to be used by other targets and would like to simply add them as a group to an

adhoc_tool

runnable dependencies.

Hi everyone, I’m using the

2.16rc1

version and I wonder if I have run into a bug. I’m telling pytest to install from a resolve: pants.toml

[python.resolves]
...
pytest = "resolves/pytest.lock"

[python.resolves_to_interpreter_constraints]
pytest= [">=2.7,!=3.0.*,!=3.1.*,!=3.2.*,!=3.3.*,!=3.4.*,!=3.5.*,!=3.6.*"]

[pytest]
install_from_resolve = "pytest"

and the corresponding lock file contains the versions specified in the requirements: resolves/pytest.lock

//   "version": 3,
//   "valid_for_interpreter_constraints": [
//     "CPython!=3.0.*,!=3.1.*,!=3.2.*,!=3.3.*,!=3.4.*,!=3.5.*,!=3.6.*,>=2.7"
//   ],
//   "generated_with_requirements": [
//     "future==0.18.3",
//     "pytest-cov!=2.12.1,<3.1,>=2.12",
//     "pytest-xdist>=1",
//     "pytest==4.6.11"
//   ],

yet, when running

pants test ...

it seems to be ignoring the lock file and using the default one that ships with pants (

pytest-cov>=2.12,!=2.12.1,<3.1, pytest-xdist>=2.5,<3, pytest==7.0.1

):

17:30:15.95 [INFO] Completed: Building 3 requirements for pytest.pex from the resolves/pytest.lock resolve: pytest-cov>=2.12,!=2.12.1,<3.1, pytest-xdist>=2.5,<3, pytest==7.0.1
17:30:15.95 [INFO] Canceled: Building requirements.pex
17:30:15.95 [ERROR] 1 Exception encountered:

Engine traceback:
  in `test` goal

ProcessExecutionFailure: Process 'Building 3 requirements for pytest.pex from the resolves/pytest.lock resolve: pytest-cov>=2.12,!=2.12.1,<3.1, pytest-xdist>=2.5,<3, pytest==7.0.1' failed with exit code 1.
...

I have tried using another name for my resolve and lockfile but it hasn’t worked

Hey, trying to write some tests for my plugin - how do I use a subsystem in the rule runner? I'm following the guide here: https://www.pantsbuild.org/docs/rules-api-testing#approach-3-rulerunner-integration-tests-for-rules

I'm getting

Exception message: Could not initialize store for cache: "Error making env for store at \"/Users/zach/.cache/pants/lmdb_store/cache/f\": No space left on device"

whenever I try to run any pants commands. I tried both changing the

local_store_dir

to a new directory and also just deleting

/Users/zach/.cache/pants/lmdb_store

entirely, but neither seemed to fix my problem. my hard drive still has 700GB free, and I don't think there's any weird partitioning scheme (at least I never set up any partitions). on a Mac M1. any ideas?

does it make sense for pex to support Python 3.12? I use pex to create nice reproducible virtual environments for tests from lockfiles and right now I cannot test Python 3.12 because pex itself does not support it. It's not a blocker for me but maybe it would be feasible to add support when the first beta comes out?

Also I can't wait for https://github.com/pantsbuild/pex/issues/1752 to be released. I would love to know how pex creating venvs can be used with nox, currently I just have some janky scripts.

I'm seeing some interesting behavior. If I change a yaml file, I can run

./pants --changed-since=master list

and I see the yaml show up in the changes. If I run

./pants --changed-since=master --changed-dependees=direct list

, I still only see the yaml file. When I run

./pants --changed-since=master dependees

, I see the

resources

target that corresponds to my yaml file. Since the language of changed-dependees says

Include direct or transitive dependees of changed targets.

I'm not really understanding why

--changed-since=master --changed-dependees=direct

doesn't show the same thing as

--changed-since=master dependees

.

Hi all, I'm having problems with setting up pants to work with a git submodule project - looks like dependencies specified in BUILD files for python sources using "overrides" are not prefixed with the subproject root? We are migrating to adopt pants in our project, before moving all of them to a single monorepo. We have one common code library already moved to pants. In our service, we use that common library as a git submodule, having a layout similar to the following:

<root>
└── serviceA
    └──  (python sources, importing lib/moduleA.py)
└── common-lib (git submodule, already migrated to pants)
    └── src
        └── moduleA.py
        └── moduleB.py
    └── tests
        └── test_moduleA.py
    └── BUILD
    └── pants.toml
└── pants.toml
└── BUILD

pants.toml in submodule is ignored by pants (as expected). pants.toml in the root of the service repo contains the following:

subproject_roots = [
  "/common-lib",
]

[source]
root_patterns = [
  "/service",
  "/common-lib",
]

Now, I understand that subproject_roots are intended to tweak the dependencies paths used in submodule's BUILD files by appending /common-lib to the path, if required. But it looks like this mechanism doesn't work, when I use "overrides" as below:

python_tests(
    name="tests",
    overrides={
        "test_moduleA.py": {
            "dependencies": [
                "src/moduleB.py"
            ]
        }
    },
)

When running

pants test ::

from the root dir, I get the following error:

ResolveError: The file or directory 'src/moduleB.py' does not exist on disk in the workspace, so the address 'src/moduleB.py' from the `dependencies` field from the target common-lib/tests/test_moduleA.py:tests cannot be resolved.

Now, I can change directory to common-lib and run

pants test ::

from there, and it works fine. But I was wondering if that could become a problem, if I will have at some point to specify some dependency using "overrides" in build file for some lib module that I actually need in the service. Is this the expected behavior? Thanks in advance for any help!

I was reading the pantsbuild blog, and noticed the post on internal python interpreter (https://blog.pantsbuild.org/two-hermetic-pythons/). I was interested in how I can build docker images that use the pants-provided python interpreter; that is, does not rely on the existence of a compatible interpreter on the docker base image. Is there any documentation on that?

Is there a way to override

python_distribution

's behavior of referring to other internal distributions and just build a hermetic sdist including all the necessary source files?

Hi folks! I am building a Docker image using Pants on an M1 Max Mac machine using Colima as the backend. When I run

pants package /path/to/Dockerfile

, I get:

11:45:59.60 [ERROR] 1 Exception encountered:

Engine traceback:
  in `package` goal

ProcessExecutionFailure: Process 'Building docker image docker:latest' failed with exit code 1.
stdout:

stderr:
ERROR: no builder "colima" found

But when I run

docker buildx ls

on the machine, I get:

$ docker buildx ls
NAME/NODE DRIVER/ENDPOINT STATUS  BUILDKIT PLATFORMS
colima *  docker
  colima  colima          running 20.10.20 linux/arm64, linux/amd64

and

docker buildx use colima

seems to work fine:

$ docker buildx use colima

Is there something I’m missing here / some reason Pants can’t find the

colima

builder?

Now that

unowned_dependency_behaviour

(link) = WARN by default, how should we be setting the owner for

pytest

?

pytest

is pinned outside of

requirements.txt

in a different file. Can I point to this file in config somewhere?

We have sphinx website built with each release. What's the state of Sphinx in Pants? I see a lot of discussion (on github, and here in the slack archives), but nothing conclusive that I can find. Is anyone building with Sphinx and Pants here?

Pants started misbehaving today, any clues?

12:48:02.56 [INFO] Initializing scheduler...
12:48:02.79 [INFO] Scheduler initialized.
Traceback (most recent call last):
  File "/Users/theoribeiro/Library/Caches/nce/142382d6e3ac0960e7d4f81b9df9de9559b7ac698db13d20cbd0e4aa57f68c1d/bindings/venvs/2.16.0rc1/bin/pants", line 8, in <module>
    sys.exit(main())
  File "/Users/theoribeiro/Library/Caches/nce/142382d6e3ac0960e7d4f81b9df9de9559b7ac698db13d20cbd0e4aa57f68c1d/bindings/venvs/2.16.0rc1/lib/python3.9/site-packages/pants/bin/pants_loader.py", line 123, in main
    PantsLoader.main()
  File "/Users/theoribeiro/Library/Caches/nce/142382d6e3ac0960e7d4f81b9df9de9559b7ac698db13d20cbd0e4aa57f68c1d/bindings/venvs/2.16.0rc1/lib/python3.9/site-packages/pants/bin/pants_loader.py", line 110, in main
    cls.run_default_entrypoint()
  File "/Users/theoribeiro/Library/Caches/nce/142382d6e3ac0960e7d4f81b9df9de9559b7ac698db13d20cbd0e4aa57f68c1d/bindings/venvs/2.16.0rc1/lib/python3.9/site-packages/pants/bin/pants_loader.py", line 92, in run_default_entrypoint
    exit_code = runner.run(start_time)
  File "/Users/theoribeiro/Library/Caches/nce/142382d6e3ac0960e7d4f81b9df9de9559b7ac698db13d20cbd0e4aa57f68c1d/bindings/venvs/2.16.0rc1/lib/python3.9/site-packages/pants/bin/pants_runner.py", line 89, in run
    return remote_runner.run(start_time)
  File "/Users/theoribeiro/Library/Caches/nce/142382d6e3ac0960e7d4f81b9df9de9559b7ac698db13d20cbd0e4aa57f68c1d/bindings/venvs/2.16.0rc1/lib/python3.9/site-packages/pants/bin/remote_pants_runner.py", line 123, in run
    return self._connect_and_execute(pantsd_handle, executor, start_time)
  File "/Users/theoribeiro/Library/Caches/nce/142382d6e3ac0960e7d4f81b9df9de9559b7ac698db13d20cbd0e4aa57f68c1d/bindings/venvs/2.16.0rc1/lib/python3.9/site-packages/pants/bin/remote_pants_runner.py", line 165, in _connect_and_execute
    return PyNailgunClient(port, executor).execute(command, args, modified_env)
native_engine.PantsdClientException: The pantsd process was killed during the run.

If this was not intentionally done by you, Pants may have been killed by the operating system due to memory overconsumption (i.e. OOM-killed). If you keep seeing this error message, try the troubleshooting steps below. If none of th
ose help, please consider filing a GitHub issue or reaching out on Slack so that we can investigate the possible memory overconsumption (<https://www.pantsbuild.org/docs/getting-help>).
 - Exit other applications, including applications running in the background.
 - Set the global option `--pantsd-max-memory-usage` to reduce Pantsd's memory consumption by retaining less in its in-memory cache (run `./pants help-advanced global`).
 - Disable pantsd with the global option `--no-pantsd` to avoid persisting memory across Pants runs, although you will miss out on additional caching.

Hello, I hope this is the right place to ask: today I was setting up our (new, experimental) Pants project on someone else's machine (mac) for the first time, and encountered this surprising error. We build some PEX files in a

docker_environment

for linux, and we resolved this issue by installing

go

on their machine. Is it expected that pants depends on go to be able to do this?

16:58:44.50 [ERROR] 1 Exception encountered:

Engine traceback:
  in `package` goal
  in Resolve transitive targets
  in Resolve direct dependencies of target - im-ingestion:docker

ProcessExecutionFailure: Process 'Building dockerfile_parser.pex from dockerfile-parser_default.lock' failed with exit code 1.
stdout:

stderr:
pid 27139 -> /Users/x/.cache/pants/named_caches/pex_root/venvs/41379f21c5956cff0d35ab1ef014bb72e1fe14dc/c83d7c7091f6e9df0ecd4a9b2f0e26696a4873cc/bin/python -sE /Users/x/.cache/pants/named_caches/pex_root/venvs/41379f21c5956cff0d35ab1ef014bb72e1fe14dc/c83d7c7091f6e9df0ecd4a9b2f0e26696a4873cc/pex --disable-pip-version-check --no-python-version-warning --exists-action a --no-input --use-deprecated legacy-resolver --isolated -q --cache-dir /Users/x/.cache/pants/named_caches/pex_root/pip_cache wheel --no-deps --wheel-dir /Users/x/.cache/pants/named_caches/pex_root/built_wheels/sdists/dockerfile-3.2.0.tar.gz/e13fd3768216a788189e0667521e1435a273a4129119a8453085d897fc34aac8/cp311-cp311-macosx_11_0_arm64.fe97d14875ad4e3bb9c900ebabe3432d /Users/x/.cache/pants/named_caches/pex_root/downloads/e13fd3768216a788189e0667521e1435a273a4129119a8453085d897fc34aac8/dockerfile-3.2.0.tar.gz --index-url <https://pypi.org/simple/> --retries 5 --timeout 15 exited with 1 and STDERR:
  ERROR: Command errored out with exit status 1:
   command: /Users/x/.cache/pants/named_caches/pex_root/venvs/41379f21c5956cff0d35ab1ef014bb72e1fe14dc/c83d7c7091f6e9df0ecd4a9b2f0e26696a4873cc/bin/python -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/Users/x/.cache/pants/named_caches/pex_root/pip_cache/.tmp/pip-req-build-_m05w_ks/setup.py'"'"'; __file__='"'"'/Users/x/.cache/pants/named_caches/pex_root/pip_cache/.tmp/pip-req-build-_m05w_ks/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' bdist_wheel -d /Users/x/.cache/pants/named_caches/pex_root/pip_cache/.tmp/pip-wheel-115d_a4i
       cwd: /Users/x/.cache/pants/named_caches/pex_root/pip_cache/.tmp/pip-req-build-_m05w_ks/
  Complete output (5 lines):
  running bdist_wheel
  running build
  running build_ext
  $ GOPATH=/Users/x/.cache/pants/named_caches/pex_root/pip_cache/.tmp/tmpsynsnw6g go get -d
  error: [Errno 2] No such file or directory: 'go'
  ----------------------------------------
  ERROR: Failed building wheel for dockerfile
ERROR: Failed to build one or more wheels

Hi, is it possible to provide

.proto

file

dependencies

to

protobuf_sources

target? Basically, I want to compile some proto files, but not their import dependencies which live in a different directory.

What do people do for tests that run on some dataset that that is not part of the repo? We have a test that first downloads data (using some shell script) from our blob store before running a test suite. How can I do this in Pants?

Hi, I am kinda new to

pants

, and had a question about optionally doing builds during run time. So we have a repo that has Python and Go code, and we run one of the Python script in CI (Buildkite) using

pants

. But it fails while looking for Go binaries. Is there a way to tell

pants

to skip

Go

build during run time?

./pants run <path to python file> --args1 args1

Hello everyone, I am running a django project with pants and I seem not to have migrations consistently updated in my postgres database. That is sometimes it works and other it shows no migrations to apply yet I have run

./pants run <path to file>:manage -- makemigrations accounts

which shows migrations have happened but when I run

./pants run <path to file>:manage -- migrate

or

./pants run <path to file>:manage -- migrate accounts

where accounts is the name of the app the models are located, it shows that

Operations to perform:
  Apply all migrations: accounts, admin, auth, contenttypes, sessions
Running migrations:
  No migrations to apply.

I need more clarity on how django migrations work with pants because I am thinking pants is building from scratch everytime try to migrate it may be dealing with sort of a cache that it picks from and doesn't constantly update

Hi, I'm trying out the Pants launcher in CI, after a long time of successful local usage. However, I'm getting an error which I think is related to my setup, but I'm not sure about why it happens on how to avoid it. Will appreciate any help. Details in 🧵.

I’m looking at the Docker repository naming scheme, is there a way currently to get the relevant source root using these placeholders? Some root projects may have a single default image, some may have multiple which could be nested in directories - so it would be good to be able to do something like:

{default_repository}/{source_root}/{[docker_image_]name}

xargs

provides a really nice mechanism to pipe results into successive

pants

calls. Is there a similar pattern for joining results of two

pants

calls? My use case is pretty non-standard, but I would like to run something like

pants <find a buch of targets> && pants <find a bunch of different targets> | xargs pants test

.

Hello! I’m trying to figure out how to get a python monorepo working within the context of an existing k8s cluster, specifically for development. Right now, at my company, we use k8s in prod and dev. In dev, we mount our host paths as volumes into our containers, so that we can dev locally, and the code is synced to the pod. I’m working on a python monorepo that will house all of our python projects and python libs. Right now I have a Dockerfile in each project’s directory, and that Dockerfile copies over its code to the docker image, and also copies over the shared libs code. Each project and lib uses poetry right now. Then I use a k8s volume to mount on top of that code, so that I can dev locally. I’ve been looking at pants as a way to simplify this process. Am I looking in the right direction?

Would I just set the target format to sdist then mount my host path on top of that?

Is it possible to chain

Get

calls together, like

await Get(<something>).and_then(lambda gr: Get(<something else>, gr))

?

Is it possible to make

list

not give a warning if nothing matches? I didn't see an option and

--level=error

seems like a pretty heavy hammer to supress.

ok, got a simpler question. When I run

./pants tailor ::

, it creates

BUILD

files in unexpected locations. If I have a python project at

src/apps/foo

, I would expect the file to be at

src/apps/foo/BUILD

, but instead, it puts it in my module at

src/apps/foo/foo/BUILD

.