Q17: If I clone another Python repo that uses Pants onto a subdirectory of a “mother” repo thst also uses Pants, can I refer the targets of the “child” repo? (e.g., when using submodules, just manually cloning some plugin codes, etc.) In such cases, can I inject the dependency without modifying BUILD files in the mother repo?

Q18: When building & running pants from sources, how do I override the system Python with a pyenv-provided one?

Q19: When I run

./pants run src/ai/backend/manager:server

(a

pex_binary()

target which depends on

python_sources()

target) or

./pants run src/ai/backend/manager/server.py

, it says

importlib.metadata.PackageNotFoundError: No package metadata was found for backend.ai-manager

with the following

__init__.py

. How do I make the package version information available in this case?

Q20: If I set

--no-run-cleanup

to

run

commands, should I manually clean up the temporary dirs? Is there any command to clean up all remaining stuffs at once?

Q21: If I don't set

--no-run-cleanup

, the finalization process seems to interfere with my own cleanup procedures, crashing Python. Can I keep my program to handle signals transparently?

Please join me in wishing good luck to Pants maintainer @ancient-vegetable-10556 on his Pycon talk today delving into how the project uses hermetic environments to ensure speed and reproducibility. 👖 https://pantsbuild.slack.com/archives/C046T6TA4/p1651429472777939

I have a toy fastapi app, and i want to bundle uvicorn into the pex. If my requirements are

python_requirement(
    name="fastapi",
    requirements=["fastapi==0.75.2"],
)

python_requirement(
    name="uvicorn",
    requirements=["uvicorn==0.17.6"],
)

Then I can get a package without problems (at least during build, it doesn’t quite run how I want to yet, but that’s another question) If I update the uvicorn dependency to be

uvicorn[standard]==0.17.6

then I see something like this (with different packages failing on each run:

22:13:20.62 [INFO] Completed: Building app/app_binary.pex with 2 requirements: fastapi==0.75.2, uvicorn[standard]==0.17.6
22:13:20.62 [ERROR] 1 Exception encountered:

  ProcessExecutionFailure: Process 'Building app/app_binary.pex with 2 requirements: fastapi==0.75.2, uvicorn[standard]==0.17.6' failed with exit code 1.
stdout:

stderr:
ERROR: Could not find a version that satisfies the requirement websockets>=10.0; extra == "standard" (from uvicorn[standard])
ERROR: No matching distribution found for websockets>=10.0; extra == "standard"

What am I missing?

Is there a way to configure a default for an environment variable if it's not configured in the main process?

A question on the interpreter chosen for the PEX shebang: I have a pex defined as

pex_binary(
    name="app_binary",
    execution_mode="venv",
    dependencies=[
        ":app",
        "3rdparty/python:uvicorn",  # This isn't imported anywhere, so dependency inference doesn't find it.
    ],
    entry_point="main.py",
    platforms=["linux_x86_64-cp-3.8.13-cp38m"],
    interpreter_constraints=["==3.8.13"]
)

but when I build the image and try to run it, the error is

/app_binary.pex 
/usr/bin/env: 'python3.10': No such file or directory

my expectation was that the shebang would pick an interpreter that matches the constraints. Besides setting shebang explicitly, how can I ensure a proper interpreter gets picked?

Q22: My projects inside the mono repo expose their own package entrypoints to detect its own features along with plugins (just like pytest plugins). Previously I could make them to recognize self-exported entrypoints because they ran as an editable installation. It seems that pants/pex does not make editable installations but just sets up python's path/environment to recognize the dependencies archived as pex and the source roots as the package import paths. This means that there is no package metadata available (as in my Q19) when my projects run via pants. How could I expose self-exported entrypoints to pants-bootstrapped Python programs?

Q23: Is there a way to display/visualize what rules are executed in what order when running a

./pants

goal?

Quick update on trying to run pants inside a container on my M1 Mac. Pants doesn't have binary wheels on Linux arm, so I had to build from source. Here are a few notes.

Hi team, a question on pex/pants cache. We would like to use cache in our jenkins CI pipeline. Currently, we config

PEX_ROOT

,

PANT_LOCAL_STORE_DIR

and

PANTS_NAMED_CACHES_DIR

in our CI. With those, we can see the pex cache are used within builds across a single branch, i.e, after the first builds, the following builds on that branch will re-use the pex cache. The question is, given we config the

PEX_ROOT

,

PANT_LOCAL_STORE_DIR

and

PANTS_NAMED_CACHES_DIR

to be the same place across all build, won’t the cache be shared across all builds and branches? For example, if one build on a branch resolves the ‘python-default’ from build-support/lockfile.txt, can it be shared?

Hello, I'm having an issue with using the

googleapiclient

Python package when running

./pants test ...

. It seems to be an issue with the underlying

pyparsing

module. I was wondering if anyone has seen anything like this before. The error I'm getting is as follows.

AttributeError: module 'pyparsing' has no attribute 'downcaseTokens'

Here's the part of the stack trace that seems relevant.

front_porch/modules/common/google_drive/google_drive.py:11: in <module>
    from googleapiclient.discovery import build
/Users/hughhan/.cache/pants/named_caches/pex_root/venvs/8dccf2108884fe3898a8bc04279cb74a330a1bbd/e83365d7bd577d2fac6d4ec70eb77f4d4607d634/lib/python3.9/site-packages/googleapiclient/discovery.py:48: in <module>
    import httplib2
/Users/hughhan/.cache/pants/named_caches/pex_root/venvs/8dccf2108884fe3898a8bc04279cb74a330a1bbd/e83365d7bd577d2fac6d4ec70eb77f4d4607d634/lib/python3.9/site-packages/httplib2/__init__.py:52: in <module>
    from . import auth
/Users/hughhan/.cache/pants/named_caches/pex_root/venvs/8dccf2108884fe3898a8bc04279cb74a330a1bbd/e83365d7bd577d2fac6d4ec70eb77f4d4607d634/lib/python3.9/site-packages/httplib2/auth.py:20: in <module>
    auth_param_name = token.copy().setName("auth-param-name").addParseAction(pp.downcaseTokens)
E   AttributeError: module 'pyparsing' has no attribute 'downcaseTokens'

Pants team invites you to add your organization or company to the Who Uses Pants page. We'd love to feature your team on this page! It helps showcase the diverse set of organizations that Pants is helping, and it demonstrates to your own users, employees and candidates that you're on the cutting-edge of the latest and greatest in build system technology. Click here to get started.

Regarding https://www.pantsbuild.org/docs/python-third-party-dependencies#lockfiles, what is the reason that lockfiles like

Pipfile.lock

cannot be used as the lockfile? Generating lockfiles via Pants seems to be very slow...

➜  ./pants generate-lockfiles
⠠ 461.47s Determine all Python interpreter versions used by Pytest in your proje
⠠ 461.47s Determine if MyPy should use Python 3.8+ (for lockfile usage)
⠠

Probably a dumb question, but I'm working on upgrading to pants 1.30 and was wondering if there's a way to have pants return output like it did before the engine upgrade. On engine v2, a lot of the output that i need for troubleshooting the upgrade gets hidden by the quieter logging for engine 2, but I can't see if one of the steps completed or not due to the lack of output

I tried the

--debug

flag but that didn't give me what I needed

Is there a way to get Pants to work properly, even if we don't check in

BUILD

files via Git? I noticed that if I have

*/**/BUILD

in the

.gitignore

, then Pants commands will complain that a

BUILD

file doesn't exist, even though it does exist in the local filesystem.

open question for the community: has anyone worked with PEX files and Data Bricks? we’re currently trying to adopt Data Bricks to run our spark jobs. we’ve already been packaging jobs into individual PEX files which contain all the necessary first- and third-party code needed, and i really like how cleanly this works. however, it looks like Data Bricks doesn’t support PEX files and would require us to use wheels instead. does anyone here have any experience with data bricks? were you able to use PEX files in any way or were you forced to use wheels instead?

There are a number of things in our repo that still require the use of a venv. I'm curious if I could use the power of

export

to reduce our bootstrap times. This bootstrapping does something like

pip install -U -e . -r requirements.txt -c constraints.txt

. Even with a warm pip cache, this takes usually around 2 minutes or so while

./pants export

is only like 15 seconds. The missing details are that the venv written by

export

sensibly doesn't have the

-e .

install step. Anyone tried to do something like this before?

Hi all, as mentioned in the #welcome channel, I'm co-responsible for a large python package at Bosch. One problem we have encountered when trying to switch to pants is the coverage report for our setup. Specifically, we run our tests in our CI pipeline three times, more or less like so:

COVERAGE_FILE=.coverage.1 pytest -rfs -v --cov=source/mypkg --option1 tests/
COVERAGE_FILE=.coverage.2 pytest -rfs -v --cov=source/mypkg --option2 tests/
COVERAGE_FILE=.coverage.3 pytest -rfs -v --cov=source/mypkg --option3 tests/
coverage combine .coverage.*
coverage report
coverage xml -o coverage.xml

The custom options

--option1

,

--option2

and

--option3

affect both the code paths in the tests and the selection of which tests to run. The subsets running for the three options are to a large extent overlapping, so it is not possible to split them into directories. Moving to pants, I additionally used tags for the selection of the tests, to avoid the error when a file has no tests selected due to one of the options. The relevant environment is as follows:

# pants.toml
[coverage-py]
report = ["console", "raw"]

# pyproject.toml
[tool.coverage.report]
ignore_errors = true

And the commands look like this:

./pants --tag='+subset1' test --test-use-coverage --coverage-py-output-dir=dist/coverage/1 :: -- --option1
./pants --tag='+subset2' test --test-use-coverage --coverage-py-output-dir=dist/coverage/2 :: -- --option2
./pants --tag='+subset3' test --test-use-coverage --coverage-py-output-dir=dist/coverage/3 :: -- --option3

Here is the main problem: Even though at the end of each one of the commands, the report on the console is accurate, trying to combine or even just report the coverage from the

dist/coverage/*/.coverage

files results in 0% coverage, which means I cannot export the combined report to xml. Also a minor problem, which might be helpful/related, but is solved on its own: If I leave out the

ignore_errors = true

in

pyproject.toml

, I get the following error, although there is no file with the name

config-3.py

in the project:

ProcessExecutionFailure: Process 'Generate Pytest report coverage report.' failed with exit code 1.
stdout:
No source for code: '/tmp/process-executionSiZHX7/config-3.py'.
Aborting report output, consider using -i.

For pants I used version 2.10.0 for experimenting/debugging and just tried 2.11.0, which results in the same behavior. I would try to create a minimal example of a repo causing this behavior, but since it would take quite a bit of effort to reduce it accordingly, I wanted to ask the community if anything comes to mind. Thanks in advance!

I assume yes, but just to confirm — does Pants know how to handle caching when there are multiple Pants processes that will have access to the cache? This is applicable for the CI use case — there is a single cache directory

$HOME/.cache/pants

that is shared by multiple builds that happen on the same node (each inside a Docker container with the mounted cache directory) at the same time.

Q24: When running tests via

./pants test

,

pyproject.toml

with

tool.pytest

section seems not to be applied. Adding explicit dependency from

python_tests

target to

resource(name="pyproject", source="pyproject.toml")

as

//:pyproject

makes the pants engine to crash:

08:40:29.91 [ERROR] panic at 'called `Result::unwrap()` on an `Err` value: PyErr { type: <class 'TypeError'>, value: TypeError("format_exception() got an unexpected keyword argument 'etype'"), traceback: Some(<traceback object at 0xffff7fa3c2c0>) }', src/python.rs:432
08:40:29.91 [ERROR] Please set RUST_BACKTRACE=1, re-run, and then file a bug at <https://github.com/pantsbuild/pants/issues>.
08:40:29.92 [INFO] Filesystem changed during run: retrying `Test` in 500ms...

How do I let my tests recognize

pyproject.toml

in the repository root? (It's not used for distribution builds but only to configure tools)

Q25: How do I pass additional arguments to

pytest

without modifying

pants.toml

? (e.g., Just temporarily add

-s

to capture console outputs in the tests)

trying out the new pex-based lockfiles in 2.11, and running into a roadblock. not sure if a bug in pants/pex or a side effect of my repo’s ugly dependency tree 🤔 I’m running on an intel mac in my

requirements.txt

I have:

numpy==1.19.5
opencv-python-headless==4.5.5.62

this works fine with vanilla

pip install

, but when I run

./pants generate-lockfiles

I get the failure:

ERROR: Cannot install numpy==1.19.5 and opencv-python-headless==4.5.5.62 because these package versions have conflicting dependencies.
ERROR: ResolutionImpossible: for help visit <https://pip.pypa.io/en/latest/user_guide/#fixing-conflicting-dependencies>
The conflict is caused by:
     The user requested numpy==1.19.5
     opencv-python-headless 4.5.5.62 depends on numpy>=1.21.2; python_version >= "3.6" and platform_system == "Darwin" and platform_machine == "arm64"

I checked the

setup.py

for that version of

opencv-python-headless

and confirmed that the newer numpy version is only required when running on ARM (see here). Is it a mistake that pex lockfile generation is trying to resolve a dependency for

platform_machine == "arm64"

when I’m running on an intel mac? or is this the correct behavior because the lockfile is trying to support both intel and m1 macs?

So I've done a bunch of fixup and am on Pants

2.11

now, but I'm back to running into the problem of a bunch of these errors:

uuid
    Required by:
      FingerprintedDistribution(distribution=<redacted> 2022.123.dev0+7d7bc1d951ada7b3f3d312e8850cace595786fdb (/home/noah/.cache/pants/named_caches/pex_root/installed_wheels/f39032808de7e6b03a697160fa392a8859ad782ea7bab0a2b192e58d91398627/<redacted>-2022.123.dev0+7d7bc1d951ada7b3f3d312e8850cace595786fdb-py3-none-any.whl), fingerprint='5c17f902db49407ef6ca4fa477de58d0e1ebf00cbbdd75d4b2770554cd068006')

Hi team, a (hopefully) quick question on

--changed-dependees

, suppose I have two tests in

unit/A_test.py

and

unit/B_test.py

and I only modified the

A_test.py

. It seem, if I run .

/pants --changed-since=origin/master --changed-dependees=transitive test

, the test in B_test.py also got triggered even

B_test.py

does not depend on

A_test.py

. I am wondering why? it is because they shared the same BUILD file

unit/BUILD

?

Trying to bump from

2.10.1rc0

to

2.11.0

and I'm seeing some funny behavior. When I make the bump and run something simple like

./pants help test

, I get the warning logs about deprecated macros and how to fix (I already have

use_deprecated_python_macros = false

from the previous bump from

2.9

to

2.10

). So, I delete that option and run the

update-build-files --fix-python-macros

and I get an error saying

use_deprecated_python_macros

is already set to false (first issue with the help logs since it says to just delete the option). So, I set to

true

and I get

11:53:37.02 [ERROR] 1 Exception encountered:

  MappingError: Failed to parse ./BUILD.pants:
__call__() got an unexpected keyword argument 'name'

I take it this is for

python_requirements()

so I delete the name field in there and then run it again and I get spammed with

* `python_requirements` in BUILD.pants: add `name="reqs"
11:54:34.40 [ERROR] You must manually add the `name=` field to the following targets. This is not done automatically by the `update-build-files` goal

So I'm in some kind of circular issue here and not sure how to resolve.