I have a situation where

pants tailor --check ::

fails in ci but not locally =( how does one debug this?

this has gotta be some small platform-specific setting / install that I'm missing but I haven't been able to find anything in the docs or online. when I attempt to build a docker image using the

docker_image

target and a simple Dockerfile on a MAC M1, it looks like the

dockerfile

module used by pants in my setup was compiled for

x86_64

but I need the

arm64

one. this feels like it might have something to do with this blurb on Mac interpreters. but changing the

interpreter_search_paths

property doesn't seem to have any affect, I suspect because that is probably referring to setting the python interpreter for the python backend tools and doesn't affect the docker backend tools. how do I get pants to create and use the

arm64

dockerfile module? here's the actual error:

Traceback (most recent call last):
  File "/private/var/folders/0k/c0crrpz142n_h62r7nnmpnsr0000gn/T/pants-sandbox-IbPeOY/.cache/pex_root/venvs/3d84392e96bb69d7972ace4055cb0144a30b9e8e/a58c33d44cdb8a7bd8160aa976f1a158437bee6a/pex", line 243, in <module>
    runpy.run_module(module_name, run_name="__main__", alter_sys=True)
  File "/Users/zach/.pyenv/versions/3.7.16/lib/python3.7/runpy.py", line 205, in run_module
    return _run_module_code(code, init_globals, run_name, mod_spec)
  File "/Users/zach/.pyenv/versions/3.7.16/lib/python3.7/runpy.py", line 96, in _run_module_code
    mod_name, mod_spec, pkg_name, script_name)
  File "/Users/zach/.pyenv/versions/3.7.16/lib/python3.7/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/Users/zach/.cache/pants/named_caches/pex_root/venvs/s/b9c7c433/venv/lib/python3.7/site-packages/__pants_df_parser.py", line 202, in <module>
    print(json.dumps([asdict(info) for info in main(*sys.argv[1:])]))
  File "/Users/zach/.cache/pants/named_caches/pex_root/venvs/s/b9c7c433/venv/lib/python3.7/site-packages/__pants_df_parser.py", line 202, in <listcomp>
    print(json.dumps([asdict(info) for info in main(*sys.argv[1:])]))
  File "/Users/zach/.cache/pants/named_caches/pex_root/venvs/s/b9c7c433/venv/lib/python3.7/site-packages/__pants_df_parser.py", line 61, in main
    from dockerfile import Command, parse_file, parse_string  # pants: no-infer-dep
ImportError: dlopen(/Users/zach/.cache/pants/named_caches/pex_root/venvs/s/b9c7c433/venv/lib/python3.7/site-packages/dockerfile.abi3.so, 0x0002): tried: '/Users/zach/.cache/pants/named_caches/pex_root/venvs/s/b9c7c433/venv/lib/python3.7/site-packages/dockerfile.abi3.so' (mach-o file, but is an incompatible architecture (have 'x86_64', need 'arm64')), '/System/Volumes/Preboot/Cryptexes/OS/Users/zach/.cache/pants/named_caches/pex_root/venvs/s/b9c7c433/venv/lib/python3.7/site-packages/dockerfile.abi3.so' (no such file), '/Users/zach/.cache/pants/named_caches/pex_root/venvs/s/b9c7c433/venv/lib/python3.7/site-packages/dockerfile.abi3.so' (mach-o file, but is an incompatible architecture (have 'x86_64', need 'arm64')), '/Users/zach/.cache/pants/named_caches/pex_root/venvs/3d84392e96bb69d7972ace4055cb0144a30b9e8e/a58c33d44cdb8a7bd8160aa976f1a158437bee6a/lib/python3.7/site-packages/dockerfile.abi3.so' (mach-o file, but is an incompatible architecture (have 'x86_64', need 'arm64')), '/System/Volumes/Preboot/Cryptexes/OS/Users/zach/.cache/pants/named_caches/pex_root/venvs/3d84392e96bb69d7972ace4055cb0144a30b9e8e/a58c33d44cdb8a7bd8160aa976f1a158437bee6a/lib/python3.7/site-packages/dockerfile.abi3.so' (no such file), '/Users/zach/.cache/pants/named_caches/pex_root/venvs/3d84392e96bb69d7972ace4055cb0144a30b9e8e/a58c33d44cdb8a7bd8160aa976f1a158437bee6a/lib/python3.7/site-packages/dockerfile.abi3.so' (mach-o file, but is an incompatible architecture (have 'x86_64', need 'arm64'))

I feel like with

scie-pants

mypy checks are slower. I am using it as a pre commit hook and don’t remember it being so slow before I am using with transitive dependees option.

Checking source files type hints.........................................Passed
- hook id: type-hint-check
- duration: 65.29s

Hello, I want to work with jupyter notebooks for some data science stuff in pants. I followed this work-around and it almost solves everything. However, how do I access local code that is shared? My folder structure looks like the following structure:

src/
├─ notebooks/
├─ libraries/
   ├─ library1/

I would like to access

library1

in my notebook. I tried to set source roots but that did not work out for me. Do you have any hints?

Is there a way to provide Azure container registry login credentials on the command line for

pants publish

for docker images? Can I just run

docker login

before running

pants publish

? Azure CR does not support credential helpers as far as I can tell.

Hello, We have some issues with out Pants (2.14) jobs sometimes get stuck when running in Jenkins.

./pants tailor --check update-build-files --check ::

Logs:

11:58:10    360.38s   Test binary /bin/python.
11:58:10    360.38s   Test binary /data/env/py3.9.13/bin/python.
11:58:10    360.38s   Test binary /opt/conda/bin/python.

Why is it "testing" multiple Python binaries? The only one that matters is one py3.9.13. One interesting observation is that if I connect to node and check processes I see two pantd running. What could be the issue? What are the steps to troubleshoot it?

I want to create a

pex_binary

that gets built based on a set of

python_distribution

targets so that the pex includes all of the entry_points metadata. The purpose of this pex is not to run a binary or script (so,

entry_point=None

); its purpose is to create a virtualenv from its contents (so,

include_tools=True

). I tried just depending on the

python_distribution

targets, but then pants looks at the transitive deps which includes things like a

relocated_file(source="LICENSE")

target. Because the

python_distribution

has transitive deps that are

files

instead of

resources

, I get a warning telling me to use

resources

instead. If I use

include_sources=False

, I still get the warning about transitive

files

targets. Oddly enough, the generated wheels still get installed in the pex, which I guess is fine since they are not sources (they are generated). So, is there a way to prevent the warning for this

pex_binary

target? Having that warning on other

pex_binary

targets that are meant to be executable is fine. But, in this case, I want an “I know what I’m doing” flag to signal that I expect pex to just use the generated wheels and install whatever the wheel includes, whether or not that wheel happened to somehow include a

files

target. So, is there a way to prevent that warning for a particular

pex_binary

target?

Hi all, I'm having trouble getting isort to work correctly with multiple "projects" within a monorepo: isort classifies my python modules as third-party. I posted a question on SO: https://stackoverflow.com/questions/75869900/pants-with-isort-thinks-my-python-modules-are-third-party. Thanks in advance!

Will

pants publish

only deploy previously packaged targets or will it package anything that hasn't been already?

I’m getting the dreaded

Failed to digest inputs: "Error storing Digest: Input/output error"

. My understanding is this comes from my Pex file being too large. Is there a good way to debug/understand how big the things going into it are? I assume that this is because I’m using a GPU version of PyTorch which is a massive

whl

Hi, I have a Python test that uses Spark, which depends on having a JVM. Is there a way to set a dependency on a JVM with a specific version? (for instance,

openjdk 11

)

Share my MLOPS open source project here: https://github.com/komprenilo/liga It is now a relatively complicated open source project using Pants!

Hey, am seeing something weird and wanted to see if anyone had any ideas. We're trying to structure a python package like this:

mypackage/__init__.py
mypackage/hello.py
mypackage/tests/test_hello.py

However, when we try to run the test file using pants, it complains it cannot import module

mypackage.hello

. However,

pants dependencies --transitive mypackage/tests/test_hello.py

shows all `mypackage/hello.py`'s dependencies as well, only the module itself is missing. So it seems like pants is correctly inferring that

test_hello.py

depends on

hello.py

but then not adding

hello.py

in as a dependency?

Hey Pants people, What is the best way to run module level scripts using the correct Pex environment? For example, I’m trying to use Mosaic’s composer tool, and to use the tool you install the mosaicml module and then run

composer training_script.py

In Pants is there a way to inject a Python environment into a command like this? For example, perhaps I create a shell script that runs this command then Pants creates the Pex environment somehow?

I have some more organizational questions for people who have successfully integrated pants for managing their monorepo(s) • do you have roles / people specifically responsible for developing and maintaining the build system? or are most developers expected to learn Pants and be able to integrate their code with it when they need to have artifacts built / deployed? Pants is quite powerful but also seems quite complex, I worry a bit about it being an impediment to developers on my team being able to deploy code. maybe it's not that big of a deal once it gets set up? • do you build additional tooling on top of Pants? I can already see some strong use cases for creating custom targets for deploying artifacts to Databricks and Dagster so they can be used there. are there other kinds of Pants-related tooling you've created or use to help make it easier for developers to get building? • any recommended reading for supporting and structuring monorepos and teams that work with them?

Hello, We are running into "Resource temporary unavailable" error when using Pants (2.14) in the concurrent CI environment (Jenkins, docker, kubernetes). The example command and debug logs are attached. CC: @gray-shoe-19951

Genuine question: why is the pants build system called pants?

Hi, I have problems setting up the experimental ruff linter. It seems to work but my config gets ignored. I have a the following setup: pyproject.toml

[tool.ruff]
cache_dir = ".cache_and_logs/.ruff"
line-length=40
select = ["E", "F", "B", "Q"]

pants.toml:

[ruff]
lockfile = "lockfiles/ruff.lock"

The line-length limit is set to 40, but when running pants fmt or fix ruff does not throw an error or changes anything. My python file contains a long list in one line (>120 chars)

How can I insert environment variables, variables defined in

pants.toml

or in a

BUILD

files into a string? Is there a f-string style syntax I can use?

docker_image(
    name="my_service",
    dependencies=[":my_pex"],
    instructions=[
        f"FROM python:{PYTHON_VERSION_MAJOR}.{PYTHON_VERSION_MINOR}.{PYTHON_VERSION_MICRO}-slim-bullseye",
        "COPY my_pex.pex .",
    ],
)

Hi! Im evaluating pants for use on existing projects, we want to move to a monorepo architecture and use pants to help us manage dependencies. The question we have is how does Pants handle multiple github deployment targets for different projects within the same repo?

Greetings! I'm attempting to test multiple interpreters. Using pants 2.15.0. Want to support python 3.8, 3.9 and 3.10 in testing. Ubuntu 22.04 with 3.10 native. Used Deadsnake to install python3.8 and python3.9. pants.toml

[python]
  interpreter_constraints = [">=3.8,  <3.11"]

From project tests BUILD

python_tests(
    name="tests",
    dependencies=[
        "src/python/project-lib:lib",
    ],
    sources=[
        "**/*_test.py",
    ],
    interpreter_constraints=parametrize(py38=["==3.8.*"], py39=["==3.9.*"], py310=["==3.10.*"]),
)

My general flow:

pants --keep-sandboxes=on_failure lint src/python/project-lib:lib
pants --keep-sandboxes=on_failure check src/python/project-lib:lib
pants --keep-sandboxes=on_failure test test/python/project-lib:tests -- -s

The lint and check goals during execution I'm getting errors when pants attempts to build the requirements.pex and the pip.log doesn't exist (only showing error from *lint*; the check goal it's during mypy.pex building):

11:05:27.77 [INFO] Canceled: Building pylint.pex from pylint_default.lock
11:05:28.38 [INFO] Preserving local process execution dir /tmp/pants-sandbox-3ffJgp for Building requirements.pex with 7 requirements: backoff~=2.2.1; python_version >= "3.7" and python_version < "4.0", jsonpickle~=3.0.1; python_version >= "3.7", loguru~=0.6.0; python_version >= "3.5", pandas~=1.5.3; python_version >= "3.8", pendulum~=2.1.2; (python_version >= "2.7" and python_full_version < "3.0.0") or python_full_version >= "3.5.0", python-magic~=0.4.27; (python_version >= "2.7" and python_full_version < "3.0.0") or python_full_version >= "3.5.0", requests~=2.28.2; python_version >= "3.7" and python_version < "4"
11:05:28.38 [INFO] Completed: Building requirements.pex with 7 requirements: backoff~=2.2.1; python_version >= "3.7" and python_version < "4.0", jsonpickle~=3.0.1; python_version >= "3.7", loguru~=0.6.0; python_version >= "3.5", p... (334 characters truncated)
11:05:28.39 [ERROR] 1 Exception encountered:

Engine traceback:
  in `lint` goal
  in Lint using Pylint

ProcessExecutionFailure: Process 'Building requirements.pex with 7 requirements: backoff~=2.2.1; python_version >= "3.7" and python_version < "4.0", jsonpickle~=3.0.1; python_version >= "3.7", loguru~=0.6.0; python_version >= "3.5", pandas~=1.5.3; python_version >= "3.8", pendulum~=2.1.2; (python_version >= "2.7" and python_full_version < "3.0.0") or python_full_version >= "3.5.0", python-magic~=0.4.27; (python_version >= "2.7" and python_full_version < "3.0.0") or python_full_version >= "3.5.0", requests~=2.28.2; python_version >= "3.7" and python_version < "4"' failed with exit code 1.
stdout:

stderr:
[Errno 2] No such file or directory: '/tmp/pants-sandbox-3ffJgp/.tmp/pex-pip-log.2s_u5pjx/pip.log'

The test goal is a bit different, failing on building *pytest.pe*x with a downloading issue:

11:06:21.00 [INFO] Canceled: Building pytest.pex from pytest_default.lock
11:06:21.67 [INFO] Preserving local process execution dir /tmp/pants-sandbox-TEAYuo for Building pytest.pex from pytest_default.lock
11:06:21.67 [INFO] Completed: Building pytest.pex from pytest_default.lock
11:06:21.67 [ERROR] 1 Exception encountered:

Engine traceback:
  in `test` goal
  in Run Pytest - test/python/hank-ai-lib/util_util_test.py:tests@interpreter_constraints=py39

ProcessExecutionFailure: Process 'Building pytest.pex from pytest_default.lock' failed with exit code 1.
stdout:

stderr:
There was 1 error downloading required artifacts:
1. coverage 6.4.3 from <https://files.pythonhosted.org/packages/a5/6a/5a201c1b85c5c406c5ee5be4d17c223ad71a5f77937fe9a680b02e6a1fb3/coverage-6.4.3-cp39-cp39-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl>
    ModuleNotFoundError: No module named 'distutils.util'

First time attempting to test multiple interpreters. Pants is supposed to pull in all the deps in the sandbox for the particular interpreter version correct? Am I grossly missing something to support multiple interpreter versions for testing? Any and all help is greatly appreciated. Thanks.

Is there a way to just get the populated values file from a

helm_deployment

without the chart being installed?

I'm trying to create a pex for a python CLI script in pants 2.15 (upgrading from 1.30), and if I use the

script

tag in my build file for the pex binary then pants complains that it cannot find the file that's in the same directory as the build file. If I use the

entry_point

tag I get an import error saying there was no module main found. I'l curious to see what I could be doing wrong

Is there a way to pass arguments to the docker build command during a

docker_image

target invocation? Context is I need to pass networking mode and a host override to use an existing ssh tunnel. I see

run_args

and

build_args

but I don’t think those work for this

Is there a way to parametrize such that 2 fields change, but without producing a cartesian product of the two fields? ie, the docs say (https://www.pantsbuild.org/v2.16/docs/targets#parametrizing-targets):

If multiple fields are parametrized, a target will be created for each value in the Cartesian product, with

,

as the delimiter in the address. See the next example.

But, I want to parametrize two

pex_binary

fields:

interpreter_constraints

and

output_path

. I want to use the named interpreter constraint in the

output_path

. So, IC

py36

has

output_path="st2-py36.pex"

I tried the following, but that (of course) did the Cartesian product which does not make sense:

pex_binary(
    ...
    interpreter_constraints=parametrize(
        py36=["CPython==3.6.*"],
        py37=["CPython==3.7.*"],
        py38=["CPython==3.8.*"],
    ),
    output_path=parametrize(
        py36="st2-py36.pex",
        py37="st2-py37.pex",
        py38="st2-py38.pex",
    ),
)

pip install

supports a

--target /path/to/dir

argument to install the reqs directly into that directory, without the usual venv structure. This is very handy for building lambda functions/layers, because doing that generally gives a directory that works as the root of a function/layer (e.g. it's what AWS's CDK does in their lambda packaging construct). Is there way to achieve something similar within when given a PEX file (or via the PEX CLI)? I'm going to experiment with

PEX_TOOLS=1 file.pex venv $(mktemp)

and using the contents of the

lib/python*/site-packages/

directory alone, dropping the rest of the venv structure, but that doesn't seem particularly right, on the surface...

I'm building a pex binary that excludes some dependencies, but they're still showing up in the artifact.

pex_binary (
    name="predictor",
    inherit_path="fallback",
    layout="zipapp",
    dependencies=[
        "src/dz/recsys/on_lambda.py",
        "3rdparty/python:psycopg2-binary",
        "3rdparty/python#scikit-learn",
        "!!3rdparty/python#boto3",
        "!!3rdparty/python#aws-lambda-powertools",
        "!!3rdparty/python#boto3-stubs",
        "!!3rdparty/python:jmespath",
        "!!3rdparty/python:pydantic",
        "!!3rdparty/python:urllib3",
        "!!3rdparty/python#psycopg2",
    ]
)

❯ pants dependencies --transitive src/dz/recsys:predictor
3rdparty/python#cloudpathlib
3rdparty/python#numpy
3rdparty/python#pandas
3rdparty/python#pandas-stubs
3rdparty/python#scikit-learn
3rdparty/python#sentry-sdk
3rdparty/python/pyproject.toml
3rdparty/python:psycopg2-binary
lockfiles/python-default.lock:python-default
src/...

but if I package and inspect the contents of deps, I'm still seeing boto3 in the packaged file

❯ du -h -d2
2.4M	./.bootstrap/pex
2.4M	./.bootstrap
1.1M	./.deps/boto3-1.26.103-py3-none-any.whl
76M	./.deps/botocore-1.29.103-py3-none-any.whl
312K	./.deps/certifi-2022.12.7-py3-none-any.whl
184K	./.deps/cloudpathlib-0.10.0-py3-none-any.whl
112K	./.deps/jmespath-1.0.1-py3-none-any.whl
1.3M	./.deps/joblib-1.2.0-py3-none-any.whl
59M	./.deps/numpy-1.22.4-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
43M	./.deps/pandas-1.5.1-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
1.2M	./.deps/pandas_stubs-1.4.3.220710-py3-none-any.whl
7.2M	./.deps/psycopg2_binary-2.9.5-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
524K	./.deps/python_dateutil-2.8.2-py2.py3-none-any.whl
2.7M	./.deps/pytz-2023.3-py2.py3-none-any.whl
336K	./.deps/s3transfer-0.6.0-py3-none-any.whl
32M	./.deps/scikit_learn-1.2.2-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
123M	./.deps/scipy-1.7.3-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
820K	./.deps/sentry_sdk-1.18.0-py2.py3-none-any.whl
60K	./.deps/six-1.16.0-py2.py3-none-any.whl
72K	./.deps/threadpoolctl-3.1.0-py3-none-any.whl
528K	./.deps/urllib3-1.26.15-py2.py3-none-any.whl
346M	./.deps
4.0K	./__pex__
428K	./src/customers
44K	./src/data
84K	./src/dz
8.0K	./src/learning
16K	./src/meta
580K	./src
440M	.

jmespath, boto3, and urllib3 are still present in the pex despite being explicitly excluded. Any idea how I can diagnose this?

Hi, I'd appreciate some help on how to handle requirements that are implicitly present when running certain targets, but I still need to import them in code. In particular I mean importing

pytest

in files that belong to

python_tests

. It's there when running

./pants test

and works fine, but it's rightfully not recognized by the IDE when using the env from

./pants export

How should I approach this? Should I add

pytest

to

requirements

? If so, is there a way to keep the versions synchronized?

For the Pants

test

goal to run Python tests (https://www.pantsbuild.org/docs/python-test-goal), is there an option to do more than one run per test? The context is that I want to do periodic runs in CI where all unit tests are reran forcefully (i.e., with the

--force

option) in order to help catch unit test flakiness. I also would like to do more than one run per test. For comparison, in Bazel one can do

--runs_per_test=<n> --cache_test_results=no

, which fits the ask perfectly. However, Pants doesn't seem to have the option to do more than forceful run per test. I suppose one could do something such as wrap

./pants test --force ::

in a Bash loop (

for i in {1..<n>}; do ./pants test --force ::; done

), but that feels clunky.

In the prerequisites page, it mentions:

Apple Silicon (M1/M2) support
If you have Python code, you may need to set your interpreter constraints to Python 3.9+, as many tools, such as Black, will not install correctly when using earlier Python versions.

Is this saying pants itself just needs to run with py3.9+ or the python project needs to run with python3.9+? Our project currently runs on python 3.7.16 and a lot of our eng use M1/M2 macs. Wondering if I should try to set pants up using a container from the get go.