Slight bug with

export

in 2.15?

(3.8.10) ~/devel/monorepo (new-pants)$ ./pants export --resolve=pants-plugins
16:39:20.32 [INFO] Completed: chosen_resolve.name
16:39:20.94 [INFO] Completed: Get interpreter version
Wrote mutable virtualenv for pants-plugins (using Python 3.8.10) to dist/export/python/virtualenvs/pants-plugins

Looks like

chosen_resolve.name

isn't unpacked as intended. Happy to move to a ticket if that seems like the case.

Ooops yes!

I think

If I run

./pants export --resolve=<some resolve>

, it seems to wipe away any other exports I've done. Is it possible to keep existing exports in tact?

Sorry for the rapid fire, but I got a new thing after upgrading to 2.15 that I'm not really sure how to interpret.

Hi, everyone. 🙂 Thanks for being here. I'm struggling to migrate an existing Python/poetry project into a pants monorepo. It's not as simple as copy/paste/and run

pants tailor

. Does anyone have a link to a "Migrate to Pants (for dummies)" doc/video?

Is someone building aws lambda layers using pants?

Just a funny thought I had because of a typo I keep making repeatedly, when I build my first guess is to run

pants build <target>

but even though this is called

pantsbuild

there's actually no

pants build

but

pants package

Just thought this was funny

Hi! This is for python projects. Would it be a difficult task to change the implementation of vcs_version? I would need a different format for the output. Also, if the version is generated in a virtual file, how do I include it in a project's metadata? Regarding python packaging, I find it confusing and I am wandering rather aimlessly around....

Is pants 2.15 compatible with python 3.11?

Hi everyone, I'm new to Pants and I'm having some trouble with setting up the resolves. I would appreciate some advice! In my root dir, I have two submodules, let's call them

project

and

utils

. Both contain python code and have their own requirements.txt files.

project/requirements.txt

includes utils' requirements:

-r ../utils/requirements.txt

since project's code imports utility functions. This setup has been working fine (I have been able to run

pants test

command) until I decided to set up the resolves for the two submodules. When I run

pants test

, I'm constantly getting the following error:

ModuleNotFoundError: No module named utils

. This comes from one of the `project`'s test scripts which tries to import a function from the utils directory. I have been able to include `utils`' dependencies to `project`'s resolve by using the

parametrize

keyword, but I'm unable to include the

utils

submodule itself. Thanks in advance for any help!

What should I do for dealing with dependency issues due to the usage of an apple silicon mac? My error message:

Failed to resolve requirements from PEX environment @ /root/.pex/unzipped_pexes/ce56906b160237e1254e7999de724a1bdd708b54.
  Needed cp311-cp311-manylinux_2_31_aarch64 compatible dependencies for:
   1: MarkupSafe>=2.0
      Required by:
        Jinja2 3.1.2
      But this pex had no ProjectName(raw='MarkupSafe', normalized='markupsafe') distributions.
   2: pyrsistent!=0.17.0,!=0.17.1,!=0.17.2,>=0.14.0
      Required by:
        jsonschema 4.17.3
      But this pex had no ProjectName(raw='pyrsistent', normalized='pyrsistent') distributions.
   3: numpy
      Required by:
        altair 4.2.2
        streamlit 1.19.0
      But this pex had no ProjectName(raw='numpy', normalized='numpy') distributions.
   4: pandas>=0.18
      Required by:
        altair 4.2.2
      But this pex had no ProjectName(raw='pandas', normalized='pandas') distributions.
   5: pandas>=0.25
      Required by:
        streamlit 1.19.0
      But this pex had no ProjectName(raw='pandas', normalized='pandas') distributions.
   6: pillow>=6.2.0
      Required by:
        streamlit 1.19.0
      But this pex had no ProjectName(raw='pillow', normalized='pillow') distributions.
   7: pyarrow>=4.0
      Required by:
        streamlit 1.19.0
      But this pex had no ProjectName(raw='pyarrow', normalized='pyarrow') distributions.
   8: charset-normalizer<4,>=2
      Required by:
        requests 2.28.2
      But this pex had no ProjectName(raw='charset-normalizer', normalized='charset-normalizer') distributions.
   9: numpy>=1.16.4
      Required by:
        pydeck 0.8.0
      But this pex had no ProjectName(raw='numpy', normalized='numpy') distributions.
   10: tornado>=6.0.3
      Required by:
        streamlit 1.19.0
      But this pex had no ProjectName(raw='tornado', normalized='tornado') distributions.
   11: watchdog; platform_system != "Darwin"
      Required by:
        streamlit 1.19.0
      But this pex had no ProjectName(raw='watchdog', normalized='watchdog') distributions.
   12: watchdog==2.3.0
      But this pex had no ProjectName(raw='watchdog', normalized='watchdog') distributions.

Does anyone else use

aws-cdk-lib

in pexes? After v2.50 it seems to be broken for me. They broke out

aws_cdk.asset*

from the main

aws_cdk

provided by

aws-cdk-lib

and now whenever I try to run the PEX, it's unable to find

aws_cdk.asset*

packages.

I have a question about pex venvs and hard links. We unpack the same large pex file into multiple venvs on the same machine, and I want to make sure I'm doing everything possible to make the venv unpacking is fast and efficient. Do I understand correctly that the pex file itself is unpacked into PEX_ROOT once and the same package files are then shared via hard links into every venv that is created from that pex file? Is there anything I can do to make the second

pex-tools my-pex.pex venv /here

faster?

Got to use the new batching features from the new release with great success! We had some test setup that has a large one-time cost setting up some mocks. By batching, we can limit how often we incur that cost and reduced a huge chunk of test execution time by 7x

(3.8.10) ~/devel/monorepo (batching)$ time ./pants test --force ops/unittests/::
....
Wrote test reports to dist/test_results

real	1m16.427s
user	0m0.719s
sys	0m0.073s

Without batching configured

(3.8.10) ~/devel/monorepo (master)$ time ./pants test ops/unittests/::
....
Wrote test reports to dist/test_results

real	7m52.670s
user	0m0.701s
sys	0m0.076s

The docs also made it very clear how best to set up this feature for our needs by using the default flags in our build files.

does “SHA256 of https://github.com/pantsbuild/pex/releases/download/v2.1.103/pex is not as expected. Aborting.” mean anything to anyone here? i’m getting that in a github action. project is using 2.14.0 with

interpreter_constraints = ["CPython>=3.10"]

. i get the error during “Downloading the Pex PEX.” when I run

./pants tailor --check ::

in a github action

Anyone seeing:

exceeded timeout of 60 seconds while waiting for pantsd to start

?

Wanted to share a story about some issues we ran into with Pants and (primarily)

pex

. We recently integrated

pants

into a codebase and pushed some changes to production (we didn't catch these issues in our deployed development environment). In our

Dockerfile

we naively ran

pants package ::

and then just utilized the

pex

files generated without doing anything else. We have an

Argo Workflow

that spins up a bunch of pods (about 10 at launch and then ~ 100 more) and executes commands. We ended up with a lot of these pods stuck in "pod initializing" and a workflow that used to take 15 minutes ended up taking about an hour. Searching through metrics on what changed I found that our use of "ephemeral pod storage" sky rocketed. I downloaded the Docker images associated with each release and noticed when I ran the pex command and checked via

docker container ls -s

I was able to confirm that

pex

was using about 500MB of storage rather than 100B (what it was before the change). After following the instructions about [optimizing Docker deploys](https://blog.pantsbuild.org/optimizing-python-docker-deploys-using-pants/) I was able to get disk usage down. After deploying these changes, the workflow is now running smoothly. To be honest I don't understand the connection between ephemeral pod storage and pods failing to spin up (Argo Workflows was failing to spin up the

wait

container even after

main

had ran) I'm glad that blog article exists but looking at

pex

documentation I couldn't find anything about

PEX_TOOLS

or unpacking into a

venv

other than in the recipes area. Am I looking in the wrong place?

What's the best way for me to configure my project build based on the target environment (dev, qa, prod)? I want to know how to choose the right Docker repositories and similar components. Right now, I can create and distribute Docker images to the development system, but I need to figure out how to adjust the system for different environments. Can you recommend any relevant documentation?

I'm running into a problem running

./pants :: test --test-use-coverage --print-stacktrace

. After my tests pass I get an error saying invalid syntax. It seems like python3.7 which was used to bootstrap pants doesn't understand my

with (
 open(...) as a,
 open(...) as b
):

syntax. I would have expected coverage to use

python3.10

as I set

[python]
interpreter_constraints = ["CPython==3.10.*"]

Full Error is

Engine traceback:
  in select
  in pants.core.goals.test.run_tests
  in pants.backend.python.goals.coverage_py.generate_coverage_reports
Traceback (most recent call last):
  File "/home/yjabri/.cache/pants/setup/bootstrap-Linux-x86_64/2.14.1_py37/lib/python3.7/site-packages/pants/engine/internals/selectors.py", line 653, in native_engine_generator_send
    res = func.send(arg)
  File "/home/yjabri/.cache/pants/setup/bootstrap-Linux-x86_64/2.14.1_py37/lib/python3.7/site-packages/pants/backend/python/goals/coverage_py.py", line 572, in generate_coverage_reports
    keep_sandboxes=keep_sandboxes,
pants.engine.process.ProcessExecutionFailure: Process 'Generate Pytest report coverage report.' failed with exit code 1.
stdout:
Couldn't parse '/tmp/pants-sandbox-QX6Xdr/foo/tests/bar.py' as Python source: 'invalid syntax' at line 190

stderr:



Use `--keep-sandboxes=on_failure` to preserve the process chroot for inspection.

Continuing my journey into trying to make streamlit applications to be built from a mac M1 into linux containers. Trying to use environments now as pointed into the discussions. Some feedback: The developer experience has been driven mostly by error messages. Like:

ValueError: `[python-bootstrap].search_paths` is configured to use local Python discovery tools, which do not work in DockerEnvironmentTarget runtime environments. To fix this, set the value of `python_bootstrap_search_path` in the `docker_environment` defined at `//:docker` to contain only hardcoded paths or the `<PATH>` special string.

This is good but having this into the examples will avoid back and forth. Another thing is the plural hint for iterable parameters. Some parameters like

environment

and

environments

give you a hint if the value is an iterable or a simple string.

python_bootstrap_search_path

is singular but its value is an iterable. So it could be

python_bootstrap_search_paths

instead. I will keep posting here any roadblocks I find.

I’m having trouble getting the git auth to work with pants for installing python packages from a private github repo. It works fine using SSH but not all developers have SSH access set up and want it to be able to work with a github token. More details in 🧵

One more roadblock. What am I doing wrong? My environment: Macbook Pro M1 Max BUILD root:

poetry_requirements(
    name="poetry",
    module_mapping={
        "python-box": ["box"],
        "azure-storage-file-datalake": ["azure.storage.filedatalake.DataLakeServiceClient"]
    }
)
local_environment(
  name="local_linux",
  compatible_platforms=["linux_x86_64"],
  fallback_environment="docker_linux",
)
docker_environment(
    name="docker_linux",
    platform="linux_x86_64",
    image="python:3.11-slim",
    python_bootstrap_search_path=["<PATH>"]
)

src/streamlit/template/BUILD:

python_sources()
pex_binary(
    name="bin",
    script="streamlit",
    dependencies=["./main.py", "//:poetry#watchdog"],
    environment="docker_linux"
)
docker_image(name="docker")

src/streamlit/template/Dockerfile:

FROM python:3.11-slim
COPY src.streamlit.template/bin.pex /bin/streamlit
CMD /bin/streamlit

src/streamlit/template/main.py:

import streamlit as st

st.title("My Streamlit App")
st.text("Hello, world!")
st.balloons()

root pants.toml:

[GLOBAL]
pants_version = "2.15.0"
backend_packages = [
  "pants.backend.python",
  "pants.backend.docker"
]

[anonymous-telemetry]
enabled = false

[python]
tailor_pex_binary_targets = false
interpreter_constraints = ["CPython==3.11.2"]
enable_resolves = true

[python-bootstrap]
search_path = ["<PYENV>"]

[docker]
env_vars = ["DOCKER_CONFIG=%(homedir)s/.docker"]
default_repository = "{parent_directory}/{directory}"

[environments-preview.names]
docker_linux = "//:docker_linux"
local_linux = "//:local_linux"

Shell command:

pants run src/streamlit/template:docker

Error Output:

09:41:51.03 [INFO] Completed: Building docker image streamlit/template:latest
Failed to find compatible interpreter on path /usr/local/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin.

Examined the following interpreters:
1.) /usr/local/bin/python3.11 CPython==3.11.2

No interpreter compatible with the requested constraints was found:

  Failed to resolve requirements from PEX environment @ /root/.pex/unzipped_pexes/c411c8338ca0b0531952668a33219de7ba52349e.
  Needed cp311-cp311-manylinux_2_31_aarch64 compatible dependencies for:
   1: MarkupSafe>=2.0
      Required by:
        Jinja2 3.1.2
      But this pex had no ProjectName(raw='MarkupSafe', normalized='markupsafe') distributions.
   2: numpy
      Required by:
        altair 4.2.2
        streamlit 1.19.0
      But this pex had no ProjectName(raw='numpy', normalized='numpy') distributions.
   3: pandas>=0.18
      Required by:
        altair 4.2.2
      But this pex had no ProjectName(raw='pandas', normalized='pandas') distributions.
   4: pandas>=0.25
      Required by:
        streamlit 1.19.0
      But this pex had no ProjectName(raw='pandas', normalized='pandas') distributions.
   5: pillow>=6.2.0
      Required by:
        streamlit 1.19.0
      But this pex had no ProjectName(raw='pillow', normalized='pillow') distributions.
   6: pyarrow>=4.0
      Required by:
        streamlit 1.19.0
      But this pex had no ProjectName(raw='pyarrow', normalized='pyarrow') distributions.
   7: charset-normalizer<4,>=2
      Required by:
        requests 2.28.2
      But this pex had no ProjectName(raw='charset-normalizer', normalized='charset-normalizer') distributions.
   8: numpy>=1.16.4
      Required by:
        pydeck 0.8.0
      But this pex had no ProjectName(raw='numpy', normalized='numpy') distributions.
   9: tornado>=6.0.3
      Required by:
        streamlit 1.19.0
      But this pex had no ProjectName(raw='tornado', normalized='tornado') distributions.
   10: watchdog; platform_system != "Darwin"
      Required by:
        streamlit 1.19.0
      But this pex had no ProjectName(raw='watchdog', normalized='watchdog') distributions.
   11: watchdog==2.3.0
      But this pex had no ProjectName(raw='watchdog', normalized='watchdog') distributions.

So far it seems that adding the

extra_type_stubs

in the

pants.toml

did the trick, and it’s worth pinning the versions of you extra_types_stubs in the config. I’m surprised though, as it looks like the stubs specified in the different dependencies universes would never be used by

mypy

: how to specify different stubs version for different package versions? For example yesterday, a new version of the pandas-stubs has been released and caused errors to appear (most likely because of the pre-release of

pandas==2.0.0rc0

). Let’s image I want to have 2 resolves with different pandas versions for which the stubs are not compatible, how shall I proceed?

Hello! Is it possible to configure

docker_image

in such a way that the build root / context is at the repository's root? I have the following directory structure:

/
  utils 
  project
    sub_dir
      Dockerfile
      BUILD (with docker_image target)

In the Dockerfiles, I want to be able to

COPY  utils utils

and

COPY project project

and execute code in the container from the root direcotry (e.g. python files might

from utils import ...

) I tried to set

dependencies=["project", "utils"]

in

docker_image

, but when trying to

pants package

I keep getting the error stating that project and utils were not found in Docker build context. What am I doing wrong?

is there a way to call any files that are inside of the pex binary? like running a script that access files inside of the pex bundle

Is there a way to

./pants run

something indefinitely in a way that doesn't block other pants invocations? I'd like to avoid needing separate package and execution steps if possible.

When I ran

pants export ::

, it hanged. (m1) Is there way to find a problem here?

12:15:05.39 [DEBUG] Completed: pants.backend.python.util_rules.pex.build_pex
12:15:05.39 [DEBUG] Completed: Scheduling: Get interpreter version
12:15:05.39 [DEBUG] Completed: Determine all Python interpreter versions used by Pytest in your project (for `export` goal)
12:15:05.40 [DEBUG] Completed: Scheduling: Building pytest.pex from pytest_default.lock
12:15:05.40 [DEBUG] Completed: pants.backend.python.util_rules.pex.build_pex
12:15:05.40 [DEBUG] Completed: Scheduling: Get interpreter version
12:15:05.52 [DEBUG] Completed: setup_sandbox

When stops

^CTraceback (most recent call last):
  File "/Users/szto/.pyenv/versions/3.9.15/lib/python3.9/runpy.py", line 197, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/Users/szto/.pyenv/versions/3.9.15/lib/python3.9/runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "/Users/szto/.cache/pants/named_caches/pex_root/unzipped_pexes/404b7939ac9a4cbb33f728a5723c1f4b67d9a803/__main__.py", line 103, in <module>
    bootstrap_pex(__entry_point__, execute=__execute__, venv_dir=__venv_dir__)
  File "/Users/szto/.cache/pants/named_caches/pex_root/unzipped_pexes/404b7939ac9a4cbb33f728a5723c1f4b67d9a803/.bootstrap/pex/pex_bootstrapper.py", line 599, in bootstrap_pex
    pex.PEX(entry_point).execute()
  File "/Users/szto/.cache/pants/named_caches/pex_root/unzipped_pexes/404b7939ac9a4cbb33f728a5723c1f4b67d9a803/.bootstrap/pex/pex.py", line 551, in execute
    sys.exit(self._wrap_coverage(self._wrap_profiling, self._execute))
  File "/Users/szto/.cache/pants/named_caches/pex_root/unzipped_pexes/404b7939ac9a4cbb33f728a5723c1f4b67d9a803/.bootstrap/pex/pex.py", line 458, in _wrap_coverage
    return runner(*args)
  File "/Users/szto/.cache/pants/named_caches/pex_root/unzipped_pexes/404b7939ac9a4cbb33f728a5723c1f4b67d9a803/.bootstrap/pex/pex.py", line 489, in _wrap_profiling
    return runner(*args)
  File "/Users/szto/.cache/pants/named_caches/pex_root/unzipped_pexes/404b7939ac9a4cbb33f728a5723c1f4b67d9a803/.bootstrap/pex/pex.py", line 583, in _execute
    return self.execute_entry(
  File "/Users/szto/.cache/pants/named_caches/pex_root/unzipped_pexes/404b7939ac9a4cbb33f728a5723c1f4b67d9a803/.bootstrap/pex/pex.py", line 775, in execute_entry
    return self.execute_module(entry_point.module)
  File "/Users/szto/.cache/pants/named_caches/pex_root/unzipped_pexes/404b7939ac9a4cbb33f728a5723c1f4b67d9a803/.bootstrap/pex/pex.py", line 783, in execute_module
    runpy.run_module(module_name, run_name="__main__", alter_sys=True)
  File "/Users/szto/.pyenv/versions/3.9.15/lib/python3.9/runpy.py", line 225, in run_module
    return _run_module_code(code, init_globals, run_name, mod_spec)
  File "/Users/szto/.pyenv/versions/3.9.15/lib/python3.9/runpy.py", line 97, in _run_module_code
    _run_code(code, mod_globals, init_globals,
  File "/Users/szto/.pyenv/versions/3.9.15/lib/python3.9/runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "/Users/szto/.cache/pants/named_caches/pex_root/installed_wheels/71c50e03e63506d9e56add7cd5a4d6bb9fcc7985f39e7d2e152233fc4d6fd74d/pex-2.1.111-py2.py3-none-any.whl/pex/tools/__main__.py", line 10, in <module>
    sys.exit(tools())
  File "/Users/szto/.cache/pants/named_caches/pex_root/installed_wheels/71c50e03e63506d9e56add7cd5a4d6bb9fcc7985f39e7d2e152233fc4d6fd74d/pex-2.1.111-py2.py3-none-any.whl/pex/tools/main.py", line 93, in main
    result = catch(pex_command.run, pex)
  File "/Users/szto/.cache/pants/named_caches/pex_root/installed_wheels/71c50e03e63506d9e56add7cd5a4d6bb9fcc7985f39e7d2e152233fc4d6fd74d/pex-2.1.111-py2.py3-none-any.whl/pex/result.py", line 105, in catch
    return func(*args, **kwargs)
  File "/Users/szto/.cache/pants/named_caches/pex_root/installed_wheels/71c50e03e63506d9e56add7cd5a4d6bb9fcc7985f39e7d2e152233fc4d6fd74d/pex-2.1.111-py2.py3-none-any.whl/pex/tools/commands/venv.py", line 202, in run
    venv.install_pip()
  File "/Users/szto/.cache/pants/named_caches/pex_root/installed_wheels/71c50e03e63506d9e56add7cd5a4d6bb9fcc7985f39e7d2e152233fc4d6fd74d/pex-2.1.111-py2.py3-none-any.whl/pex/venv/virtualenv.py", line 352, in install_pip
    self._interpreter.execute(args=["-m", "ensurepip", "-U", "--default-pip"])
  File "/Users/szto/.cache/pants/named_caches/pex_root/installed_wheels/71c50e03e63506d9e56add7cd5a4d6bb9fcc7985f39e7d2e152233fc4d6fd74d/pex-2.1.111-py2.py3-none-any.whl/pex/interpreter.py", line 1292, in execute
    stdout, stderr = Executor.execute(cmd, stdin_payload=stdin_payload, env=env, **kwargs)
  File "/Users/szto/.cache/pants/named_caches/pex_root/installed_wheels/71c50e03e63506d9e56add7cd5a4d6bb9fcc7985f39e7d2e152233fc4d6fd74d/pex-2.1.111-py2.py3-none-any.whl/pex/executor.py", line 93, in execute
    stdout_raw, stderr_raw = process.communicate(input=stdin_payload)
  File "/Users/szto/.pyenv/versions/3.9.15/lib/python3.9/subprocess.py", line 1134, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/Users/szto/.pyenv/versions/3.9.15/lib/python3.9/subprocess.py", line 1979, in _communicate
    ready = selector.select(timeout)
  File "/Users/szto/.pyenv/versions/3.9.15/lib/python3.9/selectors.py", line 416, in select
    fd_event_list = self._selector.poll(timeout)
KeyboardInterrupt
Interrupted by user.
12:21:15.08 [DEBUG] Completed: interactive_process

12:21:15.08 [DEBUG] Async completion is enabled: workunit callbacks will complete in the background.

Is there any way globally run pex? I am trying to create pex binary and want to add bin folder to execute anywhere in my mac. What is the option that I can do?

I've found some cases in our repo now that sandboxing is hiding the fact that some folks writing tests are leaving around artifacts that aren't getting cleaned up. While this isn't necessarily a problem since things are sandboxed and eventually get killed off, I think it's enabling bad habits. Are there any mechanisms available to help detect this beyond code review?