pantsbuild #github-notifications

Channels

welcome

github-notifications

development

cool-easter-32542

04/10/2023, 2:44 PM

#18706 Can't use new version of docformatter. Issue created by asherf Describe the bug Noticed that in an internal repo but this also happens on the pants repo itself. #18705 Pants version latest main branch OS This only repos on macos, seems to work fine on linux (ubuntu)

Copy code

pants lint --only=docformatter src/python/pants/option/::
10:37:44.09 [INFO] Initializing scheduler...
10:37:50.73 [INFO] Scheduler initialized.
10:37:53.35 [ERROR] 1 Exception encountered:

Engine traceback:
  in select
    ..
  in pants.core.goals.lint.lint
    `lint` goal

Traceback (most recent call last):
  File "/Users/asher/projects/pants/src/python/pants/engine/internals/selectors.py", line 626, in native_engine_generator_send
    res = rule.send(arg) if err is None else rule.throw(throw or err)
  File "/Users/asher/projects/pants/src/python/pants/core/goals/lint.py", line 470, in lint
    all_batch_results = await MultiGet(
  File "/Users/asher/projects/pants/src/python/pants/engine/internals/selectors.py", line 361, in MultiGet
    return await _MultiGet(tuple(__arg0))
  File "/Users/asher/projects/pants/src/python/pants/engine/internals/selectors.py", line 168, in __await__
    result = yield self.gets
  File "/Users/asher/projects/pants/src/python/pants/engine/internals/selectors.py", line 626, in native_engine_generator_send
    res = rule.send(arg) if err is None else rule.throw(throw or err)
  File "/Users/asher/projects/pants/src/python/pants/backend/python/lint/docformatter/rules.py", line 42, in docformatter_fmt
    result = await Get(
  File "/Users/asher/projects/pants/src/python/pants/engine/internals/selectors.py", line 118, in __await__
    result = yield self
pants.engine.process.ProcessExecutionFailure: Process 'Run Docformatter on 29 files.' failed with exit code 3.
stdout:

stderr:



Use `--keep-sandboxes=on_failure` to preserve the process chroot for inspection.

https://app.toolchain.com/organizations/pantsbuild/repos/pants/builds/pants_run_2023_04_10_10_44_00_268_d1675374c4614e38932846f365ec2ee0/type/run/?subTab=logs pantsbuild/pants

cool-easter-32542

04/10/2023, 8:00 PM

#11812 dirty python packages in dist Issue created by kaos When running

./pants package ::

any previous packages in

./dist/

ought to be removed first in order to clean out old files for any target packages. I have noticed that I have to manually remove them in case I re-organize my code, the now no longer referenced files are otherwise still left in the tree. pantsbuild/pants

cool-easter-32542

04/12/2023, 9:15 AM

#2112 How to specify a python interpreter in a virtualenv on the run using PEX_PYTHON？ New discussion created by zeyuwang615 Hi, there! I am trying to run a packed python .pex file with the command

./xxx.pex main.py

. However, I just discover that there are a few necessary packages missing in the pex file. So I try to specify the custom python interpreter in a virtualenv with all the necessary packages with the

PEX_PYTHON

environment variable. Strangely,

PEX_PYTHON=/path/to/venv/python ./xxx.pex main.py

does not work, and so does

/path/to/venv/python ./xxx.pex main.py

. I wonder how I can specify the python environment or add the necessary packages without needing to regenerate or change the .pex file? Thanks a lot! pantsbuild/pex

cool-easter-32542

04/12/2023, 4:23 PM

#18430 Ruff integration in pantsbuild only fails on auto-fixable errors Issue created by siggi84 Describe the bug The Ruff linter integration in Pantsbuild system only fails the build when it encounters a linting rule that it can automatically fix. If it encounters a rule that it cannot fix on its own, it allows the build to pass, which is inconsistent with the Flake8 integration. This behavior can lead to unnoticed linting errors. Pants version 2.16.0.dev7 OS Ubuntu 22.10 Additional info A minimal repository demonstrating the error is available. It highlights the different handling of two linting errors (one fixable F405 and another unfixable F401) by Ruff and Flake8. pantsbuild/pants

cool-easter-32542

04/12/2023, 6:33 PM

#18726 Assertion 'rc == 0' failed in mdb_page_dirty() Issue created by jtilahun Describe the bug Attempting the

export

goal with the

--no-pantsd

argument results in failure on my system with the below error message:

Copy code

jtilahun@JTN86G3:~/devel/monorepo$ ./pants --print-stacktrace --no-export-symlink-python-virtualenv --no-pantsd export --resolve=python-default
/github/home/.cargo/git/checkouts/lmdb-rs-369bfd26153a2575/6ae7a55/lmdb-sys/lmdb/libraries/liblmdb/m:2126: Assertion 'rc == 0' failed in mdb_page_dirty()
Aborted (core dumped)

The error message indicates a possible relation to other bug reports, e.g. bmatsuo/lmdb-go/issues/131. Pants version 2.15.0 OS Linux Additional info The context is that attempting the

export

goal without the

--no-pantsd

argument results in failure on my system with the below error message:

Copy code

jtilahun@JTN86G3:~/devel/monorepo$ ./pants --print-stacktrace --no-export-symlink-python-virtualenv export --resolve=python-default
11:30:06.43 [INFO] Initializing scheduler...
11:30:07.37 [INFO] Scheduler initialized.
Traceback (most recent call last):
  File "/home/jtilahun/.cache/pants/setup/bootstrap-Linux-x86_64/2.15.0_py38/bin/pants", line 8, in <module>
    sys.exit(main())
  File "/home/jtilahun/.cache/pants/setup/bootstrap-Linux-x86_64/2.15.0_py38/lib/python3.8/site-packages/pants/bin/pants_loader.py", line 123, in main
    PantsLoader.main()
  File "/home/jtilahun/.cache/pants/setup/bootstrap-Linux-x86_64/2.15.0_py38/lib/python3.8/site-packages/pants/bin/pants_loader.py", line 110, in main
    cls.run_default_entrypoint()
  File "/home/jtilahun/.cache/pants/setup/bootstrap-Linux-x86_64/2.15.0_py38/lib/python3.8/site-packages/pants/bin/pants_loader.py", line 92, in run_default_entrypoint
    exit_code = runner.run(start_time)
  File "/home/jtilahun/.cache/pants/setup/bootstrap-Linux-x86_64/2.15.0_py38/lib/python3.8/site-packages/pants/bin/pants_runner.py", line 89, in run
    return remote_runner.run(start_time)
  File "/home/jtilahun/.cache/pants/setup/bootstrap-Linux-x86_64/2.15.0_py38/lib/python3.8/site-packages/pants/bin/remote_pants_runner.py", line 123, in run
    return self._connect_and_execute(pantsd_handle, executor, start_time)
  File "/home/jtilahun/.cache/pants/setup/bootstrap-Linux-x86_64/2.15.0_py38/lib/python3.8/site-packages/pants/bin/remote_pants_runner.py", line 161, in _connect_and_execute
    return PyNailgunClient(port, executor).execute(command, args, modified_env)
native_engine.PantsdClientException: The pantsd process was killed during the run.

If this was not intentionally done by you, Pants may have been killed by the operating system due to memory overconsumption (i.e. OOM-killed). If you keep seeing this error message, try the troubleshooting steps below. If none of those help, please consider filing a GitHub issue or reaching out on Slack so that we can investigate the possible memory overconsumption (<https://www.pantsbuild.org/docs/getting-help>).
 - Exit other applications, including applications running in the background.
 - Set the global option `--pantsd-max-memory-usage` to reduce Pantsd's memory consumption by retaining less in its in-memory cache (run `./pants help-advanced global`).
 - Disable pantsd with the global option `--no-pantsd` to avoid persisting memory across Pants runs, although you will miss out on additional caching.

According to the error message, disabling pantsd with the global option

--no-pantsd

is one possible troubleshooting step. However, attempting the

export

goal with the

--no-pantsd

argument also results in failure on my system. pantsbuild/pants

cool-easter-32542

04/13/2023, 2:03 AM

#18729 Stabilize the Python implementation of the `publish` goal Issue created by benjyw Publishing via Twine is currently experimental (implemented in

pants.backend.experimental.python

). This ticket is to track any open issues that must be addressed before moving this functionality out of

experimental

and into

pants.backend.python

. pantsbuild/pants

cool-easter-32542

04/13/2023, 8:40 AM

#18730 Read Interpreter Constraints from `.python-version` pyenv file Issue created by rhuanbarreto Is your feature request related to a problem? Please describe. Today we set the python version to work with our pants enabled repo by using the pyenv

.python-version

file. Every time we update our python version in the file, we need to remember to update the

[python].interpreter_constraints

["==new_version"]

in pants.toml or leave the constraint open like

[">=3.11"]

Describe the solution you'd like Accepting a value like

"<PYENV_LOCAL>"

as pants does in

[python-bootstrap].search_path

would simplify this process. Describe alternatives you've considered Today the process can only be made manually. pantsbuild/pants

cool-easter-32542

04/13/2023, 4:08 PM

#18736 JVM stabilization ticket Issue created by chrisjrn Currently, JVM features have been released for a year, but are still marked experimental. It would be good to release these as "stable" in the short-term. This ticket will accumulate a list of other tickets that block stabilization, and will remain open for a week unless stability-blocking bugs are added here. pantsbuild/pants

cool-easter-32542

04/14/2023, 11:27 AM

#18743 testing vcs_version target New discussion created by AliGouta I am new to pants, and this is something we want to adopt within our organization. I was testing different features and was not able to run :

pants export-codegen src/foo:version

as mentionned in the doc: https://www.pantsbuild.org/docs/generating-version-tags I tried it on a simple project with pants 2.15.0 and also tried to run :

./pants export-codegen src:vcs_version

on this example: https://github.com/benjyw/black/tree/pantsify3 Each time I get the same error. Example of the error :

Copy code

12:54:43.82 [INFO] Completed: Building setuptools_scm.pex from setuptools-scm_default.lock
12:54:44.21 [INFO] Completed: Run setuptools_scm for helloworld/greet:vcs_version
12:54:44.22 [ERROR] 1 Exception encountered:

Engine traceback:
  in `export-codegen` goal
  in Hydrate the `sources` field - helloworld/greet:vcs_version

ProcessExecutionFailure: Process 'Run setuptools_scm for helloworld/greet:vcs_version' failed with exit code 1.
stdout:

stderr:
Traceback (most recent call last):
  File "/private/var/folders/q8/ww__896s4rqfpt6mhtnblkvh0000gn/T/pants-sandbox-DgFRwT/.cache/pex_root/venvs/cf4a883a550c4c8a817e41f476854a593bfff6da/185060ca67f07b2fc4906338d0f05e510b3e8247/pex", line 243, in <module>
    runpy.run_module(module_name, run_name="__main__", alter_sys=True)
  File "/Users/aligouta/.pyenv/versions/3.8.5/lib/python3.8/runpy.py", line 207, in run_module
    return _run_module_code(code, init_globals, run_name, mod_spec)
  File "/Users/aligouta/.pyenv/versions/3.8.5/lib/python3.8/runpy.py", line 97, in _run_module_code
    _run_code(code, mod_globals, init_globals,
  File "/Users/aligouta/.pyenv/versions/3.8.5/lib/python3.8/runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "/Users/aligouta/.cache/pants/named_caches/pex_root/venvs/s/7c73d32a/venv/lib/python3.8/site-packages/setuptools_scm/__main__.py", line 83, in <module>
    main()
  File "/Users/aligouta/.cache/pants/named_caches/pex_root/venvs/s/7c73d32a/venv/lib/python3.8/site-packages/setuptools_scm/__main__.py", line 30, in main
    assert version is not None
AssertionError

Use --keep-sandboxes=on_failure to preserve the process chroot for inspection.

The foolowing commande works fine:

/Users/aligouta/.pyenv/versions/3.8.5/bin/python3.8 -m setuptools_scm

Am i missing something ? Thanks ! pantsbuild/pants

cool-easter-32542

04/14/2023, 2:46 PM

#18745 `InteractiveProcess` with `run_in_workspace=True` requires `Workspace` parameter on goal rule Issue created by ndellosa95 Describe the bug Trying to call an interactive process that runs in workspace requires an unused workspace parameter on the goal rule. See here. Otherwise you get the error:

Copy code

IntrinsicError: Side-effects are not allowed in this context: SideEffecting types must be acquired via parameters to `@rule`s.

Pants version 2.16.0rc0 OS MacOS pantsbuild/pants

cool-easter-32542

04/14/2023, 7:57 PM

#2066 Intermittent / nondeterministic failure building `requirements.pex` due to missing `site-packages` Issue created by danxmoran Our CI will occasionally fail with an error like:

Copy code

Traceback (most recent call last):
  File "/home/color/.cache/nce/99421dee8fedc336d5a6bb8322fbfa602bc68fe63303ae702ec7b9c5672cd086/bindings/venvs/2.15.0rc6/lib/python3.9/site-packages/pants/engine/process.py", line 289, in fallible_to_exec_result_or_raise
    raise ProcessExecutionFailure(
pants.engine.process.ProcessExecutionFailure: Process 'Building 1 requirement for requirements.pex from the 3rdparty/lockfiles/resolves/pants-plugins.lockfile resolve: pantsbuild.pants<2.16,>=2.15.0a0' failed with exit code 1.
stdout:

stderr:
The virtualenv at /tmp/pants/named_caches/pex_root/venvs/4c85013f478e0393bbf8db8fcf02e1def7ff5031/ba7a55164c2afb363895254bbb1063124dd74d5b.lck.work is not valid. No site-packages directory was found in its sys.path:
/opt/python/3.9.16/lib/python39.zip
/opt/python/3.9.16/lib/python3.9
/opt/python/3.9.16/lib/python3.9/lib-dynload
/opt/python/3.9.16/lib/python3.9/site-packages

We've seen it hit across different resolves, Python interpreter versions, and Pex versions - the error above happened on Pex

v2.1.122

. It happens very infrequently (once every few weeks). I captured the Pants caches & execution dir for the error above. The archive is too big to attach via GitHub, but I can share it via Slack / GDrive upload. pantsbuild/pex

cool-easter-32542

04/16/2023, 2:43 AM

#18749 Exporting Python tools with `::` and `--resolve` creates venvs in different paths Issue created by achimnol Describe the bug

./pants export ::

is the to-be-deprecated way to export venvs from all resolves including tools.

./pants export --resolve=black --resolve=isort ...

is the new recommended explicit way to do the same thing. I'd expect both way should create venvs in the same location but they differ: •

./pants export ::

dist/export/python/virtualenvs/tools/black

•

./pants export --resolve=black

dist/export/python/virtualenvs/black/3.11.3

Pants version 2.16.0.dev7 OS macOS 13.3.1 pantsbuild/pants

cool-easter-32542

04/17/2023, 12:45 AM

#1969 pex build fails due to existing work-directory Issue created by christopherfrieler Beginning with version 2.1.105 building the pex file in our CI pipeline fails with the following message:

.../python3.8/site-packages/pex/atomic_directory.py176 PEXWarning: [pid:XX, tid:XXX, cwd...] After obtaining an exclusive lock on <PEX_ROOT>/isolated/.2f4fc85fa2be055a2975ce1147100c0d5c7e663a.atomic_directory.lck, failed to establish a work directory at <PEX_ROOT>/isolated/2f4fc85fa2be055a2975ce1147100c0d5c7e663a.workdir due to: [Errno 17] File exists: '<PEX_ROOT>/isolated/2f4fc85fa2be055a2975ce1147100c0d5c7e663a.workdir'

pex_warnings.warn(

.../python3.8/site-packages/pex/atomic_directory.py187 PEXWarning: [pid:XX, tid:XXX, cwd...] Continuing to forcibly re-create the work directory at <PEX_ROOT>/isolated/2f4fc85fa2be055a2975ce1147100c0d5c7e663a.workdir.

pex_warnings.warn(

Failed to spawn a job for .../bin/python: [Errno 17] File exists: '<PEX_ROOT>/isolated/2f4fc85fa2be055a2975ce1147100c0d5c7e663a.workdir/pex/./venv'

It seems to have to do with #1905 introduced in version 2.1.105, but we have no clue, why this is happening in our CI pipeline, while building the .pex file on MacOS developer machines works. It looks like something else is creating that directory, but there is only one pex command in the pipeline job and the PEX_ROOT is not cached. Our build environment uses: • the Red Hat UBI 8.4 Docker image • Python 3.8 • poetry 1.1, which manages pex as a dev-dependency Then we build the pex with

poetry run pex --inherit-path --python=python3.8 --requirement requirements.txt --find-links dist/ our_module --output-file dist/final.pex

Any idea why this is happening or what else we could check would be helpful. pantsbuild/pex

cool-easter-32542

04/17/2023, 12:36 PM

#18752 [feature] Ability to verify pants installation before initalizing a repo Issue created by joe733 Is your feature request related to a problem?

please read the heading...

Currently after installing pants launcher I'm unable to run any commands unless I initialize the current repository. Describe the solution you'd like Provide any commands like

pants help

pants version

with

exit 0

so as to verify that installation is successful. We should be able to run it right after the following command

Copy code

$ curl --proto '=https' --tlsv1.2 -fsSL <https://static.pantsbuild.org/setup/get-pants.sh>

is complete, but before initializing a directory as pants repository. Describe alternatives you've considered Can

scie-pants

help? * * *

Additional context

Working on providing pants as an `asdf-plugin`.

pantsbuild/pants

cool-easter-32542

04/17/2023, 1:52 PM

#2115 Release 2.1.133 Issue created by jsirois On the docket: ☑︎ python<=3.8 symlink with a suffix (eg 3.7m) can create a venv without a pythonX.Y symlink which breaks pex assumptions that pythonX.Y is always available #2119 pantsbuild/pex

cool-easter-32542

04/17/2023, 6:23 PM

#2119 python<=3.8 symlink with a suffix (eg `3.7m`) can create a venv without a `pythonX.Y` symlink which breaks pex assumptions that `pythonX.Y` is always available Issue created by cognifloyd I get this error when running

./pants version

on pants 2.16.x branch. Pants is using pex v2.1.131 because I have overriden

[pex-cli].version = "v2.1.131"

and

[ex-cli].known_versions

~/.pants.rc

Copy code

$ ./pants --keep-sandboxes=always version
11:48:27.51 [INFO] waiting for pantsd to start...
11:48:30.01 [INFO] pantsd started
11:48:30.13 [INFO] Preserving local process execution dir /tmp/pants-sandbox-tP7l8W for Searching for `bash` on PATH=/usr/bin:/bin:/usr/local/bin
11:48:30.14 [INFO] Preserving local process execution dir /tmp/pants-sandbox-X9MYvB for Test binary /bin/bash.
11:48:30.16 [INFO] Starting: Resolving plugins: hdrhistogram, toolchain.pants.plugin==0.27.0
11:48:30.16 [INFO] Preserving local process execution dir /tmp/pants-sandbox-M3MXvy for Resolving plugins: hdrhistogram, toolchain.pants.plugin==0.27.0
11:48:32.16 [INFO] Completed: Resolving plugins: hdrhistogram, toolchain.pants.plugin==0.27.0
11:48:32.16 [ERROR] 1 Exception encountered:

[snip]

pants.engine.process.ProcessExecutionFailure: Process 'Resolving plugins: hdrhistogram, toolchain.pants.plugin==0.27.0' failed with exit code 1.
stdout:

stderr:
Failed to spawn a job for /home/cognifloyd/.cache/pants/pants_dev_deps/Linux.x86_64.Intel(R).Core(TM).i7-3610QM.CPU.@.2.30GHz.py37.venv/bin/python3.7: pid 1531475 -> /home/cognifloyd/.cache/pants/named_caches/pex_root/venvs/80f046e10efd2c6688f5a7df4d079c8b9816ab6f/acea27d2dbcc288eac2b093081739a3575b60708/bin/python -sE /home/cognifloyd/.cache/pants/named_caches/pex_root/venvs/80f046e10efd2c6688f5a7df4d079c8b9816ab6f/acea27d2dbcc288eac2b093081739a3575b60708/pex --disable-pip-version-check --no-python-version-warning --exists-action a --no-input --isolated -q --cache-dir /home/cognifloyd/.cache/pants/named_caches/pex_root/pip_cache --log /tmp/pants-sandbox-M3MXvy/.tmp/pex-pip-log.4f462xf5/pip.log download --dest /home/cognifloyd/.cache/pants/named_caches/pex_root/downloads/resolver_download.ho5or1_z/home.cognifloyd..cache.pants.pants_dev_deps.Linux.x86_64.Intel(R).Core(TM).i7-3610QM.CPU.@.2.30GHz.py37.venv.bin.python3.7 pip==23.0.1 setuptools==67.4.0 wheel==0.38.4 --index-url <https://pypi.org/simple/> --retries 5 --timeout 15 exited with 1 and STDERR:
Re-execing from /home/cognifloyd/.cache/pants/named_caches/pex_root/venvs/80f046e10efd2c6688f5a7df4d079c8b9816ab6f/acea27d2dbcc288eac2b093081739a3575b60708/bin/python
Traceback (most recent call last):
  File "/home/cognifloyd/.cache/pants/named_caches/pex_root/venvs/80f046e10efd2c6688f5a7df4d079c8b9816ab6f/acea27d2dbcc288eac2b093081739a3575b60708/pex", line 50, in <module>
    os.execv(python, argv)
FileNotFoundError: [Errno 2] No such file or directory

Looking in

/home/cognifloyd/.cache/pants/named_caches/pex_root/venvs/80f046e10efd2c6688f5a7df4d079c8b9816ab6f/acea27d2dbcc288eac2b093081739a3575b60708/pex

, it lists a python binary that does not exist.

Copy code

$ cd /home/cognifloyd/.cache/pants/named_caches/pex_root/venvs/80f046e10efd2c6688f5a7df4d079c8b9816ab6f/acea27d2dbcc288eac2b093081739a3575b60708/
$ ls -l pex __main__.py bin/python*
lrwxrwxrwx 1 cognifloyd cognifloyd    10 Apr 17 12:04 bin/python -> python3.7m
lrwxrwxrwx 1 cognifloyd cognifloyd    10 Apr 17 12:04 bin/python3 -> python3.7m
lrwxrwxrwx 1 cognifloyd cognifloyd    19 Apr 17 12:04 bin/python3.7m -> /usr/bin/python3.7m
-rwxr-xr-x 1 cognifloyd cognifloyd 11256 Apr 17 12:04 __main__.py
lrwxrwxrwx 1 cognifloyd cognifloyd    11 Apr 17 12:04 pex -> __main__.py

The shebang python is wrong in

__main__.py

because it is using

python3.7

when the virtualenv contains `python3.7m`:

Copy code

#!/home/cognifloyd/.cache/pants/named_caches/pex_root/venvs/s/0a5a863d/venv/bin/python3.7 -sE
...
    shebang_python = '/home/cognifloyd/.cache/pants/named_caches/pex_root/venvs/s/0a5a863d/venv/bin/python3.7'

Debugging results Using

--keep-sandboxes=always

I got into the pants sandbox for pex to see what's going on:

/tmp/pants-sandbox-M3MXvy

Copy code

$ cd /tmp/pants-sandbox-M3MXvy
$ ./pex --version
2.1.131

Here is the full pex command line pants is running (

__run.sh

Copy code

#!/bin/bash
# This command line should execute the same process as pants did internally.
export CPPFLAGS= LANG=en_US.UTF-8 LDFLAGS= PATH=$'/home/cognifloyd/.cache/pants/pants_dev_deps/Linux.x86_64.Intel(R).Core(TM).i7-3610QM.CPU.@.2.30GHz.py37.venv/bin:/home/cognifloyd/.cargo/bin:/home/cognifloyd/g/github/pyenv/pyenv.git/shims:/home/cognifloyd/.local/bin:/home/cognifloyd/.local/npm/bin:/usr/local/bin:/usr/bin:/bin:/opt/bin:/home/cognifloyd/p/gcloud/google-cloud-sdk/bin:/opt/android-sdk-update-manager/tools:/opt/android-sdk-update-manager/platform-tools:/opt/nvidia-cg-toolkit/bin:/home/cognifloyd/g/github/pyenv/pyenv.git/bin:/home/cognifloyd/go/bin' PEX_IGNORE_RCFILES=true PEX_PYTHON=$'/home/cognifloyd/.cache/pants/pants_dev_deps/Linux.x86_64.Intel(R).Core(TM).i7-3610QM.CPU.@.2.30GHz.py37.venv/bin/python' PEX_ROOT=.cache/pex_root
cd /tmp/pants-sandbox-M3MXvy
$'/home/cognifloyd/.cache/pants/pants_dev_deps/Linux.x86_64.Intel(R).Core(TM).i7-3610QM.CPU.@.2.30GHz.py37.venv/bin/python' ./pex --tmpdir .tmp --jobs 2 --pip-version 23.0.1 --python-path $'/home/cognifloyd/g/github/pyenv/pyenv.git/versions/3.4.10/bin:/home/cognifloyd/.cache/pants/pants_dev_deps/Linux.x86_64.Intel(R).Core(TM).i7-3610QM.CPU.@.2.30GHz.py37.venv/bin:/home/cognifloyd/.cargo/bin:/home/cognifloyd/g/github/pyenv/pyenv.git/shims:/home/cognifloyd/.local/bin:/home/cognifloyd/.local/npm/bin:/usr/local/bin:/usr/bin:/bin:/opt/bin:/home/cognifloyd/p/gcloud/google-cloud-sdk/bin:/opt/android-sdk-update-manager/tools:/opt/android-sdk-update-manager/platform-tools:/opt/nvidia-cg-toolkit/bin:/home/cognifloyd/g/github/pyenv/pyenv.git/bin:/home/cognifloyd/go/bin' --output-file pants_plugins.pex --no-emit-warnings --venv --seed verbose --no-venv-site-packages-copies --python $'/home/cognifloyd/.cache/pants/pants_dev_deps/Linux.x86_64.Intel(R).Core(TM).i7-3610QM.CPU.@.2.30GHz.py37.venv/bin/python' $'--sources-directory=source_files' hdrhistogram $'toolchain.pants.plugin==0.27.0' --no-pypi $'--index=<https://pypi.org/simple/>' --manylinux manylinux2014 --resolver-version pip-2020-resolver --constraints __constraints.txt --layout packed

So, pex is using the pants bootstrap venv, the interpreters available are:

Copy code

$ cd $'/home/cognifloyd/.cache/pants/pants_dev_deps/Linux.x86_64.Intel(R).Core(TM).i7-3610QM.CPU.@.2.30GHz.py37.venv/bin'
$ ls -l python*
lrwxrwxrwx 1 cognifloyd cognifloyd  9 Apr 16 00:08 python -> python3.7
lrwxrwxrwx 1 cognifloyd cognifloyd  9 Apr 16 00:08 python3 -> python3.7
lrwxrwxrwx 1 cognifloyd cognifloyd 18 Apr 16 00:08 python3.7 -> /usr/bin/python3.7

python versions available on my machine Here are all of the versions of python I have available:

Copy code

$ ls -l /usr/bin/python3.[1-9]{,[0-9m]}
-rwxr-xr-x 1 root root 14304 Mar 15 12:46 /usr/bin/python3.10
-rwxr-xr-x 1 root root 14304 Mar 15 12:51 /usr/bin/python3.11
lrwxrwxrwx 1 root root    10 Aug  8  2022 /usr/bin/python3.6 -> python3.6m
-rwxr-xr-x 1 root root 14280 Aug  8  2022 /usr/bin/python3.6m
lrwxrwxrwx 1 root root    10 Aug  8  2022 /usr/bin/python3.7 -> python3.7m
-rwxr-xr-x 1 root root 14224 Aug  8  2022 /usr/bin/python3.7m
-rwxr-xr-x 1 root root 14304 Dec 12 04:09 /usr/bin/python3.8
-rwxr-xr-x 1 root root 14304 Mar 15 12:49 /usr/bin/python3.9

I have some older builds of python3.6 and python3.7 on my gentoo linux machine. 3.7 is the oldest that pants supports, so it is using that. (If you're familiar with gentoo: I have ebuilds for these in my local overlay that I copied from

::gentoo

when they were dropped because I still needed 3.6 and 3.7) These versions of python include the optional

abi tag described in https://peps.python.org/pep-3149/ (

means

--with-pymalloc

), so the binaries are

python3.6m

and `python3.7m`: What's wrong with the pex venv? As noted above, the venv is getting built with

bin/python3.7m

but `… pantsbuild/pex

cool-easter-32542

04/17/2023, 10:55 PM

#18756 Some recipes for AWS Lambda and Docker Python artefacts (including cross-building from macOS) New discussion created by huonw Questions often come up about building AWS Lambda and docker artefacts effectively on Slack. I've answered a few based on my experience using pants to deploy AWS Lambda (via zips) and Fargate for a while. This discussion post collects what is currently working for us to deploy dozens of lambdas and a handful of Fargate containers, to be a reference that I can link to. None of this is perfect, but it's been good enough, and, once we worked through the initial set-up, pants packaging has never been the cause of a deploy failure. We develop on ARM64 macOS, and deploy to Linux x86. We run pants natively, and try to do as much as work natively too as reasonable because that is noticeably more efficient than emulating the OS, and very much more efficient than emulating the architecture. We also implemented all this well before Pants 2.15 and its experimental environments feature (https://blog.pantsbuild.org/environments-simpler-multi-platform-workflows/): that can likely be used too, but I cannot report on that. Basic setup We're currently using Pants 2.14 and PEX v2.1.128. Upgrading the latter is required to pick up a few bug fixes that were affecting our Lambda and docker builds. Our

pants.toml

looks something like:

Copy code

[GLOBAL]
pants_version = "2.14.0"

backend_packages = [
  "pants.backend.python",
  "pants.backend.awslambda.python",
  "pants.backend.docker",
  ...
]

...

[pex-cli]
version = "v2.1.128"
known_versions = [
    "v2.1.128|macos_arm64|a48a461a9e9f490476aa75976dbe58d3263691a8895a53d11fdaaf8a3bc223a5|4081603",
    "v2.1.128|macos_x86_64|a48a461a9e9f490476aa75976dbe58d3263691a8895a53d11fdaaf8a3bc223a5|4081603",
    "v2.1.128|linux_x86_64|a48a461a9e9f490476aa75976dbe58d3263691a8895a53d11fdaaf8a3bc223a5|4081603",
    "v2.1.128|linux_arm64|a48a461a9e9f490476aa75976dbe58d3263691a8895a53d11fdaaf8a3bc223a5|4081603"
]

...

Python AWS Lambda zip artefacts Most important: use the
complete_platforms=
parameter, NOT
runtime=
• We build dozens of lambdas via the built-in

python_awslambda

target: https://www.pantsbuild.org/docs/awslambda-python • The most important part of getting this working reliably is using a 'complete platform' JSON file instead of the

runtime

parameter, to tell PEX about the environment: • This allows pants/PEX to select the appropriate dependencies, even when building on a different platform (like macOS vs. Linux). This does require that dependencies are available as a wheel (bdist) and/or are pure Python, because platform-specific dependencies cannot be cross-built. This has worked fine for us in practice, with the dependencies we use. • We've generated an appropriate file (many months ago) by running the following in a Lambda (in retrospect, I imagine it could've been run in a one of the AWS-provided docker images, but... this post is just describing what is working for us):

Copy code

import subprocess
        
        def lambda_handler(event, context):
            subprocess.run(
                """
                pip install --target=/tmp/subdir pex
                PYTHONPATH=/tmp/subdir /tmp/subdir/bin/pex3 interpreter inspect --markers --tags
                """,
                shell=True
            )
            return {'statusCode': 200, 'body': "{}"}

• The output is put into

3rdparty/platforms/aws_lambda_python_3_9.json

with a

BUILD

file:

Copy code

# 3rdparty/platforms/BUILD
        file(
            name="aws_lambda_python_3_9",
            source="aws_lambda_python_3_9.json",
        )

• For convenience, here's the literal file we're using (NB. this is Python 3.9 in

ap-southeast-2

, as of 8 months ago, so your environment may differ): lambda-complete-platform.zip • To actually build the lambdas we have macro:

Copy code

def your_name_here_python_awslambda(*, **kwargs):
        kwargs.setdefault("dependencies", []).extend(
            [
                # FIXME <https://github.com/pantsbuild/pants/issues/15454>
                "!!path/to/requirements#mypy",
                "!!path/to/requirements#sqlalchemy-stubs",
            ]
        )
        kwargs["complete_platforms"] = ["3rdparty/platforms:aws_lambda_python_3_9"]  
        python_awslambda(**kwargs)

• The exclusions are working around #15454, to reduce the size of the deployed package. You may need to adjust them, based on what packages you use. • This can be called like the normal

python_awslambda

target, and generates the same sort of output (a zip in

dist/

): (As above, note that the

runtime=

parameter is not used.)

Copy code

# some/code/BUILD
    your_name_here_python_awslambda(
        name="some_name",
        handler="path/to/file.py:your_handler",
    )

• We deploy these via CDK, outside of pants, using a construct that takes constructs the appropriate

dist/some.code/some_name.zip

path to pass to `lambda.Code.fromAsset`:

Copy code

pants package ::
    cdk deploy path/to/stack.js

• Things we'd like to improve, eventually: • We'd prefer to deploy dependencies in a layer and then only our first-party source in the direct lambda package • #15454 • Using the fact that the target environment is static to layout an appropriate .zip file, and avoid the lambdex/PEX dynamic environment detection at start up (this adds about 1s to the cold start, it seems) • Improving build-time/cacheability: the current all-in-one zip files means that a minor first-party code that affects a bunch of lambdas change might have to generate a fresh 300MB of .zip, which both takes time and a lot of cache space. The first point above likely improves this: splitting 1st and 3rd party packages into separate packages. Docker image artefacts from PEX • We build docker images inspired by https://blog.pantsbuild.org/optimizing-python-docker-deploys-using-pants/ • We only build them via pants, and do not push them, and instead re-tag them outside pants before pushing, as we haven't had time/strong-need to work through parametrising different registries for prod vs. development images. • Again, we use

complete_platforms

to be able to build PEX files natively on macOS: • we currently run things in

python:3.9-bullseye

containers, and thus have added

docker_python_3_9_bullseye_arm64.json

and

docker_python_3_9_bullseye_amd64.json

files to `3rdparty/platforms`; here's the

BUILD

file including how we generate them:

Copy code

# docker run -it --platform linux/arm64 python:3.9.10-bullseye bash -c 'pip install pex; pex3 interpreter inspect --markers --tags'
        # docker run -it --platform linux/amd64 python:3.9.10-bullseye bash -c 'pip install pex; pex3 interpreter inspect --markers --tags'
        files(
            name="docker_python_3_9_bullseye",
            sources=["docker_python_3_9_bullseye_*.json"],
        )

• We run Linux/arm64 images on dev machines and Linux/amd64 images on CI/in production, and thus include complete platforms for both (running amd64 images on arm64 dev machines is very slow) • To actually build images we have a macro: ``` BASE_IMAGE = "python:3.9-bullseye" COMPLETE_PLATFORMS = ("3rdparty/platforms:docker_python_3_9_bullseye",) def xo_docker_image_from_pex( *, name, pex_entry_point, extra_instructions, pex_dependencies=None, **kwargs, ): """ Build a docker image using PEX… pantsbuild/pants

cool-easter-32542

04/18/2023, 6:57 AM

#18758 `generate-lockfiles` fails when using `pants_requirements` New discussion created by jmontalt Dear Pants community, I cannot generate lockfiles when I use the

pants_requirements

target. I have a BUILD file at

tools/pants/plugins/BUILD

that looks as follows:

Copy code

pants_requirements(
    name="pants",
    resolve="pants-plugins",
    testutil=False,
)

When running

pants generate-lockfiles --resolve=pants-plugins

, I get the following error:

Copy code

06:37:53.86 [INFO] Initializing scheduler...
06:37:54.96 [INFO] Scheduler initialized.
06:38:01.67 [INFO] Preserving local process execution dir /tmp/pants-sandbox-6hqGo4 for Generate lockfile for pants-plugins
06:38:01.67 [INFO] Completed: Generate lockfile for pants-plugins
06:38:01.68 [ERROR] 1 Exception encountered:

Engine traceback:
  in `generate-lockfiles` goal
  in Generate Python lockfile

ProcessExecutionFailure: Process 'Generate lockfile for pants-plugins' failed with exit code 1.
stdout:

stderr:
pid 12622 -> /home/vscode/.cache/pants/named_caches/pex_root/venvs/ed021e31e182bf825e539d57941092529371abb9/ddab8011daaee380698ac2fb9701af18c90c03f6/bin/python -sE /home/vscode/.cache/pants/named_caches/pex_root/venvs/ed021e31e182bf825e539d57941092529371abb9/ddab8011daaee380698ac2fb9701af18c90c03f6/pex --disable-pip-version-check --no-python-version-warning --exists-action a --no-input --isolated -q --cache-dir /home/vscode/.cache/pants/named_caches/pex_root/pip_cache --log /tmp/pants-sandbox-6hqGo4/.tmp/pex-pip-log.ffqwbc89/pip.log download --dest /tmp/pants-sandbox-6hqGo4/.tmp/tmpd7u1iumy/usr.bin.python3.8 pantsbuild.pants<2.16,>=2.15.0a0 --index-url <https://pypi.org/simple/> --retries 5 --timeout 15 exited with 1 and STDERR:
ERROR: Could not find a version that satisfies the requirement pantsbuild.pants<2.16,>=2.15.0a0
ERROR: No matching distribution found for pantsbuild.pants<2.16,>=2.15.0a0

For other Python dependencies, the lockfile generation works just fine. The OS is Ubuntu 20.04. Any ideas? Thank you for your help. pantsbuild/pants

cool-easter-32542

04/18/2023, 10:02 AM

#18759 Installtion via `pipx` fails due to old version of `pystache` Issue created by joe733 Describe the bug

please read the heading

Pants version

Latest

Linux & MacOS

Additional info: Culprit: sarnold/pystache#16 (comment)

Log

Copy code

───────┬────────────────────────────────────────────────────────────────────────────────────────────────
       │ File: /home/user-name/.local/pipx/logs/cmd_2023-04-18_15.17.23_pip_errors.log
───────┼────────────────────────────────────────────────────────────────────────────────────────────────
   1   │ PIP STDOUT
   2   │ ----------
   3   │ Collecting pantsbuild-pants
   4   │   Using cached pantsbuild.pants-1.28.0-cp36-abi3-manylinux2014_x86_64.whl (49.1 MB)
   5   │ Collecting setproctitle==1.1.10 (from pantsbuild-pants)
   6   │   Using cached setproctitle-1.1.10.zip (34 kB)
   7   │   Preparing metadata (setup.py): started
   8   │   Preparing metadata (setup.py): finished with status 'done'
   9   │ Collecting ansicolors==1.1.8 (from pantsbuild-pants)
  10   │   Using cached ansicolors-1.1.8-py2.py3-none-any.whl (13 kB)
  11   │ Collecting typing-extensions==3.7.4 (from pantsbuild-pants)
  12   │   Using cached typing_extensions-3.7.4-py3-none-any.whl (20 kB)
  13   │ Collecting dataclasses==0.6 (from pantsbuild-pants)
  14   │   Using cached dataclasses-0.6-py3-none-any.whl (14 kB)
  15   │ Collecting packaging==16.8 (from pantsbuild-pants)
  16   │   Using cached packaging-16.8-py2.py3-none-any.whl (23 kB)
  17   │ Collecting psutil==5.7.0 (from pantsbuild-pants)
  18   │   Using cached psutil-5.7.0.tar.gz (449 kB)
  19   │   Preparing metadata (setup.py): started
  20   │   Preparing metadata (setup.py): finished with status 'done'
  21   │ Collecting fasteners==0.15.0 (from pantsbuild-pants)
  22   │   Using cached fasteners-0.15-py2.py3-none-any.whl (23 kB)
  23   │ Collecting python-Levenshtein==0.12.0 (from pantsbuild-pants)
  24   │   Using cached python-Levenshtein-0.12.0.tar.gz (48 kB)
  25   │   Preparing metadata (setup.py): started
  26   │   Preparing metadata (setup.py): finished with status 'done'
  27   │ Collecting PyYAML<5.4,>=5.3.1 (from pantsbuild-pants)
  28   │   Using cached PyYAML-5.3.1.tar.gz (269 kB)
  29   │   Preparing metadata (setup.py): started
  30   │   Preparing metadata (setup.py): finished with status 'done'
  31   │ Collecting setuptools==44.0.0 (from pantsbuild-pants)
  32   │   Using cached setuptools-44.0.0-py2.py3-none-any.whl (583 kB)
  33   │ Collecting toml==0.10.1 (from pantsbuild-pants)
  34   │   Using cached toml-0.10.1-py2.py3-none-any.whl (19 kB)
  35   │ Collecting twitter.common.dirutil<0.4,>=0.3.11 (from pantsbuild-pants)
  36   │   Using cached twitter.common.dirutil-0.3.11.tar.gz (9.5 kB)
  37   │   Preparing metadata (setup.py): started
  38   │   Preparing metadata (setup.py): finished with status 'done'
  39   │ Collecting pathspec==0.8.0 (from pantsbuild-pants)
  40   │   Using cached pathspec-0.8.0-py2.py3-none-any.whl (28 kB)
  41   │ Collecting requests[security]>=2.20.1 (from pantsbuild-pants)
  42   │   Using cached requests-2.28.2-py3-none-any.whl (62 kB)
  43   │ Collecting py-zipkin==0.18.4 (from pantsbuild-pants)
  44   │   Using cached py_zipkin-0.18.4-py2.py3-none-any.whl (47 kB)
  45   │ Collecting pystache==0.5.3 (from pantsbuild-pants)
  46   │   Using cached pystache-0.5.3.tar.gz (74 kB)
  47   │   Preparing metadata (setup.py): started
  48   │   Preparing metadata (setup.py): finished with status 'error'
  49   │ 
  50   │ PIP STDERR
  51   │ ----------
  52   │   error: subprocess-exited-with-error
  53   │   
  54   │   × python setup.py egg_info did not run successfully.
  55   │   │ exit code: 1
  56   │   ╰─> [3 lines of output]
  57   │       Warning: 'classifiers' should be a list, got type 'tuple'
  58   │       error in pystache setup command: use_2to3 is invalid.
  59   │       pystache: using: version '67.6.1' of <module 'setuptools' from '/home/user-name/.local/
       │ pipx/shared/lib/python3.10/site-packages/setuptools/__init__.py'>
  60   │       [end of output]
  61   │   
  62   │   note: This error originates from a subprocess, and is likely not a problem with pip.
  63   │ error: metadata-generation-failed
  64   │ 
  65   │ × Encountered error while generating package metadata.
  66   │ ╰─> See above for output.
  67   │ 
  68   │ note: This is an issue with the package mentioned above, not pip.
  69   │ hint: See above for details.
───────┴────────────────────────────────────────────────────────────────────────────────────────────────

Upgrading to
0.6.0
works.

pantsbuild/pants

cool-easter-32542

04/18/2023, 8:12 PM

#2120 Generated Pex File works on some hosts, not others New discussion created by erik-at-techsanity Hi All, I've created a Pex file that works when run within a running Podman container (hosted on RedHat 8) that I do my development work in, but when I try and run it on a proper VM (either RedHat 7 or 8), it fails to find the bundled python dependencies (it complains about typer in this case, that's just the first that it tries to load). ModuleNotFoundError: No module named 'typer' I ran with the verbose flag on all instances, and didn't see any glaring differences when using vimdiff, other then the addresses that libraries were being loaded into or from. I've generated the pexfile on both the RedHat 7 vm and the Podman container and the same result occurs. I'm using a script to handle the generation so that it's repeatable. How can I debug this? Has anyone else seen this? Thanks for any suggestions! pantsbuild/pex

cool-easter-32542

04/18/2023, 10:03 PM

#18762 Go build constraints New discussion created by joneshf Is it possible to specify build constraints to the go compiler with Pants? For example, if I have a file `foo.go`:

Copy code

//go:build sometimes
package foo

The go compiler will only include that file in the build if I specify the

sometimes

build tag:

Copy code

$ go build -tags=sometimes ./...

I couldn't see a way to specify the

-tags

flag, so this might be a feature request. But figured I'd make a discussion first, in case I missed how to do it. pantsbuild/pants

cool-easter-32542

04/19/2023, 12:33 AM

#2121 Release 2.1.134 Issue created by jsirois On the docket: ☑︎ A pex3 lock create can fail to gather metadata from sdists due to Pex being PEP-517 non-compliant. #2125 pantsbuild/pex

cool-easter-32542

04/19/2023, 12:38 AM

#18764 Docker environment variables set in pants.toml appear to not get passed through Issue created by zyd14 First off, thanks for the really cool tool, it's been a trip learning how to use it. Describe the bug I had a number of

docker_image

targets which were working fine for some time. Today I when to run one and encountered this error:

Copy code

Failed to fire hook: while creating logrus local file hook: user: Current requires cgo or $USER, $HOME set in environment
[2023-04-18T20:58:17.637886000Z][docker-credential-desktop][F] get system info: exec: "sw_vers": executable file not found in $PATH
[goroutine 1 [running, locked to thread]:
[common/pkg/system.init.0()
[       common/pkg/system/os_info.go:32 +0x1bc
#3 ERROR: rpc error: code = Unknown desc = error getting credentials - err: exit status 1, out: ``
#5 [auth] sharing credentials for <http://123456789.dkr.ecr.us-east-1.amazonaws.com|123456789.dkr.ecr.us-east-1.amazonaws.com>
#5 sha256:00484c1754ca42cdfd60e0asdf234asdf8a41a33b1f30d70412c4254
#5 ERROR: error getting credentials - err: exit status 1, out: ``
------
 > [internal] load metadata for <http://123456789.dkr.ecr.us-east-1.amazonaws.com/databricks-base:0.0.2|123456789.dkr.ecr.us-east-1.amazonaws.com/databricks-base:0.0.2>:
------
------
 > [auth] sharing credentials for <http://123456789.dkr.ecr.us-east-1.amazonaws.com|123456789.dkr.ecr.us-east-1.amazonaws.com>:
------
failed to solve with frontend dockerfile.v0: failed to create LLB definition: rpc error: code = Unknown desc = error getting credentials - err: exit status 1, out: ``

I've encountered this error before when I was first setting up pants in the repo and fixed it by adding the USER, HOME, and PATH variables to

[docker.env_vars]

in my

pants.toml

file. But now those variables are set (and appeared to be working for some time), leaving me with an unclear path to exposing these variables to the pants environment. I've also confirmed that

sw_vers

is in my PATH, and that USER and HOME are actually set.

Copy code

# pants.toml
[docker]
env_vars = [
  "HOME",
  "USER",
  "PATH",
  "GIT_COMMIT"
]

With

pants package -ldebug <target>

I noticed this message in the logs:

Copy code

spawned local process as Some(88403) for Process { argv: ["/usr/local/bin/docker", "build", "--pull=False", "--tag", "<http://12345667.dkr.ecr.us-east-1.amazonaws.com/test_image:latest|12345667.dkr.ecr.us-east-1.amazonaws.com/test_image:latest>", "--file", "src/docker/another/Dockerfile", "."], env: {"__UPSTREAM_IMAGE_IDS": ""}, working_directory: None, input_digests: InputDigests { complete: DirectoryDigest { digest: Digest { hash: Fingerprint<4bdb5dc82ac35992d61254127dd65421235d2e5c32dd9b87d8084144190c76>, size_bytes: 77 }, tree: "Some(..)" }, nailgun: DirectoryDigest { digest: Digest { hash: Fingerprint<e3b0c4429f8fc1c149afbf4c8996fb12441e4649b934ca495991c8852b855>, size_bytes: 0 }, tree: "Some(..)" }, input_files: DirectoryDigest { digest: Digest { hash: Fingerprint<4bdb5dc82ac35992fd67ed341274d123e5d2e5c32dd9b87d8084144190c76>, size_bytes: 77 }, tree: "Some(..)" }, immutable_inputs: {}, use_nailgun: {} }, output_files: {}, output_directories: {}, timeout: None, execution_slot_variable: None, concurrency_available: 0, description: "Building docker image <http://123454560.dkr.ecr.us-east-1.amazonaws.com/test_image:latest|123454560.dkr.ecr.us-east-1.amazonaws.com/test_image:latest>", level: Info, append_only_caches: {}, jdk_home: None, platform: Macos_arm64, cache_scope: PerSession, execution_strategy: Local, remote_cache_speculation_delay: 0ns }

Which seems to indicate to me that no environment variables from my

pants.toml

file are being passed through to the underlying

docker build

call (

__UPSTREAM_IMAGE_IDS

isn't one of my env vars). I also tried clearing out my docker and pants caches to no avail. Any potential workarounds are more than welcome in lieu of a direct fix, this is a bit of a blocker for me. Thanks for all your help and the really awesome tool. Pants version 0.2.15 OS MacOS, arm64 pantsbuild/pants

cool-easter-32542

04/19/2023, 5:50 AM

Comment on #18758 `generate-lockfiles` fails when using `pants_requirements` Discussion answered by jsirois An interpreter constraint broader than Pants own would go boom. So if the ICs for this resolve stray outside of the range

>=3.7,<3.10

(i.e.: say your ICs include Python 3.10), that's the problem. pantsbuild/pants

cool-easter-32542

04/20/2023, 7:43 AM

#18772 Missing transitive dependencies of pex_binary depending on python_dist running inside a docker_image Issue created by idan-at Describe the bug I am using a

docker_image

target to test a CLI.

Copy code

docker_image(
  name="docker",
  skip_push=True,
  dependencies=[":bin"],
  repository="e2e-tests-image",
  instructions=[
    "FROM python:3.9",
    "COPY cli.tests/bin.pex /bin/cli"
  ]
)

Although I wanted to directly copy the python distribution into the image, I couldn't find a way to do so, so I am using a

pex_binary

that depends on the

python_distribution

Copy code

pex_binary(
  name="bin",
  entry_point="cli.main",
  dependencies=["cli/src:dist"]
)

When I run the generated pex file manually on my machine, it works fine. When I try to run

cli

inside a container (

docker run -it e2e-tests-image bash

), I get the following error (no transitive dependencies found)

Copy code

Failed to find compatible interpreter on path /usr/local/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin.

Examined the following interpreters:
1.) /usr/local/bin/python3.9 CPython==3.9.16
2.)       /usr/bin/python3.9 CPython==3.9.2

No interpreter compatible with the requested constraints was found:

  Failed to resolve requirements from PEX environment @ /root/.pex/unzipped_pexes/e1302300f7c34e2335e6337901d8efccd5b1a826.
  Needed cp39-cp39-manylinux_2_31_aarch64 compatible dependencies for:
   1: protobuf!=3.20.0,!=3.20.1,!=4.21.1,!=4.21.2,!=4.21.3,!=4.21.4,!=4.21.5,<5.0.0dev,>=3.19.5
      Required by:
        googleapis-common-protos 1.59.0
        grpc-google-iam-v1 0.12.6
      But this pex had no ProjectName(raw='protobuf', normalized='protobuf') distributions.
   2: protobuf!=3.20.0,!=3.20.1,!=4.21.0,!=4.21.1,!=4.21.2,!=4.21.3,!=4.21.4,!=4.21.5,<5.0.0dev,>=3.19.5
      Required by:
        google-api-core 2.11.0
      But this pex had no ProjectName(raw='protobuf', normalized='protobuf') distributions.
   3: charset-normalizer<4,>=2
      Required by:
        requests 2.28.2
      But this pex had no ProjectName(raw='charset-normalizer', normalized='charset-normalizer') distributions.
   4: grpcio<2.0dev,>=1.33.2; extra == "grpc"
      Required by:
        google-api-core 2.11.0
      But this pex had no ProjectName(raw='grpcio', normalized='grpcio') distributions.
   5: protobuf>=4.21.6
      Required by:
        grpcio-status 1.53.0
      But this pex had no ProjectName(raw='protobuf', normalized='protobuf') distributions.
   6: grpcio>=1.53.0
      Required by:
        grpcio-status 1.53.0
      But this pex had no ProjectName(raw='grpcio', normalized='grpcio') distributions.
   7: grpcio<2.0.0dev,>=1.44.0
      Required by:
        grpc-google-iam-v1 0.12.6
      But this pex had no ProjectName(raw='grpcio', normalized='grpcio') distributions.
   8: grpcio<2.0.0dev,>=1.44.0; extra == "grpc"
      Required by:
        googleapis-common-protos 1.59.0
      But this pex had no ProjectName(raw='grpcio', normalized='grpcio') distributions.
   9: protobuf<5.0.0dev,>=3.19.0
      Required by:
        proto-plus 1.22.2
        google-cloud-secret-manager 2.12.4
      But this pex had no ProjectName(raw='protobuf', normalized='protobuf') distributions.

  Failed to resolve requirements from PEX environment @ /root/.pex/unzipped_pexes/e1302300f7c34e2335e6337901d8efccd5b1a826.
  Needed cp39-cp39-manylinux_2_31_aarch64 compatible dependencies for:
   1: protobuf!=3.20.0,!=3.20.1,!=4.21.1,!=4.21.2,!=4.21.3,!=4.21.4,!=4.21.5,<5.0.0dev,>=3.19.5
      Required by:
        googleapis-common-protos 1.59.0
        grpc-google-iam-v1 0.12.6
      But this pex had no ProjectName(raw='protobuf', normalized='protobuf') distributions.
   2: protobuf!=3.20.0,!=3.20.1,!=4.21.0,!=4.21.1,!=4.21.2,!=4.21.3,!=4.21.4,!=4.21.5,<5.0.0dev,>=3.19.5
      Required by:
        google-api-core 2.11.0
      But this pex had no ProjectName(raw='protobuf', normalized='protobuf') distributions.
   3: charset-normalizer<4,>=2
      Required by:
        requests 2.28.2
      But this pex had no ProjectName(raw='charset-normalizer', normalized='charset-normalizer') distributions.
   4: grpcio<2.0dev,>=1.33.2; extra == "grpc"
      Required by:
        google-api-core 2.11.0
      But this pex had no ProjectName(raw='grpcio', normalized='grpcio') distributions.
   5: protobuf>=4.21.6
      Required by:
        grpcio-status 1.53.0
      But this pex had no ProjectName(raw='protobuf', normalized='protobuf') distributions.
   6: grpcio>=1.53.0
      Required by:
        grpcio-status 1.53.0
      But this pex had no ProjectName(raw='grpcio', normalized='grpcio') distributions.
   7: grpcio<2.0.0dev,>=1.44.0
      Required by:
        grpc-google-iam-v1 0.12.6
      But this pex had no ProjectName(raw='grpcio', normalized='grpcio') distributions.
   8: grpcio<2.0.0dev,>=1.44.0; extra == "grpc"
      Required by:
        googleapis-common-protos 1.59.0
      But this pex had no ProjectName(raw='grpcio', normalized='grpcio') distributions.
   9: protobuf<5.0.0dev,>=3.19.0
      Required by:
        proto-plus 1.22.2
        google-cloud-secret-manager 2.12.4
      But this pex had no ProjectName(raw='protobuf', normalized='protobuf') distributions.

Pants version 2.15.0 OS MacOS Additional info My python package is using

click==8.1.3

and

google-cloud-secret-manager==2.12.4

as depenedencies. Thanks. pantsbuild/pants

cool-easter-32542

04/20/2023, 7:57 AM

#18773 Unable to import 'pytest' (import-error) Issue created by idan-at Describe the bug When enabling those two backends together: •

pants.backend.python

•

pants.backend.python.lint.pylint

And importing

pytest

in one of the test files (for example for using

@pytest.mark.skip

pylint

complains with the error above. Pants version 2.15.0 OS MacOS pantsbuild/pants

cool-easter-32542

04/20/2023, 2:24 PM

#16772 ./pants run <python_file> does not get invalidated / restarted if the file is changed Issue created by wisechengyi on main:

Copy code

% ./pants list not-a-source-root/slow.py
not-a-source-root:bin
not-a-source-root/slow.py:lib

on release branch (#16768):

Copy code

% ./pants list not-a-source-root/slow.py
not-a-source-root/slow.py:lib

not-a-source-root/

content:

Copy code

BUILD:
python_sources(name='lib')
pex_binary(name='bin', entry_point='slow.py', restartable=True)

slow.py:
import time
print("hello")
# time.sleep(30)
# raise Exception("Should have been restarted by now!")

pantsbuild/pants

cool-easter-32542

04/20/2023, 9:07 PM

#18775 Union-izing test's `extra_env_vars` and `runtime_package_dependencies` New discussion created by xlevus Background I have a suite of tests that do integrations against an external environment. Currently our CICD will deploy the packaged dependencies to the system and then run the tests on the local machine. Which means our 'test' step is a bit more involved than just

./pants test ::

. One suggested solution to this was to create a codegen target that deploys the resources, but this would then cause things to be deployed unnecessarily every time

export-codegen

is run. It also doesn't solve how to inject the location of the deployed resources into the tests. idea Currently you can define [test].extra_env_vars to inject extra stuff into the env, and runtime_package_dependencies to add packageable targets to the test process digest. But from a quick skim of the code, all of this is replicated in each backends subprocess call. e.g.: https://github.com/pantsbuild/pants/blob/main/src/python/pants/backend/python/goals/pytest_runner.py#L245 https://github.com/pantsbuild/pants/blob/main/src/python/pants/backend/go/goals/test.py#L595 It would be ideal if

extra_env_vars

and

runtime_package_dependencies

were somehow Unionized. Allowing 3rd party plugins to provide ways to add additional data into test runs. pantsbuild/pants

cool-easter-32542

04/21/2023, 1:17 PM

#2124 Universal lock file might not contain platform-dependent dependencies Issue created by misberner • Sample command:

pex3 lock create --style=universal 'torch==2.0.0'

• Expected behavior: A lockfile is created with all (transitive) dependencies of packages matching the

torch==2.0.0

for all platforms. • Observed behavior: Lockfile does not reference dependencies of packages for Linux (

nvidia-*

packages,

triton-2.0.0

). Without knowing too much about the internals of Pex, it looks like only the first wheel is taken into account (in my case, that's https://files.pythonhosted.org/packages/9f/cd/670e5e178db87065ee60f60fb35b040abbb819a1f686a91d9ff799fc5048/torch-2.0.0-1-cp310-cp310-manylinux2014_aarch64.whl ) and inspected for dependencies. The

torch-2.0.0.dist-info/METADATA

file for this wheel lists:

Copy code

Requires-Dist: filelock
Requires-Dist: typing-extensions
Requires-Dist: sympy
Requires-Dist: networkx
Requires-Dist: jinja2
Provides-Extra: opt-einsum
Requires-Dist: opt-einsum (>=3.3) ; extra == 'opt-einsum'

which matches the lockfile output:

Copy code

"project_name": "torch",
          "requires_dists": [
            "filelock",
            "jinja2",
            "networkx",
            "opt-einsum>=3.3; extra == \"opt-einsum\"",
            "sympy",
            "typing-extensions"
          ],

However, one of the linux/amd64 wheels (https://files.pythonhosted.org/packages/5f/24/16e94ac3a470027a2f6cf56dbbe2ce1b2742fa0ac98844f039fad103e142/torch-2.0.0-cp39-cp39-manylinux1_x86_64.whl) lists the following in its

METADATA

file:

Copy code

Requires-Dist: filelock
Requires-Dist: typing-extensions
Requires-Dist: sympy
Requires-Dist: networkx
Requires-Dist: jinja2
Requires-Dist: nvidia-cuda-nvrtc-cu11 (==11.7.99) ; platform_system == "Linux" and platform_machine == "x86_64"
Requires-Dist: nvidia-cuda-runtime-cu11 (==11.7.99) ; platform_system == "Linux" and platform_machine == "x86_64"
Requires-Dist: nvidia-cuda-cupti-cu11 (==11.7.101) ; platform_system == "Linux" and platform_machine == "x86_64"
Requires-Dist: nvidia-cudnn-cu11 (==8.5.0.96) ; platform_system == "Linux" and platform_machine == "x86_64"
Requires-Dist: nvidia-cublas-cu11 (==11.10.3.66) ; platform_system == "Linux" and platform_machine == "x86_64"
Requires-Dist: nvidia-cufft-cu11 (==10.9.0.58) ; platform_system == "Linux" and platform_machine == "x86_64"
Requires-Dist: nvidia-curand-cu11 (==10.2.10.91) ; platform_system == "Linux" and platform_machine == "x86_64"
Requires-Dist: nvidia-cusolver-cu11 (==11.4.0.1) ; platform_system == "Linux" and platform_machine == "x86_64"
Requires-Dist: nvidia-cusparse-cu11 (==11.7.4.91) ; platform_system == "Linux" and platform_machine == "x86_64"
Requires-Dist: nvidia-nccl-cu11 (==2.14.3) ; platform_system == "Linux" and platform_machine == "x86_64"
Requires-Dist: nvidia-nvtx-cu11 (==11.7.91) ; platform_system == "Linux" and platform_machine == "x86_64"
Requires-Dist: triton (==2.0.0) ; platform_system == "Linux" and platform_machine == "x86_64"
Provides-Extra: opt-einsum
Requires-Dist: opt-einsum (>=3.3) ; extra == 'opt-einsum'

where all the platform-specific dependencies are absent from the universal lockfile. Is this a bug, or is my understanding of what should be in the universal lockfile just wrong? Many thanks! pantsbuild/pex

cool-easter-32542

04/21/2023, 2:51 PM

#18779 `pex_binary().args` are not properly escaped. Issue created by kaos Describe the bug If you attempt to inject any

--args

pex

will complain like:

Copy code

pex: error: argument --inject-args: expected one argument

Pants version 2.16.0rc0 Additional info Slack thread These args needs to be escaped: pants/src/python/pants/backend/python/util_rules/pex.py Lines 586 to 587 in </pantsbuild/pants/commit/1aca34961105f98c2dd63addd37f9e5cfc49ee9b|1aca349> Introduced in #17905 pantsbuild/pants