#18488 Terraform 1.0.7 (the default) file download length is incorrect on macOS_x86_64 Issue created by ndellosa95 Describe the bug Title. Error message:

10:08:19.28 [ERROR] 1 Exception encountered:

Engine traceback:
  in `fmt` goal
  in Format with `terraform fmt`

Exception: Error hashing/capturing URL fetch response: Downloaded file was larger than expected digest

Pants version 2.15 OS Intel MacOS Additional info Probably just need to rerun

build-support/bin/terraform_tool_versions.py

pantsbuild/pants

#18489 `pants check` fails on Terraform modules due to lack of `terraform init` Issue created by ndellosa95 Describe the bug After setting up some basic Terraform in my repo, running

pants check ::

fails with the following error message:

16:48:40.39 [ERROR] Completed: pants.backend.terraform.goals.check.terraform_check - terraform-validate failed (exit code 1).
Partition #1 - `terraform validate` on `myrepo/terraform/environments/dev`:
╷
│ Error: Module not installed
│ 
│   on myrepo/terraform/environments/dev/main.tf line 23:
│   23: module "my_module" {
│ 
│ This module is not yet installed. Run "terraform init" to install all
│ modules required by this configuration.
╵

Partition #2 - `terraform validate` on `myrepo/terraform/modules/infrastructure`:
╷
│ Error: Missing required provider
│ 
│ This configuration requires provider
│ <http://registry.terraform.io/hashicorp/azurerm|registry.terraform.io/hashicorp/azurerm>, but that provider isn't available.
│ You may be able to install it automatically by running:
│   terraform init
╵



✕ terraform-validate failed.

Pants version 2.15 OS Intel Mac Additional info Ideally,

pants check

would run

init

,

validate

, and

plan

. I'm going to open a feature request for this exact thing. It would also be super useful if you could run

terraform apply

through Pants, either using the

publish

or

experimental-deploy

goals (another issue, the difference between these two should be elaborated on as they're both from the core backend). pantsbuild/pants

#18490 Run `terraform plan` on check after validate Issue created by ndellosa95 Is your feature request related to a problem? Please describe. I created a bug issue here related to

pants check

not running

terraform init

before

terraform validate

. IMO, check should also run terraform plan. Describe the solution you'd like

pants check

runs

terraform plan

after running

init

and

validate

. Describe alternatives you've considered Check feels like the right goal to me but I'm fairly ambivalent about what the actual goal is called so long as some mechanism to call

terraform plan

is in place. pantsbuild/pants

#18491 Run `terraform apply` on `publish` Issue created by ndellosa95 Is your feature request related to a problem? Please describe. Similar to my issue for

terraform plan

here, it would be great if we could use

pants publish

to call terraform apply. Describe the solution you'd like

pants publish

calls

terraform apply

. Describe alternatives you've considered

pants experimental-deploy

also seems like an appropriate goal for this feature. pantsbuild/pants

#17927 Disambiguate inferred deps by source root Issue created by benjyw A repo may sometimes have multiple providers of the same symbol, across different source roots. A common case is when multiple projects are "naively" merged into a single monorepo, so that each project is in its own source root. Multiple such projects may provide the same symbol, either as first-party source (via .py files with the same relative path from the source root) or as third-party (via multiple requirements.txt with overlapping project names). E.g.,

src/project_1/mycompany/config.py
src/project_2/mycompany/config.py

Where src/project_1 and src/project_2 are source roots. In practice, we can resolve this ambiguity by (optionally) preferring the provider in the same source root as the consumer. This is very likely to be what the user intends. pantsbuild/pants

#2096 Introduce PAR support / add an option to ship a PEX with Python included. Issue created by jsirois With the advent of scies, Pex's can now be shipped as standalone native executables with the Python interpreter included without much effort. If Pex were to incorporate this feature, it might be spelled like:

--par {eager,lazy}

This would cause Pex to generate one or more binaries using scie-jump that either directly include the appropriate PBS Python archive or else lazily fetch it using ptex. The fiddly bit here will be selecting an appropriately matched PBS per Pex target Python / abbreviated

--platform

/

--complete-platform

. pantsbuild/pex

#15171 Poetry locking tries to combine a global Poetry PEX with per-resolve `Requires-Python` (ICs), which can fail. Issue created by jsirois Reported as:

% python --version           
Python 3.10.3

% pyenv version              
global-3.10.3 (set by PYENV_VERSION environment variable)

% ./pants generate-lockfiles 
Bootstrapping Pants using /Users/mike/.pyenv/shims/python3.9
Creating the virtualenv PEX.
Downloading the Pex PEX.
SHA256 fingerprint of <https://github.com/pantsbuild/pex/releases/download/v2.1.62/pex> verified.
Installing pantsbuild.pants==2.11.0rc3 into a virtual environment at /Users/mike/.cache/pants/setup/bootstrap-Darwin-x86_64/2.11.0rc3_py39
New virtual environment successfully created at /Users/mike/.cache/pants/setup/bootstrap-Darwin-x86_64/2.11.0rc3_py39.
09:21:04.25 [INFO] Completed: Generate lockfile for python-default
09:21:04.26 [ERROR] 1 Exception encountered:

  ProcessExecutionFailure: Process 'Generate lockfile for python-default' failed with exit code 1.
stdout:
The currently activated Python version 3.7.12 is not supported by the project (>=3.10).
Trying to find and use a compatible version. 

  NoCompatiblePythonVersionFound

Where

pants.toml

has:

[python]
interpreter_constraints = ["CPython>=3.10"]

The workaround was to align the Poetry tool ICs:

[poetry]
version = "poetry==1.1.13"
interpreter_constraints = ["CPython>=3.10,<4"]

That workaround though, besides violating DRY, is unworkable the minute there is >1 resolve. pantsbuild/pants

#14020 Lockfile support generates an incorrect lockfile for any direct reference requirement. Issue created by jsirois Pants, via

pkg_resources.Requirement

, handles direct reference requirements as described here: https://www.python.org/dev/peps/pep-0440/#direct-references These requirements parse with no version as demonstrated here:

$ pex setuptools -- -c 'from pkg_resources import Requirement; req = Requirement.parse("darglint @ git+<https://github.com/thejcannon/darglint@XYZ%22|https://github.com/thejcannon/darglint@XYZ">); print(f"req: {req} spec: {req.specifier}")'
req: darglint@ git+<https://github.com/thejcannon/darglint@XYZ> spec:

This is problematic with our current lock file implementaion using Poetry since: pants/src/python/pants/backend/python/subsystems/poetry.py Line 97 in </pantsbuild/pants/commit/315dd5c37a3e3394dee363e232d3c516583d4ead|315dd5c> pants/src/python/pants/backend/python/subsystems/poetry.py Line 107 in </pantsbuild/pants/commit/315dd5c37a3e3394dee363e232d3c516583d4ead|315dd5c> pants/src/python/pants/backend/python/subsystems/poetry.py Lines 68 to 76 in </pantsbuild/pants/commit/315dd5c37a3e3394dee363e232d3c516583d4ead|315dd5c> That generates a

pyproject.toml

with an incorrect dependency entry for every direct reference requirement we are trying to lock using

poetry lock

on the

pyproject.toml

. Instead of getting a lock on the direct reference requirement you specify and instead of failing to indicate we can't generate a proper lock for these, we silently generate a lock against the latest publically available version of the project on PyPI (since we say

version = "*"

). pantsbuild/pants

#18500 `help backends` and `help-advanced backends` show the same content (both include preview/experimental backends) Issue created by cognifloyd Describe the bug Under the

help

goal:

help backends

and

help-advanced backends

show the same content (both include preview/experimental backends). Only the

-advanced

variant should show the preview/experimental backends (according to the

help

output).

Usage:
  ...
  ./pants help backends                      List all available backends.
  ./pants help-advanced backends             List all backends, including experimental/preview.
  ...

Either remove the experimental/preview backends from the

help backends

view, or just remove the

help-advanced

version. Pants version 2.16.0a0 OS Both pantsbuild/pants

#18502 `help backends` should indicate which backends are used by `update-build-files` Issue created by cognifloyd Is your feature request related to a problem? Please describe.

./pants help backends

shows that

pants.backend.build_files.fix.deprecations

and

pants.backend.build_files.fmt.black

are NOT enabled by default. That surprised me because the

update-build-files

goal definitely uses black. Describe the solution you'd like In the

help backends

output:

Enabled backends are marked with `*`. To enable a backend add it to `[GLOBAL].backend_packages`.

Let's add another character, maybe

-

, to show that a backend is conditionally enabled. The

update-build-files

goal will use a formatter backend without regard to the

[GLOBAL].backend_packages

setting. By default, it uses

pants.backend.build_files.fmt.black

and

pants.backend.build_files.fix.deprecations

. The formatter can be switched from

black

to

yapf

or

buildifier

based on the `[update-build-files].formatter` setting. Using

fix.deprecations

backend is controlled by the `[update-build-files].fix_safe_deprecations` setting (either

True

or

False

). So, these goal-specific backends should get a

-

indicator to show that it is enabled, but only for that one special-purpose goal based on different settings. Describe alternatives you've considered Just remove the backends from the output? That would be confusing too, because the backends exist. Also, you can enable the

pants.backend.build_files.fix.deprecations

backend so that it ALSO runs during the

fix

goal, not just during

update-build-files

. Additional context current output

% ./pants help backends

Backends
--------

List with all known backends for Pants.

Enabled backends are marked with `*`. To enable a backend add it to `[GLOBAL].backend_packages`. 

...
[ ] pants.backend.build_files.fix.deprecations                                              [pants]
[ ] pants.backend.build_files.fmt.black                                                     [pants]
[ ] pants.backend.build_files.fmt.buildifier                                                [pants]
[ ] pants.backend.build_files.fmt.yapf                                                      [pants]
...
[*] pants.backend.python                                                                    [pants]
     Support for Python.
     
     See <https://www.pantsbuild.org/docs/python-backend>.
[ ] pants.backend.python.lint.autoflake                                                     [pants]
     Autoformatter for removing unused Python imports.
     
     See <https://github.com/myint/autoflake> for details.
[*] pants.backend.python.lint.bandit                                                        [pants]
     Security linter for Python.
     
     See <https://www.pantsbuild.org/docs/python-linters-and-formatters> and
     <https://bandit.readthedocs.io/en/latest/>.
[*] pants.backend.python.lint.black                                                         [pants]
     Autoformatter for Python.
     
     See <https://www.pantsbuild.org/docs/python-linters-and-formatters> and
     <https://black.readthedocs.io/en/stable/>.
...

proposed output

% ./pants help backends

Backends
--------

List with all known backends for Pants.

Enabled backends are marked with `*`. To enable a backend add it to `[GLOBAL].backend_packages`. 
The `update-build-files` goal uses backends marked with `-` based on `[update-build-files]` settings.

...
[-] pants.backend.build_files.fix.deprecations                                              [pants]
[-] pants.backend.build_files.fmt.black                                                     [pants]
[ ] pants.backend.build_files.fmt.buildifier                                                [pants]
[ ] pants.backend.build_files.fmt.yapf                                                      [pants]
...
[*] pants.backend.python                                                                    [pants]
     Support for Python.
     
     See <https://www.pantsbuild.org/docs/python-backend>.
[ ] pants.backend.python.lint.autoflake                                                     [pants]
     Autoformatter for removing unused Python imports.
     
     See <https://github.com/myint/autoflake> for details.
[*] pants.backend.python.lint.bandit                                                        [pants]
     Security linter for Python.
     
     See <https://www.pantsbuild.org/docs/python-linters-and-formatters> and
     <https://bandit.readthedocs.io/en/latest/>.
[*] pants.backend.python.lint.black                                                         [pants]
     Autoformatter for Python.
     
     See <https://www.pantsbuild.org/docs/python-linters-and-formatters> and
     <https://black.readthedocs.io/en/stable/>.
...

pantsbuild/pants

#18503 Add descriptions for all modules in `help backends` output Issue created by cognifloyd Is your feature request related to a problem? Please describe. In the

./pants help backends

output, some backends have a description, and others don't. Describe the solution you'd like Add descriptions for all modules so it shows up in

help backends

output. The help output pulls this from the module docstring in

register.py

like this one for python. All of the descriptions should be written for the pants end user audience: Why would they want to enable this backend? Some of the existing descriptions should be updated so that the backends are documented consistently. pantsbuild/pants

#18504 `help` output does not mention `symbols` Issue created by cognifloyd Describe the bug

./pants help symbols

was added in #18378. But

./pants help

and

./pants help-advanced

do not mention it. Also,

./pants help [name]

should also mention

symbols

as a valid type of name. Pants version 2.16.0a0 OS both pantsbuild/pants

#18505 `preamble` backend: Add support for more complex templates Issue created by cognifloyd Is your feature request related to a problem? Please describe.

preamble

cannot currently handle the range of valid copyright headers in the StackStorm repo. I need a template that lets me have: • 0 or 1 shebang • 1 or more copyright lines • 1 license block Describe the solution you'd like When the preamble plugin adds the template to the top of a file, it should add only one copyright line, inserting it after the shebang line if present. If a file has a header with more than one copyright line, the preamble plugin should accept that. I'm not sure how to extend the template file format to include the additional information about the repeatable portions of the template. Describe alternatives you've considered regex-lint plugin can help with validating the header, but it can't automatically add it when missing. I really want to use

preamble

so it can auto-add the header for me. The regex-lint config looks like:

content_patterns:
  - name: python_header
    pattern: |+
      ^(?:#\!\/usr\/bin\/env python3
      )?# Copyright 20\d\d The StackStorm Authors.
      (?:# Copyright 20\d\d .*
      )*#
      # Licensed under the Apache License, Version 2.0 (the "License");
      # you may not use this file except in compliance with the License.
      # You may obtain a copy of the License at
      #
      #     <http://www.apache.org/licenses/LICENSE-2.0>
      #
      # Unless required by applicable law or agreed to in writing, software
      # distributed under the License is distributed on an "AS IS" BASIS,
      # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
      # See the License for the specific language governing permissions and
      # limitations under the License.

Along those lines, maybe the

template_by_globs

file could be something like:

<globs>:
  match: <regex string>
  new: "bla bla $year bla bla"

Another ideas for `template_by_globs`: What if it was an ordered list: each template must match in the order provided. Implicitly, that's what the

preamble

plugin does now: 0 or 1 shebang, and then the template. Except then it becomes odd figuring out how many times it can repeat... How would I tag to say that 0-1 shebang or 1+ for copyright, or only once for the rest. Additional context Here is an example of a header that should pass:

# Copyright 2020 The StackStorm Authors.
# Copyright 2019 Extreme Networks, Inc.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
#     <http://www.apache.org/licenses/LICENSE-2.0>
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.

This config, however, will add an extra preamble above what is already there:

"**/*.py:!**/__init__.py": |+
  # Copyright $year The StackStorm Authors.
  #
  # Licensed under the Apache License, Version 2.0 (the "License");
  # you may not use this file except in compliance with the License.
  # You may obtain a copy of the License at
  #
  #     <http://www.apache.org/licenses/LICENSE-2.0>
  #
  # Unless required by applicable law or agreed to in writing, software
  # distributed under the License is distributed on an "AS IS" BASIS,
  # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
  # See the License for the specific language governing permissions and
  # limitations under the License.

pantsbuild/pants

#18506 `preamble` backend: Use target type, not just path, to select files preamble should work on. Issue created by cognifloyd Is your feature request related to a problem? Please describe. I have extension-less executables that need the header. Writing a glob for extension-less files can be problematic. Describe the solution you'd like In the preamble config, allow me to specify target types, not just paths. Then anything that is owned by a

python_source*

target would get picked up, no matter if it matches the

*.py

pattern typical of python files. Describe alternatives you've considered Fiddly globs pantsbuild/pants

#18507 Investigate use of `/tmp` / `tmpfs` / persistent storage for execution roots Issue created by stuhood #18153 currently proposes switching from hardlinking to symlinking of large input files, because in some cases

/tmp

is mounted on a different filesystem as a

tmpfs

(check

df -h $(mktemp)

on the relevant machine). But from a quick survey of developers, mounting

/tmp

as a

tmpfs

has become more rare (recent Ubuntu and macOS mount it on a traditional filesystem). Too small a

tmpfs

can also result in

Os { code: 28, kind: StorageFull, message: "No space left on device" }

if enough concurrent processes are running. Benchmarks of using for `--local-execution-root-dir`: •

tmpfs

• a persistent directory •

/tmp

- the current default (on a machine with it mounted persistently) ... show that using a

tmpfs

is ~25% faster than using a persistent directory, and 45% faster than using

/tmp

(for an unknown reason). This is before the actual optimization suggested by #18153 is implemented though (which will reduce the actual IO that results from creating large files, by using hardlinks/symlinks in more cases). * * * Based on this, I'm proposing that we: 1. default

--local-execution-root-dir

to the same filesystem as the store 2. add explicit GC for

--local-execution-root-dir

• likely by nesting it under its pid, and periodically deleting orphaned directories (rather than relying on tmp cleanup) 3. continue to use hardlinking in #18153 • with detection/fallback to either copying or symlinking (TBD after benchmarking, since copies would avoid visibly changing rule sandbox behavior) if

--local-execution-root-dir

has been set to a different filesystem This will allow for a good default on machines that don't have an available

tmpfs

(which seems common), while still allowing folks to move the

--local-execution-root-dir

if they have explicitly created a large enough

tmpfs

and would like to use it. pantsbuild/pants

#18162 Testing in Docker environment fails with pex FileNotFoundError Issue created by psontag Describe the bug When trying to run some tests in my

docker_environment

I get the below error:

Traceback (most recent call last):
  File "/usr/local/lib/python3.8/runpy.py", line 187, in _run_module_as_main
    mod_name, mod_spec, code = _get_main_module_details(_Error)
  File "/usr/local/lib/python3.8/runpy.py", line 221, in _get_main_module_details
    return _get_module_details(main_name)
  File "/usr/local/lib/python3.8/runpy.py", line 155, in _get_module_details
    code = loader.get_code(mod_name)
  File "<frozen zipimport>", line 159, in get_code
  File "<frozen zipimport>", line 709, in _get_module_code
  File "<frozen zipimport>", line 536, in _get_data
FileNotFoundError: [Errno 2] No such file or directory: './pex'

I have included the full traceback below. If you need any more information please let me know. Pants version 2.16.0.dev5 OS I am developing on a Mac but the tests are supposed to be executed in the

python:3.8-slim

linux Docker container. Additional info In my root level

BUILD

file I have configured the

docker_environment

docker_environment(
    name="dev_docker",
    platform="linux_x86_64",
    image="python:3.8-slim",
    python_bootstrap_search_path=["<PATH>"],
)

The test

BUILD

file has set the environment

python_tests(
    name="tests", environment="dev_docker"
)

pants.toml

[GLOBAL]
pants_version = "2.16.0.dev5"
dynamic_ui = true
backend_packages = [
  "pants.backend.build_files.fix.deprecations",
  "pants.backend.build_files.fmt.black",
  "pants.backend.python",
  "pants.backend.python.lint.autoflake",
  "pants.backend.python.lint.black",
  "pants.backend.python.lint.flake8",
  "pants.backend.python.lint.isort",
  "pants.backend.python.lint.pyupgrade",
  "pants.backend.python.typecheck.mypy",
  "pants.backend.docker",
]

[python]
enable_resolves = true
interpreter_constraints = [">=3.8.*"]

[python.resolves_to_only_binary]
__default__= [":all:"]

[python-bootstrap]
search_path = [
  "<PYENV>",
  "<PATH>"
]

[test]
batch_size = 16

[pytest]
args = ["--no-header"]
execution_slot_var = "PANTS_RUNNER_ID"
config = "pyproject.toml"

[docker]
tailor = false

[environments-preview.names]
dev_docker = "//:dev_docker"

full error traceback ``` pants test src/project:: --print-stacktrace --keep-sandboxes=on_failure 132353.31 [INFO] Completed: Pulling Docker image

python:3.8-slim

because the image is missing locally. 132354.77 [INFO] Preserving local process execution dir /private/var/folders/8b/0w6lmjjx7fsdm_6hhcrjfdfw0000gq/T/pants-sandbox-oEZLFk for Find interpreter for constraints: CPython>=3.8.* 132354.78 [ERROR] 1 Exception encountered: Engine traceback: in select .. in pants.core.goals.test.run_tests

test

goal Traceback (most recent call last): File "/Users/me/Library/Caches/nce/a7a7e5c8d0659936773c836830336c0e3743f47588d74ecdce11e40b499a7003/bindings/venvs/2.16.0.dev5/lib/python3.9/site-packages/pants/engine/internals/selectors.py", line 626, in native_engine_generator_send res = rule.send(arg) if err is None else rule.throw(throw or err) File "/Users/me/Library/Caches/nce/a7a7e5c8d0659936773c836830336c0e3743f47588d74ecdce11e40b499a7003/bindings/venvs/2.16.0.dev5/lib/python3.9/site-packages/pants/core/goals/test.py", line 836, in run_tests results = await MultiGet( File "/Users/me/Library/Caches/nce/a7a7e5c8d0659936773c836830336c0e3743f47588d74ecdce11e40b499a7003/bindings/venvs/2.16.0.dev5/lib/python3.9/site-packages/pants/engine/internals/selectors.py", line 360, in MultiGet return await _MultiGet(tuple(__arg0)) File "/Users/me/Library/Caches/nce/a7a7e5c8d0659936773c836830336c0e3743f47588d74ecdce11e40b499a7003/bindings/venvs/2.16.0.dev5/lib/python3.9/site-packages/pants/engine/internals/selectors.py", line 167, in await result = yield self.gets File "/Users/me/Library/Caches/nce/a7a7e5c8d0659936773c836830336c0e3743f47588d74ecdce11e40b499a7003/bindings/venvs/2.16.0.dev5/lib/python3.9/site-packages/pants/engine/internals/selectors.py", line 626, in native_engine_generator_send res = rule.send(arg) if err is None else rule.throw(throw or err) File "/Users/me/Library/Caches/nce/a7a7e5c8d0659936773c836830336c0e3743f47588d74ecdce11e40b499a7003/bindings/venvs/2.16.0.dev5/lib/python3.9/site-packages/pants/backend/python/goals/pytest_runner.py", line 475, in run_python_tests setup = await Get( File "/Users/me/Library/Caches/nce/a7a7e5c8d0659936773c836830336c0e3743f47588d74ecdce11e40b499a7003/bindings/venvs/2.16.0.dev5/lib/python3.9/site-packages/pants/engine/internals/selectors.py", line 120, in await result = yield self File "/Users/me/Library/Caches/nce/a7a7e5c8d0659936773c836830336c0e3743f47588d74ecdce11e40b499a7003/bindings/venvs/2.16.0.dev5/lib/python3.9/site-packages/pants/engine/internals/selectors.py", line 626, in native_engine_generator_send res = rule.send(arg) if err is None else rule.throw(throw or err) File "/Users/me/Library/Caches/nce/a7a7e5c8d0659936773c836830336c0e3743f47588d74ecdce11e40b499a7003/bindings/venvs/2.16.0.dev5/lib/python3.9/site-packages/pants/backend/python/goals/pytest_runner.py", line 475, in run_python_tests setup = await Get( File "/Users/me/Library/Caches/nce/a7a7e5c8d0659936773c836830336c0e3743f47588d74ecdce11e40b499a7003/bindings/venvs/2.16.0.dev5/lib/python3.9/site-packages/pants/engine/internals/selectors.py", line 120, in await result = yield self File "/Users/me/Library/Caches/nce/a7a7e5c8d0659936773c836830336c0e3743f47588d74ecdce11e40b499a7003/bindings/venvs/2.16.0.dev5/lib/python3.9/site-packages/pants/engine/internals/selectors.py", line 626, in native_engine_generator_send res = rule.send(arg) if err is None else rule.throw(throw or err) File "/Users/me/Library/Caches/nce/a7a7e5c8d0659936773c836830336c0e3743f47588d74ecdce11e40b499a7003/bindings/venvs/2.16.0.dev5/lib/python3.9/site-packages/pants/backend/python/goals/pytest_runner.py", line 262, in setup_pytest_for_target ) = await MultiGet( File "/Users/me/Library/Caches/nce/a7a7e5c8d0659936773c836830336c0e3743f47588d74ecdce11e40b499a7003/bindings/venvs/2.16.0.dev5/lib/python3.9/site-packages/pants/engine/internals/selectors.py", line 451, in MultiGet return await _MultiGet((__arg0, __arg1, __arg2, __arg3, __arg4, __arg5)) File "/Users/me/Library/Caches/nce/a7a7e5c8d0659936773c836830336c0e3743f47588d74ecdce11e40b499a7003/bindings/venvs/2.16.0.dev5/lib/python3.9/site-packages/pants/engine/internals/selectors.py", line 167, in await result = yield self.gets File "/Users/me/Library/Caches/nce/a7a7e5c8d0659936773c836830336c0e3743f47588d74ecdce11e40b499a7003/bindings/venvs/2.16.0.dev5/lib/python3.9/site-packages/pants/engine/internals/selectors.py", line 626, in native_engine_generator_send res = rule.send(arg) if err is None else rule.throw(throw or err) File "/Users/me/Library/Caches/nce/a7a7e5c8d0659936773c836830336c0e3743f47588d74ecdce11e40b499a7003/bindings/venvs/2.16.0.dev5/lib/python3.9/site-packages/pants/backend/python/goals/pytest_runner.py", line 475, in run_python_tests setup = await Get( File "/Users/me/Library/Caches/nce/a7a7e5c8d0659936773c836830336c0e3743f47588d74ecdce11e40b499a7003/bindings/venvs/2.16.0.dev5/lib/python3.9/site-packages/pants/engine/internals/selectors.py", line 120, in await result = yield self File "/Users/me/Library/Caches/nce/a7a7e5c8d0659936773c836830336c0e3743f47588d74ecdce11e40b499a7003/bindings/venvs/2.16.0.dev5/lib/python3.9/site-packages/pants/engine/internals/selectors.py", line 626, in native_engine_generator_send res = rule.send(arg) if err is None else rule.throw(throw or err) File "/Users/me/Library/Caches/nce/a7a7e5c8d0659936773c836830336c0e3743f47588d74ecdce11e40b499a7003/bindings/venvs/2.16.0.dev5/lib/python3.9/site-packages/pants/backend/python/goals/pytest_runner.py", line 262, in setup_pytest_for_target ) = await MultiGet( F… pantsbuild/pants

#18508 Symlinked export pointed to a directory that didn't exist Issue created by engnatha Describe the bug Using the new

--export-symlink-python-virtualenv

option in 2.15.0 and got some interesting results. I had messed up the symlinked venv so I was working on getting back to a known state. In the code block below, you can see I deleted the entire pants cache and had the symlinked export symlink to a venv that didn't exist.

~/devel/monorepo (move-dev-reqs)$ rm -rf ~/.cache/pants/
~/devel/monorepo (move-dev-reqs)$ ./pants export --export-symlink-python-virtualenv --resolve=python-default
Bootstrapping Pants using /usr/bin/python3.9
Creating the virtualenv PEX.
Downloading the Pex PEX.
SHA256 fingerprint of <https://github.com/pantsbuild/pex/releases/download/v2.1.103/pex> verified.
Installing pantsbuild.pants==2.15.0 into a virtual environment at /home/nathanael/.cache/pants/setup/bootstrap-Linux-x86_64/2.15.0_py39
New virtual environment successfully created at /home/nathanael/.cache/pants/setup/bootstrap-Linux-x86_64/2.15.0_py39.
Wrote symlink to immutable virtualenv for python-default (using Python 3.8.10) to dist/export/python/virtualenvs/python-default
~/devel/monorepo (move-dev-reqs)$ ls dist/export/python/virtualenvs/python-default/
3.8.10
~/devel/monorepo (move-dev-reqs)$ ls -lh dist/export/python/virtualenvs/python-default/3.8.10 
lrwxrwxrwx 1 nathanael nathanael 138 Mar 14 18:47 dist/export/python/virtualenvs/python-default/3.8.10 -> /home/nathanael/.cache/pants/named_caches/pex_root/venvs/c978a68158a371b508ccdfb785cf9438c7adcf3b/ddab8011daaee380698ac2fb9701af18c90c03f6
~/devel/monorepo (move-dev-reqs)$ ls ~/.cache/pants/named_caches/
ls: cannot access '/home/nathanael/.cache/pants/named_caches/': No such file or directory

This is problematic since I was expecting the export to fix things. Using

--no-local-cache

sets up a venv properly, but without that flag, it falls back to the non-existent directory. Pants version 2.15.0 OS Ubuntu 20.04 pantsbuild/pants

#18509 `pants update-build-files` does not fix ersc Issue created by kaos Describe the bug

$ pants update-build-files ::
10:47:16.19 [WARN] DEPRECATED: the target name experimental_run_shell_command is scheduled to be removed in version 2.18.0.dev0.

Instead, use `run_shell_command`, which behaves the same. Run `pants update-build-files` to automatically fix your BUILD files.
10:47:16.20 [WARN] DEPRECATED: the target name experimental_run_shell_command is scheduled to be removed in version 2.18.0.dev0.

Instead, use `run_shell_command`, which behaves the same. Run `pants update-build-files` to automatically fix your BUILD files.
10:47:27.13 [INFO] No required changes to BUILD files found.However, there may still be deprecations that `update-build-files` doesn't know how to fix. See <https://www.pantsbuild.org/v2.16/docs/upgrade-tips> for upgrade tips.

Pants version 2.16.0a0 OS Mac M1 Additional info The target is in a macro file. pantsbuild/pants

#18510 Add python version to `Could not find a version that satisfies the requirement` error message. Issue created by asherf when the user sees an error like:

ERROR: Could not find a version that satisfies the requirement pandas==1.5.2
ERROR: No matching distribution found for pandas==1.5.2

it can be confusing, because the python version (interpreter) is not mentioned here. which is usually the cause for this error. see: https://pantsbuild.slack.com/archives/C046T6T9U/p1678901897308199 so the error message should have more information to make it clear the cause of for this failure. pantsbuild/pants

#2097 Option to exclude dependencies from pex file Issue created by JoostvDoorn Pex currently relies on Pip to resolve dependencies and build the PEX file. However, there are cases where users may want to exclude certain packages, either because they are already available on the target environment or because they are too large to include in the PEX file. The proposed solution is to add an exclude flag to Pex, which would allow users to specify a list of packages to exclude from the PEX file. During the assembly phase, Pex would exclude these packages and their transitive dependencies, resulting in a smaller PEX file. I believe this would be a useful feature for many users. We would want to use it with Pants to exclude dependencies such as pyspark for an environment that includes pyspark already. This issue follows up on the comment here: #2082 (comment) Other related issues have been filed previously in pants and pex as well. #1146 pantsbuild/pants#11324 pantsbuild/pex

#18511 Expand `generate-lockfiles --diff` to report when VCS deps change Issue created by cognifloyd Is your feature request related to a problem? Please describe. When a VCS dep uses a branch name (like `st2-rbac-backend @ git+https://github.com/StackStorm/st2-rbac-backend.git@master`), and that branch gets new commits, the artifact hash changes even though the version does not. This update does not get reflected in the

generate-lockfiles --diff

output. Describe the solution you'd like Include a line in the diff output for VCS deps when the hash changes, even though the version stays the same. Describe alternatives you've considered Manually review the git diff when committing the lockfile (well, I do this anyway - the

--diff

summarized output is just so much nicer!) Additional context Here is the lockfile's diff for a recent update to the st2-rbac-backend repo where nothing but the hash changed:

{
           "artifacts": [
             {
               "algorithm": "sha256",
-              "hash": "3b40f91ab4d13271cb1049423b54f103710ded6c8469cac3e39c63ad8b3ece09",
+              "hash": "9141e9e388ab036f9c6ecc055666e900de86f0e30ac818ca37af86d8cb6fb309",
               "url": "git+<https://github.com/StackStorm/st2-rbac-backend.git@master>"
             }
           ],
           "project_name": "st2-rbac-backend",
           "requires_dists": [],
           "requires_python": null,
           "version": "3.9.dev0"
         },

pantsbuild/pants

#18512 Internal error in `mypy` during `pants check` Issue created by kaos Describe the bug Once every third blue moon I observe mypy crashing with "internal error" for perfectly valid sources. Pants version 2.16.0a0 (but have observed this since ~2.14 or before) OS Mac Additional info

# from mypy sandbox
./__run.sh
src/.../x.py:205: error: INTERNAL ERROR -- Please try using mypy master on GitHub:
<https://mypy.readthedocs.io/en/stable/common_issues.html#using-a-development-mypy-build>
Please report a bug at <https://github.com/python/mypy/issues>
version: 0.961
Traceback (most recent call last):
  File "mypy/checker.py", line 431, in accept
  File "mypy/nodes.py", line 747, in accept
  File "mypy/checker.py", line 782, in visit_func_def
  File "mypy/checker.py", line 793, in _visit_func_def
  File "mypy/checker.py", line 1500, in check_method_override
  File "mypy/checker.py", line 1528, in check_method_or_accessor_override_for_base
  File "mypy/checker.py", line 1614, in check_method_override_for_base_with_name
  File "mypy/checker.py", line 1776, in check_override
  File "mypy/messages.py", line 845, in signature_incompatible_with_supertype
  File "mypy/messages.py", line 867, in pretty_callable_or_overload
  File "mypy/messages.py", line 1979, in pretty_callable
AttributeError: attribute 'arguments' of 'FuncDef' undefined
src/.../x.py:205: : note: use --pdb to drop into pdb

Reading the comment in

__mypyrunner.sh

my suspicion falls on a corrupt mypy cache, as this issue goes away every time I wipe the pants caches and there are some trickery going on with it. The

cache.db

sqlite file seems to not be corrupt though. (as in it is usable, not sure about the state of the contents in the DB itself) Minimal work-around To preserve as much cache as possible, this wipes the mypy caches only:

rm -rf ~/.cache/pants/named_caches/mypy_cache/

pantsbuild/pants

#18515 Set `GOPRIVATE` for golang build &amp; test New discussion created by misterikkit Is there any way to specify environment variables to the go toolchain? Specifically, I work across a set of git repos hosted on a private server, and must set

GOPRIVATE=my.server.domain.name

for builds to work at all. I would assume this gets configured in

pants.toml

, but I did not find a concise listing of the configuration options in go support. (I also saw nothing relevant in go_binary) Thanks in advance! pantsbuild/pants

#18516 Move Docker images off Docker Hub Issue created by jsirois Email from Docker:

Docker is sunsetting Free Team organizations

Free Team organizations are a legacy subscription tier that no longer exists. This tier included many of the same features, rates, and functionality as a paid Docker Team subscription.

After reviewing the list of accounts that are members of legacy Free Team organizations, we’ve identified yours as potentially being one of them.

If you own a legacy Free Team organization, access to paid features — including private repositories — will be suspended on April 14, 2023 (11:59 pm UTC). Upgrade your subscription before April 14, 2023 to continue accessing your organization.

If you don’t upgrade to a paid subscription, Docker will retain your organization data for 30 days, after which it will be subject to deletion. During that time, you will maintain access to any images in your public repositories, though rate limitations will apply. At any point during the 30-day period, you can restore access to your organization account if you upgrade to a paid subscription. Visit our FAQ for more information.

One alternative is GitHub packages: https://docs.github.com/en/packages/learn-github-packages/introduction-to-github-packages pantsbuild/pants

#18517 Pants on Fedora Silverblue New discussion created by jgarte Hi, This is an open question for anyone who might know that answer on it and would like to share knowledge on it. What would be required to run pants on Fedora Silverblue? https://silverblue.fedoraproject.org/ pantsbuild/pants

#18519 bug with loading dependencies when running mypy Issue created by ryanking Pants version 2.16, but also was happening on 2.15 OS Mac I have been trying to hunt down a bug in pants for a few weeks and finally have a reproduction, but don't know what do do with it. From time to time, a mypy run will fail due to missing dependencies. Dependencies which are definitely configured and findable. Often a re-run will succeed, which has made this very hard to reproduce. I now found a case where running a single file will fail, but running with

::

will succeed. I have both sandboxes and when I edit __files.txt in the latter it passes. However I don't know what to look for in these sandboxes to figure out what's going on. Failure from the "bad" sandbox:

.cache/pex_root/venvs/e1862d6434ffd1c4230c239d7d17d05591664247/4e8b276cfa3c3338d49f8656ee098b9ee6162291/lib/python3.8/site-packages/botocore-stubs/compat.pyi:20:1: error: Library stubs not installed for "six.moves.http_client"  [import]
    from six.moves import http_client
    ^
.cache/pex_root/venvs/e1862d6434ffd1c4230c239d7d17d05591664247/4e8b276cfa3c3338d49f8656ee098b9ee6162291/lib/python3.8/site-packages/botocore-stubs/compat.pyi:20:1: note: Hint: "python3 -m pip install types-six"
.cache/pex_root/venvs/e1862d6434ffd1c4230c239d7d17d05591664247/4e8b276cfa3c3338d49f8656ee098b9ee6162291/lib/python3.8/site-packages/botocore-stubs/compat.pyi:20:1: note: (or run "mypy --install-types" to install all missing stub packages)
.cache/pex_root/venvs/e1862d6434ffd1c4230c239d7d17d05591664247/4e8b276cfa3c3338d49f8656ee098b9ee6162291/lib/python3.8/site-packages/botocore-stubs/compat.pyi:20:1: note: See <https://mypy.readthedocs.io/en/stable/running_mypy.html#missing-imports>
.cache/pex_root/venvs/e1862d6434ffd1c4230c239d7d17d05591664247/4e8b276cfa3c3338d49f8656ee098b9ee6162291/lib/python3.8/site-packages/botocore-stubs/compat.pyi:20:1: error: Library stubs not installed for "six.moves"  [import]
    from six.moves import http_client
    ^
src/django_apps/counseling/google_calendar.py:74:14: error: Module has no attribute "parser"  [attr-defined]
                (dateutil.parser.parse(e['start']), dateutil.parser.parse(e['end']))
                 ^~~~~~~~~~~~~~~
Found 3 errors in 2 files (checked 1 source file)

Success from the good sandbox, edited to run only the file in question:

Success: no issues found in 1 source file

A fun complication is that the mypy cache appears to get poisoned. If I have a failed run on the "bad" sandbox, the good one will then fail. If I reset the cache it will work. pantsbuild/pants

#12449 Allow `[python_tool].version` and `[python-tool].extra_requirements` to point to `python_requirement` targets? Issue created by Eric-Arellano A weird thing with our tool support is that you can have the same requirement both to run a tool and as a library imported by your code. Classic example: Pytest. Right now, you have to be careful to set the pytest version in both your requirements.txt and

[pytest].version

, violating DRY. Instead, this proposal would update

PythonToolBase

to allow either using a raw requirement string or a target address for a

python_requirement_library

, so you could do this:

# requirements.txt
pytest==5.6.1

[pytest]
version = "//:pytest"

pantsbuild/pants

#16044 Unable to guarantee consistent versions for (transitive) dependencies between tools (particularly mypy) and runtime resolves Issue created by huonw Describe the bug We've got a codebase that uses mypy extensively, and various plugins, and we'd prefer if we had good methods for having the versions of our mypy plugin libraries match the ones used at runtime. For instance:

# pyproject.toml
[tool.mypy]
plugins = ["pydantic.mypy", "sqlmypy", "strawberry.ext.mypy_plugin"]
...

Two of these come from libraries also used at runtime (

pydantic

and

strawberry-graphql

) and thus it'd be good to be using a consistent version globally: if we have mypy using plugins for different versions of our runtime libraries who knows if type checking passing is meaningful. There's a few options: • write

extra_requirements = ["pydantic==...", ...]

etc. but that duplicates the version configuration, and doesn't guarantee identical versions of transitive dependencies without listing them all out by hand. • use

source_plugins

pointing to the relevant

python_requirement

target (which isn't what it's documented to be used for), which at least deduplicates version configuration for those direct requirements, but also doesn't guarantee identical versions of transitive dependencies. The second is slightly better than the first, so that's what we've tried so far despite it seemingly being not a proper use of

source_plugins

. For instance:

[GLOBAL]
pants_version = "2.12.0"

backend_packages = [
  "pants.backend.python",
  "pants.backend.python.typecheck.mypy",
]

[python]
interpreter_constraints = ["CPython==3.9.*"]
enable_resolves = true

[mypy]
lockfile = "3rdparty/python/mypy.lock"
source_plugins = [
  "//:pydantic",
]

[anonymous-telemetry]
enabled = false

With top-level

BUILD

that simulates a transitive dependency

typing-extensions

that's stuck on an older version:

python_requirement(name="pydantic", requirements=["pydantic==1.9.0"])
python_requirement(name="typing-extensions", requirements=["typing-extensions==4.0.0"])

This mostly works, after running

./pants generate-lockfiles

, we have

mypy.lock

containing:

...
          "project_name": "pydantic",
          "version": "1.9"
...
          "project_name": "typing-extensions",
          "version": "4.3"
...
  "requirements": [
    "mypy==0.910",
    "pydantic==1.9.0"
  ],
...

However, note that

typing-extensions

is missing from the requirements, and indeed the pinned version of the transitive dependency of

pydantic

is different to the one in the runtime default resolve (which is just

"version": "4"

). Notes: • For this particular

typing-extensions

dependency, this probably doesn't matter so much, but other transitive dependencies of some plugins may make a difference if the version is significantly different (e.g. hypothetically a plugin that validates that a particular string literal with strict formatting requirements (e.g. "the arg is a valid regex" in a call like

re.compile("...")

), in addition to the basic runtime check: if the plugin uses a newer version that might support more features, mypy will incorrectly be successful). • This potentially relates to general issues about

./pants generate-lockfiles

always choosing the latest version (if there's been new releases), such as #12880, #15704 and #15723. • This also potentially relates to #12449. • Maybe we want to be saying that mypy can pull plugins from a particular resolve, but that's potentially weird (e.g. the docs for

source_plugins

call out using a separate resolve for deps of such plugins https://www.pantsbuild.org/docs/reference-mypy#section-source-plugins) As always, thanks for pants! Pants version 2.12.0 OS macOS Additional info Reproducing example: https://github.com/huonw/pants-tool-pinning pantsbuild/pants

#14320 How multiple lockfiles &amp; first-party source plugins work for Pylint, MyPy, Flake8 Issue created by Eric-Arellano We allow you to create first-party source plugins and set via

[pylint].source_plugins

etc, which expects addresses to

python_source

targets. Those can depend on third-party requirement like any normal code, via

python_requirement

. We currently consume the requirements via the tool lockfile: pants/src/python/pants/backend/python/lint/pylint/rules.py Lines 103 to 113 in </pantsbuild/pants/commit/f258544f8cf9a3fe135a40e587d27ff8fa0e42c8|f258544> Our tool lockfile generation knows to include those requirements: pants/src/python/pants/backend/python/lint/pylint/subsystem.py Lines 292 to 296 in </pantsbuild/pants/commit/f258544f8cf9a3fe135a40e587d27ff8fa0e42c8|f258544> That means that that tool lockfile codepath is ignoring the

resolve

field of those

python_requirements

. -- However,

resolve

is still relevant! You can run Pants goals on that first party plugin code directly, like anything else. When you do that, those targets will be the inputs to our

pex_from_targets.py

code, which determines the

resolve

by inspecting the input targets: pants/src/python/pants/backend/python/lint/pylint/rules.py Lines 91 to 102 in </pantsbuild/pants/commit/f258544f8cf9a3fe135a40e587d27ff8fa0e42c8|f258544> -- Practically, what this all means is that multiple resolves is very useful for source plugins. You can define a distinct resolve for your plugin code, which lets you avoid introducing dependencies just needed for your plugin into your main projects. When you run Pants on that plugin code directly, it will do the right thing. When you run Pants on code that uses a different resolve than your plugin, Pants can still handle that. It won't complain about conflicting resolves, and the tool lockfile will still have the requirements you need to make things work. If you're fine with using a single resolve for everything, you can ignore all this nuance and Pants will do the right thing. pantsbuild/pants

#18524 `javascript_test` target, and support running `npm test &lt;file&gt;` via `pants test` Issue created by tobni Is your feature request related to a problem? Please describe. The javascript backend does not support

pants test

goal. Describe the solution you'd like Add a

javascript_test

target, and let

pants test

run via

npm test <file>

, let users install test runner via

package.json#devDependencies

Describe alternatives you've considered Special subsystem for different test-runners. Additional context This means a "loose"

javascript_source

without a

package_json

cannot be tested via pants. pantsbuild/pants