#19082 Auto cherry-picker always errors with merge conflicts Issue created by benjyw It failed on #19079 with a zillion merge conflicts: https://github.com/pantsbuild/pants/actions/runs/5039766268/jobs/9038117307 But actually is a super-trivial merge that a manual run of the cherry-pick.sh script did with no trouble. pantsbuild/pants

#19085 src/python/pants/backend/javascript/package/rules_integration_test.py fails intermittently in CI Issue created by benjyw E.g., https://github.com/pantsbuild/pants/actions/runs/5040314797/jobs/9040567788 The error is:

error <https://registry.yarnpkg.com/caniuse-lite/-/caniuse-lite-1.0.30001485.tgz>: Extracting tar content of undefined failed, the file appears to be corrupt: "EEXIST: file already exists, mkdir '/tmp/pants-sandbox-N5M0ON/._yarn_cache/v6/npm-caniuse-lite-1.0.30001485-026bb7319f1e483391872dc303a973d4f513f619-integrity/node_modules/caniuse-lite/data/features'"

So possibly a race condition while writing to yarn cache? pantsbuild/pants

#18489 `pants check` fails on Terraform modules due to lack of `terraform init` Issue created by ndellosa95 Describe the bug After setting up some basic Terraform in my repo, running

pants check ::

fails with the following error message:

16:48:40.39 [ERROR] Completed: pants.backend.terraform.goals.check.terraform_check - terraform-validate failed (exit code 1).
Partition #1 - `terraform validate` on `myrepo/terraform/environments/dev`:
╷
│ Error: Module not installed
│ 
│   on myrepo/terraform/environments/dev/main.tf line 23:
│   23: module "my_module" {
│ 
│ This module is not yet installed. Run "terraform init" to install all
│ modules required by this configuration.
╵

Partition #2 - `terraform validate` on `myrepo/terraform/modules/infrastructure`:
╷
│ Error: Missing required provider
│ 
│ This configuration requires provider
│ <http://registry.terraform.io/hashicorp/azurerm|registry.terraform.io/hashicorp/azurerm>, but that provider isn't available.
│ You may be able to install it automatically by running:
│   terraform init
╵



✕ terraform-validate failed.

Pants version 2.15 OS Intel Mac Additional info Ideally,

pants check

would run

init

,

validate

, and

plan

. I'm going to open a feature request for this exact thing. It would also be super useful if you could run

terraform apply

through Pants, either using the

publish

or

experimental-deploy

goals (another issue, the difference between these two should be elaborated on as they're both from the core backend). pantsbuild/pants

#19098 docker_image secrets doesn't expanduser Issue created by idan-at Describe the bug When defining secrets on a

docker_image

target, paths starting with

~

are not respected. For example:

docker_image(
  secrets={
    mysecret="~/.secret"
  }
)

Leads to an error:

ERROR: failed to stat <pwd>/~/.secret: stat <pwd>/~/.secret: no such file or directory

. Pants version 2.15.0 OS MacOS pantsbuild/pants

#19105 Blog about VS Code plugin Issue created by cczona pantsbuild/pants

#19106 Rewrite intro docs to bring more focus onto dependency inference Issue created by cczona Paraphrasing @Eric-Arellano: Right now, Pants intro docs give a laundry list of benefits like fine-grained caching. One of the benefits is dep inference. Arguably the most important part of explaining Pants is dep inference. It's what unlocks other benefits such fine-grained caching, and it's what differentiates us from competing systems. It's a distinguishing feature that the competition has never caught up with, and enables a lot of ease-of-use. We should rewrite the home page to be more narrative-driven, where dep inference is first presented and then it's illustrated how this unlocks key benefits. pantsbuild/pants

#12240 Add how to guides to Pants docs Issue created by Eric-Arellano differentiates between how tos vs. guides vs. API references. Right now, Pants is missing dedicated how tos for things like: • adding a linter • publishing lots of wheels out of a monorepo • setting up a test to run with Pants • should probably include handling hermeticity and

files

• building a PEX • finding your most used code Our current docs do have elements of how-tos, but they should be more dedicated and tutorial style, e.g. that you can follow along with. A major part of this ticket is deciding how to organize the sidebar of our docs. pantsbuild/pants

#19117 Blog about Django support Issue created by cczona Per @happy-kitchen-89482 on Slack: "We have several features custom-built to work well with Django (e.g., so that makemigrations and runserver work properly). I should probably write up a blog post about those, since we have more and more Django users interested in Pants" pantsbuild/pants

#19118 Blog about how we use PyO3 and the gains it's delivering Issue created by cczona pantsbuild/pants

#19121 Getting UnicodeDecodeError from stdout.decode when xdist is disabled Issue created by TalAmuyal When setting

[pytest].xdist_enabled = false

in

pants.toml

, I get:

15:53:04.53 [ERROR] 1 Exception encountered:

Engine traceback:
  in select
    ..
  in pants.core.goals.test.run_tests
    `test` goal

Traceback (most recent call last):
  File "/opt/pantsbuild/.cache/nce/ae553d873185fa750061b1a3d4226442fb99fb55d4403b0227ec972406554da3/bindings/venvs/2.16.0rc1/lib/python3.9/site-packages/pants/engine/internals/selectors.py", line 623, in native_engine_generator_send
    res = rule.send(arg) if err is None else rule.throw(throw or err)
  File "/opt/pantsbuild/.cache/nce/ae553d873185fa750061b1a3d4226442fb99fb55d4403b0227ec972406554da3/bindings/venvs/2.16.0rc1/lib/python3.9/site-packages/pants/core/goals/test.py", line 832, in run_tests
    results = await MultiGet(
  File "/opt/pantsbuild/.cache/nce/ae553d873185fa750061b1a3d4226442fb99fb55d4403b0227ec972406554da3/bindings/venvs/2.16.0rc1/lib/python3.9/site-packages/pants/engine/internals/selectors.py", line 358, in MultiGet
    return await _MultiGet(tuple(__arg0))
  File "/opt/pantsbuild/.cache/nce/ae553d873185fa750061b1a3d4226442fb99fb55d4403b0227ec972406554da3/bindings/venvs/2.16.0rc1/lib/python3.9/site-packages/pants/engine/internals/selectors.py", line 165, in __await__
    result = yield self.gets
  File "/opt/pantsbuild/.cache/nce/ae553d873185fa750061b1a3d4226442fb99fb55d4403b0227ec972406554da3/bindings/venvs/2.16.0rc1/lib/python3.9/site-packages/pants/engine/internals/selectors.py", line 623, in native_engine_generator_send
    res = rule.send(arg) if err is None else rule.throw(throw or err)
  File "/opt/pantsbuild/.cache/nce/ae553d873185fa750061b1a3d4226442fb99fb55d4403b0227ec972406554da3/bindings/venvs/2.16.0rc1/lib/python3.9/site-packages/pants/backend/python/goals/pytest_runner.py", line 507, in run_python_tests
    return TestResult.from_batched_fallible_process_result(
  File "/opt/pantsbuild/.cache/nce/ae553d873185fa750061b1a3d4226442fb99fb55d4403b0227ec972406554da3/bindings/venvs/2.16.0rc1/lib/python3.9/site-packages/pants/core/goals/test.py", line 172, in from_batched_fallible_process_result
    stdout=process_result.stdout.decode(),
UnicodeDecodeError: 'utf-8' codec can't decode byte 0x84 in position 1437: invalid start byte

Pants version Tried with both 2.16.0rc1 and 2.16.0rc3. OS Linux. Additional info @tdyas suggested in Slack that we shouldn't assume

ProcessResult.stdout

is in any particular format. Slack thread: link. pantsbuild/pants

#14427 Pants Daily extened testing breaks for python 3.9 & 3.8 Issue created by asherf There seems to be an issue with one of the tests:

tests/python/pants_test/integration/pep561_integration_test.py:pep561_integration

it seems that there might be an issues with how wheels are generated.

E               pants.engine.process.ProcessExecutionFailure: Process 'Building requirements.pex with 2 requirements: pantsbuild.pants.testutil@ file:///tmp/process-executionhjsTuh/pantsbuild.pants.testutil-2.10.0rc0-py37.py38.py39-none-any.whl, pantsbuild.pants@ file:///tmp/process-executionhjsTuh/pantsbuild.pants-2.10.0rc0-cp39-cp39-linux_x86_64.whl' failed with exit code 1.
E               stdout:
E
E               stderr:
E               ERROR: pantsbuild.pants-2.10.0rc0-cp39-cp39-linux_x86_64.whl is not a supported wheel on this platform.
E               pid 1740 -> /data/home/asher/.cache/pants/named_caches/pex_root/venvs/deb35be568d133a27ee7892e2e4d92cdaec27960/6d236475afbb7effab5a8fa5c138ffa500fb50b5/pex --disable-pip-version-check --no-python-version-warning --exists-action a --isolated -q --cache-dir /data/home/asher/.cache/pants/named_caches/pex_root --log /tmp/process-executionauGYaH/.tmp/tmpddl0jvx4/pip.log download --dest /tmp/process-executionauGYaH/.tmp/tmpu1558erc/data.home.asher..pyenv.versions.3.7.10.bin.python3.7 pantsbuild.pants.testutil@ file:///tmp/process-executionhjsTuh/pantsbuild.pants.testutil-2.10.0rc0-py37.py38.py39-none-any.whl pantsbuild.pants@ file:///tmp/process-executionhjsTuh/pantsbuild.pants-2.10.0rc0-cp39-cp39-linux_x86_64.whl --index-url <https://pypi.org/simple/> --retries 5 --timeout 15 exited with 1 and STDERR:
E               None
E
E
E
E               Use `--no-process-cleanup` to preserve process chroots for inspection.
E
E       assert 1 == 0
E        +  where 1 = PantsResult(command=['/data/home/asher/.cache/pants/named_caches/pex_root/venvs/s/1f8cedfe/venv/bin/python3.9', '-m', ...rocess chroots for inspection.\n\n', workdir='/tmp/process-executionhjsTuh/.pants.d/tmp/tmpzcddf302.pants.d', pid=1409).exit_code
<https://github.com/pantsbuild/pants/runs/5122573691?check_suite_focus=true#step:14:594>
src/python/pants/testutil/pants_integration_test.py:53: AssertionError

This can be repo'ed on an Ubuntu machine like so:

PANTS_PYTHON_INTERPRETER_CONSTRAINTS="['==3.9.*']"  ./pants test  tests/python/pants_test/integration/pep561_integration_test.py

@witty-crayon-22786 @Eric-Arellano @enough-analyst-54434 pantsbuild/pants

#19126 Change options declarations to use subclasses New discussion created by thejcannon I want to fix #18856, which has a secondary benefit of making rule inputs more fine-grained. The only way to accomplish that is by converting options declarations to be types, so that they are valid annotations. As a strawman, that'd look something like:

class GlobalOptions(BootstrapOptions, Subsystem):
    options_scope = GLOBAL_SCOPE
    help = "Options to control the overall behavior of Pants."

    @bool_option
    class Colors:
        default = sys.stdout.isatty()
        help = softwrap(
            """
            Whether Pants should use colors in output or not. This may also impact whether
            some tools Pants runs use color.

            When unset, this value defaults based on whether the output destination supports color.
            """
        )

which was previously:

colors = BoolOption(
        default=sys.stdout.isatty(),
        help=softwrap(
            """
            Whether Pants should use colors in output or not. This may also impact whether
            some tools Pants runs use color.

            When unset, this value defaults based on whether the output destination supports color.
            """
        ),
    )

Then rule code needing the option would use:

@rule
def my_rule(colors_container: GlobalOptions.Colors) -> ...:
    colors_container.value  # known to be type `bool`
    # And we'd also have `colors_container.is_default_value` of type `bool`
    #    instead of `subsystem.options.is_default("<name>")`

This would extend to environment-aware options as well (although the details are fuzzy to me) pantsbuild/pants

#19131 Upgrade default nodejs distribution to current LTS Issue created by tobni Is your feature request related to a problem? Please describe. Pants should default to a currently supported LTS version of nodejs. Describe the solution you'd like Bump the default to either 18 or 20. I lean 18 because that is current defaults by cloud providers 🤷‍♂️ Describe alternatives you've considered None Additional context nodejs 16 stops receiving security updates in 3 months https://endoflife.date/nodejs, pants should not default to use EOL releases. pantsbuild/pants

#19132 Golang: Missing packages when compiling binary in docker environment Issue created by hardishw Describe the bug When compiling go binary in a docker environment, some packages are not provided when the OS differs between the host and docker environment, despite existing in the go.mod. See below.

[ERROR] Completed: Compile with Go - (environment:linux_docker, <http://google.golang.org/grpc/internal/syscall|google.golang.org/grpc/internal/syscall>) - <http://google.golang.org/grpc/internal/syscall|google.golang.org/grpc/internal/syscall> failed (exit code 1).
gopath/pkg/mod/google.golang.org/grpc@v1.55.0/internal/syscall/syscall_linux.go:29:2: could not import <http://golang.org/x/sys/unix|golang.org/x/sys/unix> (open : no such file or directory)

Pants version 2.16.0rc0 OS MacOS pantsbuild/pants

#18488 Terraform 1.0.7 (the default) file download length is incorrect on macOS_x86_64 Issue created by ndellosa95 Describe the bug Title. Error message:

10:08:19.28 [ERROR] 1 Exception encountered:

Engine traceback:
  in `fmt` goal
  in Format with `terraform fmt`

Exception: Error hashing/capturing URL fetch response: Downloaded file was larger than expected digest

Pants version 2.15 OS Intel MacOS Additional info Probably just need to rerun

build-support/bin/terraform_tool_versions.py

pantsbuild/pants

#17615 NpxToolBase should support lockfiles Issue created by sureshjoshi From: #17557 NpxToolBase supports changing versions, but needs lockfile support. Quoting myself:

npx is a bit weird (as is npm exec in my opinion).

It maintains its own cache and looks for local node_modules wherever it can. So, npx cowsay will ask to install 1.5.0 - but if I want to lock that to using a lockfile, I need to do npm install cowsay (generates package-lock.json) and then npx cowsay will use that local version in the node_modules.

So, lockfiles == good, but a bit of work here, unless there is an npx input to use a lockfile directly - but I just ran some tests that didn't work, nor can I find any docs on this subject right now.

pantsbuild/pants

#13066 remote-apis-testing fails with Pants v2.7.0 and Buildgrid Issue created by tdyas The remote-apis-testing project fails when I update it to use Pants v2.7.0 (via a version of example-python with Pants v2.7.0):

client_1  |   ProcessExecutionFailure: Process 'Generating Python sources from helloworld/util/proto/config.proto.' failed with exit code 127.
client_1  | stdout:
client_1  |
client_1  | stderr:
client_1  | 2021-10-01T18:22:10.461+0200 [455:140532379025216] [buildboxcommon_systemutils.cpp:65] [ERROR] Error while calling `execv(./bin/protoc --python_out _generated_files --plugin=protoc-gen-mypy=./mypy_protobuf.pex_bin_protoc-gen-mypy_shim.sh --mypy_out _generated_files helloworld/util/proto/config.proto )`: No such file or directory
client_1  | ./bin/protoc: No such file or directory

pantsbuild/pants

#19145 What is the best method to install PyTorch? New discussion created by minato-ellie I have tried using the

index

and

find_links

options in the

[python-repos]

section of the pants.toml file, but they prevent the generation of the lock file. The build process gets stuck indefinitely and eventually times out. As a workaround, I decided not to specify a CUDA version, and PyTorch installation was successful. It worked.However, I am unable to utilize the 'export' and 'package' goals.. What is the best method to install PyTorch in Pants and export the virtual environment for IDE usage (in my case, PyCharm)? Is this even possible, and if so, what would be the recommended approach? This is almost the final (yet most unavoidable) hurdle I face in migrating my ML project to Pants. pantsbuild/pants

#19147 Exporting only the used 3rd party dependencies of a given target New discussion created by james-hollingsworth Hi, We have a Python monorepo and use a single requirements.txt in order to prevent version drift between projects (I mention this as I realise using multiple lock files/resolves would be one way to solve what I'm asking..) We would like to create a Python virtual env for one of / each of our monorepo projects/targets in order to carry out certain testing / debugging locally with confidence the dependencies used match those of the deployed target. Ideally we would like a way to include a few additional dev tool packages in the virtual env required for testing such as pytest, behave etc. I've found a way to do it with the

export

goal, but a warning message states my approach will be deprecated. Is there another approach I can use to achieve the same thing while keeping the single lock file? Or.. :) .. any chance the feature can not become deprecated? Thanks, James Some more detail: We can create a virtual env containing only the 3rd party dependencies needed by the target by doing e.g.:

pants export projects/myproject:

Additionally we can get additional dev tool packages to be included in that env by specifying them as source dependencies in the project's

BUILD

, e.g.:

python_sources(
    name="myproject",
    dependencies=["//:root#pytest", "//:root#behave"],
)

However when I run pants export I see this warning: [WARN] DEPRECATED: exporting resolves without using the --resolve option is scheduled to be removed in version 2.23.0.dev0. If instead we export using resolves, then the virtual env contains the dependencies of all projects which is not what we want. Also, if we keep the tooling dependencies in a separate resolve - they end up in a separate virtual env which is not so good for IDE integration (though I note this is covered in another discussion). pantsbuild/pants

#19156 TypeError when using macro in root BUILD file Issue created by thejcannon Describe the bug Given `BUILD.pants`:

docker_environment(
  name="focal-docker",
  image=wo_get_pinned_name("ubuntu:focal")
)

and macro:

def wo_get_pinned_name(x):
    return x

Pants version 2.16.0rc2 OS Linux Additional info

Exception caught: (pants.engine.internals.scheduler.ExecutionError)
  File "/home/josh/.cache/nce/ae553d873185fa750061b1a3d4226442fb99fb55d4403b0227ec972406554da3/bindings/venvs/2.16.0rc2/bin/pants", line 8, in <module>
    sys.exit(main())
  File "/home/josh/.cache/nce/ae553d873185fa750061b1a3d4226442fb99fb55d4403b0227ec972406554da3/bindings/venvs/2.16.0rc2/lib/python3.9/site-packages/pants/bin/pants_loader.py", line 123, in main
    PantsLoader.main()
  File "/home/josh/.cache/nce/ae553d873185fa750061b1a3d4226442fb99fb55d4403b0227ec972406554da3/bindings/venvs/2.16.0rc2/lib/python3.9/site-packages/pants/bin/pants_loader.py", line 110, in main
    cls.run_default_entrypoint()
  File "/home/josh/.cache/nce/ae553d873185fa750061b1a3d4226442fb99fb55d4403b0227ec972406554da3/bindings/venvs/2.16.0rc2/lib/python3.9/site-packages/pants/bin/pants_loader.py", line 92, in run_default_entrypoint
    exit_code = runner.run(start_time)
  File "/home/josh/.cache/nce/ae553d873185fa750061b1a3d4226442fb99fb55d4403b0227ec972406554da3/bindings/venvs/2.16.0rc2/lib/python3.9/site-packages/pants/bin/pants_runner.py", line 99, in run
    runner = LocalPantsRunner.create(
  File "/home/josh/.cache/nce/ae553d873185fa750061b1a3d4226442fb99fb55d4403b0227ec972406554da3/bindings/venvs/2.16.0rc2/lib/python3.9/site-packages/pants/bin/local_pants_runner.py", line 90, in create
    build_config = options_initializer.build_config(options_bootstrapper, env)
  File "/home/josh/.cache/nce/ae553d873185fa750061b1a3d4226442fb99fb55d4403b0227ec972406554da3/bindings/venvs/2.16.0rc2/lib/python3.9/site-packages/pants/init/options_initializer.py", line 111, in build_config
    return _initialize_build_configuration(self._plugin_resolver, options_bootstrapper, env)
  File "/home/josh/.cache/nce/ae553d873185fa750061b1a3d4226442fb99fb55d4403b0227ec972406554da3/bindings/venvs/2.16.0rc2/lib/python3.9/site-packages/pants/init/options_initializer.py", line 48, in _initialize_build_configuration
    working_set = plugin_resolver.resolve(options_bootstrapper, env)
  File "/home/josh/.cache/nce/ae553d873185fa750061b1a3d4226442fb99fb55d4403b0227ec972406554da3/bindings/venvs/2.16.0rc2/lib/python3.9/site-packages/pants/init/plugin_resolver.py", line 137, in resolve
    for resolved_plugin_location in self._resolve_plugins(
  File "/home/josh/.cache/nce/ae553d873185fa750061b1a3d4226442fb99fb55d4403b0227ec972406554da3/bindings/venvs/2.16.0rc2/lib/python3.9/site-packages/pants/init/plugin_resolver.py", line 161, in _resolve_plugins
    params = Params(request, determine_bootstrap_environment(session))
  File "/home/josh/.cache/nce/ae553d873185fa750061b1a3d4226442fb99fb55d4403b0227ec972406554da3/bindings/venvs/2.16.0rc2/lib/python3.9/site-packages/pants/core/util_rules/environments.py", line 443, in determine_bootstrap_environment
    session.product_request(ChosenLocalEnvironmentName, [Params()])[0],
  File "/home/josh/.cache/nce/ae553d873185fa750061b1a3d4226442fb99fb55d4403b0227ec972406554da3/bindings/venvs/2.16.0rc2/lib/python3.9/site-packages/pants/engine/internals/scheduler.py", line 569, in product_request
    return self.execute(request)
  File "/home/josh/.cache/nce/ae553d873185fa750061b1a3d4226442fb99fb55d4403b0227ec972406554da3/bindings/venvs/2.16.0rc2/lib/python3.9/site-packages/pants/engine/internals/scheduler.py", line 513, in execute
    self._raise_on_error([t for _, t in throws])
  File "/home/josh/.cache/nce/ae553d873185fa750061b1a3d4226442fb99fb55d4403b0227ec972406554da3/bindings/venvs/2.16.0rc2/lib/python3.9/site-packages/pants/engine/internals/scheduler.py", line 497, in _raise_on_error
    raise ExecutionError(

Exception message: 1 Exception encountered:

MappingError: Failed to parse ./BUILD.pants:
TypeError: _unrecognized_symbol_func() takes 0 positional arguments but 1 was given

pantsbuild/pants

#19157 `CompletePexEnvironment.environment_dict` implictly disallows `PATH` in `subprocess-environment` Issue created by thejcannon Describe the bug Using https://www.pantsbuild.org/v2.16/docs/reference-subprocess-environment#env_vars

[subprocess-environment]
env_vars.add = ["PATH=.localbinaries/bin"]

you'll get:

File "/home/josh/.cache/nce/ae553d873185fa750061b1a3d4226442fb99fb55d4403b0227ec972406554da3/bindings/venvs/2.16.0rc2/lib/python3.9/site-packages/pants/backend/python/util_rules/pex_cli.py", line 186, in setup_pex_cli_process
    **complete_pex_env.environment_dict(python=bootstrap_python),
  File "/home/josh/.cache/nce/ae553d873185fa750061b1a3d4226442fb99fb55d4403b0227ec972406554da3/bindings/venvs/2.16.0rc2/lib/python3.9/site-packages/pants/backend/python/util_rules/pex_environment.py", line 222, in environment_dict
    d = dict(
TypeError: dict() got multiple values for keyword argument 'PATH'

Pants version

2.16.0rc2

OS Linux Additional info N/A pantsbuild/pants

#19158 `__defaults__` silently allows-and-ignores fallthrough kwargs Issue created by thejcannon Describe the bug I was trying to do:

__defaults__(python_test=dict(environment="focal-docker"))

and getting frustrated it wasn't working. Turns out it just gets collected and ignored: pants/src/python/pants/engine/internals/defaults.py Line 117 in </pantsbuild/pants/commit/3ce90948d2dc49dd525ca4a10daeaba98068a8e0|3ce9094> Pants version

2.16.0rc2

OS N/A Additional info N/A pantsbuild/pants

#19159 `__defaults__` isn't applied to generated targets Issue created by thejcannon Describe the bug Given:

__defaults__({python_test:dict(tags=["focal-docker"])})

in the root

BUILD

, then somewhere else:

python_tests(name="tests")

then

pants peek <some_test>

shows an empty

tags

😢 Pants version

main

and

2.16.0rc2

OS N/A Additional info N/A pantsbuild/pants

#19160 `BinaryNotFoundError` complaining about Python, even though `pants.backend.python.providers.experimental.pyenv` is enabled Issue created by thejcannon Describe the bug • Enable

pants.backend.python.providers.experimental.pyenv

• Use a

docker_environment

with a Python that isn't Py3, like

ubuntu:bionic

You'll see the error. This is likely because

get_pyenv_install_info

is using

PythonBinary

, when it should likely be

PythonBuildStandaloneBinary

Pants version

2.16.0rc2

OS (Docker linux) Additional info :/ pantsbuild/pants

#12430 The `InvalidEntryPoint` should subclass `InvalidFieldException` Issue created by kaos That sounds good to have

InvalidEntryPoint

subclass

InvalidFieldException

, definitely. And makes sense if you want to do in a followup to keep this PR better scoped. _Originally posted by @Eric-Arellano in #12414 (comment)_ pantsbuild/pants

#19170 Unexpected invalidation of all paths hindering pytest runs due to excessive retries by filesystem changes Issue created by achimnol Describe the bug When I run

pants test :: -- -m 'not integration'

on a checkout of https://github.com/lablup/backend.ai on macOS, pants detects filesystem changes too frequently and restarts many pytest processes in the middle, causing resource occupation issues like overlapping of TCP port numbers calculated from execution slots, file-based locks, and etc. You could try by yourself by just cloning the repo and run the above command. No long installation is required because the test suite spawns their own database containers with fixtures locally using Docker and Docker Compose. Just ensure that

pyenv install 3.11.3 && mkdir .tmp

and have Docker Compose v2 (either Docker Desktop or OrbStack). Pants version 2.16.0rc4 (installed via scie-pants) OS It happens on macOS Ventura 13.4 only. I'm not sure about the previous macOS versions, but potentially some internal filesystem changes in recent updates like this 107550080 issue might be related somehow (this particular issue has blocked double-click opening of most Korean-named files in 13.3's Finder!). I have an Ubuntu 22.04 aarch64 VM on this Mac, but there is no symptoms like this with the same repository setup. Additional info An excerpt from `.pants.d/pants.log`:

20:25:48.16 [INFO] handling request: `--python-repos-find-links=-['file:///Users/joongi/Library/Caches/nce/142382d6e3ac0960e7d4f81b9df9de9559b7ac698db13d20cbd0e4aa57f68c1d/bindings/find_links/2.16.0rc4/920d0b88/index.html']
 test :: -- -m not integration`
20:26:03.15 [INFO] notify invalidation: cleared 0 and dirtied 0 nodes for all paths
20:26:03.21 [INFO] notify invalidation: cleared 7766 and dirtied 27565 nodes for all paths
20:26:03.34 [INFO] notify invalidation: cleared 0 and dirtied 0 nodes for all paths
20:26:03.36 [INFO] notify invalidation: cleared 4 and dirtied 27565 nodes for all paths
20:26:03.36 [INFO] notify invalidation: cleared 0 and dirtied 0 nodes for all paths
20:26:03.38 [INFO] notify invalidation: cleared 4 and dirtied 27565 nodes for all paths
20:26:08.07 [INFO] notify invalidation: cleared 0 and dirtied 0 nodes for all paths
20:26:08.08 [INFO] notify invalidation: cleared 7698 and dirtied 27565 nodes for all paths
20:26:08.16 [INFO] notify invalidation: cleared 0 and dirtied 0 nodes for all paths
20:26:08.16 [INFO] notify invalidation: cleared 0 and dirtied 0 nodes for all paths
20:26:08.25 [INFO] notify invalidation: cleared 4 and dirtied 27565 nodes for all paths
20:26:09.00 [INFO] Extending leases
20:26:09.07 [INFO] notify invalidation: cleared 0 and dirtied 0 nodes for all paths
20:26:09.08 [INFO] notify invalidation: cleared 0 and dirtied 0 nodes for all paths
20:26:09.26 [INFO] notify invalidation: cleared 7698 and dirtied 27565 nodes for all paths
20:26:09.26 [INFO] notify invalidation: cleared 0 and dirtied 0 nodes for all paths
20:26:09.34 [INFO] notify invalidation: cleared 0 and dirtied 0 nodes for all paths
20:26:09.34 [INFO] notify invalidation: cleared 0 and dirtied 0 nodes for all paths
20:26:09.36 [INFO] Done extending leases
20:26:09.39 [INFO] notify invalidation: cleared 0 and dirtied 0 nodes for all paths
20:26:09.40 [INFO] notify invalidation: cleared 0 and dirtied 0 nodes for all paths

pantsbuild/pants

#19171 Pyenv test failing over directory already existing Issue created by benjyw On my macbook, running

./pants test src/python/pants/backend/python/providers/pyenv/rules_integration_test.py

consistently errors with the underlying issue being:

E       pants.engine.process.ProcessExecutionFailure: Process 'Install Python 3.9' failed with exit code 1.
E       stdout:
E       
E       stderr:
E       pyenv: /Users/benjyw/.cache/pants/named_caches/pyenv/44136fa355b3678a1146ad16f7e8649e94fb4fc21fe77e8310c060f61caaff8a/versions/3.9.16 already exists
E       Traceback (most recent call last):
E         File "/private/tmp/pants-sandbox-TjtzIT/install_python_shim.py", line 29, in <module>
E           main()
E         File "/private/tmp/pants-sandbox-TjtzIT/install_python_shim.py", line 21, in main
E           subprocess.run(["./pyenv-2.3.13/bin/pyenv", "install", SPECIFIC_VERSION], check=True)
E         File "/Users/benjyw/.pyenv/versions/3.10.6/lib/python3.10/subprocess.py", line 524, in run
E           raise CalledProcessError(retcode, process.args,
E       subprocess.CalledProcessError: Command '['./pyenv-2.3.13/bin/pyenv', 'install', '3.9.16']' returned non-zero exit status 1.

The key line is

/Users/benjyw/.cache/pants/named_caches/pyenv/44136fa355b3678a1146ad16f7e8649e94fb4fc21fe77e8310c060f61caaff8a/versions/3.9.16 already exists

- which is true! In fact it has to be true since we create the DONE file and its lock file in there. In that dir there is a DONE.lock but no DONE. SO I guess a previous pyenv install failed and left things in an intermediate state. Probably we need to keep the DONE and lockfile outside that dir, so we can clean it prior to attempting to pyenv-install into it? pantsbuild/pants

#16261 Deprecate and remove run `--no-cleanup`? Issue created by thejcannon Now that

run

respects

--no-process-cleanup

, Interactive processes use normal-looking sandboxes (

/tmp/pants-sandbox...

), and that

test --debug

should ideally respect

--no-process-cleanup

(#16260 ).

--no-cleanup

feels like it doesn't really belong because it isn't adding much. Alternatively I think this and #16260 could be fixed with a

--no-interactive-process-cleanup

option if we wanted to give the user further control (but I would also think the ship has sailed since

run

respects

--no-process-cleanup

for the interactive process 🤷‍♂️ ) pantsbuild/pants

#11187 Add shell completion for Pants 2.0+ Issue created by chiefnoah Pants 1.x had bash completion, but it was lost with the 2.0 rewrite. This is an extremely helpful feature and it would be nice if it was added back in. pantsbuild/pants

#19182 CI: collect/aggregate results to highlight flakiest tests Issue created by huonw Is your feature request related to a problem? Please describe. Pants' CI is quite flaky at the moment, with a lot of failures due to exceeding timeouts, and some other types of flakes too. I don't think there's a currently a good way to get insight into what's flaky other than experiencing them for yourself (and/or clicking through a lot of builds), and so people might do spot fixes, but it's not easy to get a systematic view. Describe the solution you'd like Some sort of 'dashboard' (even an adhoc/on-demand one) summarising the most common test timeouts and failures over the past month (or some other threshold). Describe alternatives you've considered None Additional context N/A pantsbuild/pants