#9399 HTTP 429 too many requests for Pex PEX Issue created by jsirois Seen in TravisCI "Build OSX wheels (Python 3.6)" shard. Looks like:

curl: (22) The requested URL returned error: 429 too many requests
Curl failed with args: -s <https://github.com/pantsbuild/pex/releases/download/v2.1.6/pex>

We may need to investigate whether we can bump rate limits with GitHub, host in some other manner or correct some endemic problem if we have one with asking for the binary too often. pantsbuild/pants

#19183 python_requirement not working in a plugin Issue created by idan-at Describe the bug I am writing a custom pants plugin, and I fail to import 3rd party dependencies. Here is my setup:

python_sources(
  name="sources",
  sources=["**/*.py"],
  dependencies=[":requirements", ":pants_requirements"]
)

python_requirement(
  name="requirements",
  requirements=[
    "GitPython==3.1.12",
    "semver==2.13.0",
  ]
)

pants_requirements(
  name="pants_requirements",
)

Pants version 2.15.0 OS MacOS Additional info I haven't found any pants plugin that consumes 3rd party dependencies. Is that supported? Thanks! pantsbuild/pants

#2155 Non-deterministic output due to non-deterministic os.walk and unix file permissions of original files Issue created by ento Context I'm using Pants to package up a GitHub Actions action I started writing in Python. The packaged pex file is checked in to the repo, so that when the action gets run, all dependencies get pulled down as part of the action itself, similar to how actions/typescript-action uses ncc. Problem and current workaround It appeared to work fine until I added a CI check that builds a pex file in the CI environment and verifies the checked in pex file is identical to the one that just got built: even though none of the source files nor dependencies changed, the check didn't pass. I tracked down the causes and was able to use a forked pex repo with a few changes to make the check pass: main...ento:pex:deterministic-bootstrap 1.

os.walk

was yielding directories in a non-deterministic order, apparently. Fixed by applying

.sort()

in a few places. • I wrote a few tests that reproduces this issue: main...ento:pex:deterministic-os-walk 2. My local environment and the CI environment had different umask values, and because pex preserves filesystem permissions, the resulting zip archive ended up being different too. Fixed by applying something like a umask when adding entries to the zip archive. • Although this worked in my case, I don't think the way I did it really guarantees reproducibility: umask at the OS level works fine when it comes to reproducibility because the default permission is something constant when a new filesystem entry is created (although I don't know if that's the same across all (POSIX-compliant?) OS'es), but in the case of pex, there's no default permission per se and it uses the permission of the entry on disk as the starting point. Questions I can work on opening a PR or two to address these sources of non-determinism if that's something good to incorporate into pex. 1. For

os.walk

, I'm thinking of adding a wrapper around

os.walk

like

deterministic_os_walk

and using it at least in the two places I needed to patch. Would it be a good idea to use it everywhere, or should it be limited to just these two places? Or is this something out of scope of guarantees that pex is willing to provide? 2. Is providing deterministic builds when it comes to unix file permissions something within scope of pex? If so, what would be a good approach? • I did think of specifying the umask in the CI environment and not handling it within pex when I was trying to come up with a fix, but enforcing the umask value in local dev environments is going to be hard, unless all contributors' machines are set up the same way, as the value would need to be set before you clone the GitHub actions repo. • I noticed there's a

--use-system-time

flag for allowing non-deterministic timestamps. With file permissions, we could similarly have a non-deterministic mode and deterministic mode, where deterministic mode creates new zipfile entries with

u=rw,g=rw,o=rw

for files and

u=rwx,g=rwx,o=rwx

for directories and with a configurable umask applied, but.. that will mean files that were originally executable will lose that bit. The determinisitc mode could, instead, limit what it controls to

r

and

w

permissions and let

x

pass through, which might work good enough in practice. pantsbuild/pex

#19186 `protobuf_sources` interacts badly with multiple Python resolves Issue created by tgolsson Describe the bug I found two issues while trying to get a few protobuf sources to be part of multiple resolves. I'm not sure if it's a setup-specific thing, haven't set up a full repro yet. The base setup is the following: We have a set of resolves that apply to the exact same set of code. To solve this we have the following snippet in the root:

__defaults__(
    {
        python_source: dict(resolve=parametrize("cpu", "gpu", "reqs")),
        python_sources: dict(resolve=parametrize("cpu", "gpu", "reqs")),
        pex_binary: dict(resolve=parametrize("cpu", "gpu", "reqs")),
    }
)

However; parametrizing

protobuf_sources.python_resolve

here fails with the following error: pants/src/python/pants/engine/internals/graph.py Lines 437 to 441 in </pantsbuild/pants/commit/a34feabef5400eb52472c34900ed39438dfa1d55|a34feab> Then; I tried to simply generate all the variants in a loop:

for resolve in ['reqs', 'cpu', 'gpu']:
    protobuf_sources(
        name=f'proto_{resolve}',
        python_resolve=resolve,
        ...,
    )

However; we have two files in our set:

foo_v2

, which imports

foo_v1

. Thus, this approach to ambiguity of which

foo_v1.proto

it would be importing. Thus, I had to expand to two

protobuf_source

statements with explicit dependencies:

for resolve in ['reqs', 'cpu', 'gpu']:
    protobuf_source(
        name=f"foo_v1_{resolve}",
        source="foo_v1.proto",
        grpc=True,
        python_resolve=resolve,
    )

    protobuf_source(
        name=f"foo_v2_{resolve}",
        source="foo_v2.proto",
        grpc=True,
        python_resolve=resolve,
        dependencies=[f":foo_v1_{resolve}"],
    )

Pants version Stable, 2.15.0 OS Linux Additional info Will try to isolate a repro soon! pantsbuild/pants

#13288 include path to BUILD file where an invalid target name was used Issue created by tdyas This BUILD file snippet:

java_sources(
    dependencies=[
        "//core:lockfile"
        "//core:com.google.guava_guava",
    ],
)

led to this error due to the missing command causing the strings to be concatenated:

InvalidTargetName: The target name lockfile//core:com.google.guava_guava (defined in directory core) contains banned characters (`:`,`/`). Please replace these characters with another separator character like `_` or `-`.

The error would have been more actionable if it had included the path to the BUILD file where the target name had been referenced. Happens with Pants

main

branch as of today. pantsbuild/pants

#17660 Report all dependency errors for all targets at once in visibility backend Issue created by danxmoran Is your feature request related to a problem? Please describe. #17612 improved the reporting of dependency rule errors, so that all errors for a single target will be reported. However if multiple targets have errors, the system will still only report errors for one target (and after you fix those and re-run, it'll report the errors for the next target, and so on...). This one-at-a-time approach is frustrating to developers - we've specifically heard it about other linters/formatters in the past, and I expect to receive the same feedback if we adopt the visibility backend with its current behavior. Describe the solution you'd like It would be ideal for the dependency-rule system to report all errors for all targets. Describe alternatives you've considered Keep using

import-linter

instead of the visibility backend 🤢 Additional context AIUI, this would require changing the API of the dependency-rule system. Currently the core dependency resolution rule requests

ValidatedDependencies

but throws away the result here: pants/src/python/pants/engine/internals/graph.py Lines 1304 to 1315 in </pantsbuild/pants/commit/a4054868c11690874005999cb80100e4da107538|a405486> So implementations of

ValidateDependenciesRequest

->

ValidatedDependencies

have no choice but to raise an exception on a dependency error, and the first error raised is the one that gets reported. IMO a better API would always return a value - conceptually something like Rust's

Result

, though I'm not sure how best to model that in Python. The core logic could then loop over all the results, log errors, and raise some error to bail out if needed. pantsbuild/pants

#2156 Pex stripping querystring from artifact downloads Issue created by jake-sigtech Pex strips querystring from artifact URL code We are using pypicloud for a private python package index. This is backed by S3 and returns package links as s3 pre-signed urls, which contain querystrings for authentication with s3. Hence the artifact download fails, when the querystrings are stripped (403 error). Would it be possible not to strip the querystring? The querystring changes each time, so would be best for caching if the url without the querystring is the one used for the cache. pantsbuild/pex

#14128 Improve output when pants-generated targets are used with goals that don't match any files Issue created by thejcannon Is your feature request related to a problem? Please describe. If you're used to running

./pants --changed-since=main fmt lint check test

(or have an alias or whatever) but don't touch any files that match

test

(just as an example). You'll see:

09:23:20.41 [WARN] No applicable files or targets matched. The `test` goal works with these target types:

  * go_package
  * junit_test
  * python_test
  * scala_junit_test
  * scalatest_test
  * shunit2_test

However, you only specified targets with these target types:

  * python_source

This could happen as well if you supply the "target" as a dir which doesn't contain tests as well (so not limited to

--changed-since

). Describe the solution you'd like In these cases, perhaps a no-op output (so maybe neither a green check or a red X). Or the ability to configure what happens. Describe alternatives you've considered N/A Additional context N/A pantsbuild/pants

#15532 Heisenbug: color and dynamic UI turning off using pantsd Issue created by thejcannon Describe the bug Inexplicably sometimes when running Pants, I'll see I've lost color and dynamic UI. Everything else seems A-OK. Pants version

main

as of 2.13.devX OS Linux Additional info I use VS Code terminal pantsbuild/pants

#19194 Modernize our cherry-picking workflow/scripts Issue created by thejcannon With the recent shift of cherry-picking workflow to GitHub Actions, we should change our scripts/docs to lean into the new workflow. Specifically I think there's a few cherry-picking instances: Instances: • Cherry-picking done by GHA • This is most likely triggered by merging a "needs-cherry-pick" PR • But also can be triggered in other cases. Like if the milestone was missing, or someone decides after merging it should be picked • Cherry-picking done by people • Ideally, this should ONLY be done in the case of a merge conflict With those in mind, I suggest that we shift the logic of our scripts to focus on GHA, as that's the default and much more common case. For local picking, we can have a different/slimmer set of scripts more dedicated to the use-case. Additionally, I think it'd be excellent if the GHA picker: • Labeled PRs it failed to pick • Commented on them to notify the author • This comment can and should explain the steps require to fixing So in a world where we comment on PR, we may not need dedicated scripts for the user if the resolution steps are straightforward. pantsbuild/pants

#19199 Connecting to pantsd fails when any non-utf8 env variables are in the ambient environment. Issue created by jsirois There are two cases brought to light by fixing pantsbuild/scie-pants#198 via fixing a-scie/jump#105 and one is ~the same as that issue - using Rust's

std::env::vars()

which has docs that it panics instead of a signature / API that makes the fallibility manifest. So the only way to run Pants in an ambient env with non-utf8 entries currently is to use

--no-pantsd

(and turn off the native client when available with

PANTS_NO_NATIVE_CLIENT=1

). The 1st issue is in the native client here: pants/src/rust/engine/client/src/main.rs Line 88 in </pantsbuild/pants/commit/00a82f030fe625ee85c1c7a031743a6c6f927e9a|00a82f0> Errors looks like the error above in

scie-pants

/ `scie-jump`:

$ env PANTS_VERSION=2.17.0a1 $'FOO=B\xa5R' ./scie-pants --version
thread 'main' panicked at 'called `Result::unwrap()` on an `Err` value: "B\xA5R"', library/std/src/env.rs:171:83

The 2nd issue is in the

PyNailgunClient

used when starting pantsd for the first time or when no native client is present or active: pants/src/rust/engine/src/externs/nailgun.rs Lines 44 to 51 in </pantsbuild/pants/commit/00a82f030fe625ee85c1c7a031743a6c6f927e9a|00a82f0> The error looks like:

$ env $'FOO=B\xa5R' ./scie-pants --version
Traceback (most recent call last):
  File "/home/jsirois/.cache/nce/ff338ba9a8dd5abb25f774b6bf232ee14f99d98720d9c34bb15042fbe70eeb70/bindings/venvs/2.15.0/bin/pants", line 8, in <module>
    sys.exit(main())
  File "/home/jsirois/.cache/nce/ff338ba9a8dd5abb25f774b6bf232ee14f99d98720d9c34bb15042fbe70eeb70/bindings/venvs/2.15.0/lib/python3.9/site-packages/pants/bin/pants_loader.py", line 123, in main
    PantsLoader.main()
  File "/home/jsirois/.cache/nce/ff338ba9a8dd5abb25f774b6bf232ee14f99d98720d9c34bb15042fbe70eeb70/bindings/venvs/2.15.0/lib/python3.9/site-packages/pants/bin/pants_loader.py", line 110, in main
    cls.run_default_entrypoint()
  File "/home/jsirois/.cache/nce/ff338ba9a8dd5abb25f774b6bf232ee14f99d98720d9c34bb15042fbe70eeb70/bindings/venvs/2.15.0/lib/python3.9/site-packages/pants/bin/pants_loader.py", line 92, in run_default_entrypoint
    exit_code = runner.run(start_time)
  File "/home/jsirois/.cache/nce/ff338ba9a8dd5abb25f774b6bf232ee14f99d98720d9c34bb15042fbe70eeb70/bindings/venvs/2.15.0/lib/python3.9/site-packages/pants/bin/pants_runner.py", line 89, in run
    return remote_runner.run(start_time)
  File "/home/jsirois/.cache/nce/ff338ba9a8dd5abb25f774b6bf232ee14f99d98720d9c34bb15042fbe70eeb70/bindings/venvs/2.15.0/lib/python3.9/site-packages/pants/bin/remote_pants_runner.py", line 123, in run
    return self._connect_and_execute(pantsd_handle, executor, start_time)
  File "/home/jsirois/.cache/nce/ff338ba9a8dd5abb25f774b6bf232ee14f99d98720d9c34bb15042fbe70eeb70/bindings/venvs/2.15.0/lib/python3.9/site-packages/pants/bin/remote_pants_runner.py", line 161, in _connect_and_execute
    return PyNailgunClient(port, executor).execute(command, args, modified_env)
UnicodeEncodeError: 'utf-8' codec can't encode character '\udca5' in position 1: surrogates not allowed

pantsbuild/pants

#7399 generate unstable package versions which can be ordered by timestamp in release.sh Issue created by cosmicexplorer CI on master was broken after merging 0672383, because when building wheels, our release script sets

PANTS_UNSTABLE_VERSION

to be the version in

src/python/pants/VERSION

, along with the first 8 characters of the commit sha. This should be fine, but not mentioned in https://www.python.org/dev/peps/pep-0440/#local-version-identifiers is that local build tags which consist of only numbers will have leading zeroes truncated, e.g.:

X> pex packaging setuptools
Python 2.7.15 (default, Jun 20 2018, 17:40:09)
[GCC 4.2.1 Compatible Apple LLVM 9.0.0 (clang-900.0.39.2)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
(InteractiveConsole)
>>> from packaging.version import Version
>>> Version('1.15.0.dev4+06723833')
<Version('1.15.0.dev4+6723833')>
>>> str(Version('1.15.0.dev4+06723833'))
'1.15.0.dev4+6723833'

This required a fix in #7400 by prefacing the sha with a constant non-numeric string to avoid it getting parsed as a number. While that was fixed, it might also be pretty useful to further modify the

PANTS_UNSTABLE_VERSION

, to add a timestamp before the SHA. This would allow

ls

to ensure the most recently produced wheel would always be at the bottom of the output. pantsbuild/pants

#19201 Any example repos that have successfully integrated Pants with Jenkins CI? New discussion created by AtrikDas I have read the article https://www.pantsbuild.org/docs/using-pants-in-ci but I would find it very helpful if anyone could share an example repo of how they have achieved integrating pants successfully with Jenkins CI. Specifically I am looking for examples in a monorepo where one Jenkinsfile at the root of the project folder is handling the CI pipeline for every project in the monorepo. I believe this is where Pants would shine because it is able to detect the files which contain any changes and only run the lint/tests on those projects instead of running the CI pipeline on all the projects in the monorepo. Any help on understanding this further is appreciated. pantsbuild/pants

#19212 Spec a developer hub for plugins etc Issue created by cczona A lot of projects have a central hub for contributed code, such as extensions. Things that aren't necessarily supported or endorsed by the project, but brings together in one place all the 3rd-party things that try to extend the featureset in some way. For Pants, this might be something as basic as a docs page for listing one's plugin. It could prove to be a way to surface surprises earlier, and help people collaborate on such 3rd party additions without necessarily leaning on the maintainers. pantsbuild/pants

#19219 Helm unittest subsystem version &gt;0.3.3 support Issue created by begemotik Is your feature request related to a problem? Please describe. It seems that the current project used for helm_unittests is no longer supported. Instead the maintainer proceeds the development in <https://github.com/helm-unittest/helm-unittest|the following project

.

Describe the solution you'd like Does it make sense to switch unittesting to the project which is actually maintained? Tried to do it within the configfile

[helm-unittest]
version = "0.3.3"
url_template = "<https://github.com/helm-unittest/helm-unittest/releases/download/v{version}/helm-unittest-{platform}-{version}.tgz>"
use_unsupported_version = "warning"
known_versions = [
  "0.3.3|linux_x86_64|8ebe20f77012a5d4e7139760cabe36dd1ea38e40b26f57de3f4165d96bd486ff|21685365",
  "0.3.3|linux_arm64 |7f5e4426428cb9678f971576103df410e6fa38dd19b87fce4729f5217bd5c683|19944514",
  "0.3.3|macos_x86_64|b2298a513b3cb6482ba2e42079c93ad18be8a31a230bd4dffdeb01ec2881d0f5|21497144",
  "0.3.3|macos_arm64 |2365f5b3a99e6fc83218457046378b14039a3992e9ae96a4192bc2e43a33c742|20479438"
]

Errors occured:

16:07:27.77 [ERROR] Completed: Run Helm Unittest - pupkins/tests/pup_test.yaml - failed (exit code 1).
unknown flag: --helm3

Error: unknown flag: --helm3

pantsbuild/pants

Comment on #19035 how to generate requirements.txt from BUILD file? Discussion answered by benjyw Interesting question. Typically there is already a requirements.txt from which the targets are generated. But it sounds like you have manual

python_requirement

targets or something like that? Anyway, in v2 you can do something like:

./pants peek 3rdparty/python/:: | jq -r ".[].requirements | select( . != null ) | flatten[]"

pantsbuild/pants

#19227 Document Pants uninstall Issue created by AIGeneratedUsername Is your feature request related to a problem? Please describe. Uninstalling is not documented. Describe the solution you'd like Update documentation: 1. describe how to uninstall Pants 2. describe how to clean up any Pants cache or config files after uninstall Describe alternatives you've considered I was searching issues and discussions for the "uninstall" word and I found nothing. pantsbuild/pants

#11269 Explore replacing monomorphization with implicit conversion Issue created by stuhood We currently "monomorphize" `@rule`s in the

RuleGraph

by creating a distinct copy of a

@rule

per set of

Param

types that it might be used with. This is quite complicated, and a lot of time has been invested in making it "mostly work"... but the implementation is not perfect, and we've needed to apply constraints that can be difficult to reason about. A motivation for monomorphization is that it avoids needing

eq

or

hash

implementations for positional arguments to

@rules

which are computed from `Param`s, rather than being `Param`s themselves. That is of questionable value relative to the complexity it incurs. And additionally, because monomorphization creates additional copies of

@rules

, it's likely that we are memoizing less than we could be. * * * An alternative to monomorphization would be to instead treat the preparation of the positional arguments to a

@rule

as similar to "implicit conversion" (a native concept in Scala, and possible via the

Deref

trait in Rust). This would involve optionally recursing to do "type conversion" via other rules (such as from

Addresses

to

Targets

) before invoking a

@rule

. For the purposes of these conversions, it's likely possible to keep a global lookup table of conversions that are possible, rather than actually encoding the conversion as a dependency of the

RuleGraph

entry. pantsbuild/pants

#19231 Run tracker failed to create directory for logs New discussion created by JunyuanZhangCoin Hi, I am getting this error when running pants. Looks like it failed to create the folder by calling

self.run_logs_file.parent.mkdir(exist_ok=True, parents=True)

but parents=True should work no matter the parents exist or not. I am confused.

docker exec -ti udas_webserver sh -c "./pants --print-stacktrace -ldebug run udas/src/python:launcher -- manage runserver 0.0.0.0:8888 --noreload"
20:50:13.38 [DEBUG] acquiring lock: <pants.pantsd.lock.OwnerPrintingInterProcessFileLock object at 0x7f791fcee970>
20:50:13.41 [DEBUG] terminating pantsd
20:50:13.41 [DEBUG] sending signal 15 to pid 151
20:50:13.62 [DEBUG] successfully terminated pid 151
20:50:13.62 [DEBUG] purging metadata directory: /opt/pynest/.pids/7fe3dfe26c48/pantsd
20:50:13.64 [DEBUG] Launching pantsd
20:50:13.64 [DEBUG] purging metadata directory: /opt/pynest/.pids/7fe3dfe26c48/pantsd
20:50:13.64 [DEBUG] pantsd command is: PANTS_DAEMON_ENTRYPOINT=pants.pantsd.pants_daemon:launch_new_pantsd_instance PYTHONPATH=/root/.cache/pants/setup/bootstrap-Linux-x86_64/pants.t2vWUx/install/bin:/usr/lib/python38.zip:/usr/lib/python3.8:/usr/lib/python3.8/lib-dynload:/root/.cache/pants/setup/bootstrap-Linux-x86_64/2.13.0_py38/lib/python3.8/site-packages /root/.cache/pants/setup/bootstrap-Linux-x86_64/2.13.0_py38/bin/python /root/.cache/pants/setup/bootstrap-Linux-x86_64/2.13.0_py38/bin/pants --pants-bin-name=./pants --pants-version=2.13.0 --print-stacktrace -ldebug run udas/src/python:launcher -- manage runserver 0.0.0.0:8888 --noreload
20:50:14.96 [DEBUG] pantsd is running at pid 612, pailgun port is 33789
20:50:14.96 [DEBUG] releasing lock: <pants.pantsd.lock.OwnerPrintingInterProcessFileLock object at 0x7f791fcee970>
20:50:14.96 [DEBUG] Connecting to pantsd on port 33789
20:50:14.96 [DEBUG] Connecting to pantsd on port 33789 attempt 1/3
20:50:14.97 [DEBUG] Connected to pantsd
20:50:14.99 [DEBUG] Launching 1 roots (poll=false).
20:50:15.03 [DEBUG] computed 1 nodes in 0.047116 seconds. there are 7 total nodes.
20:50:15.20 [INFO] Initializing scheduler...
20:50:15.21 [DEBUG] File handle limit is: 1048576
20:50:15.22 [DEBUG] Using [cache::CommandRunner { inner: bounded::CommandRunner { inner: nailgun::CommandRunner { inner: local::CommandRunner { .. }, .. }, .. }, .. }, cache::CommandRunner { inner: bounded::CommandRunner { inner: nailgun::CommandRunner { inner: local::CommandRunner { .. }, .. }, .. }, .. }] for process execution.
20:50:15.41 [DEBUG] Changes to /root/.cache/pants/setup/bootstrap-Linux-x86_64/pants.t2vWUx/install/bin, outside of the buildroot, will not be invalidated.
20:50:15.41 [DEBUG] Changes to /root/.cache/pants/setup/bootstrap-Linux-x86_64/pants.t2vWUx/install/bin, outside of the buildroot, will not be invalidated.
20:50:15.41 [DEBUG] Changes to /usr/lib/python38.zip, outside of the buildroot, will not be invalidated.
20:50:15.41 [DEBUG] Changes to /usr/lib/python3.8, outside of the buildroot, will not be invalidated.
20:50:15.41 [DEBUG] Changes to /usr/lib/python3.8/lib-dynload, outside of the buildroot, will not be invalidated.
20:50:15.41 [DEBUG] Changes to /root/.cache/pants/setup/bootstrap-Linux-x86_64/2.13.0_py38/lib/python3.8/site-packages, outside of the buildroot, will not be invalidated.
20:50:15.41 [DEBUG] setting up service <pants.pantsd.service.scheduler_service.SchedulerService object at 0x7f5c2af00fd0>
20:50:15.41 [DEBUG] setting up service <pants.pantsd.service.store_gc_service.StoreGCService object at 0x7f5c2af12850>
20:50:15.41 [DEBUG] starting service <pants.pantsd.service.scheduler_service.SchedulerService object at 0x7f5c2af00fd0>
20:50:15.41 [DEBUG] starting service <pants.pantsd.service.store_gc_service.StoreGCService object at 0x7f5c2af12850>
20:50:15.42 [INFO] Scheduler initialized.
20:50:15.42 [DEBUG] Launching 1 roots (poll=false).
20:50:15.42 [DEBUG] computed 1 nodes in 0.000391 seconds. there are 7 total nodes.
20:50:15.46 [ERROR] [Errno 2] No such file or directory: '/opt/pynest/.pants.d/run-tracker/pants_run_2023_06_02_20_50_15_463_684ee4e61f3c465db30f03ff385e5b42'
Traceback (most recent call last):
  File "/usr/lib/python3.8/pathlib.py", line 1288, in mkdir
    self._accessor.mkdir(self, mode)
FileNotFoundError: [Errno 2] No such file or directory: '/opt/pynest/.pants.d/run-tracker/pants_run_2023_06_02_20_50_15_463_684ee4e61f3c465db30f03ff385e5b42'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/root/.cache/pants/setup/bootstrap-Linux-x86_64/2.13.0_py38/lib/python3.8/site-packages/pants/bin/daemon_pants_runner.py", line 131, in single_daemonized_run
    runner = LocalPantsRunner.create(
  File "/root/.cache/pants/setup/bootstrap-Linux-x86_64/2.13.0_py38/lib/python3.8/site-packages/pants/bin/local_pants_runner.py", line 130, in create
    run_tracker = RunTracker(options_bootstrapper.args, options)
  File "/root/.cache/pants/setup/bootstrap-Linux-x86_64/2.13.0_py38/lib/python3.8/site-packages/pants/goal/run_tracker.py", line 95, in __init__
    self.run_logs_file.parent.mkdir(exist_ok=True, parents=True)
  File "/usr/lib/python3.8/pathlib.py", line 1293, in mkdir
    self.mkdir(mode, parents=False, exist_ok=exist_ok)
  File "/usr/lib/python3.8/pathlib.py", line 1288, in mkdir
    self._accessor.mkdir(self, mode)
FileNotFoundError: [Errno 2] No such file or directory: '/opt/pynest/.pants.d/run-tracker/pants_run_2023_06_02_20_50_15_463_684ee4e61f3c465db30f03ff385e5b42'


See <https://www.pantsbuild.org/v2.13/docs/troubleshooting> for common issues.
Consider reaching out for help: <https://www.pantsbuild.org/v2.13/docs/getting-help>

pantsbuild/pants

#19235 Integrity files collide when generating node modules Issue created by tobni Describe the bug Newer versions of package managers generate integrity files (e.g

.package-lock.json

, notice the leading

.

). The current strategy of merging node_modules installations across workspaces will collide in these integrity files, because the digests were generated from subsets, and but the integrity file is in the same location on disk. Pants version Which version of Pants are you using? OS Are you encountering the bug on MacOS, Linux, or both? Additional info pantsbuild/pants

#19236 Install entire node project when creating node_modules digest, subset in sandboxes Issue created by tobni Is your feature request related to a problem? Please describe. Currently, node_modules installations are crafted from installing the "owning" workspace, and all its deps seperately, followed by a digest merge. This is problematic, especially considering #19235. Even more so is the fact that the total runtime of pants not benefit from the "incremental" approach, as node_modules is resolved at the project level by the package manager, not per workspace. Describe the solution you'd like Install entire project once. Describe alternatives you've considered Additional context pantsbuild/pants

#19237 Port javascript dep inference to Rust Issue created by tobni Is your feature request related to a problem? Please describe. Dep inference currently runs in node. It is very slow, and installing babel bloats the cache. Describe the solution you'd like Rewrite it in rust. Describe alternatives you've considered Keep as is. Additional context pantsbuild/pants

#19238 Support test report generation in `node_build_script` Issue created by tobni Is your feature request related to a problem? Please describe. Add some kind of support for capturing test reports generated by a node test runner. Describe the solution you'd like Extend

node_test_script

with configuration to capture test reports Describe alternatives you've considered A clear and concise description of any alternative solutions or features you've considered. Additional context Add any other context or screenshots about the feature request here. pantsbuild/pants

#19239 Implement support for `pants export` exporting `node_modules` Issue created by tobni Is your feature request related to a problem? Please describe. In order to satisfy IDEs,

node_modules

need to be accessible Describe the solution you'd like Mirroring the python backend, the JS backend should support a

pants export

that can produce node_modules, either in the repo tree or to some other location- Describe alternatives you've considered A clear and concise description of any alternative solutions or features you've considered. Additional context Add any other context or screenshots about the feature request here. pantsbuild/pants

#19240 Stabilization Tracking: Javascript Issue created by tobni This is a stabilization tracking ticket for the Javascript backend. Items here are intended to be resolved in order to remove the "experimental" label from Javascript support: For stabilization ☐ #19236 ☐ #19235 ☑︎ #19237 ☐ #19238 ☐ #19239 ☐ #19241 ☐ #19242 ...This list is currently not exhaustive. pantsbuild/pants

#19241 Write documentation for javascript backend Issue created by tobni Is your feature request related to a problem? Please describe. Describe the solution you'd like Describe alternatives you've considered Additional context pantsbuild/pants

#19242 Create example repositories leveraging javascript backend Issue created by tobni Is your feature request related to a problem? Please describe. 1. Example building a FaaS compatible package 2. Example building frontend and backend with JS 3. Example building frontend with pex backend (python) 4. ??? Describe the solution you'd like Describe alternatives you've considered Additional context pantsbuild/pants

#19244 Run tracker failed to create directory for logs Issue created by benjyw Turning this into an issue, as it is not really a "discussion". Originally posted by JunyuanZhangCoin June 2, 2023 Hi, I am getting this error when running pants. Looks like it failed to create the folder by calling

self.run_logs_file.parent.mkdir(exist_ok=True, parents=True)

but parents=True should work no matter the parents exist or not. I am confused.

`
docker exec -ti udas_webserver sh -c "./pants --print-stacktrace -ldebug run udas/src/python:launcher -- manage runserver 0.0.0.0:8888 --noreload"
20:50:13.38 [DEBUG] acquiring lock: <pants.pantsd.lock.OwnerPrintingInterProcessFileLock object at 0x7f791fcee970>
20:50:13.41 [DEBUG] terminating pantsd
20:50:13.41 [DEBUG] sending signal 15 to pid 151
20:50:13.62 [DEBUG] successfully terminated pid 151
20:50:13.62 [DEBUG] purging metadata directory: /opt/pynest/.pids/7fe3dfe26c48/pantsd
20:50:13.64 [DEBUG] Launching pantsd
20:50:13.64 [DEBUG] purging metadata directory: /opt/pynest/.pids/7fe3dfe26c48/pantsd
20:50:13.64 [DEBUG] pantsd command is: PANTS_DAEMON_ENTRYPOINT=pants.pantsd.pants_daemon:launch_new_pantsd_instance PYTHONPATH=/root/.cache/pants/setup/bootstrap-Linux-x86_64/pants.t2vWUx/install/bin:/usr/lib/python38.zip:/usr/lib/python3.8:/usr/lib/python3.8/lib-dynload:/root/.cache/pants/setup/bootstrap-Linux-x86_64/2.13.0_py38/lib/python3.8/site-packages /root/.cache/pants/setup/bootstrap-Linux-x86_64/2.13.0_py38/bin/python /root/.cache/pants/setup/bootstrap-Linux-x86_64/2.13.0_py38/bin/pants --pants-bin-name=./pants --pants-version=2.13.0 --print-stacktrace -ldebug run udas/src/python:launcher -- manage runserver 0.0.0.0:8888 --noreload
20:50:14.96 [DEBUG] pantsd is running at pid 612, pailgun port is 33789
20:50:14.96 [DEBUG] releasing lock: <pants.pantsd.lock.OwnerPrintingInterProcessFileLock object at 0x7f791fcee970>
20:50:14.96 [DEBUG] Connecting to pantsd on port 33789
20:50:14.96 [DEBUG] Connecting to pantsd on port 33789 attempt 1/3
20:50:14.97 [DEBUG] Connected to pantsd
20:50:14.99 [DEBUG] Launching 1 roots (poll=false).
20:50:15.03 [DEBUG] computed 1 nodes in 0.047116 seconds. there are 7 total nodes.
20:50:15.20 [INFO] Initializing scheduler...
20:50:15.21 [DEBUG] File handle limit is: 1048576
20:50:15.22 [DEBUG] Using [cache::CommandRunner { inner: bounded::CommandRunner { inner: nailgun::CommandRunner { inner: local::CommandRunner { .. }, .. }, .. }, .. }, cache::CommandRunner { inner: bounded::CommandRunner { inner: nailgun::CommandRunner { inner: local::CommandRunner { .. }, .. }, .. }, .. }] for process execution.
20:50:15.41 [DEBUG] Changes to /root/.cache/pants/setup/bootstrap-Linux-x86_64/pants.t2vWUx/install/bin, outside of the buildroot, will not be invalidated.
20:50:15.41 [DEBUG] Changes to /root/.cache/pants/setup/bootstrap-Linux-x86_64/pants.t2vWUx/install/bin, outside of the buildroot, will not be invalidated.
20:50:15.41 [DEBUG] Changes to /usr/lib/python38.zip, outside of the buildroot, will not be invalidated.
20:50:15.41 [DEBUG] Changes to /usr/lib/python3.8, outside of the buildroot, will not be invalidated.
20:50:15.41 [DEBUG] Changes to /usr/lib/python3.8/lib-dynload, outside of the buildroot, will not be invalidated.
20:50:15.41 [DEBUG] Changes to /root/.cache/pants/setup/bootstrap-Linux-x86_64/2.13.0_py38/lib/python3.8/site-packages, outside of the buildroot, will not be invalidated.
20:50:15.41 [DEBUG] setting up service <pants.pantsd.service.scheduler_service.SchedulerService object at 0x7f5c2af00fd0>
20:50:15.41 [DEBUG] setting up service <pants.pantsd.service.store_gc_service.StoreGCService object at 0x7f5c2af12850>
20:50:15.41 [DEBUG] starting service <pants.pantsd.service.scheduler_service.SchedulerService object at 0x7f5c2af00fd0>
20:50:15.41 [DEBUG] starting service <pants.pantsd.service.store_gc_service.StoreGCService object at 0x7f5c2af12850>
20:50:15.42 [INFO] Scheduler initialized.
20:50:15.42 [DEBUG] Launching 1 roots (poll=false).
20:50:15.42 [DEBUG] computed 1 nodes in 0.000391 seconds. there are 7 total nodes.
20:50:15.46 [ERROR] [Errno 2] No such file or directory: '/opt/pynest/.pants.d/run-tracker/pants_run_2023_06_02_20_50_15_463_684ee4e61f3c465db30f03ff385e5b42'
Traceback (most recent call last):
  File "/usr/lib/python3.8/pathlib.py", line 1288, in mkdir
    self._accessor.mkdir(self, mode)
FileNotFoundError: [Errno 2] No such file or directory: '/opt/pynest/.pants.d/run-tracker/pants_run_2023_06_02_20_50_15_463_684ee4e61f3c465db30f03ff385e5b42'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/root/.cache/pants/setup/bootstrap-Linux-x86_64/2.13.0_py38/lib/python3.8/site-packages/pants/bin/daemon_pants_runner.py", line 131, in single_daemonized_run
    runner = LocalPantsRunner.create(
  File "/root/.cache/pants/setup/bootstrap-Linux-x86_64/2.13.0_py38/lib/python3.8/site-packages/pants/bin/local_pants_runner.py", line 130, in create
    run_tracker = RunTracker(options_bootstrapper.args, options)
  File "/root/.cache/pants/setup/bootstrap-Linux-x86_64/2.13.0_py38/lib/python3.8/site-packages/pants/goal/run_tracker.py", line 95, in __init__
    self.run_logs_file.parent.mkdir(exist_ok=True, parents=True)
  File "/usr/lib/python3.8/pathlib.py", line 1293, in mkdir
    self.mkdir(mode, parents=False, exist_ok=exist_ok)
  File "/usr/lib/python3.8/pathlib.py", line 1288, in mkdir
    self._accessor.mkdir(self, mode)
FileNotFoundError: [Errno 2] No such file or directory: '/opt/pynest/.pants.d/run-tracker/pants_run_2023_06_02_20_50_15_463_684ee4e61f3c465db30f03ff385e5b42'


See <https://www.pantsbuild.org/v2.13/docs/troubleshooting> for common issues.
Consider reaching out for help: <https://www.pantsbuild.org/v2.13/docs/getting-help>
```</div>
`

pantsbuild/pants

#19247 Proposal: Shift changelog content generation awar from releaser onto PR author New discussion created by thejcannon Proposal In order to help automate the release process, and shift the burden of changelog "news fragments" from the releaser onto the PR author, I propose we shift our process to have the PR which proposes a change also update the changelog. Manually curating the changelog is a breeding ground for merge conflicts, so I suggest we leverage `towncrier` to aid us as well. Towncrier is able to generate changelog entries from in-repo "news fragments" which exist as markdown files on disk with a very short summary of the change. This essentially eliminates

changelog.py

. It is customizable enough to adapt to our current categorizations. * * * How PRs • PR authors, in addition to making code changes (and possibly docs changes) will also add a file whose name is the category (unless it is "internal") with suffix ".md" in a new directory under

src/python/pants/notes

. If the PR requires a cherrypick, the directory will be the name of the milestone (e.g.

2.15.x

) otherwise the directory will be

main

. E.g. #19184 would've added

bugfix.md

to

src/python/pants/notes/main

. #18917 would've added

performance.md

to

src/python/pants/notes/2.15.x

. • CI will validate that the file has been added and matches the category label. It will also validate the directory is correct based on the milestone. • CI will immediately rename the file to include the PR number as the prefix and push to the branch (CI will skip testing). This serves two purposes: • towncrier requires all news fragments start with a unique slug (this avoids merge conflicts). The PR is a unique identifier for the change. • When generating the changelog, this is used to reference the PR which introduced the change. • (Cherry-picking happens normally) • If a cherry-pick is determined to be needed after merging. A new PR should be made that moves the file into the relevant subdirectory from

main

. This PR MUST be cherry-picked, and must be picked after the original cherry-pick is merged. • (This can be automated) Branching • When

main

is bumped to

.a0

, the directory for the feature branch is created (e.g.

2.17.x

), and the relevant

towncrier

configs are added/updated. • Now cherry-pick PRs can target 2.17.x with their docs • Open PRs not label for cherrypick will be using the directory

main

, which will point to the next version when they are released • (This can be automated) Releasing • When preparing a release for a feature-branch, run

towncrier build --config src/python/pants/notes/<branch>/towncrier.toml --version <new version> --date '<human date>'

• (This will be automated) • This will delete the news fragments on disk and update the relevant changelog

.md

. • Make a PR which should be labeled for cherry-pick targeting the right branch • When preparing a

.devN

release, the only differences are: • Use the config for

main

• Don't label for cherry-pick * * * (We can decide after the fact if we want to keep the PR labels, or just use the fragment name. I consider that out of scope for this proposal though. The category labels will continue to be required and must match the filename.) pantsbuild/pants

#19248 `from ..foo import *` glob imports are silently ignored by Rust parser Issue created by huonw Describe the bug The new Rust parser doesn't seem to understand a 'plain' parent import, like

from ..file1 import *

. It does understand

from ..file1 import func

, though. This reproducer constructs a directory structure like:

.
├── pants.toml
└── src
    └── a
        ├── BUILD
        ├── file1.py
        └── file2.py

where

file2.py

has

from .file1 import *

. It runs

pants peek src/a/file2.py

with both the python parser (which shows a dependency on

src/a/file1.py

) and with the Rust parser (which doesn't). Unlike #19173, there's no inference warnings either.

cd $(mktemp -d)

cat > pants.toml <<EOF
[GLOBAL]
pants_version = "2.17.0rc0"

backend_packages = ["pants.backend.python"]

[python]
interpreter_constraints = [">=3.8"]

[python-infer]
use_rust_parser = false
EOF

# set up the file structure:
mkdir -p src/a
touch src/a/file1.py
echo "from .file1 import *" > src/a/file2.py

echo "python_sources()" > src/a/BUILD

tree .

# OK: with Python parser `"dependencies": ["src/a/file1.py"]`
pants peek src/a/file2.py

# flip to the rust parser
sed -i '' 's/use_rust_parser = false/use_rust_parser = true/' pants.toml

# BUG: with Rust parser `"dependencies": []`
pants peek src/a/file2.py

Pants version 2.17.0rc0 OS macOS Additional info #19173 pantsbuild/pants