I think https://github.com/tj-actions/changed-files isn't behaving how we expect... https://github.com/pantsbuild/pants/pull/19021 changed two files, but the GHA output has:

affected=$(python build-support/bin/classify_changed_files.py ".github/workflows/auto-cherry-picker.yaml|build-support/bin/cherry_pick.sh|docs/markdown/Python/python/python-lockfiles.md|pants.toml|src/python/pants/backend/python/framework/django/BUILD|src/python/pants/backend/python/framework/django/dependency_inference.py|src/python/pants/backend/python/framework/django/dependency_inference_test.py|src/python/pants/backend/python/framework/django/detect_apps.py|src/python/pants/backend/python/framework/django/detect_apps_test.py|src/python/pants/backend/python/framework/django/scripts/BUILD|src/python/pants/backend/python/framework/django/scripts/dependency_visitor.py|src/python/pants/backend/python/subsystems/python_tool_base.py|src/python/pants/backend/python/subsystems/python_tool_base_test.py|src/python/pants/backend/python/util_rules/BUILD|src/python/pants/backend/python/util_rules/pex.py|src/python/pants/backend/python/util_rules/pex_from_targets.py|src/python/pants/backend/python/util_rules/pex_from_targets_test.py|src/python/pants/backend/python/util_rules/pex_requirements.py|src/python/pants/backend/python/util_rules/pex_test.py|src/python/pants/init/plugin_resolver.py")

Relatedly I think the check that we are in the pantsbuild org may not be strong enough. The GitHub UI, when making edits, prefers those to be in the pantsbuild repo: https://github.com/pantsbuild/pants/pull/19026 Those count as pushes to the repo, and therefore are currently building wheels (the PR status is skipped, but the

push

status isn't)

So, GitHub Actions apparently uses the version of the action for PR closed events from the PR branch? 😞 https://github.com/pantsbuild/pants/actions/runs/5009700926 failed with an error that has been fixed on

main

. "Workflow File" leads to https://github.com/pantsbuild/pants/actions/runs/5009700926/workflow

hey folks, do we have existing code to handle cleanup of passthrough args? ex

./pants lint testprojects/src/helm:: -- --debug

?

ArgsListOption

will handle the passthrough, I'm hoping to have some existing code for passing different args to subtasks for where it makes sense. For a concrete case, for deploying helm charts, we render the chart with

helm template

and then deploy it with `helm upgrade`; some args are only valid for the

upgrade

part, and we can filter them out of the ones we pass to

template

. Currently we only pass them through to

upgrade

, which makes sense for, ex,

--force

. But it's possible for the

template

step to fail, and then

--debug

would be helpful. See https://github.com/pantsbuild/pants/issues/18089

Not sure if it's new thing, but a

pants --all-changed fix fmt test lint

on the pants-repo just deadlocked my whole WSL instance. Never had that happen before; though it tends to happen a lot with Bazel. Didn't find anything in the issue tracker for it so not sure how common it is. Unfortunately I didn't have any other shells open so can do no post-morteming... The bazel issue manifests as a deadlock in the kernels

d_alloc

and is almost always triggered by

go mod download

in the repo where I see it, likely some fs race condition... most likely due to the shims in WSL. All the dead processes will be in

D

state in htop or similar tools, for

uninterruptible sleep

.

Well, the auto-cherry-picker is working, albeit not fully: https://github.com/pantsbuild/pants/actions/runs/5009691730 https://github.com/pantsbuild/pants/pull/19031

although it’s a little bit out of line with convention (because

2.16.0

is not released yet), i think that we should cut

2.17.0a0

and the

2.17.x

branch now to keep (closer to) our six week target.

@bitter-ability-32190: hm: does https://github.com/pantsbuild/pants/pull/19037 fix https://github.com/pantsbuild/pants/actions/runs/5017155703/jobs/8994909813?pr=19038 ?

what is the reason behind asking for a len on http_source while the sha256 already provide integrity, most of the checksum file will only include sha256?

i think that https://github.com/pantsbuild/pants/actions/runs/5018552588/jobs/9001610509?pr=19043 is a reproducible network failure on

main

seems that Pants is missing the release for

pantsbuild.pants.testutil

2.17.0a0. Getting the following error when running it from

main

and can't find it either in https://pypi.org/project/pantsbuild.pants.testutil/#history

ERROR: Could not find a version that satisfies the requirement pantsbuild.pants.testutil<2.18,>=2.17.0a0
ERROR: No matching distribution found for pantsbuild.pants.testutil<2.18,>=2.17.0a0

I’d need some help understanding this error:

pants.engine.target.InvalidFieldException: The target testprojects/src/python/native:main has the `interpreter_constraints` ('==3.9.*',), which are not a subset of the `interpreter_constraints` of some of its dependencies:

  * ('==3.7.*',): testprojects/src/python/native/main.py:main_lib
  * ('==3.7.*',): testprojects/src/python/native:dist

To fix this, you should likely adjust testprojects/src/python/native:main's `interpreter_constraints` to match the narrowest range in the above list.

from https://github.com/pantsbuild/pants/actions/runs/5018907494/jobs/9008981047?pr=17671#step:8:346 for https://github.com/pantsbuild/pants/pull/17671

https://github.com/pantsbuild/pants/issues/19054 I think this is a must have to not have massive IO in the JS backend, but I'm either missing something obvious or in need of other ideas how to achieve it.

@witty-crayon-22786 There's a... well it isn't a bug, but isn't not a bug in 2.16.x. Using the new

s3

URL Handler, I'm seeing:

File "/home/ubuntu/.cache/nce/1239dc7f2c4e6bb57836aa10d4b93b4a7bd7016bedcb8ca326ef6b803855e362/bindings/venvs/2.16.0rc2/lib/python3.9/site-packages/pants/backend/url_handlers/s3/register.py", line 85, in download_from_s3

    signer.add_auth(http_request)

  File "/home/ubuntu/.cache/pants/named_caches/pex_root/venvs/s/1968668f/venv/lib/python3.9/site-packages/botocore/auth.py", line 833, in add_auth

    signature = self.get_signature(

  File "/home/ubuntu/.cache/pants/named_caches/pex_root/venvs/s/1968668f/venv/lib/python3.9/site-packages/botocore/auth.py", line 819, in get_signature

    del headers['x-amz-security-token']

KeyError: 'x-amz-security-token'

Using

botocore-a-la-carte

as a plugin. I upstreamed the fix and is in PR: https://github.com/boto/botocore/pull/2948 who the hell knows when that goes in. ...I have no idea why this didn't fire earlier, or why the tests pass. The code in question is 9 years old. So... might wanna 🍒 a Pants workaround 😕

@bitter-ability-32190: it looks like the cherry-pick job might have picked more than it should have here: https://github.com/pantsbuild/pants/actions/runs/5026777727/jobs/9015456286

hm… branches/`push`es technically no longer need to run the wheels shards if change-detection doesn’t think they’re necessary, since the Release job is responsible for that.

https://github.com/pantsbuild/pants/issues/19022 has been approved, but had some ongoing discussion/tweaks… I think I’ve resolved them but I’m not sure it’s a good resolution. Can I merge? (Cc @happy-kitchen-89482 )

if anyone is able to rubber stamp https://github.com/pantsbuild/pants/pull/19064, it would be appreciated. that will let me kick off the next

2.17.0a

release attempt with the new job

I vaguely recall there was a change in the setup_py stuff that would allow using

cython

or

mypyc

called via a custom setup.py... Was that ever taken anywhere? I think it was a PR from @happy-kitchen-89482?

@enough-analyst-54434 Does Science currently have full support for

bindings

? Want to make sure I'm not jumping the gun before I start debugging this

ERROR: No file named :hellofastapi.pex is stored in this scie.

...
[[lift.files]]
name = ":hellofastapi-pex"

[[lift.commands]]
exe = "{scie.bindings.venv}/venv/bin/uvicorn"
args = [ "hellofastapi.main:app",]
description = "The FastAPI executable."

[[lift.bindings]]
exe = "#{cpython:python}"
args = [ "{:hellofastapi.pex}", "venv", "--bin-path", "prepend", "--compile", "--rm", "all", "{scie.bindings}/venv",]
name = "venv"
description = "Installs HelloFastAPI into a venv and pre-compiles .pyc"

[lift.bindings.env]
default = { "=PATH" = "{cpython}/python/bin:{scie.env.PATH}", "PEX_TOOLS" = "1", "PEX_ROOT" = "{scie.bindings}/pex_root" }

My first rust PR, it’s a draft still and I know I need to test this stuff, but I’m looking for feedback because I don’t know what I’m doing 😅 https://github.com/pantsbuild/pants/pull/19070

@witty-crayon-22786 Any reason I'd be getting permissions issues when trying to grab 2.17.0a1?

Bootstrapping Pants 2.17.0a1 using cpython 3.9.15
Installing pantsbuild.pants==2.17.0a1 into a virtual environment at /Users/sj/Library/Caches/nce/2351f91ce4510094a8c79af7e3578ac7f71d3918106e3b86ddf7cee196084d4a/bindings/venvs/2.17.0a1
New virtual environment successfully created at /Users/sj/Library/Caches/nce/2351f91ce4510094a8c79af7e3578ac7f71d3918106e3b86ddf7cee196084d4a/bindings/venvs/2.17.0a1.
Error: Failed to exec "/Users/sj/Library/Caches/nce/2351f91ce4510094a8c79af7e3578ac7f71d3918106e3b86ddf7cee196084d4a/bindings/venvs/2.17.0a1/lib/python3.9/site-packages/pants/bin/native_client" with argv ["/Users/sj/Library/Caches/nce/2351f91ce4510094a8c79af7e3578ac7f71d3918106e3b86ddf7cee196084d4a/bindings/venvs/2.17.0a1/lib/python3.9/site-packages/pants/bin/native_client", "--python-repos-find-links=-[\'file:///Users/sj/Library/Caches/nce/2351f91ce4510094a8c79af7e3578ac7f71d3918106e3b86ddf7cee196084d4a/bindings/find_links/2.17.0a1/a577739f/index.html\']", "--version"]: EACCES: Permission denied

Would it be a good idea to get the scie-pants launcher into mainstream Linux repos? Any real technical reason stopping us here?

so, the list of variables which `pantsd` preserves when it starts up has grown slightly in order to attempt to capture variables used in docker auth (because the rust

docker

client code needs to consume them as environment variables, without forking a few libraries to allow for explicitly passing them). but that has proved to be a losing battle. some users are still encountering variables (likely specific to particular auth backends). i’m trying to decide between: 1. adding (yet another) option to allow folks to set which variables are preserved 2. giving up on trying to sandbox env vars in

pantsd

, and/or adjusting how we sandbox so that it only applies to python

@witty-crayon-22786 you mentioned having thoughts on remote caching https://github.com/pantsbuild/pants/issues/11149 and specifically concerns about fine-grained cache requests due to individually caching each entry in a directory tree for non-REAPI implementations, like S3 or GHA. I did a quick bit of research into some of the other systems: • Bazel and Gradle seem to do fine-grained request-per-entry-in-tree in their HTTP caches • Lage and Rush seem to tar up the entire output of each command, rather than handling trees • I note that, AIUI, Pants' REAPI currently does a gRPC request per file (https://github.com/pantsbuild/pants/issues/6990) What were your thoughts? How best to have this discussion?

Hey folks, I was fiddling with implementing lockfiles for Terraform, and was hitting some design issues 🧵

This jacks us up to 10 Linux x86 test shards in CI, while I concurrently work on setting up remote caching: https://github.com/pantsbuild/pants/pull/19099

https://github.com/pantsbuild/pants/pull/19099 appears to stabilize (and speed up the wall time of) CI. Once I merge it (review plz!) I highly recommend merging it into your PR branches.

@bitter-ability-32190 is https://github.com/pantsbuild/pants/pull/19056 the last thing you have for

2.16.x

?