#18329 Docker environment may leave behind partially-root-owned sandboxes on Linux Issue created by stuhood As reported in #18306 and mostly addressed in #18327, files in the sandbox bind mount may end up owned by root on Linux. To ensure that they can still be captured, #18327 made them readable, but they will not be able to be removed. To avoid this problem, we might need to create a user in the container to chown them to. Alternatively, it's possible that a variant of the strategy originally employed by #18225 (tar-pipes) could be used to capture outputs as well, so that we skipped the bind mount entirely. pantsbuild/pants

#18331 Allow custom bin paths Issue created by GuillaumeDesforges Describe the bug Pants looks for bash, git and other programs in /bin, /usr/bin and other common locations. On my system, binaries are managed a bit differently so they are not in such locations. Pants version 2.16.0.dev7 OS NixOS (Linux) pantsbuild/pants

#18333 run_shell_command missing help Issue created by cburroughs Describe the bug

$ ./pants --version
2.16.0.dev7
$ grep shell pants.toml 
  "pants.backend.shell"
$ ./pants help run_shell_command
Unknown entity: run_shell_command
Did you mean shell_command, run_shell_command_request, or test_shell_command?

Pants version 2.16.0.dev7 OS Linux Additional info cburroughs/example-python@e27998a pantsbuild/pants

#18335 run_shell_command workdir='.' appears ignored Issue created by cburroughs Describe the bug I would expect that if I set

run_shell_command(
    name='where-am-i-workdir',
    command="pwd",
    workdir='.'
)

Then the command to run relative to the BUILD file ("`.` specifies the location of the

BUILD

file.") However,

$ cat src/py/example/BUILD  | head -18
run_shell_command(
    name='where-am-i',
    command="pwd",
)
    
run_shell_command(
    name='where-am-i-workdir',
    command="pwd",
    workdir='.'
)

run_shell_command(
    name='where-am-i-workdir-2',
    command="cd {chroot} && pwd",
    workdir='.'
)

$ ./pants run src/py/example:where-am-i-workdir
/home/ecsb/src/o/example-python

$ ./pants run src/py/example:where-am-i-workdir-2
/tmp/pants-sandbox-7zAQyl

Neither of which are the "same" (in or out of the sandbox) directory as the BUILD file. Pants version 2.16.0.dev7 OS Linux Additional info cburroughs/example-python@e27998a A less contrived pseudo-code example of what I'm trying to do:

run_shell_command(
    command="./src.py.example/foo-cli.pex --do-stuff-to-files=my-file.txt",
    execution_dependencies=['src/py/example/foo-cli.pex', ':my-text-files'],
    workdir='.'

pantsbuild/pants

#17803 Building wheel for setproctitle (pyproject.toml) did not run successfully Issue created by hackmann Describe the bug Run the one-step setup script /bin/bash -c "$(curl -fsSL https://static.pantsbuild.org/setup/one_step_setup.sh)"

Bootstrapping Pants using /usr/bin/python3
Installing pantsbuild.pants==2.14.0 into a virtual environment at /Users/hack/.cache/pants/setup/bootstrap-Darwin-arm64/2.14.0_py39
  error: subprocess-exited-with-error
  
  × Building wheel for setproctitle (pyproject.toml) did not run successfully.
  │ exit code: 1
  ╰─> [15 lines of output]
      running bdist_wheel
      running build
      running build_ext
      building 'setproctitle' extension
      creating build
      creating build/temp.macosx-10.9-universal2-cpython-39
      creating build/temp.macosx-10.9-universal2-cpython-39/src
      clang -Wno-unused-result -Wsign-compare -Wunreachable-code -fno-common -dynamic -DNDEBUG -g -fwrapv -O3 -Wall -iwithsysroot/System/Library/Frameworks/System.framework/PrivateHeaders -iwithsysroot/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.9/Headers -arch arm64 -arch x86_64 -Werror=implicit-function-declaration -DSPT_VERSION=1.2.2 -D__darwin__=1 -I/Users/hack/.cache/pants/setup/bootstrap-Darwin-arm64/pants.pDEMtV/install/include -I/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.9/Headers -c src/setproctitle.c -o build/temp.macosx-10.9-universal2-cpython-39/src/setproctitle.o
      In file included from src/setproctitle.c:14:
      In file included from src/spt.h:15:
      src/spt_python.h:16:10: fatal error: 'Python.h' file not found
      #include <Python.h>
               ^~~~~~~~~~
      1 error generated.
      error: command '/usr/bin/clang' failed with exit code 1
      [end of output]
  
  note: This error originates from a subprocess, and is likely not a problem with pip.
  ERROR: Failed building wheel for setproctitle
ERROR: Could not build wheels for setproctitle, which is required to install pyproject.toml-based projects

Pants version 2.14.0 OS MacOSX Additional info All prerequisite are installed: • macOS. • Python 3.9 discoverable on your PATH. • A C compiler, system headers and Python headers (to compile native Python modules). • Internet access (so that Pants can fully bootstrap itself). I don't have Xcode.app installed and it is not listed as a prerequisite (/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.9/Headers). Instead I have command line tools installed (

xcode-select --install

). And Python.h is located at

/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.9/Headers/Python.h

pantsbuild/pants

#2063 Create lockfile for `xmlsec` fails Issue created by kaos

$ python --version
Python 3.11.1
# but same issue on Python 3.8.13

$ pex3 --version
2.1.122

$ pex3 lock create "xmlsec==1.3.13"
Building lxml version 4.9.2.
Building without Cython.
Building against libxml2 2.9.4 and libxslt 1.1.29
running dist_info
creating /private/var/folders/y2/6t5bbk554wv5h6w_qlgpcpnm0000gr/T/tmpqn87v27s/build/lxml.egg-info
writing /private/var/folders/y2/6t5bbk554wv5h6w_qlgpcpnm0000gr/T/tmpqn87v27s/build/lxml.egg-info/PKG-INFO
writing dependency_links to /private/var/folders/y2/6t5bbk554wv5h6w_qlgpcpnm0000gr/T/tmpqn87v27s/build/lxml.egg-info/dependency_links.txt
writing requirements to /private/var/folders/y2/6t5bbk554wv5h6w_qlgpcpnm0000gr/T/tmpqn87v27s/build/lxml.egg-info/requires.txt
writing top-level names to /private/var/folders/y2/6t5bbk554wv5h6w_qlgpcpnm0000gr/T/tmpqn87v27s/build/lxml.egg-info/top_level.txt
writing manifest file '/private/var/folders/y2/6t5bbk554wv5h6w_qlgpcpnm0000gr/T/tmpqn87v27s/build/lxml.egg-info/SOURCES.txt'
reading manifest file '/private/var/folders/y2/6t5bbk554wv5h6w_qlgpcpnm0000gr/T/tmpqn87v27s/build/lxml.egg-info/SOURCES.txt'
reading manifest template '<http://MANIFEST.in|MANIFEST.in>'
writing manifest file '/private/var/folders/y2/6t5bbk554wv5h6w_qlgpcpnm0000gr/T/tmpqn87v27s/build/lxml.egg-info/SOURCES.txt'
creating '/private/var/folders/y2/6t5bbk554wv5h6w_qlgpcpnm0000gr/T/tmpqn87v27s/build/lxml.dist-info'
adding license file "LICENSE.txt" (matched pattern "LICEN[CS]E*")
adding license file "LICENSES.txt" (matched pattern "LICEN[CS]E*")
Could not gather lock metadata for 1 project with source artifacts:
1. /private/var/folders/y2/6t5bbk554wv5h6w_qlgpcpnm0000gr/T/tmpkqwg1011/Users.andreas.stenius.tmp.venv.bin.python3.11/xmlsec-1.3.13.tar.gz: 'build-backend'

Regression from previous version:

$ pex3 --version
2.1.121

$ pex3 lock create "xmlsec==1.3.13"
{"allow_builds": true, "allow_prereleases": false, "allow_wheels": true, "build_isolation": true, "constraints": [], "locked_resolves": [{"locked_requirements": [{"artifacts": [{"algorithm": "sha256", "hash": "2455cfaeb7ac70338b3257f41e21f0724f4b5b0c0e7702da67ee6c3640835b67", "url": "<https://files.pythonhosted.org/packages/06/5a/e11cad7b79f2cf3dd2ff8f81fa8ca667e7591d3d8451768589996b65dec1/lxml-4.9.2.tar.gz>"}], "project_name": "lxml", "requires_dists": ["BeautifulSoup4; extra == \"htmlsoup\"", "Cython>=0.29.7; extra == \"source\"", "cssselect>=0.7; extra == \"cssselect\"", "html5lib; extra == \"html5\""], "requires_python": "!=3.0.*,!=3.1.*,!=3.2.*,!=3.3.*,!=3.4.*,>=2.7", "version": "4.9.2"}, {"artifacts": [{"algorithm": "sha256", "hash": "916f5d78e8041f6cd9391abba659da8c94a4fef7196d126d40af1ff417f2cf86", "url": "<https://files.pythonhosted.org/packages/37/9f/342d4562eac99178d0d515c780285e107c6828cefad37d02f05b7b7d8751/xmlsec-1.3.13.tar.gz>"}], "project_name": "xmlsec", "requires_dists": ["lxml>=3.8"], "requires_python": ">=3.5", "version": "1.3.13"}], "platform_tag": ["cp311", "cp311", "macosx_13_0_arm64"]}], "path_mappings": {}, "pex_version": "2.1.121", "pip_version": "20.3.4-patched", "prefer_older_binary": false, "requirements": ["xmlsec==1.3.13"], "requires_python": [], "resolver_version": "pip-legacy-resolver", "style": "strict", "target_systems": [], "transitive": true, "use_pep517": null}

pantsbuild/pex

#2019 When used as a REPL, Pex does not support command history Issue created by benjyw A regular Python interpreter does so automatically when importing

site

, but Pex bypasses that mechanism. pantsbuild/pex

#18344 Pants repo's internal-only `pants` script not concurrency safe Issue created by stuhood Parallel bootstraps of Pants triggered by Rust integration tests can cause bootstrap failures: https://github.com/pantsbuild/pants/actions/runs/4249976339/jobs/7390639488#step:13:375 pantsbuild/pants

#18346 helm backend should at least WARN (maybe error?) if the user supplied image is not found Issue created by cburroughs Is your feature request related to a problem? Please describe.

$ cat dependencies src/deployment/BUILD 
cat: dependencies: No such file or directory
#Overrides the `image` value for the chart using the target address for the first-party docker image.
helm_deployment(dependencies=["src/chart"],
                sources=["common-values.yaml"],
                values={"image": "src/docker:docker"})
$ cat dependencies src/deployment/BUILD 
cat: dependencies: No such file or directory
#Overrides the `image` value for the chart using the target address for the first-party docker image.
helm_deployment(dependencies=["src/chart"],
                sources=["common-values.yaml"],
                values={"image": "src/docker:whale-typo"})
$ ./pants dependencies src/deployment:deployment
src/chart:chart

😿 Describe the solution you'd like If the user specifically says "use this image", but said image isn't found, pants should warn or error out or something Describe alternatives you've considered Additional context #18345 https://github.com/cburroughs/example-python/tree/17da445368d145f22693f86d4157895609968922 pantsbuild/pants

#18348 Using `overrides` with incorrect field values isn't caught immediately, and leads to confusing downstream errors Issue created by huonw Describe the bug If a target generator has some of its generated targets customised via

overrides

, it seems like the overrides values aren't validated in the same way as providing them directly to a normal target. This can lead to weird downstream errors that are hard to debug (e.g. https://pantsbuild.slack.com/archives/C046T6T9U/p1677030322791399?thread_ts=1677027041.399219&cid=C046T6T9U). Reproducer: https://gist.github.com/huonw/9c62ef32d71751af21be957ac794d67b

git clone <https://gist.github.com/9c62ef32d71751af21be957ac794d67b.git>
cd 9c62ef32d71751af21be957ac794d67b

pants dependencies ::
# UnsupportedWildcardError: The address `:` from the `dependencies` field from the target //:reqs#requests ended in a wildcard (`:`), which is not supported.

BUILD

file (the

dependencies="..."

should be

dependencies=["..."]

):

# Using overrides:
python_requirements(
    name="reqs", overrides={"requests": dict(dependencies=":reqs#setuptools")}
)
#> error: UnsupportedWildcardError: The address `:` from the `dependencies` field from the target //:reqs#requests ended in a wildcard (`:`), which is not supported.

# Equivalent manual targets:
# python_requirement(name="setuptools", requirements=["setuptools"])
# python_requirement(name="requests", requirements=["requests"], dependencies=":setuptools")
#> error: InvalidFieldTypeException: The 'dependencies' field in target //:requests must be an iterable of strings (e.g. a list of strings), but was `':setuptools'` with type `str`.

The error message hints at the problem ("the address `:`" comes from the string

":reqs#setuptools"

being treated as an iterable equivalent to

[":", "r", "e", ...]

), but is far less useful than being really specific about the problem, like the equivalent `python_requirement`s are. Pants version 2.14.0, 2.15.0rc7, 2.16.0.dev7 OS macOS Additional info Add any other information about the problem here, such as attachments or links to gists, if relevant. pantsbuild/pants

#18349 Providing a positional argument to a target leads to non-specific `TypeError: __call__() takes ...` error Issue created by huonw Describe the bug All targets take keyword arguments only, and passing a positional argument like

python-sources("x")

accidentally is an error, but this error isn't specific about the location of the problem, which makes it hard to find.

cd "$(mktemp -d)"
echo 'python_sources("x")' > BUILD

cat > pants.toml <<EOF
[GLOBAL]
pants_version = "2.14.0"
backend_packages = ["pants.backend.python"]

[anonymous-telemetry]
enabled = false
EOF

pants dependencies ::
#>   MappingError: Failed to parse ./BUILD:
#> __call__() takes 1 positional argument but 2 were given

It'd be great if the error at least pointed to the line, and would be even better if the raw Python-generated

TypeError

was translated into something meaningful for Pants users. Pants version 2.14.0, 2.16.0.dev7 OS macOS Additional info Add any other information about the problem here, such as attachments or links to gists, if relevant. pantsbuild/pants

#18351 Add support to refer to a group of targets in BUILD files Issue created by AlexTereshenkov Is your feature request related to a problem? Please describe. Imagine there is a directory containing a number of nested directories each containing JSON files. There's a

file

target in the

BUILD

file in each directory. I'd like to produce a package, say, an

archive

, that will include all files in the current directory, and all the subdirectories, recursively:

archive(name="main-archive", format="tar", files=["helloworld::"])

that's illegal

UnsupportedWildcardError: The address `helloworld::` from the `files` from the target //:main-archive 
ended in a wildcard (`::`), which is not supported.

I wonder if it would be helpful to have a way to say "include all targets in a directory, recursively"? Describe the solution you'd like To be able to refer to a group of targets in the repository without specifying their paths explicitly. Describe alternatives you've considered I could do a

list

/

peek

and get all the targets of a particular type to be placed into the

dependencies

field once. However, I'd need to update that list every time a new directory with a JSON file is added. Having a target

files(sources=["**/*/*.json"])

does the trick so you can depend on a single target, however, that's too coarse-grained; having both

file

target in the subdirectory and all-covering

files

leads to duplicate ownership which is not ideal either. pantsbuild/pants

#2065 Pex files do not allow for OpenTelemetry auto-instrumentation Issue created by Peder2911 We are using pantsbuild to set up a developer platform with an emphasis on a nice, frictionless developer experience. A part of this experience is codeless auto-instrumentation such as is provided by Opentelemetry with auto instrumentation. Specifically, we are looking to auto-instrument using the Kubernetes Operator offered by OpenTelemetry. Auto-instrumentation with OpenTelemetry works by monkey-patching libraries, creating passthrough functions that wrap existing functions with logging statements. The way this works is kind of obscure to me, and is something I have to figure out a bit more before I can give constructive input into how to support this. Auto-instrumentation of non-pexified applications can be as easy as running (for example from a clean Python3.9 docker container):

pip install flask opentelemetry-distro opentelemetry-instrumentation-flask

cat << EOF > app.py
import flask
app = flask.Flask(__name__)
@app.route("/")
def helloworld():
  return "<h1>Hello world!</h1>"
app.run()
EOF

opentelemetry-instrument \
  --traces_exporter console \
  --metrics_exporter console \
 python app.py

This will spin up a Flask server that also emits telemetry. This separation of code and logging is super useful for maintaining a clean developer experience. This does not work out-of-the-box with Pex, however. Ideally, one would be able to run the following to achieve the same thing:

pip install pex opentelemetry-distro opentelemetry-instrumentation-flask

cat << EOF > app.py
import flask
app = flask.Flask(__name__)
@app.route("/")
def helloworld():
  return "Hello from Pex"
app.run()
EOF

pex flask -o app.pex --exe ./app.py
opentelemetry-instrument \
  --metrics_exporter console \
  --traces_exporter console \
  ./app.pex

This does, as expected, not allow OpenTelemetry to inject any instrumentation ,and will not result in any telemetry being emitted. It would be pretty awesome if it did though! I would appreciate any thoughts and ideas for how to implement this. pantsbuild/pex

#18353 How to run streamlit apps in pants? New discussion created by rhuanbarreto Streamlit can only run through its CLI by doing

streamlit run /path/to/source.py

, so I wonder how can I run streamlit in pants once the run needs a pex binary as an input? pantsbuild/pants

#2066 Intermittent / nondeterministic failure building `requirements.pex` due to missing `site-packages` Issue created by danxmoran Our CI will occasionally fail with an error like:

Traceback (most recent call last):
  File "/home/color/.cache/nce/99421dee8fedc336d5a6bb8322fbfa602bc68fe63303ae702ec7b9c5672cd086/bindings/venvs/2.15.0rc6/lib/python3.9/site-packages/pants/engine/process.py", line 289, in fallible_to_exec_result_or_raise
    raise ProcessExecutionFailure(
pants.engine.process.ProcessExecutionFailure: Process 'Building 1 requirement for requirements.pex from the 3rdparty/lockfiles/resolves/pants-plugins.lockfile resolve: pantsbuild.pants<2.16,>=2.15.0a0' failed with exit code 1.
stdout:

stderr:
The virtualenv at /tmp/pants/named_caches/pex_root/venvs/4c85013f478e0393bbf8db8fcf02e1def7ff5031/ba7a55164c2afb363895254bbb1063124dd74d5b.lck.work is not valid. No site-packages directory was found in its sys.path:
/opt/python/3.9.16/lib/python39.zip
/opt/python/3.9.16/lib/python3.9
/opt/python/3.9.16/lib/python3.9/lib-dynload
/opt/python/3.9.16/lib/python3.9/site-packages

We've seen it hit across different resolves, Python interpreter versions, and Pex versions - the error above happened on Pex

v2.1.122

. It happens very infrequently (once every few weeks). I captured the Pants caches & execution dir for the error above. The archive is too big to attach via GitHub, but I can share it via Slack / GDrive upload. pantsbuild/pex

#18354 TypeError: __init__() got an unexpected keyword argument 'platform' Issue created by reven33 Describe the bug I have some protobuf go changes in PR and the PR checks failed because of the error below. Does anyone know how to fix it? Really appreciate. Run ./pants proto-compile Bootstrapping Pants using /opt/hostedtoolcache/Python/3.9.16/x64/bin/python3.9 Creating the virtualenv PEX. Downloading the Pex PEX. SHA256 fingerprint of https://github.com/pantsbuild/pex/releases/download/v2.1.103/pex verified. Installing pantsbuild.pants==2.15.0rc7 into a virtual environment at /home/runner/.cache/pants/setup/bootstrap-Linux-x86_64/2.15.0rc7_py39 New virtual environment successfully created at /home/runner/.cache/pants/setup/bootstrap-Linux-x86_64/2.15.0rc7_py39. 174041.22 [INFO] Initialization options changed: reinitializing scheduler... 174042.65 [INFO] Scheduler initialized. Error: 5.17 [ERROR] 1 Exception encountered: Engine traceback: in

proto-compile

goal TypeError: *init*() got an unexpected keyword argument 'platform' Error: Process completed with exit code 1. Pants version Which version of Pants are you using? 2.15.0rc7 OS Are you encountering the bug on MacOS, Linux, or both? The issue happens in github build &release step Additional info Add any other information about the problem here, such as attachments or links to gists, if relevant. pantsbuild/pants

#18359 support relative docker_image target syntax for helm interpolated images values Issue created by cburroughs Is your feature request related to a problem? Please describe.

docker_image(name='tmp-example')

helm_deployment(dependencies=["src/chart"],
                sources=["common-values.yaml"],
                values={"image": "src/docker:tmp-example"})

$ ./pants experimental-deploy src/deployment:deployment  -- --debug | grep image
...
      # Uses the `image` value entry from the deployment inputs
      image: corp.registry.example/tmp-example:localdev

But

docker_image(name='tmp-example')

helm_deployment(dependencies=["src/chart"],
                sources=["common-values.yaml"],
                values={"image": ":tmp-example"})

Gives

$ ./pants experimental-deploy src/deployment:deployment  -- --debug | grep image
      # Uses the `image` value entry from the deployment inputs
      image: :tmp-example

Describe the solution you'd like I'd like the

:foo

target syntax to work. Describe alternatives you've considered Always using the fully qualified targets. Version: 2.16.0.dev7 pantsbuild/pants

#18361 Rule visitor chokes on union types Issue created by kaos

It doesn't work, you get:

pants.base.exceptions.RuleTypeError: /Users/benjyw/src/pants/pants-plugins/internal_plugins/test_lockfile_fixtures/rules.py:85: Could not resolve type for `pytest.to_pex_request` in module internal_plugins.test_lockfile_fixtures.rules.

Return type annotation required for `to_pex_request` in /Users/benjyw/src/pants/src/python/pants/backend/python/subsystems/python_tool_base.py:276

Even though

to_pex_request

does have a return type annotation. _Originally posted by @happy-kitchen-89482 in #18350 (comment)_ pantsbuild/pants

#18364 Allow gathering/joining multiple `async def` coroutines ala `MultiGet` Issue created by huonw Is your feature request related to a problem? Please describe. In #18303 (comment), we identify that there's no native way for rule code to run multiple Python

async def

coroutines concurrently, similar to Python's

asyncio.gather

(or

std::future::join!

in Rust). Pants'

MultiGet

is related, but only supports `Get`s specifically. Being able to do this would allow embarrassingly parallel invocations of rule helpers in a loop to achieve maximum concurrency. Once this is done, the flake8 rule added in #18303 should be generalised to flag

await _some_helper()

inside a loop, in addition to `await Get(...)`/`await MultiGet(...)` in a loop. Describe the solution you'd like Some mechanism to schedule coroutines (and preferably arbitrary `Awaitable`/`Future` instances) to run concurrently. Describe alternatives you've considered N/A Additional context N/A pantsbuild/pants

#18365 Audit/rewrite `PNT30` flake8 lint failures to achieve more concurrency Issue created by huonw Is your feature request related to a problem? Please describe. In #18303, a new flake8 rule is added that flags constructs like

[await Get(...) for ... in ...]

. If these are embarrassingly parallel (i.e. no dependencies between iterations), they should be rewritten to use `MultiGet`: for instance,

await MultiGet(Get(...) for ... in ...)

. That PR added

noqa: PNT30: requires triage

annotations to many instances of this, rather than fix them. This issue is about going through and either fixing them or making the annotation more specific. Embarrassingly parallel: ☐

src/python/paths/backend/go/goals/debug_goals.py

☐

src/python/paths/backend/go/goals/generate.py

☐

src/python/paths/backend/go/util_rules/cgo.py

☐

src/python/paths/backend/go/util_rules/go_bootstrap.py

☐

src/python/paths/backend/go/util_rules/goroot.py

☐

src/python/paths/backend/helm/util_rules/post_renderer.py

☐

src/python/paths/backend/python/framework/django/detect_apps.py

☐

src/python/paths/bsp/util_rules/targets.py

☐

src/python/paths/core/goals/deploy.py

☐

src/python/paths/core/goals/update_build_files.py

☐

src/python/paths/engine/internals/build_files.py

☑︎

src/python/paths/engine/internals/dep_rules.py

☐

src/python/paths/jvm/goals/lockfile.py

☐

src/python/paths/source/source_root.py

Embarrassingly parallel, except there's an early return, so just moving all invocations to a

MultiGet

might result in more work total. ☐

src/python/paths/backend/cc/util_rules/toolchain.py

☐

src/python/paths/backend/python/goals/setup_py.py

☐

src/python/paths/core/subsystems/python_bootstrap.py

Describe the solution you'd like For each of the files above, rewrite the loops that call

await Get(...)

internally to use

MultiGet

. Describe alternatives you've considered N/A Additional context N/A pantsbuild/pants

#7491 Consider adding Flake8 lint for `await` statements inside `for` bodies in @rules Issue created by stuhood A code block in an

@rule

like:

results = []
for x in some_collection:
  result = await Get(Y, X, x)
  results.append(result)

...generally represents a missed opportunity for parallelism (because all of the dependency requests could be proceeding in parallel), and should be converted to:

results = await MultiGet(Get(Y, X, x) for x in some_collection)

We should examine existing usecases, and if there are no strongly motivating examples for

await-in-for

, consider making it an error. pantsbuild/pants

#2067 Release 2.1.125 Issue created by jsirois On the docket: ☑︎ Support attempts to build sdists for foreign platforms. #2073 pantsbuild/pex

#18368 Visibility rules: Support matching on target name. Issue created by kaos

Support matching on target name.

_Originally posted by @kaos in #18093 (comment)_ This also implies that the syntax is the same for both selectors and rules. pantsbuild/pants

#16727 `./pants --loop ...` with failing command continuously reruns, forever Issue created by huonw Describe the bug When running

./pants --loop ...

with a command that has an error, pants will run and rerun the command forever. Examples of failing commands: •

./pants --loop fmt ::

with black configured when there's a Python source file with invalid syntax (spelled out below) • any pants command that needs to run a `shell_command`/`adhoc_tool` with exit code != 0 #18275 • running any goal on a target that doesn't exist #18276 • running any goal when a BUILD file is invalid #18277 Reproducer of the

black

example (see linked issues for other simpler reproducers): https://gist.github.com/huonw/55bf0340a467881dc8511346f5ad2d5a has a single bad file

broken.py

. Running:

git clone <mailto:git@gist.github.com|git@gist.github.com>:55bf0340a467881dc8511346f5ad2d5a.git
cd 55bf0340a467881dc8511346f5ad2d5a
./pants --loop fmt ::

This gives endless copies of the black error message:

09:17:34.49 [ERROR] 1 Exception encountered:

  ProcessExecutionFailure: Process 'Run Black on 1 file.' failed with exit code 123.
stdout:

stderr:
error: cannot format broken.py: Cannot parse: 1:8: invalid formatting

Oh no! 💥 💔 💥
1 file failed to reformat.



Use `--keep-sandboxes=on_failure` to preserve the process chroot for inspection.

09:17:34.59 [ERROR] 1 Exception encountered:

  ProcessExecutionFailure: Process 'Run Black on 1 file.' failed with exit code 123.
stdout:

stderr:
error: cannot format broken.py: Cannot parse: 1:8: invalid formatting

Oh no! 💥 💔 💥
1 file failed to reformat.



Use `--keep-sandboxes=on_failure` to preserve the process chroot for inspection.

etc. Pants version • 2.12.0 • 2.14.0a0 • 2.16.0.dev6 OS macOS Additional info N/A pantsbuild/pants

#2069 Failed to resolve compatible distributions: `Sphinx==6.1.3` requires `docutils<0.20,>=0.18` but `docutils 0.16` was resolved Issue created by NiklasRosenstein Hi! I'm trying to build a Pex of an application that has

s3

as a dependency. But when I try that, I get the following error:

$ pex ./ -o snapper-backups.pex -e snapper_backups.main
Failed to resolve compatible distributions:
1: Sphinx==6.1.3 requires docutils<0.20,>=0.18 but docutils 0.16 was resolved

The issue can be more easily reproduced by simply trying to Pexify the

s3

[1] package:

$ pex s3
Failed to resolve compatible distributions:
1: Sphinx==6.1.3 requires docutils<0.20,>=0.18 but docutils 0.16 was resolved

[1] The version of

s3

that is being used here is

3.0.0

. This appears to be a bug with Pex because I can properly install

s3

with Pip. pantsbuild/pex

#2073 Support attempts to build sdists for foreign platforms. Issue created by jsirois At least in the

pex --lock

case, Pex has a natural crease to attempt to build sdists into wheels, even for foreign platforms. In the "yolo" case, it can turn out the wheel does build successfully and is compatible with the foreign platform. This would allow the maximum possible set of platform and complete_platform builds to succeed. Only those that truly required pre-building the sdist into a wheel on the foreign platform would fail. The standard (old) non-lockfile

--platform

resolve relies on Pip which rejects sdists and fails the

pip download

. That case can only be fixed by using

--complete-platform

instead, but that is the direction folks should move anyhow. pantsbuild/pex

#15698 export-codegen only exports Go, but not Python files Issue created by isingh Describe the bug I am trying out using python and go in the same repo. I setup a directory with proto definitions for gRPC service. Both python and Go can use these definitions in tests right now. However when I run

./pants export-codegen ::

it only exports Go files and not python. Go 1.18 and Python 3.9. Pants version 2.11 OS MacOS Additional info Sample

pants.toml

[GLOBAL]
pants_version = "2.11.0"
backend_packages = [
  "pants.backend.python",
  "pants.backend.python.lint.black",
  "pants.backend.codegen.protobuf.python",
  "pants.backend.experimental.codegen.protobuf.go",
  "pants.backend.experimental.go",
]

[golang]
expected_version = "1.18"

[source]
root_patterns = [
  '/py',
  '/protos',
  '/pkg',
]

All python code is in

/py

, grpc in

/protos

and go in

/pkg

.

/protos/BUILD

contains:

rotobuf_sources(
    name="protos",
    grpc=True,
)

python_requirements(
    name="requirements.txt",
)

python_requirement(name="grpc", requirements=["grpcio>=1.46.0"], modules=["grpc"])

protos/requirements.txt

contains:

grpcio>=1.46.0
protobuf~=3.19.0

If I run

./pants test ::

both python and go tests are able to compile the rpc protobufs and actually test (I havent tried talking between go and py yet) pantsbuild/pants

#18375 Dependency inference fails for `shellcheck` without output Issue created by kaos Describe the bug

14:05:37.08 [ERROR] Parsing build/scripts/dev_container.sh for dependency inference failed because Shellcheck's output could not be loaded as JSON. Please open a GitHub issue at <https://github.com/pantsbuild/pants/issues/new> with this error message attached.


shellcheck version: v0.8.0
process_result.stdout:

Pants version

2.16.0.dev5

OS

Mac M1

Additional info The script in question is simple, i.e. without any dependencies.

# cat build/scripts/dev_container.sh
set -euxf pipefail

if [ ! -f pants ]; then
    echo "Run this from the repo root: ./build/scripts/dev_container.sh"
    exit 1
fi

: ${IMG_NAME:=<http://ghcr.io/x-deploy-image|ghcr.io/x-deploy-image>}
: ${IMG_TAG:=@sha256:...}
: ${IMG:=$IMG_NAME$IMG_TAG}
: ${DEV_HOME:=${HOME}/.cache/x}

mkdir -p ${DEV_HOME}
cat << EOF > ${DEV_HOME}/.gitconfig
[safe]
        directory = /src
EOF

docker run \
    --platform linux/amd64 \
    -it \
    --rm \
    --network=host \
    -w /src \
    -v $(pwd):/src \
    -u $(id -u):$(id -g) \
    -v ${DEV_HOME}:/home/${USER} \
    -e HOME=/home/${USER} \
    -e USER \
    -e PANTS_WATCH_FILESYSTEM=false \
    -e PANTS_LOCAL_CACHE=false \
    -e PANTS_PANTSD=false \
    -e PANTS_LOCAL_STORE_PROCESSES_MAX_SIZE_BYTES=1600000000 \
    -e PANTS_LOCAL_STORE_FILES_MAX_SIZE_BYTES=2560000000 \
    -e PANTS_LOCAL_STORE_DIRECTORIES_MAX_SIZE_BYTES=1600000000 \
    ${IMG}

pantsbuild/pants

#18379 Including directories in docker_image Issue created by MichalOleszak Describe the bug I have the following repository structure with Pants root set to repo's root, `"/"`:

/
  - pants.toml
  - project
      - requirements.txt
      - dirs_with_python_code
      - sub_dir
          - Dockerfile 
  - utils
      - dirs_with_python_code

Dockerfile looks like this:

FROM ......
ENV PYTHONPATH .
WORKDIR /app
COPY project project
COPY utils utils
RUN pip3 install -r project/requirements.txt

My goal is to build a docker image that would include the

utils

and

project

directories, and whose root corresponds to my repository's root, so that I can install requirement as done in the dockerfile, and so that python scripts in the container can e.g.

from utils import ...

. This I can easily do without Pants, by simply running

docker build

from the repo's root. However, I'm not able to achieve the same with the

docker_image

Pants target. Here is what I have already tried and how I failed. 1. I have defined my

docker_image

in a BULID file next to the Dockerfile. I've tried setting different values for

context_root

unsure which one would correspond to the repo's root:

"."

,

"./"

,

""

. I have set

dependencies=["project:project", "utils@resolve=project"]

. Outcome of running `pants package`: the COPY instruction failed stating that project and utils don't exist. 2. Suspecting the context may not be set to the root, I've moved my

docker_image

target to a BUILD file in the root dir and modified its

source

accordingly. The effect was still the same. 3. Instead of copying things manually via a Dockerfile, I decided to try using a pex binary. In project's BUILD, I've added the following:

pex_binary(
    name="project_binary",
    dependencies=[":project", "utils:utils"],
    resolve="project",
    layout="packed",
    execution_mode="venv",
)

docker_image(
    name="project_image",
    repository="project",
    instructions=[
        "FROM ......,
        "COPY project/project_binary.pex /bin/project",
        "RUN pip3 install -r project/requirements.txt",
    ]
)

Result: it failed on not finding the requirements file. 4. I have removed the RUN instruction from the Dockerfile which allowed by to successfully build an image. I ran the container to look inside and found that

bin/project/utils

only contains the

__init__.py

file and no other files that are inside the utils dir. The same for the project dir, that's why no requirements file was found. This corresponds to the fact that running

pants dependencies

on my Dockerfile or pex binary target yields:

project/__init__.py
utils/__init__.py@resolve=project

No other files except for the inits are included. Please advise me on what am I doing wrong! My goal is as stated above: just recreate with Pants what I can already do with

docker build

. Pants version 2.14.1 OS MacOS 12.5.1 with M1 chip EDIT I've created a minimum reproducible example which is much simpler and the same problem manifests in it. At my repository's root, I've crated a Dockerfile that copies everything into the image:

FROM alpine
WORKDIR /app
COPY . .

Also at the root, I have the following BUILD file:

docker_image(
    name="root_image",
)

Then, I run:

pants package :root_image

docker run -it root_image:latest

Inside the container:

ls

This only shows a Dockerfile, no other files or directories are included in the image. I try adding different

dependencies

to

docker_image

, but whatever I pass there, no other files or directories are included in the docker image. On the contrary, when I run

docker build . -f Dockerfile -t root_image:latest

from the repo's root, I obtain an image that has all the repo content in it. I would like to recreate this with Pants. pantsbuild/pants

Comment on #18353 How to run streamlit apps in pants? Discussion answered by kaos stripped down answer of https://pantsbuild.slack.com/archives/C046T6T9U/p1677251089751089?thread_ts=1677247987.213439&cid=C046T6T9U Using Pants

<2.16.0.dev5

# some/BUILD
pex_binary(name="bin", script="streamlit", dependencies=[<streamlit app>])

$ pants run some:bin -- run <streamlit app>

Using Pants

>=2.16.0.dev5

# some/BUILD
pex_binary(name="bin", script="streamlit", args=["run", <streamlit app>], dependencies=[<streamlit app>])

$ pants run some:bin

pantsbuild/pants