#18629 Support optional 3rd party dependencies in helm Issue created by jscarrott Is your feature request related to a problem? Please describe. I am generating a helm chart that would ordinarily have chart dependencies that are optional based on a condition. The method of defining helm artifacts doesn't seem to allow for this. Describe the solution you'd like Preferably defining a 3rd party helm artifact would simply allow it to be used manually inside the chart.yaml instead of injecting the chart automatically (or at least have the option) so that you could manually configure the chart yaml as you would like. Describe alternatives you've considered I think I could make proxy 1st party chart that references the 3rd party chart and then make that optional but that is very awkward. Additional context Add any other context or screenshots about the feature request here. pantsbuild/pants

#18630 Add a configurable timeout to pantsd after some amount of no-usage Issue created by sureshjoshi While trying to figure out why I seemingly had less memory available than I should (given that most of the applications on my computer were closed), I saw this as my top memory usage: This is strange since I hadn't used or run a Pants command in these repos in over a week or two, so imagine my surprise that 7 of my top 9 processes were Pants-related. Each of them (the Pants and Python) appeared to be averaging 1.5-2% CPU as well. I'm sure that memory is all recoverable as needed, but I think having those processes stay live while doing nothing (for weeks) is a bit overkill. What is surprising is that I run processes that routinely take up all available memory for short time periods, so I thought that would inevitably cause

pantsd

to shut down due to memory pressure. pantsbuild/pants

#18631 Allow requesting multiple runs per test Issue created by jtilahun Is your feature request related to a problem? Please describe. It would be great if the Pants

test

goal had an option to do more than one run per test. For comparison, in Bazel, the

bazel test

command allows one to specify

--cache_test_results=no --runs_per_test=<n>

in order to accomplish this. I haven't found an equivalent option in Pants though. Describe the solution you'd like I would like an option similar to

--runs_per_test

to be available as part of the Pants

test

goal so that one can request multiple runs per test. Describe alternatives you've considered I've considered the

pytest-repeat

plugin for Pytest (https://pypi.org/project/pytest-repeat/). However, according to the project description page, the plugin does not support

unittest.TestCase

test classes. Additional context The context is that one may want to do multiple runs per test in order to more readily expose flaky unit tests. I want to do periodic runs in CI where all unit tests are reran forcefully (i.e., with the

--force

option). I would also like to do more than run per test. I don't know of a supported method to do multiple runs per test directly through Pants though. pantsbuild/pants

#16778 Nondeterministic `ImportError: No module named __pants_df_parser` while computing changed targets Issue created by danxmoran (NOTE: There's been a lot of discussion on this in Slack here, but I realized I never filed an issue for it) Describe the bug A few times per day (out of ~hundreds of jobs), our CI jobs will fail when trying to build

dockerfile_parser.pex

. The error always hits at the beginning of goal execution, and affects all our goals (

test

,

lint

,

check

). The error looks like:

13:28:44.08 [INFO] Starting: Resolving plugins: toolchain.pants.plugin==0.20.0
13:28:56.61 [INFO] Completed: Resolving plugins: toolchain.pants.plugin==0.20.0
13:28:57.16 [INFO] [auth-plugin] Successfully loaded Toolchain token from env var 'TOOLCHAIN_AUTH_TOKEN', expiration: 2022-12-20T20:52:09+00:00.
13:29:17.25 [INFO] Starting: Building dockerfile_parser.pex from dockerfile-parser_default.lock
13:29:18.32 [INFO] Canceled: Building dockerfile_parser.pex from dockerfile-parser_default.lock
Exception caught: (pants.engine.internals.scheduler.ExecutionError)
  File "/tmp/pants_setup/bootstrap-Linux-x86_64/2.13.0a1+gitb75932fb_py38/bin/pants", line 8, in <module>
    sys.exit(main())
  File "/tmp/pants_setup/bootstrap-Linux-x86_64/2.13.0a1+gitb75932fb_py38/lib/python3.8/site-packages/pants/bin/pants_loader.py", line 115, in main
    PantsLoader.main()
  File "/tmp/pants_setup/bootstrap-Linux-x86_64/2.13.0a1+gitb75932fb_py38/lib/python3.8/site-packages/pants/bin/pants_loader.py", line 111, in main
    cls.run_default_entrypoint()
  File "/tmp/pants_setup/bootstrap-Linux-x86_64/2.13.0a1+gitb75932fb_py38/lib/python3.8/site-packages/pants/bin/pants_loader.py", line 93, in run_default_entrypoint
    exit_code = runner.run(start_time)
  File "/tmp/pants_setup/bootstrap-Linux-x86_64/2.13.0a1+gitb75932fb_py38/lib/python3.8/site-packages/pants/bin/pants_runner.py", line 99, in run
    runner = LocalPantsRunner.create(
  File "/tmp/pants_setup/bootstrap-Linux-x86_64/2.13.0a1+gitb75932fb_py38/lib/python3.8/site-packages/pants/bin/local_pants_runner.py", line 159, in create
    specs = calculate_specs(
  File "/tmp/pants_setup/bootstrap-Linux-x86_64/2.13.0a1+gitb75932fb_py38/lib/python3.8/site-packages/pants/init/specs_calculator.py", line 94, in calculate_specs
    (changed_addresses,) = session.product_request(
  File "/tmp/pants_setup/bootstrap-Linux-x86_64/2.13.0a1+gitb75932fb_py38/lib/python3.8/site-packages/pants/engine/internals/scheduler.py", line 577, in product_request
    return self.execute(request)
  File "/tmp/pants_setup/bootstrap-Linux-x86_64/2.13.0a1+gitb75932fb_py38/lib/python3.8/site-packages/pants/engine/internals/scheduler.py", line 521, in execute
    self._raise_on_error([t for _, t in throws])
  File "/tmp/pants_setup/bootstrap-Linux-x86_64/2.13.0a1+gitb75932fb_py38/lib/python3.8/site-packages/pants/engine/internals/scheduler.py", line 498, in _raise_on_error
    raise ExecutionError(
Exception message: 1 Exception encountered:
Engine traceback:
  in select
  in pants.vcs.changed.find_changed_owners
  in pants.backend.project_info.dependees.find_dependees
  in pants.backend.project_info.dependees.map_addresses_to_dependees
  in pants.engine.internals.graph.resolve_dependencies (src/projects/rhp/operations:rhp-api)
  in pants.backend.docker.util_rules.dependencies.inject_docker_dependencies (src/projects/rhp/operations:rhp-api)
  in pants.backend.docker.subsystems.dockerfile_parser.parse_dockerfile
  in pants.engine.process.fallible_to_exec_result_or_raise
Traceback (most recent call last):
  File "/tmp/pants_setup/bootstrap-Linux-x86_64/2.13.0a1+gitb75932fb_py38/lib/python3.8/site-packages/pants/engine/process.py", line 272, in fallible_to_exec_result_or_raise
    raise ProcessExecutionFailure(
pants.engine.process.ProcessExecutionFailure: Process 'Parse Dockerfile.' failed with exit code 1.
stdout:
stderr:
Traceback (most recent call last):
  File "/tmp/pants-sandbox-kVePcL/.cache/pex_root/venvs/e2b00a6620ed72ee63184ca7a64555f5551da8c5/f679563b742633f5edaba61ca7d405616ffc49b3/pex", line 234, in <module>
    runpy.run_module(module_name, run_name="__main__", alter_sys=True)
  File "/opt/python/3.8.12/lib/python3.8/runpy.py", line 203, in run_module
    mod_name, mod_spec, code = _get_module_details(mod_name)
  File "/opt/python/3.8.12/lib/python3.8/runpy.py", line 138, in _get_module_details
    raise error("No module named %s" % mod_name)
ImportError: No module named __pants_df_parser

Pants version • The error started appearing on

PANTS_SHA=5d8a328d72209863986c8959b20305505bc068ba

, and has continued appearing as we've upgraded along the

2.13.x

branch. • As far as I can tell, the error was not appearing on

v2.13.0a0

OS Linux Additional info Through debugging on Slack, we've found: 1.

__pants_df_parser.py

truly isn't in the venv when this error occurs 2. The error doesn't occur if we use

--no-process-cleanup

3. The sandbox PEX created for the dockerfile parser contains the file, even when the pex_root venv does not. Running the PEX creates a venv that contains the file. pantsbuild/pants

#16589 `test_sigint_kills_request_waiting_for_lock` from `pantsd_integration_test.py` is flaky Issue created by Eric-Arellano https://github.com/pantsbuild/pants/runs/7920622325?check_suite_focus=true#step:13:414 pantsbuild/pants

#2110 Support equivalent of `pip install --target=...` to export distributions and source to flat directory Issue created by huonw

pip install

has a `--target`/`-t` option that installs the requested requirements directly to a specific directory, rather than into the usual structure with

site-packages

etc. It would be nice if pexes could support this too. Currently, there's two related

PEX_TOOLS=1

commands, but neither quite do exactly this: • `PEX_TOOLS=1 ./some-pex venv ...`: constructs the full venv structure • `PEX_TOOLS=1 ./some-pex repository extract ...`: only exports the distributions, not the source Motivation: Direct installation like this is handy for building artefacts for certain cases, such as FaaS platforms like Lambda: their docs recommend just splatting all dependencies into the top level of a zip file, along with whatever first party code: https://docs.aws.amazon.com/lambda/latest/dg/python-package.html#python-package-create-package-with-dependency (steps 4-6). More discussion: https://pantsbuild.slack.com/archives/C046T6T9U/p1680247328906249 pantsbuild/pex

#14793 __run.sh should strip env variables Issue created by Eric-Arellano I spent a lot of time debugging why something doesn't work with Pants, because it strips

SSL_CERT_FILE

. The

__run.sh

script does not strip though, it only sets what Pants actively sets. pantsbuild/pants

#18657 deploy pex binary on AWS lambda docker without having to unzip lambdex-zip archive Issue created by mister-rao pex is cool, my binary size for a medium size django project is around 60MB. I am using AWS container lambda to deploy pex. This works, but I have to unzip the lambdex archive which shoots up deps size to 200MB+. Is there a way to run pex directly on the lambda container, without unzipping it? pantsbuild/pants

#18658 `pants lint` gives no output when using `ruff` Issue created by cedric-fauth Describe the bug I enabled ruff as a linter and tried to lint a file. Ruff fails as expected but I don't get any information why it failed. When I run ruff (not the pants integration) I get:

file.py:1:1: D100 Missing docstring in public module
file.py:1:1: I001 [*] Import block is un-sorted or un-formatted
file.py:4:5: ANN201 Missing return type annotation for public function `hello`
file.py:4:5: D103 Missing docstring in public function

When I run

pants lint file.py

I should get the same output or? Instead I only get

ruff failed

. Pants version 2.16.0a1 OS WSL pantsbuild/pants

#18659 make it easier to interpolate artifact names Issue created by cburroughs Is your feature request related to a problem? Please describe. Right now it is easy to tell pants to "package all the things" with

pants package ::

. This fills up

dist

with all of the goodies. However, instead of having

foo.tar.gz

I'd like to have

foo-VERSION.tar.gz

so that everything is ready for uploading publishing. (Where version is maybe a constant like

v.1.2.3

or just the git sha.)

output_path

can almost do that, but then I have to re-implement the

/ --> .

logic.

output_path
    type: str | None
    default: None

    Where the built asset should be located.
    
    If undefined, this will use the path to the BUILD file, followed by the target name. For example, `src/python/project:app`
    would be `src.python.project/app.ext`.
    
    When running `pants package`, this path will be prefixed by `--distdir` (e.g. `dist/`).
    
    Warning: setting this value risks naming collisions with other package targets you may have.

Describe the solution you'd like Give all "just a file" artifacts an

output_name

to go along with

output_path

, the "path to output" logic then looks something like: • default:

build_file_dir.replace('/', '.') + name

• if

output_name

is set:

build_file_dir.replace('/', '.') + output_name

• if

output_path

is set: `output_Path Describe alternatives you've considered • set the target name to something dynamic like "foo + git_sha". Quick and dirty but then the name changes on every commit. • Write a

versioned_tarball

macro. Additional context Doing it with a macro isn't that bad. But I feel like there is some fundamental inconsistency where some "packages" like docker or python_distributions have a separate "version", but others do not. spinoff from: https://pantsbuild.slack.com/archives/C0D7TNJHL/p1680240146011409 pantsbuild/pants

#18660 Third-party library missing from `mpy` sandbox when using `[pex].venv_use_symlinks = true` Issue created by danxmoran Describe the bug In a minimized(-ish) example: `pants.toml`:

[GLOBAL]
pants_version = "2.16.0rc0"
backend_packages = [
    "pants.backend.python",
    "pants.backend.python.typecheck.mypy",
]

[source]
root_patterns = ["/"]

[python]
interpreter_constraints = ["==3.8.*"]
enable_resolves = true
default_resolve = "default"

[python.resolves]
default = "default.lockfile"

`requirements.txt`:

opentelemetry-exporter-otlp-proto-http==1.15.0
opentelemetry-instrumentation-grpc==0.36b0
opentelemetry-sdk==1.15.0

`default.lockfile`: ``` // This lockfile was autogenerated by Pants. To regenerate, run: // // pants generate-lockfiles --resolve=default // // --- BEGIN PANTS LOCKFILE METADATA: DO NOT EDIT OR REMOVE --- // { // "version": 3, // "valid_for_interpreter_constraints": [ // "CPython==3.8.*" // ], // "generated_with_requirements": [ // "opentelemetry-exporter-otlp-proto-http==1.15.0", // "opentelemetry-instrumentation-grpc==0.36b0", // "opentelemetry-sdk==1.15.0" // ], // "manylinux": "manylinux2014", // "requirement_constraints": [], // "only_binary": [], // "no_binary": [] // } // --- END PANTS LOCKFILE METADATA --- { "allow_builds": true, "allow_prereleases": false, "allow_wheels": true, "build_isolation": true, "constraints": [], "locked_resolves": [ { "locked_requirements": [ { "artifacts": [ { "algorithm": "sha256", "hash": "63579f9a0628e06278f7e47b7d7d5b6ce20dc65c5e96a6f3ca99a6adca0396e8", "url": "https://files.pythonhosted.org/packages/df/73/b6e24bd22e6720ca8ee9a85a0c4a2971af8497d8f3193fa05390cbd46e09/backoff-2.2.1-py3-none-any.whl" }, { "algorithm": "sha256", "hash": "03f829f5bb1923180821643f8753b0502c3b682293992485b0eef2807afa5cba", "url": "https://files.pythonhosted.org/packages/47/d7/5bbeb12c44d7c4f2fb5b56abce497eb5ed9f34d85701de869acedd602619/backoff-2.2.1.tar.gz" } ], "project_name": "backoff", "requires_dists": [], "requires_python": "<4.0,>=3.7", "version": "2.2.1" }, { "artifacts": [ { "algorithm": "sha256", "hash": "4ad3232f5e926d6718ec31cfc1fcadfde020920e278684144551c91769c7bc18", "url": "https://files.pythonhosted.org/packages/71/4c/3db2b8021bd6f2f0ceb0e088d6b2d49147671f25832fb17970e9b583d742/certifi-2022.12.7-py3-none-any.whl" }, { "algorithm": "sha256", "hash": "35824b4c3a97115964b408844d64aa14db1cc518f6562e8d7261699d1350a9e3", "url": "https://files.pythonhosted.org/packages/37/f7/2b1b0ec44fdc30a3d31dfebe52226be9ddc40cd6c0f34ffc8923ba423b69/certifi-2022.12.7.tar.gz" } ], "project_name": "certifi", "requires_dists": [], "requires_python": ">=3.6", "version": "2022.12.7" }, { "artifacts": [ { "algorithm": "sha256", "hash": "3d9098b479e78c85080c98e1e35ff40b4a31d8953102bb0fd7d1b6f8a2111a3d", "url": "https://files.pythonhosted.org/packages/ef/81/14b3b8f01ddaddad6cdec97f2f599aa2fa466bd5ee9af99b08b7713ccd29/charset_normalizer-3.1.0-py3-none-any.whl" }, { "algorithm": "sha256", "hash": "3573d376454d956553c356df45bb824262c397c6e26ce43e8203c4c540ee0acb", "url": "https://files.pythonhosted.org/packages/01/c7/0407de35b70525dba2a58a2724a525cf882ee76c3d2171d834463c5d2881/charset_normalizer-3.1.0-cp38-cp38-manylinux_2_17_aarch64.manylinux2014_aarch64.whl" }, { "algorithm": "sha256", "hash": "f645caaf0008bacf349875a974220f1f1da349c5dbe7c4ec93048cdc785a3326", "url": "https://files.pythonhosted.org/packages/13/b7/21729a6d512246aa0bb872b90aea0d9fcd1b293762cdb1d1d33c01140074/charset_normalizer-3.1.0-cp38-cp38-musllinux_1_1_aarch64.whl" }, { "algorithm": "sha256", "hash": "1c60b9c202d00052183c9be85e5eaf18a4ada0a47d188a83c8f5c5b23252f649", "url": "https://files.pythonhosted.org/packages/2c/2f/ec805104098085728b7cb610deede7195c6fa59f51942422f02cc427b6f6/charset_normalizer-3.1.0-cp38-cp38-manylinux_2_5_i686.manylinux1_i686.manylinux_2_17_i686.manylinux2014_i686.whl" }, { "algorithm": "sha256", "hash": "80d1543d58bd3d6c271b66abf454d437a438dff01c3e62fdbcd68f2a11310d4b", "url": "https://files.pythonhosted.org/packages/31/8b/81c3515a69d06b501fcce69506af57a7a19bd9f42cabd1a667b1b40f2c55/charset_normalizer-3.1.0-cp38-cp38-musllinux_1_1_ppc64le.whl" }, { "algorithm": "sha256", "hash": "ea9f9c6034ea2d93d9147818f17c2a0860d41b71c38b9ce4d55f21b6f9165a11", "url": "https://files.pythonhosted.org/packages/33/10/c87ba15f779f8251ae55fa147631339cd91e7af51c3c133d2687c6e41800/charset_normalizer-3.1.0-cp38-cp38-musllinux_1_1_i686.whl" }, { "algorithm": "sha256", "hash": "e89df2958e5159b811af9ff0f92614dabf4ff617c03a4c1c6ff53bf1c399e0e1", "url": "https://files.pythonhosted.org/packages/67/30/dbab1fe5ab2ce5d3d517ad9936170d896e9687f3860a092519f1fe359812/charset_normalizer-3.1.0-cp38-cp38-manylinux_2_17_ppc64le.manylinux2014_ppc64le.whl" }, { "algorithm": "sha256", "hash": "73dc03a6a7e30b7edc5b01b601e53e7fc924b04e1835e8e407c12c037e81adbd", "url": "https://files.pythonhosted.org/packages/68/77/af702eba147ba963b27eb00832cef6b8c4cb9fcf7404a476993876434b93/charset_normalizer-3.1.0-cp38-cp38-musllinux_1_1_s390x.whl" }, { "algorithm": "sha256", "hash": "de5695a6f1d8340b12a5d6d4484290ee74d61e467c39ff03b39e30df62cf83a0", "url": "https://files.pythonhosted.org/packages/74/5f/361202de730532028458b729781b8435f320e31a622c27f30e25eec80513/charset_normalizer-3.1.0-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl" }, { "algorithm": "sha256", "hash": "3a06f32c9634a8705f4ca9946d667609f52cf130d5548881401f1eb2c39b1e2c", "url": "https://files.pythonhosted.org/packages/ac/c5/990bc41a98b7fa2677c665737fdf278bb74ad4b199c56b6b564b3d4cbfc5/charset_normalizer-3.1.0-cp38-cp38-macosx_10_9_x86_64.whl" }, { "algorithm": "sha256", "hash": "6f5c2e7bc8a4bf7c426599765b1bd33217ec84023033672c1e9a8b35eaeaaaf8", "url": "https://files.pythonhosted.org/packages/b0/55/d8ef4c8c7d2a8b3a16e7d9b03c59475c2ee96a0e0c90b14c99faaac0ee3b/charset_normalizer-3.1.0-cp38-cp38-musllinux_1_1_x86_64.whl" }, { "algorithm": "sha256", "hash": "7381c66e0561c5757ffe616af869b916c8b4e42b367ab29fedc98481d1e74e14", "url": "https://files.pythonhosted.org/packages/d8/ca/a7ff600781bf1e5f702ba26bb82f2ba1d3a873a3f8ad73cc44c79dfaefa9/charset_normalizer-3.1.0-cp38-cp38-macosx_11_0_arm64.whl" }, { "algorithm": "sha256", "hash": "e633940f28c1e913615fd624fcdd72fdba807bf53ea6925d6a588e84e1151531", "url": "https://files.pythonhosted.org/packages/ea/38/d31c7906c4be13060c1a5034087966774ef33ab57ff2eee76d71265173c3/charset_normalizer-3.1.0-cp38-cp38-macosx_10_9_universal2.whl" }, { "algorithm": "sha256", "hash": "78cacd03e79d009d95635e7d6ff12c21eb89b894c354bd2b2ed0b4763373693b", "url": "<https://files.pythonhosted.org/packages/f8/ed/500609cb2457b002242b090c814549997424d72690ef3058cfdfca91f68b/charset_normalizer-… pantsbuild/pants

#18662 Wrong version of Python used to build `pex_binary` targets in `2.16.0rc0` Issue created by danxmoran Describe the bug • Our CI image contains both Python 3.8 and 3.9. • We set

[python].interpreter_constraints = ["==3.8.*"]

in

pants.toml

. • At least one

pex_binary

depends on a version of

numpy

that (for reasons we haven't dug into) only works with Python 3.8, not Python 3.9 • We haven't investigated the build failures because we expect everything to run against Python 3.8 as configured by

[python].interpreter_constraints

After upgrading to Pants 2.16.0rc0 we see failures building the

pex_binary

in CI, with errors that indicate the build process is trying to build a dependency (

numpy

) against Python 3.9 instead of the expected/configured Python 3.8 This is very concerning. We still run Python 3.8 everywhere in production, so I don't want Pexes to be building against Python 3.9. I've downgraded us back to 2.16.0a1 for now and confirmed this fixes the problem. Pants version 2.16.0rc0 OS Linux Additional info I am suspicious of d3d3257 and e8d387b pantsbuild/pants

#18664 pantsd_integration_test.py fails intermittently but frequently in CI Issue created by benjyw One or more of these tests frequently fail in CI, for poorly-understood reasons. E.g., https://github.com/pantsbuild/pants/actions/runs/4591012030/jobs/8106925253 https://github.com/pantsbuild/pants/actions/runs/4586835505/jobs/8104870643 For now the tests are marked as skipped in #18665. pantsbuild/pants

#18667 Nested directories which match source roots can lead to slightly confusing behaviour Issue created by cbwl It would be helpful if pants could warn when package structures contain directories which match with source roots. I recently implemented a plugin under a package called something like

myorg.pants.plugins.myplugin

. The plugin worked fine but various pants goals (

lint

,

test

, etc.) failed with pants errors along the lines of:

❯ pants test ::
......
Engine traceback:
  in `test` goal
  in Run Pytest - pants/plugins/myorg/pants/plugins/myplugin/foo_test.py:tests
  in Resolve transitive targets
  in Resolve direct dependencies of target - pants/plugins/myorg/pants/plugins/myplugin/foo_test.py:tests
  in Inferring Python dependencies by analyzing source

UnownedDependencyError: Pants cannot infer owners for the following imports in the target pants/plugins/myorg/pants/plugins/myplugin/foo_test.py:tests:

  * myorg.pants.plugins.myplugin.foo.FooClass (line: 11)

If you do not expect an import to be inferrable, add `# pants: no-infer-dep` to the import line. Otherwise, see <https://www.pantsbuild.org/v2.15/docs/troubleshooting#import-errors-and-missing-dependencies> for common problems.

It took a bit of poking around to realise that the problem here was that a part of my package name (the

pants.plugins

part) was identical in directory structure to the source root in which my plugin was defined (the top-level directory

/pants/plugins

). Pants is fine with this if source roots are well-defined, but in my case my

pants.toml

had this incorrect configuration:

[source]
root_patterns = [
  "pants/plugins", # missing leading slash
  ...
]

The problem here was resolved by adding the leading slash to my

root_pattern

so that Pants no longer (quite reasonably) thought that the directory

/pants/plugins/myorg/pants/plugins

was, in fact a source root. It would be convenient if pants could report a warning about cases where multiple nested paths both correspond to a

root_pattern

. That would have immediately pointed to the issue here and helped me more quickly arrive at the solution of replacing my

root_pattern

with

"/pants/plugins"

. I don't think that any change in behaviour around treatment of

root_patterns

or dependency inference is strictly needed beyond that warning. Related slack posts: • https://pantsbuild.slack.com/archives/C01CQHVDMMW/p1680281281374909 • https://pantsbuild.slack.com/archives/C046T6T9U/p1680518068155799 pantsbuild/pants

#15935 Support other top-level gitignore locations Issue created by rshin Is your feature request related to a problem? Please describe. Pants uses patterns in the

.gitignore

to find additional files to ignore: https://www.pantsbuild.org/docs/reference-global#section-pants-ignore-use-gitignore However, Git itself also looks at

$GIT_DIR/info/exclude

(usually

.git/info/exclude

) and

$XDG_CONFIG_DIR/git/ignore

(usually

~/.config/git/ignore

, unless

core.excludeFiles

in

~/.gitconfig

is set differently) to get additional ignore patterns: https://git-scm.com/docs/gitignore. Describe the solution you'd like It would be good if Pants supported these locations too. This request is similar to #5682 but potentially simpler, as the interpretation of the patterns in these other ignore files is the same (they are relative to the repository root rather than some other location) and there is a fixed number of them. Describe alternatives you've considered You can replicate the additional ignore patterns in

.pants.rc

pantsbuild/pants

#17342 Simplify our APIs to leverage `Get()` with no arguments Issue created by Eric-Arellano We have some types like

AllTargetsRequest

and

ZipBinaryRequest

that solely existed to work around the lack of #12946. That is, it allowed us to "lazily" evaluate things, e.g. inside if statements, by doing

await Get(ZipBinary, ZipBinaryRequest)

. Now that we can do

Get(ZipBinary, {})

, we should deprecate and then delete those old APIs. We only need the "singleton" rules that return

ZipBinary

without needing to have a

ZipBinaryRequest

. This change will make for less code and a tighter rule graph.

we should deprecate

We can do that by putting

warn_or_error()

inside the rules for

ZipBinaryRequest

et al. pantsbuild/pants

#2111 Allow pex venv to export dependencies/first-party code for platforms other than the current one Issue created by huonw Currently tools subcommands like

PEX_TOOLS=1 ./some.pex venv ...

work with the current OS/python interpreter as the target of interest, and give information/exports based on how the PEX file would run in this environment (potentially an error). https://gist.github.com/huonw/adbd1d685f595d10780e5f878a5245cd It would be convenient if it was possible for a venv or flat layout (#2110) to be exported for other targets, as this would make it easier to, for instance, construct simple artefacts for running in FaaS platforms like AWS Lambda: 1. construct a PEX using a complete platforms JSON for the Lambda environment 2. export the PEX's first and third party code using the same complete platforms JSON for the Lambda environment 3. zip that up and upload it (https://docs.aws.amazon.com/lambda/latest/dg/python-package.html#python-package-create-package-with-dependency steps 4-6) Currently, step 2 needs to be run in a docker container with compatible tags, and this can be awkward (e.g. pants only recently got (experimental) support for docker environments) and slow (e.g. it may even require CPU emulation, like building an x86-64 lambda on arm64 macOS, or an arm64 lambda on x86-64 linux, or whatever other combination). (NB. even with this feature, step 1 may need to be run in a docker container, if there's any platform-specific sdists to build. This feature request is just about step 2, not changing step 1. IME, we can get a long long way without this: we deploy many lambdas using only bdists and platform-independent sdists.) More discussion: https://pantsbuild.slack.com/archives/C046T6T9U/p1680247328906249 pantsbuild/pex

#18669 `archive` is not reproducible Issue created by huonw Describe the bug The

archive

target's output can change at a byte level even when its inputs is identical. This appears to be due to timestamps changing, but there may be other factors too (e.g. file sort order, permissions, groups/owners). This reproducer runs two builds of .zip and .tar archives, using a fixed file on disk. This may be worse if generating inputs to

archive

too (e.g.

packages=[...]

or using the output of

shell_command

), but I haven't tested in detail. The

--no-pantsd --no-local-cache

args ensures that the archiving process definitely runs, as might happen when running on different machines (i.e. no shared cache).

cd $(mktemp -d)

cat > pants.toml <<EOF
[GLOBAL]
pants_version = "2.16.0rc0"

backend_packages = []

[anonymous-telemetry]
enabled = false
EOF

echo 1 > foo.txt

cat > BUILD <<EOF
file(name="file", source="foo.txt")
for format in ["zip", "tar"]:
    archive(name=format, format=format, files=[":file"])
EOF

pants --no-pantsd --no-local-cache package ::
cp -r dist first

sleep 3 # ensure the invocations are separated

pants --no-pantsd --no-local-cache package ::
cp -r dist second

# BUG: these are different
shasum first/* second/*

# BUG: for tar, it is specifically the (octal) timestamp that differs
diff -U3  <(hexdump -C first/tar.tar) <(hexdump -C second/tar.tar)

If I run that, I see (nondeterministic) output like:

9c0f0769e85cc4a6acc490f92ea32b79862a953a  first/tar.tar
669f4c618dac7c3590f392f9c5f23670731456d6  first/zip.zip
81b0b4dba3c99855642ce468500c89afa8c9f862  second/tar.tar
9be595741557454a409d243aa0bf84ac91f7b5e2  second/zip.zip

--- /dev/fd/11	2023-04-04 14:53:36.000000000 +1000
+++ /dev/fd/12	2023-04-04 14:53:36.000000000 +1000
@@ -4,7 +4,7 @@
 00000060  00 00 00 00 30 30 30 36  34 34 20 00 30 30 30 37  |....000644 .0007|
 00000070  36 35 20 00 30 30 30 30  32 34 20 00 30 30 30 30  |65 .000024 .0000|
 00000080  30 30 30 30 30 30 32 20  31 34 34 31 32 37 32 36  |0000002 14412726|
-00000090  35 30 30 20 30 31 32 35  30 37 00 20 30 00 00 00  |500 012507. 0...|
+00000090  35 31 30 20 30 31 32 35  31 30 00 20 30 00 00 00  |510 012510. 0...|
 000000a0  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
 *
 00000100  00 75 73 74 61 72 00 30  30 68 75 6f 6e 00 00 00  |.ustar.00huon...|

Pants version 2.16.0rc0 OS macOS Additional info N/A pantsbuild/pants

#18671 JVM resource jar creation is broken on Mac by variation in `touch` command. Issue created by jsirois BSD touch uses

-t

to set the timestamp whereas GNU touch uses

-d

. We use / assume the GNU binary as noted here: #16950 (comment) As discovered here: https://pantsbuild.slack.com/archives/C046T6T9U/p1680604327733559?thread_ts=1680604327.733559&amp;cid=C046T6T9U Where the error looks like:

12:24:56.74 [ERROR] 1 Exception encountered:
 
 Engine traceback:
   in select
   in pants.core.goals.check.check
   in pants.backend.scala.goals.check.scalac_check (scalac)
   in pants.backend.scala.compile.scalac.compile_scala_source
   in pants.jvm.compile.compile_classpath_entries
   in pants.jvm.resources.assemble_resources_jar
   in pants.engine.process.fallible_to_exec_result_or_raise
 Traceback (most recent call last):
   File "/Users/jbenito/.cache/pants/setup/bootstrap-Darwin-x86_64/pants.1Nnv7r/install/lib/python3.9/site-packages/pants/engine/process.py", line 275, in fallible_to_exec_result_or_raise
     raise ProcessExecutionFailure(
 pants.engine.process.ProcessExecutionFailure: Process 'Build resources JAR for sdk/transport-security-web-lib/src/test/resources:resources' failed with exit code 1.
 stdout:
 
 stderr:
 /usr/bin/touch: illegal option -- d
 usage:
 touch [-A [-][[hh]mm]SS] [-acfhm] [-r file] [-t [[CC]YY]MMDDhhmm[.SS]] file ...

It appears #16950 was cherry picked back to 2.13.1 and 2.14.0; so Pants has been broken for JVM resource jars since 2.13.1. pantsbuild/pants

#18683 /usr/bin/env: &lt;python-version&gt; No such file or directory when using adhoc_tool runnable is pex_binary Issue created by cburroughs Describe the bug

python_source(
    name='lib-hello',
    source='hello.py')

pex_binary(
    name="hi",
    entry_point="hello.py",
)

adhoc_tool(
    name="local-data",
    runnable=":hi",
    args=["--foo"],
    output_directories=["data"],
    log_output=True,
)

python_source(
    name='lib-world',
    source='world.py',
    dependencies=[':local-data']
)

Results in

$ pants --keep-sandboxes=on_failure run hello_pex/world.py 
17:07:27.22 [INFO] Preserving local process execution dir /tmp/pants-sandbox-LcNOLX for Running the `adhoc_tool` at hello_pex:local-data
17:07:27.22 [INFO] Completed: Running the `adhoc_tool` at hello_pex:local-data
17:07:27.22 [ERROR] 1 Exception encountered:

Engine traceback:
  in `run` goal

ProcessExecutionFailure: Process 'the `adhoc_tool` at hello_pex:local-data' failed with exit code 127.
stdout:

stderr:
/usr/bin/env: 'python3.9': No such file or directory

Confusingly, the

__run.sh

from the sandbox runs without error when invoked directly. so I'm not sure what is going on here. Pants version 2.16.0a0, 2.16.0rc0 OS Linux Additional info https://github.com/cburroughs/example-adhoc/tree/ad-hoc-pex pantsbuild/pants

#18684 vcs_version doesn't work with go_package Issue created by SergeBakharev Describe the bug go_package doesn't load codegen'd version files from vcs_version. Pants version 2.15.0 OS Both Additional info A repo with a minimal example to demonstrate the issue is available here https://github.com/SergeBakharev/pants-go-vcs_version pantsbuild/pants

#18686 Pants ignores fully-specified `sources` values beginning with `.` unless you add a `!` line to your `pants_ignore` Issue created by chrisjrn … and that is confusing. I had a

BUILD

file:

files(
  sources=[…, ".cargo/config", …],
)

which resulted in the error:

native_engine.IntrinsicError: Unmatched glob from src/rust/engine:rust_sources's `sources` field: "src/rust/engine/.cargo/config"

Do the file(s) exist? If so, check if the file(s) are in your `.gitignore` or the global `pants_ignore` option, which may result in Pants not being able to see the file(s) even though they exist on disk. Refer to <https://www.pantsbuild.org/v2.17/docs/troubleshooting#pants-cannot-find-a-file-in-your-project>.

.cargo

was not in my

.gitignore

or my `pants_ignore`; this behaviour was not listed in the troubleshooting docs. pantsbuild/pants

#18688 Use Pants to build our rust code Issue created by chrisjrn While Pants's rust code currently uses a custom bootstrapper to build the native Pants code before executing Python, that bootstrapper is somewhat inconvenient to use: it is sensitive to branch changes and small changes in rust source code. Even without dependency inference, Pants' caching approach is well suited to this problem! We should try using

adhoc_tool

to execute the rust toolchain rather than executing it directly. A POC exists at: https://github.com/pantsbuild/pants/compare/main...chrisjrn:pants:chrisjrn/rusty-pants?expand=1 A few issues to reckon with: 1. Chicken-and-egg: how do we execute Pants the first time, when there's no native engine to be had. 2. Handling API changes during development: if the Python API for the native engine changes, the Python code will likely stop working until the rust is built. We may wish to rely on only published versions of pants for building the rust code, however, this would involve maintaining a separate pants.toml in the meantime. Maybe that's OK? pantsbuild/pants

#17962 Bootstrap python by xrepo Issue created by da-tubi Is your feature request related to a problem? Please describe. There are no Python 3.8/3.9 in Ubuntu 22.04. I tried to use pyenv to install a Python 3.9 interpreter for Pants:

pyenv install 3.9.16

But it keeps failing because I didn't install:

tk8.6-dev
liblzma-dev
...

Describe the solution you'd like I use xrepo to install python.

$ xrepo install "python 3.9.10"
$ xrepo env -b "python 3.9.10" which python
/home/da/.xmake/packages/p/python/3.9.10/d06fa80222094670880bb9934b285f43/bin/python

I added

/home/da/.xmake/packages/p/python/3.9.10/d06fa80222094670880bb9934b285f43/bin

to PATH, and it works well for Pants. I hope there is a

<XREPO>

just like

<PYENV>

available for xrepo installed Python, so that I do not need to tune the PATH. Describe alternatives you've considered Further more, if pants can not find the suitable Python interpreter, and if

xrepo

is available as a command line, pants could invoke xrepo to install the corresponding Python interpreter. Additional context • python packages in xrepo: https://xrepo.xmake.io/#/packages/linux?id=python-linux Notice that

xrepo search python

can only find the latest support python version. And all supported python can be found in the above link. xrepo users could install python in the following ways:

xrepo install "python 3.x"
xrepo install "python 3.8"
xrepo install "python >=3.11"

pantsbuild/pants

#18691 `adhoc_tool` ignores the `extra_env_vars` field Issue created by chrisjrn The code just isn't there 🤷 Affects 2.16. pantsbuild/pants

#18698 add `DurationOption` type for duration-type config options Issue created by tdyas There are several Pants options which take an integer value (of varying time units) in order to configure duration-related configuration like timeout duration. Some of the options use seconds, others use milliseconds. In some cases, the time unit is encoded into the option name (

-millis

suffix). To reduce user confusion, we should add a

DurationOption

type for these options which could take human-readable time syntax, for example

2mins30secs

or

1234ms

and just convert to a single type internally. pantsbuild/pants

#18704 Cant get started. Unable to export to single virtualenv &amp; if I even should?? New discussion created by p-baum I'm clearly missing something. My build file and dependencies are recognised. I'm trying to export for my IDE (vscode) but the export produces multiple envs (python-default, tools/black, tools/pylint, etc). What I want is everything in one env where I can point my IDE. I've read the documentation but even that states 'You can specify the --resolve flag multiple times to export multiple virtualenvs at once.' What am I missing? Why would one want to generate multiple different envs. Is there a "trick" to link them somwhow? I think I'm missing some basic understanding which the docs are not providing. pantsbuild/pants

Comment on #18704 Cant get started. Unable to export to single virtualenv &amp; if I even should?? Discussion answered by p-baum I'm try to be too clever for my own good. A solution is to use the PYTHONPATH env variable (vscode settings "python.envFile") to include multiple paths:

PYTHONPATH=src/base1:src/base2:dist/export/python/virtualenvs/tools

after running

pants export --symlink-python-virtualenv

pantsbuild/pants

Comment on #18704 Unable to export to single virtualenv &amp; if I even should?? Discussion answered by p-baum So I switched to version

2.16.0rc0

and simply added the tools I want (pylint, black, isort) to my requirements.txt and

pants generate-lockfiles --resolve=python-default

pants export --py-resolve-format=symlinked_immutable_virtualenv --resolve=python-default

set

dist/export/python/virtualenvs/python-default/3.10.10/bin/python

as Interpreter and, with the exception of isort, its working. pantsbuild/pants

Comment on #18704 Unable to export to single virtualenv &amp; if I even should?? Discussion answered by p-baum So I switched to version

2.16.0rc0

and simply added the tools I want (pylint, black, isort) to my requirements.txt and

pants generate-lockfiles --resolve=python-default

pants export --py-resolve-format=symlinked_immutable_virtualenv --resolve=python-default

set

dist/export/python/virtualenvs/python-default/3.10.10/bin/python

as Interpreter and vscode linting and black formatting is working. "Sort imports" however is not. Running the fmt goal does however sort the imports. I will sort them with the fmt goal for now. pantsbuild/pants