set the channel description: Notifications about relevant activity on GitHub (e.g., new discussions)

/github subscribe pantsbuild/pants discussions

✅ Subscribed to pantsbuild/pants. This channel will receive notifications for

issues

,

pulls

,

commits

,

releases

,

deployments

,

discussions

/github unsubscribe pantsbuild/pants pulls commits releases deployments

This channel will receive notifications from pantsbuild/pants for:

issues

,

discussions

/github subscribe pantsbuild/pex discussions

✅ Subscribed to pantsbuild/pex. This channel will receive notifications for

issues

,

pulls

,

commits

,

releases

,

deployments

,

discussions

/github unsubscribe pantsbuild/pex pulls commits releases deployments

This channel will receive notifications from pantsbuild/pex for:

issues

,

discussions

#17162 Pants native code re-compiles without any good reason Issue created by sureshjoshi As per: https://pantsbuild.slack.com/archives/C0D7TNJHL/p1664222048965349 Created this ticket since this problem just happened again, after I had finished committing

pyright

code - only python code touched. I was able to get a filesystem modification log to see if any Rust code was obviously touched. Doesn't appear so (other than the fingerprint?). Logs, filesystem dump, and context below. * * * Slack context:

SJ: Is there any reason why native code is re-built every time I run a command? For a 5 minute code change, I'm on about 45 minutes of trying to commit it - between these compilation cycles, and pre-commit taking 10-15 minutes doing... something...

sj@tinyrick pants-completions % PY=python3.9 ./pants --changed-since=origin/main fmt lint
[=== 00:00 Building native code... ===]
Compiling pyo3-build-config v0.16.6
Compiling engine v0.0.1 (/Users/sj/Developer/oss/pants-completions/src/rust/engine)
Compiling pyo3-ffi v0.16.6
Compiling pyo3 v0.16.6
Building [=======================> ] 444/445: engine

Eric: We've been hacking on the engine much more the past month for the Docker feature, but it shouldn't be every time you run a command. Maybe every time you pull main if you don't do so frequently

SJ: Nope, cloned a new repo from main, created a branch, compiled, and then when doing linting/formatting, I got hit with I think 4-5 more compiles along the way - and I've only touched 3 python files

* * *

#  ./.pants.d/pants.log
22:07:45.50 [INFO] handling request: `--changed-since=cbaf0902195c8e3bd2d7897dff2398a1a4110e2a lint`
22:07:56.33 [INFO] request completed: `--changed-since=cbaf0902195c8e3bd2d7897dff2398a1a4110e2a lint`
22:07:57.28 [INFO] handling request: `run build-support/bin/check_inits.py`
22:08:00.44 [INFO] request completed: `run build-support/bin/check_inits.py`
22:08:01.38 [INFO] handling request: `run build-support/bin/check_banned_imports.py`
22:08:03.55 [INFO] request completed: `run build-support/bin/check_banned_imports.py`

-> SJ: Recompilation somewhere in this lease section below - when I next tried to run pyright <-

22:09:09.19 [INFO] Extending leases
22:09:09.32 [INFO] Done extending leases
22:10:29.34 [INFO] Extending leases
22:10:29.46 [INFO] Done extending leases
22:11:49.48 [INFO] Extending leases
22:11:49.58 [INFO] Done extending leases

``` # find . -type f -exec stat -f "%Sm %N" -t "%Y%y%m%d%H%M" {} \; | sort [... above these are all just compiled python files and then source files ...] 20222210072206 ./src/python/pants/util/__pycache__/meta.cpython-38.pyc 20222210072206 ./src/python/pants/util/__pycache__/ordered_set.cpython-38.pyc 20222210072206 ./src/python/pants/util/__pycache__/osutil.cpython-38.pyc 20222210072206 ./src/python/pants/util/__pycache__/resources.cpython-38.pyc 20222210072206 ./src/python/pants/util/__pycache__/strutil.cpython-38.pyc 20222210072206 ./src/python/pants/util/__pycache__/value_interpolation.cpython-38.pyc 20222210072206 ./src/python/pants/vcs/__pycache__/__init__.cpython-38.pyc 20222210072206 ./src/python/pants/vcs/__pycache__/changed.cpython-38.pyc 20222210072206 ./src/python/pants/vcs/__pycache__/git.cpython-38.pyc 20222210072206 ./src/rust/engine/target/release/.fingerprint/engine-129b40209128f1a7/lib-engine.json 20222210072206 ./src/rust/engine/target/release/deps/engine.engine.196cc17b-cgu.0.rcgu.o 20222210072206 ./src/rust/engine/target/release/deps/libengine.dylib 20222210072206 ./src/rust/engine/target/release/libengine.dylib 20222210072207 ./.pants.d/exceptions.63937.log 20222210072207 ./.pants.d/run-tracker/pants_run_2022_10_07_22_06_45_199_26386121f6db475399023fab2d4f04d8/logs 20222210072207 ./.pants.d/run-tracker/pants_run_2022_10_07_22_07_49_288_e8e931fced04489ea77a6812447ae924/logs 20222210072207 ./.pants.d/run-tracker/pants_run_2022_10_07_22_07_57_570_c5d91b33517d425d8aefd78712607146/logs 20222210072207 ./.pids/2ee2736cec13/pantsd/fingerprint 20222210072207 ./.pids/2ee2736cec13/pantsd/pid 20222210072207 ./.pids/2ee2736cec13/pantsd/process_name 20222210072207 ./.pids/2ee2736cec13/pantsd/socket 20222210072208 ./.git/COMMIT_EDITMSG 20222210072208 ./.git/logs/HEAD 20222210072208 ./.git/logs/refs/heads/17141-pyright 20222210072208 ./.git/objects/83/bf540adfaca888e0a25ead871ce315e17de154 20222210072208 ./.git/refs/heads/17141-pyright 20222210072208 ./.pants.d/run-tracker/pants_run_2022_10_07_22_08_01_658_d6eb3fe9980542928814c02966b291f9/logs 20222210072210 ./src/rust/engine/target/release/.fingerprint/engine-129b40209128f1a7/lib-engine 20222210072210 ./src/rust/engine/target/release/.fingerprint/engine-bfd629cc0604aa7b/run-build-script-build-script-build 20222210072210 ./src/rust/engine/target/release/.fingerprint/engine-bfd629cc0604aa7b/run-build-script-build-script-build.json 20222210072210 ./src/rust/engine/target/release/.fingerprint/pyo3-71697e9634a45b1b/run-build-script-build-script-build 20222210072210 ./src/rust/engine/target/release/.fingerprint/pyo3-71697e9634a45b1b/run-build-script-build-script-build.json 20222210072210 ./src/rust/engine/target/release/.fingerprint/pyo3-b28cb431842f72f2/invoked.timestamp 20222210072210 ./src/rust/engine/target/release/.fingerprint/pyo3-b28cb431842f72f2/lib-pyo3 20222210072210 ./src/rust/engine/target/release/.fingerprint/pyo3-build-config-494844a142ad02e6/run-build-script-build-script-build 20222210072210 ./src/rust/engine/target/release/.fingerprint/pyo3-build-config-494844a142ad02e6/run-build-script-build-script-build.json 20222210072210 ./src/rust/engine/target/release/.fingerprint/pyo3-build-config-b3ec6e3b36bed0f2/dep-lib-pyo3-build-config 20222210072210 ./src/rust/engine/target/release/.fingerprint/pyo3-build-config-b3ec6e3b36bed0f2/invoked.timestamp 20222210072210 ./src/rust/engine/target/release/.fingerprint/pyo3-build-config-b3ec6e3b36bed0f2/lib-pyo3-build-config 20222210072210 ./src/rust/engine/target/release/.fingerprint/pyo3-build-config-b3ec6e3b36bed0f2/lib-pyo3-build-config.json 20222210072210 ./src/rust/engine/target/release/.fingerprint/pyo3-f7e59450ab3ea08c/build-script-build-script-build 20222210072210 ./src/rust/engine/target/release/.fingerprint/pyo3-f7e59450ab3ea08c/build-script-build-script-build.json 20222210072210 ./src/rust/engine/target/release/.fingerprint/pyo3-f7e59450ab3ea08c/dep-build-script-build-script-build 20222210072210 ./src/rust/engine/target/release/.fingerprint/pyo3-f7e59450ab3ea08c/invoked.timestamp 20222210072210 ./src/rust/engine/target/release/.fingerprint/pyo3-ffi-a55b4f70a5b394b9/build-script-build-script-build 20222210072210 ./src/rust/engine/target/release/.fingerprint/pyo3-ffi-a55b4f70a5b394b9/build-script-build-script-build.json 20222210072210 ./src/rust/engine/target/release/.fingerprint/pyo3-ffi-a55b4f70a5b394b9/dep-build-script-build-script-build 20222210072210 ./src/rust/engine/target/release/.fingerprint/pyo3-ffi-a55b4f70a5b394b9/invoked.timestamp 20222210072210 ./src/rust/engine/target/release/.fingerprint/pyo3-ffi-ab77610b1a8d5439/run-build-script-build-script-build 20222210072210 ./src/rust/engine/target/release/.fingerprint/pyo3-ffi-ab77610b1a8d5439/run-build-script-build-script-build.json 20222210072210 ./src/rust/engine/target/release/.fingerprint/pyo3-ffi-e9ac07c84d5cc2b8/dep-lib-pyo3-ffi 20222210072210 ./src/rust/engine/target/release/.fingerprint/pyo3-ffi-e9ac07c84d5cc2b8/invoked.timestamp 20222210072210 ./src/rust/engine/target/release/.fingerprint/pyo3-ffi-e9ac07c84d5cc2b8/lib-pyo3-ffi 20222210072210 ./src/rust/engine/target/release/.fingerprint/pyo3-ffi-e9ac07c84d5cc2b8/lib-pyo3-ffi.json 20222210072210 ./src/rust/engine/target/release/build/engine-bfd629cc0604aa7b/invoked.timestamp 20222210072210 ./src/rust/engine/target/release/build/engine-bfd629cc0604aa7b/output 20222210072210 ./src/rust/engine/target/release/build/engine-bfd629cc0604aa7b/root-output 20222210072210 ./src/rust/engine/target/release/build/engine-bfd629cc0604aa7b/stderr 20222210072210… pantsbuild/pants

#11605 Fatal Python error: Illegal instruction Issue created by jsirois Seen for #11604 which is based on top of 9dd433d. Presumably this has nothing to do with the actual test being run here but is some sort of more generic engine bug:

23:59:55.88 [WARN] Completed: test - src/python/pants/backend/awslambda/python/rules_test.py:tests failed (exit code -4).
============================= test session starts ==============================
collected 1 item
src/python/pants/backend/awslambda/python/rules_test.py 
Fatal Python error: Illegal instruction
Thread 0x00007f1acc343700 (most recent call first):
Thread 0x00007f1ad76f3680 (most recent call first):
  File "/tmp/process-execution7mBvjs/src/python/pants/engine/internals/scheduler.py", line 326 in _run_and_return_roots
  File "/tmp/process-execution7mBvjs/src/python/pants/engine/internals/scheduler.py", line 498 in execute
  File "/tmp/process-execution7mBvjs/src/python/pants/engine/internals/scheduler.py", line 588 in product_request
  File "/tmp/process-execution7mBvjs/src/python/pants/testutil/rule_runner.py", line 176 in request
  File "/tmp/process-execution7mBvjs/src/python/pants/backend/awslambda/python/rules_test.py", line 42 in create_python_awslambda
  File "/tmp/process-execution7mBvjs/src/python/pants/backend/awslambda/python/rules_test.py", line 82 in test_create_hello_world_lambda
  File "/home/travis/.cache/pants/named_caches/pex_root/venvs/short/6b904442/lib/python3.7/site-packages/_pytest/python.py", line 183 in pytest_pyfunc_call
  File "/home/travis/.cache/pants/named_caches/pex_root/venvs/short/6b904442/lib/python3.7/site-packages/pluggy/callers.py", line 187 in _multicall
  File "/home/travis/.cache/pants/named_caches/pex_root/venvs/short/6b904442/lib/python3.7/site-packages/pluggy/manager.py", line 87 in <lambda>
  File "/home/travis/.cache/pants/named_caches/pex_root/venvs/short/6b904442/lib/python3.7/site-packages/pluggy/manager.py", line 93 in _hookexec
  File "/home/travis/.cache/pants/named_caches/pex_root/venvs/short/6b904442/lib/python3.7/site-packages/pluggy/hooks.py", line 286 in __call__
  File "/home/travis/.cache/pants/named_caches/pex_root/venvs/short/6b904442/lib/python3.7/site-packages/_pytest/python.py", line 1641 in runtest
  File "/home/travis/.cache/pants/named_caches/pex_root/venvs/short/6b904442/lib/python3.7/site-packages/_pytest/runner.py", line 162 in pytest_runtest_call
  File "/home/travis/.cache/pants/named_caches/pex_root/venvs/short/6b904442/lib/python3.7/site-packages/pluggy/callers.py", line 187 in _multicall
  File "/home/travis/.cache/pants/named_caches/pex_root/venvs/short/6b904442/lib/python3.7/site-packages/pluggy/manager.py", line 87 in <lambda>
  File "/home/travis/.cache/pants/named_caches/pex_root/venvs/short/6b904442/lib/python3.7/site-packages/pluggy/manager.py", line 93 in _hookexec
  File "/home/travis/.cache/pants/named_caches/pex_root/venvs/short/6b904442/lib/python3.7/site-packages/pluggy/hooks.py", line 286 in __call__
  File "/home/travis/.cache/pants/named_caches/pex_root/venvs/short/6b904442/lib/python3.7/site-packages/_pytest/runner.py", line 255 in <lambda>
  File "/home/travis/.cache/pants/named_caches/pex_root/venvs/short/6b904442/lib/python3.7/site-packages/_pytest/runner.py", line 311 in from_call
  File "/home/travis/.cache/pants/named_caches/pex_root/venvs/short/6b904442/lib/python3.7/site-packages/_pytest/runner.py", line 255 in call_runtest_hook
  File "/home/travis/.cache/pants/named_caches/pex_root/venvs/short/6b904442/lib/python3.7/site-packages/_pytest/runner.py", line 215 in call_and_report
  File "/home/travis/.cache/pants/named_caches/pex_root/venvs/short/6b904442/lib/python3.7/site-packages/_pytest/runner.py", line 126 in runtestprotocol
  File "/home/travis/.cache/pants/named_caches/pex_root/venvs/short/6b904442/lib/python3.7/site-packages/_pytest/runner.py", line 109 in pytest_runtest_protocol
  File "/home/travis/.cache/pants/named_caches/pex_root/venvs/short/6b904442/lib/python3.7/site-packages/pluggy/callers.py", line 187 in _multicall
  File "/home/travis/.cache/pants/named_caches/pex_root/venvs/short/6b904442/lib/python3.7/site-packages/pluggy/manager.py", line 87 in <lambda>
  File "/home/travis/.cache/pants/named_caches/pex_root/venvs/short/6b904442/lib/python3.7/site-packages/pluggy/manager.py", line 93 in _hookexec
  File "/home/travis/.cache/pants/named_caches/pex_root/venvs/short/6b904442/lib/python3.7/site-packages/pluggy/hooks.py", line 286 in __call__
  File "/home/travis/.cache/pants/named_caches/pex_root/venvs/short/6b904442/lib/python3.7/site-packages/_pytest/main.py", line 348 in pytest_runtestloop
  File "/home/travis/.cache/pants/named_caches/pex_root/venvs/short/6b904442/lib/python3.7/site-packages/pluggy/callers.py", line 187 in _multicall
  File "/home/travis/.cache/pants/named_caches/pex_root/venvs/short/6b904442/lib/python3.7/site-packages/pluggy/manager.py", line 87 in <lambda>
  File "/home/travis/.cache/pants/named_caches/pex_root/venvs/short/6b904442/lib/python3.7/site-packages/pluggy/manager.py", line 93 in _hookexec
  File "/home/travis/.cache/pants/named_caches/pex_root/venvs/short/6b904442/lib/python3.7/site-packages/pluggy/hooks.py", line 286 in __call__
  File "/home/travis/.cache/pants/named_caches/pex_root/venvs/short/6b904442/lib/python3.7/site-packages/_pytest/main.py", line 323 in _main
  File "/home/travis/.cache/pants/named_caches/pex_root/venvs/short/6b904442/lib/python3.7/site-packages/_pytest/main.py", line 269 in wrap_session
  File "/home/travis/.cache/pants/named_caches/pex_root/venvs/short/6b904442/lib/python3.7/site-packages/_pytest/main.py", line 316 in pytest_cmdline_main
  File "/home/travis/.cache/pants/named_caches/pex_root/venvs/short/6b904442/lib/python3.7/site-packages/pluggy/callers.py", line 187 in _multicall
  File "/home/travis/.cache/pants/named_caches/pex_root/venvs/short/6b904442/lib/python3.7/site-packages/pluggy/manager.py", line 87 in <lambda>
  File "/home/travis/.cache/pants/named_caches/pex_root/venvs/short/6b904442/lib/python3.7/site-packages/pluggy/manager.py", line 93 in _hookexec
  File "/home/travis/.cache/pants/named_caches/pex_root/venvs/short/6b904442/lib/python3.7/site-packages/pluggy/hooks.py", line 286 in __call__
  File "/home/travis/.cache/pants/named_caches/pex_root/venvs/short/6b904442/lib/python3.7/site-packages/_pytest/config/__init__.py", line 163 in main
  File "/home/travis/.cache/pants/named_caches/pex_root/venvs/short/6b904442/lib/python3.7/site-packages/_pytest/config/__init__.py", line 185 in console_main
  File "/home/travis/.cache/pants/named_caches/pex_root/venvs/short/6b904442/lib/python3.7/site-packages/pytest/__main__.py", line 5 in <module>
  File "/opt/python/3.7.6/lib/python3.7/runpy.py", line 85 in _run_code
  File "/opt/python/3.7.6/lib/python3.7/runpy.py", line 208 in run_module
  File "/home/travis/.cache/pants/named_caches/pex_root/venvs/7e44d39a11d9894dd62c2e62abfc5e9b171c7d99/7b8c0ebf3fd10e0026e748337ba25c8779e42ef3/pex", line 97 in <module>

pantsbuild/pants

#17335 Cache at a testcase level - similar to pytest-testmon New issue created by AbdealiJK Is your feature request related to a problem? Please describe. pytest-testmon is a plugin that helps in figuring out the relationship between the "covered line of code" and the testcase. This changes the concept of rerunning and aching pytest results from a file level to a line or testcase level. i.e. when a line of code is modified, you can figure out which testcases need to be rerun. Describe the solution you'd like If pants can run only the testcases based on the lines that I have modified, that would be cool and much faster Describe alternatives you've considered • pytest-testmon (But it does not have accurate coverage information) • pytest-tia (This is not maintaled any longer) • https://python-tia.readthedocs.io/en/latest/alternatives.html Additional context In my project we have 100+ tests defined in a single test file (mostly via pytest.mark.parametrize). We could also just break this up into smaller files. But not sure if that is ideal pantsbuild/pants

#17336 Should we expose internal (python) tools as targets? New discussion created by thejcannon OK my last big request/idea for 2.15.x I promise (but also not really 😉) Sometimes I find myself wanting to run a tool which Pants is currently wrapping. • pex - For hopefully obvious reasons. • also tools like

black

or

pytest

or similar I could of course declare a

pex_binary

and run that but it requires me ensuring the version and configuration is similar to that of Pants (boo double-config). One big reason for this that I find myself wanting is knowing what CLI args are possible (especially for pytest). There's no easy way invoke

pytest --help

and get all the CLI knobs/levers including third-party and first-party ones. A second reason is wanting to run

pex lock ...

using the exact same version of

pex

Pants is. So what would it look like to have explicit/implicit targets for each tool? Then I could

run

them. AND

export

wouldn't be exporting them by default if they had addresses 😉 Thoughts? My bikeshed name was

interal_tool

or more specifically

internal_python_tool

(to mirror

external_tool

from #17277) pantsbuild/pants

Comment on #17332 Should we stop marking `XResult` as uncacheable for fmt/fix/lint/test/etc... Discussion answered by stuhood The scope of

uncacheability

for a

@rule

is a bit different:

Can see that the log is printed very quickly showing the result was cached

On the first run of

pantsd

, this would happen regardless of whether the

@rule

was marked uncacheable, because the

@rule

cannot hit a persistent cache: only the processes below it. So I think that for the case you're thinking of (lots of output in CI...?), it would not have any impact. The only time the result would cease to be printed would be when the process below it re-ran within a single session, but did not change... almost certainly because it re-ran and hit the cache, but possibly also because it failed for a second time with exactly the same output. That latter case (failing again) is problematic. No matter how many times a test fails and re-runs, you'd want to see its output: even if it was identical to the last failure.

Pollutes the output with unhelpful information

I disagree that "a test completed" is unhelpful info in a CI situation... for one thing, some CI providers will kill your process if you don't render any output for at least N minutes: Travis did at least, not sure about Actions. And at the very least, a user might reasonably expect that something has hung. To remove this information, I think that it would need to be replaced with some other ongoing summary of work ("N processes queued, M processes completed, ...")... which I would be in favor of if we could reify enough information out of

@rules

to make it useful. pantsbuild/pants

#17338 Add `skip` options to all test subsystems Issue created by danxmoran Our generic partitioning logic expects subsystems to implement a

skip

option. I added hard-coded placeholders to our existing subsystems as part of #17134. Replace those with proper `SkipOption`s. pantsbuild/pants

#17339 Add a `batch_size` option to the `test` subsystem Issue created by danxmoran The new test-batching logic of the

test

goal includes logic to rebatch partitions according to a max batch size. In #17134 I hard-coded a max size of 128. Replace that hard-coded value with an option on the

TestSubsystem

, similar to the option we already define on the

LintSubsystem

. pantsbuild/pants

#12946 Rules API: allow using a `Get` with no arguments Issue created by Eric-Arellano This rule is slow because it finds binaries that aren't actually needed. Even if you are unzipping a

.zip

file, we'll still find Tar: pants/src/python/pants/core/util_rules/archive.py Lines 168 to 206 in </pantsbuild/pants/commit/636a5c7bcd609d1a4a44770a777ceb10e0a794fb|636a5c7> Because

UnzipBinary

is a singleton, there is nothing we can pass for the input of the

Get

. So we either have to put the type in the

@rule

signature or add a new type like

UnzipBinaryRequest

, so you could do

await Get(UnzipBinary, UnzipBinaryRequest)

. Instead, I propose we allow

await Get(UnzipBinary)

. pantsbuild/pants

#17342 Simplify our APIs to leverage `Get()` with no arguments New issue created by Eric-Arellano We have some types like

AllTargetsRequest

and

ZipBinaryRequest

that solely existed to work around the lack of #12946. That is, it allowed us to "lazily" evaluate things, e.g. inside if statements, by doing

await Get(ZipBinary, ZipBinaryRequest)

. Now that we can do

Get(ZipBinary, {})

, we should deprecate and then delete those old APIs. We only need the "singleton" rules that return

ZipBinary

without needing to have a

ZipBinaryRequest

. This change will make for less code and a tighter rule graph.

we should deprecate

We can do that by putting

warn_or_error()

inside the rules for

ZipBinaryRequest

et al. pantsbuild/pants

#17343 Synthetic targets for lockfiles break when a target has the same name as the resolve Issue created by Eric-Arellano To reproduce, apply this diff:

diff --git a/3rdparty/python/BUILD b/3rdparty/python/BUILD
index 9803dba972..764a4a09e2 100644
--- a/3rdparty/python/BUILD
+++ b/3rdparty/python/BUILD
@@ -2,13 +2,14 @@
 # Licensed under the Apache License, Version 2.0 (see LICENSE).
 
 python_requirements(
+    name="python-default",
     module_mapping={
         "strawberry-graphql": ["strawberry"],
         "beautifulsoup4": ["bs4"],
         "python-gnupg": ["gnupg"],
     },
     overrides={
-        "humbug": {"dependencies": ["#setuptools"]},
+        "humbug": {"dependencies": ["python-default#setuptools"]},
     },
 )

Then run `./pants dependencies 3rdparty/python:python-default#setuptools`:

ValueError: The explicit dependency `3rdparty/python/user_reqs.lock:python-default` of the target at `3rdparty/python:python-default#setuptools` does not provide enough address parameters to identify which parametrization of the dependency target should be used.
Target `3rdparty/python:python-default` can be addressed as:
  * 3rdparty/python:python-default
  * 3rdparty/python:python-default#PyYAML
  * 3rdparty/python:python-default#ansicolors

I'm not sure what the correct solution is here. Maybe we should add something special to the synthetic target like

:python-default--resolve

to reduce the risk of collisions? This is not hypothetical - it broke Toolchain when upgrading to 2.15. -- It would also probably be useful to add an option to disable synthetic targets. Right now, the only escape hatch was to rename Toolchain's target. -- cc @kaos - thoughts? pantsbuild/pants

#17344 Disabling python-infer.imports results in incorrect dependencies / dependees New issue created by njgrisafi Describe the bug When disabling the

python-infer.imports

subsystem and run

./pants dependencies <some-empty-file>

we get a dependencies that should not be there. The behavior seems to mark sources in the same target (e.g.

python_sources

) as dependencies of each other. When creating more granular sources (e.g.

python_source

) it does not output any dependencies which is expected. Pants version 2.8 (upgrading from 2.7) OS Both MacOS / Linux Additional info How to reproduce

$ ./pants dependencies src/python/pants/base/hash_utils.py 
src/python/pants/util/ordered_set.py
src/python/pants/util/strutil.py

$ ./pants --python-infer-imports=False dependencies src/python/pants/base/hash_utils.py 
src/python/pants/base/__init__.py
src/python/pants/base/build_environment.py
src/python/pants/base/build_root.py
src/python/pants/base/deprecated.py
src/python/pants/base/exception_sink.py
src/python/pants/base/exceptions.py
src/python/pants/base/exiter.py
src/python/pants/base/glob_match_error_behavior.py
src/python/pants/base/parse_context.py
src/python/pants/base/specs.py
src/python/pants/base/specs_parser.py

Example branch https://github.com/njgrisafi/pants-example/tree/dep-inf-2.8 Run

./pants dependencies app/module_1/api.py

(an empty file) and confirm it has dependencies. Run

./pants dependees app/module_1/api.py

(unused) and confirm is has dependees. pantsbuild/pants

#1967 Unable to create a PEX with an unpublished wheel with a lock file New issue created by zmanji I have an unpublished wheel with requirements. Think a native code wheel that has requirements in pyproject.toml. Given this wheel I want to build a pex but I want the resolve to use a lock for reproducibility. What I have done is create a lockfile with all of the third party deps I need. When I try:

pex --lock pex.lock <WHL>

This fails with:

Failed to resolve compatible artifacts from lock pex.lock for 1 target:
1. /home/zmanji/.pex/venvs/ad3fcfb926d70cf748f6693d03b37390c95022dc/5985ed09b49a653d6596b0e14d134c5456cf1a9f/bin/python3.10:
    Failed to resolve all requirements for cp310-cp310-manylinux_2_35_x86_64 interpreter at /home/zmanji/.pex/venvs/ad3fcfb926d70cf748f6693d03b37390c95022dc/5985ed09b49a653d6596b0e14d134c5456cf1a9f/bin/python3.10 from pex.lock:

Configured with:
    build: True
    use_wheel: True

Dependency on <PROJECT> (via: <PROJECT>==0.1) not satisfied, no candidates found.

I have the same issue when I put all of the third party deps into a pex and ask pex to resolve from a pex repository. I also have tried building an intransitive pex with just the wheel and putting all of the other deps in another pex that's put on the PEX_PATH but pex does not traverse the entire pex path when activating a pex. I think this was discussed in #1423 Currently I resolve my issue by: 1. Building a pex with all third party dependencies. 2. Extracting all third party dependencies into a wheelhouse 3. Build a final pex with my projects wheel, with no pypi index and the above wheelhouse. My workaround is fine so I don't think this is an urgent to be fixed but I think pantsbuild/pants#15801 is the same issue in pants. I do think this is worth resolving because conceptually to me resolving from a pex-repository (or a lock file) should be the same as resolving from a fixed wheelhouse. pantsbuild/pex

#17345 `experimental_shell_command`'s sandbox and invalidation includes transitive dependencies of `experimental_shell_command` dependencies, not just outputs New issue created by huonw Describe the bug When a

experimental_shell_command

(

:second

) depends on another

experimental_shell_command

(

:first

), the sandbox for

:second

include the dependencies of

:first

, rather than just the output of

:first

. https://gist.github.com/huonw/c55d0b387ed6030cda611898ee2d0361 provides a reproducer, where

:first

generates

output.txt

by 'consuming'

input.txt

(but does nothing with it), and

:second

includes a

sleep

to demonstrate when it is running. The

archive

package includes all the

.txt

files that ended in the sandbox for `:second`: I'd expect it to only be the direct output of

:first

(

output.txt

), not the

input.txt

dependency of

:first

. BUILD file from that gist for convenience:

file(name="input", source="input.txt")
experimental_shell_command(
    name="first",
    command="""
    echo "doesn't affect output"
    echo contents > output.txt
    """,
    tools=["echo"],
    dependencies=[":input"],
    outputs=["output.txt"]
)

experimental_shell_command(
    name="second",
    command="""
    sleep 3 # make 'actually running' obvious
    """,
    dependencies=[":first"],
    tools=["sleep"],
    outputs=["*.txt"],
)
archive(name="archive", files=[":second"], format="zip")

git clone <mailto:git@gist.github.com|git@gist.github.com>:c55d0b387ed6030cda611898ee2d0361.git
cd c55d0b387ed6030cda611898ee2d0361
./pants version # 2.13.0

# initial build (takes about 3 seconds to run :second, due to sleep)
./pants package ::

# check package output:
unzip -l dist/archive.zip # two files: input.txt output.txt

# validating the cache works as expected (~instantaneous)
./pants package ::

# BUG: change to :first dependencies, that doesn't affect output... 
echo 'new contents' > input.txt
# ... reruns :second, and thus takes ~3 seconds
./pants package ::

# EXPECTED: change to :first script that doesn't affect output...
sed -i '' 's/affect output/affect output still/' BUILD
# .... doesn't rerun :second, only :first
./pants package ::

AFAICT, there's no way to break `:second`'s importing of

:input

, e.g. adjusting to

dependencies=[":first", "!:input"]

doesn't do anything: the behaviour is the same (and using

!!:input

isn't supported). The

:input

being

file

is just for convenience. This appears to apply to all other target types, e.g. a

pex_binary

or yet another

experimental_shell_command

. I haven't checked how this behaves when depending on explicit codegen targets, only these adhoc

experimental_shell_command

ones. Pants version 2.13.0 OS macOS Additional info Background: how are we using

experimental_shell_command

to hit this? We're using

experimental_shell_command

to try to bridge the gap between Pants supported code and unsupported code (JS/TS), as well as for adhoc codegen tasks (to avoid avoid having to write and maintain a plugin). This can result in 'long' chains of `experimental_shell_command`s that depend on each other (and other resources), e.g.:

# some app:
python_sources()
pex_binary(name="app", ...)

# codegen the schema itself:
experimental_shell_command(name="schema", dependencies=[":app"], outputs=["schema.json"], command="./app.pex export-schema > schema.json", tools=["python3.9"])

# set up the NPM dependencies:
experimental_shell_command(name="node_modules", outputs=["node_modules/**"], command="npm ci", ...)

# use the schema to do codegen or generate docs or whatever:
experimental_shell_command(name="codegen-from-schema", dependencies=[":node_modules", ":schema"], outputs=["codegen/**"], command="npm run do-codegen < schema.json", ...)
experimental_shell_command(name="docs-from-schema", dependencies=[":node_modules", ":schema"], outputs=["docs/**"], command="npm run generate-docs < schema.json", ...)

# continues with targets using :codegen-from-schema and :docs-from-schema... (`archive`, `experimental_shell_command` and `experimental_run_shell_command`)

The Python code that goes into

pex_binary

changes regularly and thus the PEX itself changes too, but the exported

schema.json

doesn't change so much (i.e. we're often refactoring/adding features/fixing bugs without changing the schema). In theory, if

:app

changes but the schema doesn't, `:schema`'s dependees (

:codegen-from-schema

,

:docs-from-schema

) don't need to run, but those dependees pull in the

app.pex

file from

:app

, and thus do rerun. Rerunning can take significant time. (See also: https://pantsbuild.slack.com/archives/C046T6T9U/p1666315386420799) pantsbuild/pants

#17327 isort with multiple config files does not work correctly Issue created by AbdealiJK Describe the bug I have a project like:

libs
    lib1
        pyproject.toml
        lib1
            ... files in the lib
        lib1_test
            ... tests for the lib
    lib2
        ... similar to lib1
    ... more libs

And I am using pyproject.toml for the isort configs:

[tool.isort]
profile = "pycharm"
skip_glob = ["venv*"]
line_length = 110
known_first_party = ["lib1", "lib1_test"]
combine_as_imports = true
force_grid_wrap = 0
include_trailing_comma = true
order_by_type = false
case_sensitive = true

And the pyproject.toml for every lib changes the

known_first_party = ["lib1", "lib1_test"]

to treat only the library as a first party import. When I run:

./pants fmt ::

- it was using the wrong first party imports On @happy-kitchen-89482 's suggestio - tried:

./pants fmt libs/lib1/::

And it seemed to work After a lot of digging, I realized that isort finds the config file based on the 1st file provided to it https://pycqa.github.io/isort/docs/configuration/config_files.html And order of files that pants is sending into isort is mosst likely the culprit here. Pants version: 2.13.0 OS: Windows 10 + Ubuntu 18.04 LTS Additional info Slack conversation: link Another issue related to isort I checked: #15069 The batch ID concept in this issue may have been a workaround here: #14941 pantsbuild/pants

#17351 Fetching tool requirements from a requirements file New discussion created by AbdealiJK Can I generate

[flake8].extra_requirements

from a

requirements.txt

file ? Instead of hardcoding it in the

pants.toml

? I just have a lot of requirements files I have right now (

requirements-flake8.txt

,

requirements-pytest.txt

, etc) And was wondering if I can reuse them instead of rewriting them in a config file which is pants specific pantsbuild/pants

#17221 ModuleNotFoundError: No module named 'pex' Issue created by engnatha Describe the bug When running pants in our CI system, we can have a container that has multiple clones of our repository. At various points, they will be running goals like

./pants export

or

./pants run

. At some point in the chain, we see errors like this

10:42:12.54 [INFO] Initialization options changed: reinitializing scheduler...
10:42:13.10 [INFO] Scheduler initialized.
Traceback (most recent call last):
  File "/usr/lib/python3.8/runpy.py", line 194, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/usr/lib/python3.8/runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "/home/buildbot/.pex/unzipped_pexes/7907cee4a80399696d31dbee748702a492b91a80/__main__.py", line 102, in <module>
    from pex.pex_bootstrapper import bootstrap_pex
ModuleNotFoundError: No module named 'pex'

The only way to remedy this seems to be clearing out

~/.pex

and trying the goal again. Pants version 2.13.0 OS Ubuntu 20.04 Additional info We've tried adding some steps to manage concurrency in the system and automatically wipe

~/.pex

when this error is detected, but that's unreliable to implement everywhere. Some initially debugging was kindly provided by @jsirois who filed #17176 to resolve. pantsbuild/pants

#17352 NanoPB code generation New issue created by sureshjoshi This is more of a "note to self" to add code generation for NanoPB when the

cc

backend drops. I was lamenting having to write my own script for this yesterday, while having most of the tools we use already. pantsbuild/pants

#17353 Distribution dependencies not showing up in packaged distribution New issue created by engnatha Describe the bug A pants distribution can be constructed in such a way that

dependencies

shows relevant package data in the graph. Upon building the distribution, one will find that those data files have been (silently) omitted from the built distribution. They are left to discover these missing files likely at run time. This issue has been reproduced in https://github.com/engnatha/pants-distribution-bug. The steps to reproduce are • Verify

secret_data.txt

is a dependency with

./pants dependencies --transitive //:test_distribution

• Build the distribution with

./pants package //:test_distribution

• Confirm the file is not present in the packaged distribution

unzip -l dist/test_dist-0-py3-none-any.whl

The silent dropping is coming from this line. The data file cannot be associated with a package and is dropped from

package_data

in the constructed

setup.py

. If one places a dummy file in

helloworld/

, sets up a

BUILD

file, and associates that module with the distribution target, then the data file is included again. Note that the "dummy" file can't be just

__init__.py

since

tailor

was not picking that up to make a new

BUILD

file (though that's likely a separate issue). This does not seem to conform to pep-420 standards for namespace packages that intend to remove the need for python modules within every directory. Pants version 2.14.0rc5 (though this was also the case on 2.13.0 and likely earlier versions) OS Ubuntu 20.04 Additional info To be fair, sphinx also doesn't gracefully handle namespace packages (at least for the version we use). We have had to put

__init__.py

files around our repo to make documentation building work properly. pantsbuild/pants

#17354 Improve safety of using `environments` during `BUILD` evaluation time New issue created by stuhood Currently, build graph calculations and

BUILD

file evaluation are pinned to the

__local__

environment (see #17129), meaning that it will not run in a

docker

, or

remote

environment. This is accomplished by "masking" the environment at graph and

BUILD

file evaluation boundaries, and then explicit injecting the

ChosenLocalEnvironmentName

as the current

EnvironmentName

. While that process is enforced (by the

masked_type

annotations) and fairly safe, it is not type safe for

@rule

authors who might think that they can use the

Platform

or

EnvironmentVars

during

BUILD

file evaluation time or dependency inference. Because they can access those types in those locations, but they will not get the appropriate values to use at runtime: they will instead get the values for the

__local__

environment. If you know that, then you can manually avoid consuming environment-specific types in those

@rules

, and defer the work to, say, codegen (as @thejcannon demonstrates here), but making this type safe instead would definitely be preferable. * * * One way to help guide users would be to overload all of the environment-aware intrinsics (

Process

running,

Platform

calculation,

EnvironmentVars

calculation, etc) to consume _either_: • an

EnvironmentName

(as they do today in

2.15.x

), representing the runtime environment of a target • a type specific to build graph evaluation, or explicitly for the local environment, maybe called

LocalEnvironmentName

or

BuildGraphEnvironmentName

. The latter type would be available to

@rules

which were part of

BUILD

file evaluation and dependency calculation, while the former type would be available to codegen and etc. This would bifurcate

@rules

by type, and maybe help to guide @rule authors to avoid consuming the "wrong"

Platform

. But because dependency inference still needs to be able to run processes, it would not represent true type safety. pantsbuild/pants

#17355 Stabilization Tracking: Environments New issue created by stuhood This is a tracking issue for stabilizing the "environments" feature. A few relevant probably-blocking issues: ☐ #8353 ☑︎ #17129 ☐ #17354 ☐ #16873 ☐ #17376 ☐ #17383 pantsbuild/pants

#17356 Update plugin upgrade guide with changes to `test` schema Issue created by danxmoran Document: 1. New expectations for

test

subsystems 2. The new

TestRequest

union 3. The new 2-part rule setup for test runners pantsbuild/pants