#17357 Javascript/Typescipt support New discussion created by sureshjoshi High Level

prettier

is a

fmt

plugin, with

eslint

support on the way as a

fix

and/or

lint

plugin. This leaves us with the question(s) of transpiling/bundling and how they should work. The JS ecosystem is the wild west, with a new build system coming out ... monthly? So, the prospect of adding yet another tool to that mix doesn't give me the warm and fuzzies. What would Pants's "purpose" be in TS/JS support? Would the idea be to be: A) a replacement for stuff like

webpack

,

vite

,

esbuild

(yay Go),

swc

(yay Rust),

turbopack

(yay Rust again?), etc (to which I'm a giant 👎 ) - or, B) would it be to use those as plugins and facilitate their usage in a consistent matter, and maybe throw some dep inference + caching in for good measure, or C) Alternative/middleground The prospect of using Pants and migrating config from

vite

to yet another transpiler/builder (since the JS ecosystem is the wild west), is a non-starter for me. Notable Notes and Quotable Quotes from the Slack thread

I'd like to be able to use prettier, other tools the same way I do for my python tools (pants fmt).

I'd like to have something to use with npm/yarn/webpack to do dependency management and caching.

I definitely don't think its A. There's enough choice in the JS ecosystem without rebuilding the wheel here

• Thomas

...there's lots of language specific functionality built into the bundlers (often as plugins) ...

...I suspect it'll be a hard sell for front end engineering to switch away from that normal JS/TS tooling, at least until pants has proven itself...

I suspect pants only prove itself convincingly via B (ie running outside pants gives the same behaviour as running inside) or a relatively extensive A.

• Huon

I think the idea that Pants should not get in the way is a good one. As much as possible it should just run your tools for you, not prevent you from using your tools. Basically, you should feel ~comfortable in a --keep-sandboxes chroot even if you don't know Pants.

• John

The purpose should be: 1) provide a uniform interface, 2) ability to build a JS bundle and package it as an asset into a pex, 3) perf improvements where feasible

• Benjy

pantsbuild/pants

#17363 Buf 1.9 isn't compatible with the buf backend Issue created by jyggen Running Buf 1.9 results in the following error during `pants fmt`:

Failure: exec: "diff": cannot run executable found relative to current directory

From the Go docs:

To avoid those security problems, as of Go 1.19, this package will not resolve a program using an implicit or explicit path entry relative to the current directory.

buf format

needs

diff

in order to run, and our buf backend uses

BinaryShims

to get

diff

into the sandbox. Taking a look at

BinaryShims.bin_directory

, which we add to

PATH

before running

buf

, it is very much a relative path. The easy fix would be to make

BinaryShims.bin_directory

an absolute path instead, but we don't know about the sandbox and its path at the time

bin_directory

is set. Instead we can patch the sandbox similar to the `GoSdkProcess` rule to make

PATH

absolute. pantsbuild/pants

#17368 Formalizing the distinction between "source targets" and "binary targets" New discussion created by benjyw Context: #17345 The ticket mentions that if an

experimental_shell_command

A depends on another

experimental_shell_command

B then the sandbox for A includes all the sources of B, instead of just the output of running B. This is because we naively traverse the transitive deps looking for sources. A related situation exists with the "local dists" feature of the python backend: when a python_sources() depends on a python_distribution(), we build the distribution and make it loadable on the sys.path (this is to support building native deps locally without having to publish them before consuming them). But because we traverse transitive deps naively, the python_distribution's sources are also gathered that way, so they end up on the sys.path twice. In that case we handle this specially, via the the

LocalDistsPex.remaining_sources

field, which subtracts out the sources in the dist from the sources in the transitive deps. I think the underlying problem is that we don't formally distinguish between "source targets" and "binary targets". In this schema, "source targets" would provide files from their

sources

field to their dependents. While "binary targets" would have no

sources

field, and would provide files created by "building" them (the meaning of "building" depends on the target type), in topological order. When traversing targets looking for sources, we wouldn't traverse through "binary targets". Thoughts? pantsbuild/pants

Comment on #17351 Fetching tool requirements from a requirements file Discussion answered by thejcannon So Pants uses

pex

to make lockfiles, so you should be able to lock like Pants does, and tell Pants to ignore the missing metadata. • Tell Pants to ignore missing metadata (unfortunately global): Set

[python].invalid_lockfile_behavior = "ignore"

• If you plan on just doing it yourself, you can install

pex

on your machine. Otherwise you'll need to sandbox

pex

in a

pex_binary

(which is a much longer explanation I plan on blogging about later). Note that #17336 would solve this in a very convenient way. • One caveat is to make sure you're using the version of

pex

equal or lower than Pants is, or else you might use newer metadata than Pants' PEX knows how to use. https://www.pantsbuild.org/docs/reference-pex-cli (for your version) has the

version

option and the default is what Pants is using. Alternatively you can bump that up because PEX is always backwards-compatible. • Then run

pex3 lock create -r <requirements.txt> -o <output.lock>

with your input and output (output would be your setting of

[<tool>].lockfile

pantsbuild/pants

Comment on #17325 Why is an un-inferred dependency a warning, rather than an error, and should it stay that way? Discussion answered by benjyw I'm not convinced it is an error in every case - you could, for example, be importing from a package that you know is available at runtime. Making this an error would probably require support for dependency scopes, so we can express the idea of a "provided" dependency. But allowing repos to opt-in (or out, eventually) of this being an error sounds like a good idea. All that said, I'm not sure I see the correctness issue? If you had a typo, for example, then when you fix the typo that file is invalidated. pantsbuild/pants

#17376 Consider allowing users to set a different `runtime_environment=` on test targets New issue created by stuhood The Environments design doc ended up suggesting that test targets might gain a

runtime_environment=

field which when set in addition to an

environment=

field would cause a test to be built in one environment, but then executed in another (presumably compatible) environment. We should be on the lookout for use cases for this feature before stabilizing Environments. pantsbuild/pants

#17070 Update docs for `test` plugin API Issue created by danxmoran #17044 pantsbuild/pants

#17094 Unify partitioning boilerplate between `lint`/`fmt` and `test` Issue created by danxmoran The new plugin API for

test

partitioning introduced in #17044 is a nearly-identical match to the API introduced for `lint`+`fmt` in #16735 and #16980. There is significant duplication between the two API setups - consolidate them so plugin authors have fewer concepts to juggle. pantsbuild/pants

#17383 Support partitioning by `environment` field for lint/fmt/fix/etc New issue created by chrisjrn Currently, the partitioning behavior for lint/fmt/fix involves a "planning" step, where each backend chooses how to partition its work. We're trying to make environment selection something that can be implemented without backend authors needing to think about it, so it would be useful if there were support for further partitioning each partition (e.g. by environment field on targets), such that we can dispatch each sub-partition to the appropriate environment. pantsbuild/pants

#13682 Add support for cross-building Python applications to `test`, `package` inside-of/for a Docker image Issue created by ryanking Is your feature request related to a problem? Please describe. My problem is that we have a good chunk of python code which depends on a lot of system libraries and command line tools being on the PATH. This is bioinformatics code for which this pattern is quite common. This code will run in a container, whose image is built by pants, in CI and production. When running tests in development environments I would like to be able to use the same environment. Describe the solution you'd like I would like to be able to tell pants– "when running this target, run it in a container from _this image_". Ideally that image can be built by pants automatically when needed. Describe alternatives you've considered One alternative we are considering is to move all our development work to linux machines and/or docker containers where we have everything installed. (We may do this regardless of this change). Additional context slack discussion - https://pantsbuild.slack.com/archives/C046T6T9U/p1637272879490300 * * * Notes from #14145: Cross-building Python dists (wheels) to target a different platform could be accomplished internally to Pants using Docker, to allow both cross-platform PEXes and Docker images to be built without pre-built dists. Cross-building inside of a docker image is very similar to remote execution to a different plaform (as described in #11148): at a fundamental level, it's a bit like executing all logic remotely inside of the container. Similarly, it involves isolating the subgraph in which the cross-build is occuring using a different `Platform`/`Environment`. pantsbuild/pants

#16393 Better explain in Docker docs gotcha with `platforms` and `pex_binary` Issue created by Eric-Arellano

Okay I think what is happening is that for now, Pants creates the binary on your mac beforehand and then copies it into the container. It does not create it in the container (although that is highly requested and we want to do it)

So, the binary is being built w/ macOS deps, and failing to run in Linux. The solution is to use platforms or complete_platforms

This is not very intuitive. While #13682 would be better, in the meantime, this should be better documented. pantsbuild/pants

#17386 Regression in streaming output of test runners Issue created by Eric-Arellano Describe the bug Now, when we finish a result, we stream this:

12:28:52.46 [INFO] Completed: Run Pytest - succeeded.

Before, we streamed a message like this:

Completed: Run Pytest - (environment:remote, src/python/toolchain/buildsense/management/commands/cache_history_test.py:tests) - succeeded.

To reproduce, simply run

./pants test src/python/pants/util/strutil_test.py

. Pants version I suspect this comes from #17134. It happened between `09af33db0dffa0445cd865b7a87d444de30089db 09af33d and 12e339d. pantsbuild/pants

#17387 Pants 2.14.0 `remote_auth_plugin` option causes pants error New issue created by twunderlich-grapl Describe the bug A clear and concise description of the bug. In Pants 2.14.0 if you have

remote_auth_plugin

set in your pants.toml, pants will error out with

[ERROR] remote auth plugin already provided via entry point of a plugin. `[GLOBAL].remote_auth_plugin` should not be specified in options.`

I believe that this was meant to display a deprecation warning and not an error Pants version Which version of Pants are you using? Upgrading to Pants 2.14.0 from Pants 2.13.0 OS Are you encountering the bug on MacOS, Linux, or both? Linux Additional info Add any other information about the problem here, such as attachments or links to gists, if relevant. I do have the toolchain plugin installed, you can see my upgrade PR at https://github.com/grapl-security/codecov-buildkite-plugin/pull/62/files Slack discussion: https://pantsbuild.slack.com/archives/C046T6T9U/p1666970858242799 pantsbuild/pants

#17389 Target visibility scopes New discussion created by kaos Copied from the design doc: Background Monorepos can by definition house very large code bases. In order to maintain good architectural hygiene it is important to adhere to intermodule interfaces and protocols to avoid chaos and escalating maintenance. Use Case By implementing clear boundaries for modules to interact only via public interfaces, the inner workings of the modules can be kept private, avoiding intricate and complicated dependency patterns. The visibility field offers the ability in a declarative way to specify which modules may depend on a certain target. Motivation We want to support large projects promoting good systems design. The visibility feature will be a tool that helps with that. Feature request: #13393 pantsbuild/pants

#17391 Docs for new batched-pytest execution New issue created by danxmoran Add docs for new field / semantics introduced by #17385 pantsbuild/pants

#17129 Require goals to explicitly locate/choose an environment, and pin `graph` calculations to `__local__` Issue created by stuhood Currently,

determine_bootstrap_environment

is called to compute a "default"

EnvironmentName

before the rule graph is entered, such that all `@goal_rule`s have one in scope automatically. While this was useful for getting things going and proving out our cross-compilation capabilities (demonstrated in #11148 and #13682), it is error prone. The environment design has been trending toward (eventually) giving all targets an

environment=

field (even if the vast majority of the time it is only set via

__defaults__

). And in a world where ~all targets have an

environment=

field, ever accidentally using the default would definitely represent an issue. Instead, we'd like to put the onus on each

@goal_rule

to select the relevant environment (usually via targets). Additionally, to resolve the open question in Appendix A in the design doc, we will pin effectively all calculations in

graph.py

to the

__local__

environment (to eventually potentially be made configurable). This will mean that graph-introspection goals won't need to choose environments, and other `@goal_rule`s won't need to select an environment while computing the graph. pantsbuild/pants

#14941 Run pytest on multiple files the pytest-way Issue created by kerys

pants test

triggers pytest on a per-file basis. This has different behavior compared to running

pytest <foldername>

because each pytest session only knows the single test file. This results in: • the pytest report summary on terminal is created per test file, and the only really useful test summary is the one provided by pants which provides less information compared to the pytest summary. • pytest plugins that create summary reports (such as pytest-html, pytest-json) create a single report per file (compared to a single report for all executed tests) • pytest session scope fixtures are initialized repeatedly because for each test file a new pytest session is started. I'm just speaking for myself, but I got so used to these points just working when using pytest that running python tests in pants as it is now feels like a downgrade to me. It would be great if there was a way to have the "vanilla" pytest experience also when using pants. pantsbuild/pants

#17266 TODO: Document synthetic targets API Issue created by kaos What it says on the tin. pantsbuild/pants

#17398 Running `export` for specific Python tools (or lockfiles) only Issue created by achimnol Is your feature request related to a problem? Please describe. Sometimes I modify requirements for a specific Python tool, but I need to re-export all of the virtualenvs including other Python tools and the default one (

python-default

). It takes quite some time. Describe the solution you'd like I'd like to re-run

./pants export

for a specific lockfile only. (e.g.,

./pants export --resolve=flake8,mypy

) pantsbuild/pants

#17402 PR labels New discussion created by kaos The question arose in #17390 about the proper label to select for changes to the default module mapping table in the Python backend. Is it a user-api-change or new feature? Or do we need yet another label? I'm starting this discussion so we can settle this for future PRs of this kind. pantsbuild/pants

#17403 lint fails after fix & fmt didn't changed anything New issue created by thelittlebug Describe the bug the title says it Pants version current main d05c437 OS Linux pop-os 6.0.2-76060002-generic #202210150739~1666289067~22.04~fe0ce53 SMP PREEMPT_DYNAMIC Thu O x86_64 x86_64 x86_64 GNU/Linux Additional info this is the file that didn't pass the lint but it passes the fix & fmt without any changes

import tuhls.dashboard.settings as dashboard_settings
import utils.config as cfg
from django.conf import settings

c = cfg.ConfigDict()
apps = cfg.ConfigList(
    "django.contrib.admin",
    "django.contrib.auth",
    "django.contrib.contenttypes",
    "django.contrib.sessions",
    "tuhls.dashboard",
    "tuhls.dashboard.tests.dashboard_test",
    "tuhls.icons",
)
c = cfg.base_settings(c, "dashboardtest", apps, True)
c = cfg.template_settings(c)
c = cfg.db_sqlite_test_settings(c)
c = dashboard_settings.base(c)
c.ROOT_URLCONF = "tuhls.dashboard.urls"
c.AUTH_USER_MODEL = "dashboard_test.TestUser"
c.LOGIN_URL = "/login/"
c.LOGOUT_REDIRECT_URL = "/"
settings.configure(**c)

as you can see, isort removed the double blank line after the imports (i think that's wrong) but if i add the double blank line and run only lint it fails anyway it could be a isort bug. i have ~1k python source files and it happens only in this file is there an easy way to change the python tool versions? pantsbuild/pants

#17404 `experimental_shell_command` substitutes `{chroot}` but does not set `$CHROOT` (nor have docs) New issue created by huonw Describe the bug

experimental_shell_command

seems to substitute

{chroot}

with the sandbox root, which is handy to find dependencies, but it doesn't set the

$CHROOT

environment variable. This behaviour both: • differs to

experimental_run_shell_command

, which also sets the env var • isn't documented at all (https://www.pantsbuild.org/docs/reference-experimental_shell_command doesn't mention "chroot" at all) Example: https://gist.github.com/huonw/5686e2cdccc1d49e4e1e591884574dcc

git clone <https://gist.github.com/5686e2cdccc1d49e4e1e591884574dcc.git>
cd 5686e2cdccc1d49e4e1e591884574dcc
./pants run //:b

Output (

#

are my annotations)

12:30:38.05 [INFO] Initializing scheduler...
12:30:38.13 [INFO] Scheduler initialized.

# output from "a" (experimental_shell_command):
12:30:38.18 [INFO] env var= sub=/private/var/folders/sv/vd266m4d4lvctgs2wpnhjs9w0000gn/T/pants-sandbox-wmxV9W

# output from "b" (experimental_run_shell_command):
env var=/private/var/folders/sv/vd266m4d4lvctgs2wpnhjs9w0000gn/T/pants-sandbox-uYdVez sub=/private/var/folders/sv/vd266m4d4lvctgs2wpnhjs9w0000gn/T/pants-sandbox-uYdVez

This potentially relates very tangentially to #16807. Pants version 2.15a0 OS macOS Additional info https://gist.github.com/huonw/5686e2cdccc1d49e4e1e591884574dcc pantsbuild/pants

#17405 Running Python code in `experimental_shell_command` requires building a PEX and specifying an appropriate interpreter as a tool New issue created by huonw Describe the bug When

experimental_shell_command

needs to run Python code from the repo, it seems the best way to do that is to depend on a PEX and then specify a compatible Python version as a

tool

(and

bash

too). This seems... unwieldy, and the second part in particular seems like it's liable to end up with 'works on my machine' problems, if developers have different

PATH

configurations. Example: https://gist.github.com/huonw/47bc63951eac7a05a3a3442843f040a9

git clone <https://gist.github.com/47bc63951eac7a05a3a3442843f040a9.git>
cd 47bc63951eac7a05a3a3442843f040a9
./pants run //:print

This would be better if (numbers corresponding to

BUG

comments): 1. (convenience) the shell command could just depend on

script.py

(and its venv/other dependencies) directly, similar to how

./pants run script.py

works mostly the same as

./pants run //:pex

. This would avoid needing another PEX binary that gets unnecessarily explicitly packaged on

./pants package ::

. 2. (reliability/system dependencies) we didn't have to specify both a compatible Python and Bash in

tools

3. (reliability/system dependencies) related to that, if I use

interpreter_constraints = ["CPython==3.7.*"]

(and

tools=["python3.7", "bash"]

), running

./pants run script.py

works fine, but neither

./pants run //:pex

nor

./pants run //:print

do. • I have 3.7.13 installed via pyenv, but not on my path by default (i.e. running

python3.7

hits the pyenv shim, and gets an error like ``The `python3.7' command exists in these Python versions: 3.7.13``). • Running

pyenv global 3.7.13

or

pyenv shell 3.7.13

first, and then rerunning

./pants run //:print

works fine, but would presumably break anything using other versions... Key files for posterity/convenience:

# script.py
print("hello")

# BUILD
python_sources(name="py")
pex_binary(name="pex", entry_point="script.py")

experimental_shell_command(
    name="shell",
    command="{chroot}/pex.pex > output.txt",
    tools=["python3.9", "bash"], # BUG 2
    outputs=["output.txt"],
    dependencies=[":pex"], # BUG 1
)

experimental_run_shell_command(
    name="print", command="cat {chroot}/output.txt", dependencies=[":shell"]
)

# pants.toml
[GLOBAL]
pants_version = "2.15.0a0"

backend_packages = [
  "pants.backend.shell",
  "pants.backend.python",
]

[python]
interpreter_constraints = ["CPython==3.9.*"] # BUG 3

[anonymous-telemetry]
enabled = false

Pants version 2.15.0a0 OS macOS Additional info This is related to https://pantsbuild.slack.com/archives/C046T6T9U/p1666315386420799, and has the same context as #17345: we're using

experimental_shell_command

for ad hoc code generation, where we execute part of our API code to generate a schema file. pantsbuild/pants

#1968 Parallell pex creation fails due to possible race conditions in package installation New issue created by tapetersen We're using pex with makefiles and parallell jobs in CI and are noticing spurious errors like the one below

pex --find-links out/wheels -o out/pexes/pex1.pex ./pex1 -e pex1:main
pex --find-links out/wheels -o out/pexes/pex2.pex -r pex2/requirements.txt --python-script pex2/entrypoint.py
Could not find the installation RECORD for zipp 3.10.0 under /root/.pex/installed_wheels/4fcb6f278987a6605757302a6e40e896257570d11c51628968ccb2a47e80c6c1/zipp-3.10.0-py3-none-any.whl.workdir

The other pex built fine and if I retry it will work. Problem is probably made worse by having all wheels cached so no time is spent downloading them which makes it more likely to run into races when installing. This would probably need some locking around the installed_wheels folder or more fine grained locking/atomik operations for the real installation folder. It's not urgent for us as I can avoid making those in parallell or ensure separate pexdirs but it was somewhat of a nasty issue to debug and would perhaps save someone the time and frustration in the future. pantsbuild/pex

#1969 pex build fails due to existing work-directory New issue created by christopherfrieler Beginning with version 2.1.105 building the pex file in our CI pipeline fails with the following message:

.../python3.8/site-packages/pex/atomic_directory.py176 PEXWarning: [pid:XX, tid:XXX, cwd...] After obtaining an exclusive lock on <PEX_ROOT>/isolated/.2f4fc85fa2be055a2975ce1147100c0d5c7e663a.atomic_directory.lck, failed to establish a work directory at <PEX_ROOT>/isolated/2f4fc85fa2be055a2975ce1147100c0d5c7e663a.workdir due to: [Errno 17] File exists: '<PEX_ROOT>/isolated/2f4fc85fa2be055a2975ce1147100c0d5c7e663a.workdir'

pex_warnings.warn(

.../python3.8/site-packages/pex/atomic_directory.py187 PEXWarning: [pid:XX, tid:XXX, cwd...] Continuing to forcibly re-create the work directory at <PEX_ROOT>/isolated/2f4fc85fa2be055a2975ce1147100c0d5c7e663a.workdir.

pex_warnings.warn(

Failed to spawn a job for .../bin/python: [Errno 17] File exists: '<PEX_ROOT>/isolated/2f4fc85fa2be055a2975ce1147100c0d5c7e663a.workdir/pex/./venv'

It seems to have to do with #1905 introduced in version 2.1.105, but we have no clue, why this is happening in our CI pipeline, while building the .pex file on MacOS developer machines works. It looks like something else is creating that directory, but there is only one pex command in the pipeline job and the PEX_ROOT is not cached. Our build environment uses: • the Red Hat UBI 8.4 Docker image • Python 3.8 • poetry 1.1, which manages pex as a dev-dependency Then we build the pex with

poetry run pex --inherit-path --python=python3.8 --requirement requirements.txt --find-links dist/ our_module --output-file dist/final.pex

Any idea why this is happening or what else we could check would be helpful. pantsbuild/pex

#17408 Feedback wanted re Github Discussions New discussion created by cczona In recent weeks we've been exploring use of GitHub Discussions. Its biggest advantages include persistence (Slack messages disappear after 3 months), search engine indexing (the world doesn't get to see how welcoming and responsive our Slack community is!), and excellent integration with GH Issues. Now that folks have had a chance to give it a try, thoughts please! Shall we expand the experiment a bit by increasing the number of conversations we hold on Discussions? pantsbuild/pants

#1970 Pex does not support `#subdirectory=X` in archive URLs New issue created by jsirois Although subdirectory works for VCS URLs:

$ pex "sdev_logging_utils @ git+<https://github.com/SerialDev/sdev_py_utils@bd4d36a0#subdirectory=sdev_logging_utils>" -- -c 'import sdev_logging_utils; print(sde
v_logging_utils.__file__)'
/home/jsirois/.pex/installed_wheels/17ea6d41d9e674356d65867174aedd68eebe1f27c9c3af7caa5b41220e9fa3f1/sdev_logging_utils-0.1-py3-none-any.whl/sdev_logging_utils/__init__.py

They do not work for archives:

$ pex "sdev_logging_utils @ <https://github.com/SerialDev/sdev_py_utils/archive/bd4d36a0.zip#subdirectory=sdev_logging_utils>" -- -c 'import sdev_logging_utils; p
rint(sdev_logging_utils.__file__)'
pid 27573 -> /home/jsirois/.pex/venvs/eb7094f36ecbc6fef7a9ae166bfc5e280dfd242b/5985ed09b49a653d6596b0e14d134c5456cf1a9f/bin/python -sE /home/jsirois/.pex/venvs/eb7094f36ecbc6fef7a9ae166bfc5e280dfd242b/5985ed09b49a653d6596b0e14d134c5456cf1a9f/pex --disable-pip-version-check --no-python-version-warning --exists-action a --no-input --use-deprecated legacy-resolver --isolated -q --cache-dir /home/jsirois/.pex/pip_cache wheel --no-deps --wheel-dir /home/jsirois/.pex/built_wheels/sdists/bd4d36a0.zip/1eaf75287d0a0a4a820f6fd7661ccef36c22e448b40733c8b7dab26060f9203e/cp310-cp310-manylinux_2_35_x86_64.workdir /home/jsirois/.pex/downloads/resolver_download.93vmt089/home.jsirois.bin.pex.venv.bin.python/bd4d36a0.zip --index-url <https://pypi.org/simple> --retries 5 --timeout 15 exited with 1 and STDERR:
    ERROR: Command errored out with exit status 1:
     command: /home/jsirois/.pex/venvs/eb7094f36ecbc6fef7a9ae166bfc5e280dfd242b/5985ed09b49a653d6596b0e14d134c5456cf1a9f/bin/python -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/home/jsirois/.pex/pip_cache/.tmp/pip-req-build-1lio9mtt/setup.py'"'"'; __file__='"'"'/home/jsirois/.pex/pip_cache/.tmp/pip-req-build-1lio9mtt/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' --no-user-cfg egg_info --egg-base /home/jsirois/.pex/pip_cache/.tmp/pip-pip-egg-info-sugze2us
         cwd: /home/jsirois/.pex/pip_cache/.tmp/pip-req-build-1lio9mtt/
    Complete output (5 lines):
    Traceback (most recent call last):
      File "<string>", line 1, in <module>
      File "/usr/lib/python3.10/tokenize.py", line 394, in open
        buffer = _builtin_open(filename, 'rb')
    FileNotFoundError: [Errno 2] No such file or directory: '/home/jsirois/.pex/pip_cache/.tmp/pip-req-build-1lio9mtt/setup.py'
    ----------------------------------------
ERROR: Command errored out with exit status 1: python setup.py egg_info Check the logs for full command output.

This is as opposed to Pex's vendored Pip itself, which does work in either case:

$ PYTHONPATH=pex/vendor/_vendored/pip python -mpip -V
pip 20.3.4 from /home/jsirois/dev/pantsbuild/jsirois-pex/pex/vendor/_vendored/pip/pip (python 3.10)

$ rm -rf pip-test && PYTHONPATH=pex/vendor/_vendored/pip python -mpip install --prefix pip-test "sdev_logging_utils @ git+<https://github.com/SerialDe/sdev_py_utils@bd4d36a0#subdirectory=sdev_logging_utils>"
Collecting sdev_logging_utils@ git+<https://github.com/SerialDev/sdev_py_utils@bd4d36a0#subdirectory=sdev_logging_utils>
  Cloning <https://github.com/SerialDev/sdev_py_utils> (to revision bd4d36a0) to /tmp/pip-install-92f0p4va/sdev-logging-utils_7de7920fd66b412297c724bde456f193
  Running command git clone -q <https://github.com/SerialDev/sdev_py_utils> /tmp/pip-install-92f0p4va/sdev-logging-utils_7de7920fd66b412297c724bde456f193
  WARNING: Did not find branch or tag 'bd4d36a0', assuming revision or ref.
  Running command git checkout -q bd4d36a0
Using legacy 'setup.py install' for sdev-logging-utils, since package 'wheel' is not installed.
Installing collected packages: sdev-logging-utils
    Running setup.py install for sdev-logging-utils ... done
Successfully installed sdev-logging-utils-0.1
WARNING: You are using pip version 20.3.4; however, version 22.3 is available.
You should consider upgrading via the '/usr/bin/python -m pip install --upgrade pip' command.

$ rm -rf pip-test && PYTHONPATH=pex/vendor/_vendored/pip python -mpip install --prefix pip-test "sdev_logging_utils @ <https://github.com/SerialDev/sdev_py_utils/archive/bd4d36a0.zip#subdirectory=sdev_logging_utils>"
Collecting sdev_logging_utils@ <https://github.com/SerialDev/sdev_py_utils/archive/bd4d36a0.zip#subdirectory=sdev_logging_utils>
  Downloading <https://github.com/SerialDev/sdev_py_utils/archive/bd4d36a0.zip>
     - 251 kB 353 kB/s
Using legacy 'setup.py install' for sdev-logging-utils, since package 'wheel' is not installed.
Installing collected packages: sdev-logging-utils
    Running setup.py install for sdev-logging-utils ... done
Successfully installed sdev-logging-utils-0.1
WARNING: You are using pip version 20.3.4; however, version 22.3 is available.
You should consider upgrading via the '/usr/bin/python -m pip install --upgrade pip' command.

pantsbuild/pex

Comment on #17402 PR labels for default module mapping Discussion answered by Eric-Arellano It really is internal, imo. It shouldn't break any existing users because their overrides take precedence over our default. So it's indeed not User API Change. It's also not really a new feature. The only reason I wasn't using internal in the past is that often these PRs come from first-time contributors and I wanted their change to show up in the changelog. But that doesn't seem very sound. We wouldn't do that if they made other types of internal changes like a refactor. And we already give them attribution in our Contributors.md and Slack/Google Group announcements. pantsbuild/pants

#17413 Prevent using a `@union` as a `@rule` argument New issue created by stuhood See #17412 for the pitfall of using a

@union

as a

@rule

argument. pantsbuild/pants

#17414 Better support for tools with built-in file-watching/hot-reload, for faster dev loops New issue created by huonw Is your feature request related to a problem? Please describe. A lot of dev tools (especially in the JS/TS world, #14190) have built-in hot reload in response to file changes. This hot reload is often much faster than starting the whole dev tool from scratch, and can preserve context/state. For instance, a lot of frontend UI dev work is done via https://storybook.js.org to be able to view, test and iterate on components in isolation, quickly. It seems a common dev flow is to start the storybook server as a long running process (maybe as an NPM script, e.g.

"scripts": { "storybook": "start-storybook -p 6006 -s public", ... }

in

package.json

, which can be invoked as

npm run storybook

), and then edit the code. The changes becomes visible in the storybook interface very soon after via file-watchin gand hot-reload built in to storybook. Starting storybook from scratch takes much longer (1min or more in our codebase). (There's other examples too, e.g. our frontend has a

start

script to start the react development server, that does similar hot-reloading for the whole frontend when the code changes, without needing to start from scratch, or lose (much) state.) AIUI, pants will typically interrupt a long running process when there's changes, and then start it in a new sandbox. This is a significantly worse dev experience for at least two reasons: • much higher latency (1min, rather than <1s) • potentially losing context/state (e.g. which component is visible, scroll position, any entered data) Describe the solution you'd like A flag similar to

restartable

that has almost exactly 'opposite' behaviour: instead of interrupting the process and starting in a new sandbox, just copy across any changes into the existing sandbox and let the code running within the sandbox respond to that change. (This includes copying in all transitive changes, such as codegen output changing.) For instance, in the best case, we might have something like

typescript_sources(name="frontend", sources=["**/*.ts", "**/*.tsx"])
experimental_shell_command(
   name="storybook"
   command="npx start-storybook -p 6006 -s public",
   dependencies=[":frontend"],
   tools=[...],
   update_sandbox_live=True, # NEW FEATURE
)

(A "best case" sketch relying on #16803 (for running the shell command), #16961 (for typescript) and #17405 (related to being able to run the

start-storybook

command from an external dependency).) Given this

./pants run path/to:storybook

would start a long-running storybook server (for instance, in

/tmp/pants/sandbox/123abc

), and changes to any of the frontend files would be copied into

/tmp/pants/sandbox/123abc

directly. Describe alternatives you've considered I cannot think of one 😄 Additional context Some solution here will be important for us to be able to migrate the JS/TS parts of our monorepo to pants, in addition to the Python parts. pantsbuild/pants