#17721 Support `./pants freeze --resolve=xyz` New issue created by da-tubi Is your feature request related to a problem? Please describe. import a lockfile into an existing virtualenv Describe the solution you'd like

./pants freeze --resolve=xyz.                      # like pip freeze
./pants freeze --resolve=xyz --secured     # like pex3 lock export

or

./pants import --resolve=xyz # in that way, we do not need to pip install

Describe alternatives you've considered Workaround 1 https://github.com/da-tubi/pants-minimal-freeze Workaround 2 The following command line is similar to

pip freeze

pex3 lock export 3rdparty/python/xxx.lock

Then we can use the exported

requirements.txt

in the specific virtual env:

pex3 lock export 3rdparty/python/xxx.lock > /tmp/requirements.txt
# pyenv activate xxx or conda activate xxx
pip install -r /tmp/requirements.txt

And

pex3 lock export

does not work for pants' lockfile, because there are

//

leading parts. For now, we can use command line to strip the leading

//

. Additional context None pantsbuild/pants

#14070 Use immutable_inputs for `PEX`s New issue created by stuhood #13848 (designed in #12716) added support for immutable process inputs, which are symlinked into the sandbox, rather than copied there. Once immutable inputs have stabilized (notably, once #13899 is fixed), we should provide the

repository.pex

as an immutable input to significantly reduce IO. It's also possible that in most cases where a PEX is used as an input to a process, that supplying it as an immutable input would make sense: in particular, cases where a PEX contains only thirdparty code, and firstparty code is materialized as loose files. pantsbuild/pants

#17282 Include/exclude and heuristic-based `immutable_inputs` New issue created by stuhood

Process.immutable_inputs

is currently configured as a mapping from a file path to a

Digest

to materialize at that path. It is merged into the

input_digest

of a

Process

to form a "complete"

Digest

of inputs. But this separation is a bit unfortunate, because it means that using

immutable_inputs

(which should be fairly widely used: see discussion on #14070) means a lot of change to how your

Process

is constructed. * * * From an API-changes perspective, we should: 1. remove the guarantee that inputs are mutable by default, to allow us to automatically choose the strategy that we use for each path, and symlink wherever we choose to. • Then, by default, we should automatically use symlinking for files or directories over a certain size, etc. 2. convert the existing

immutable_inputs

list into a flat list of paths which should be _forced_/includelisted to be materialized as immutable (regardless of the heuristic from above), but which must already exist in the

input_digest

(maybe "force" should go in the name?

force_immutable_paths=

...?). • This will mean that unlike today (where you need to change how your

Process

is constructed to stop adding an input to the

input_digests

and instead add it to the

immutable_inputs

as a dict), you would only need to add a path to the

immutable_inputs

, without changing your

input_digest

. 3. add an

mutable_inputs

(or

force_mutable_paths=

...?) flat list of paths to act as an excludelist that prevents a path from being materialized as a symlink. * * * From an implementation perspective, this would look like moving the support for creating the symlinks for

ImmutableInputs

from an explicit step before running a process: pants/src/rust/engine/process_execution/src/local.rs Lines 641 to 662 in </pantsbuild/pants/commit/08410e5b94d3e3aa86b2538ee1991d964e27e72c|08410e5> ...to a thing that happens inside

fn materialize_directory

when an includelisted path is encountered: pants/src/rust/engine/fs/store/src/lib.rs Lines 1188 to 1201 in </pantsbuild/pants/commit/08410e5b94d3e3aa86b2538ee1991d964e27e72c|08410e5>

materialize_directory(_helper)

would take either: 1. a reference to a collection of paths which were includelisted as immutable 2. a trait or function which implemented the heuristic for which `directory::Entry`s / paths to includelist ...and a reference to

ImmutableInputs

, and then would create a symlink to the

ImmutableInputs

whenever it encountered one of the includelisted paths. pantsbuild/pants

#17726 Python Pipfile CodeArtifact Issue created by nikwotton Describe the bug I have a project with a pipfile and pipfile.lock. The pipfile is pointing to an internal codeartifact instance, so I’ve therefore added a

indexes.add

line to pants.toml. I’m trying to run ./pants generate-lockfiles but it’s erroring in claiming it’s unable to find specific artifacts. I think it’s finding some of them, because as I tweak the pipfile.lock it moves onto other artifacts it can’t find. The logs aren’t being super helpful though, just regurgitating that it can’t find an artifact matching a long list of requirements. Is there a way to enable a verbose/debug logging so I can see if it’s finding some before it fails on others vs just not finding any? Or any other debugging tips I could try? Pants version 2.14.0 OS Linux inside docker pantsbuild/pants

#17727 Override virtualenv prompt when exporting using symlinks Issue created by danxmoran Is your feature request related to a problem? Please describe. When you export a Python virtualenv using symlinks:

./pants export --resolve=<resolve> --symlink-python-virtualenv

You end up with a link at

./dist/export/python/virtualenvs/<resolve>/<python-version>

pointing into Pants' named caches. The linked dir has a sha for a name. When you activate the virtualenv, your prompt then changes to be the sha-name of the cache dir, instead of the link:

$ source ./dist/export/python/virtualenvs/<resolve>/<python-version>/bin/activate
# Prompt changes to:
(0796ea8d0cb05fb74572064697475fd9cfdfa756) $

This takes up a ton of real estate in the console, and is confusing if you don't know how everything links up under-the-hood. Describe the solution you'd like

python -m venv

supports a

--prompt

option to set the

PS1

generated for a venv, overriding the default of using the venv's

basename

. It looks like pex's

venv

command already has the same option. It'd be nice if the export logic plumbed through the resolve name (maybe configurable via option?) to the process that generates the virtualenv in the cache dir. Describe alternatives you've considered We could disable the default virtualenv prompt and roll our own prompt generation via wrapper script, but we're trying to get away from needing so many wrappers now that we're adopting Pants. pantsbuild/pants

#17729 Allow using a shell command as part of any goal (check, lint, fmt, fix, deploy, ...) Issue created by huonw Is your feature request related to a problem? Please describe. Shell is a universal glue, that allows augmenting pants adhoc with a lower barrier to entry (and, for simple tasks, lower maintenance burden) than writing a custom plugin. It'd be nifty to generalise the current codegen

experimental_shell_command

, run

experimental_run_shell_command

and recently test

experimental_test_shell_command

(#17640) to allow using a shell script for 'anything'. For example, in a polyglot repo, pants may support some of the languages/tools natively, but not others, but it'd still be nice to have basic commands like

./pants lint ::

and

./pants fmt ::

work across the whole system. Additionally, it would allow faster experiments and migration of an existing repo into pants, if glue scripts (and/or makefiles, or whatever) can be reused, and switch to having

./pants

to more. (We 'tripped' over both of these in our migration into pants, but I personally have liked pants enough to suffer through split tooling, and convince the rest of the team too, as well 😅 ) Describe the solution you'd like Something way to have a shell command hook into other goals. For instance:

# in a directory with CFN templates
experimental_shell_command(
   name="my-linter",
   command="cfn-lint",  # <https://github.com/aws-cloudformation/cfn-lint>, installed somehow
   tools=["..."],
   goal="check",
   dependencies=["./*.yaml"]
)

# in a docs/ directory
experimental_shell_command(
   name="my-formatter",
   command="mdformat",  # <https://github.com/executablebooks/mdformat>
   tools=["..."],
   goal="fmt",
   dependencies=["./**/*.md"]
)

• The

dependencies

would be the input files (for file-based commands). • The

outputs

field could be reused for

fmt

and

fix

to write back to the codebase. Some questions that seem relevant (although some can probably be deferred/considered potential future enhancements?): 1. How do these sort of shell commands get ordered with other `fmt`/`fix` subsystems (e.g. if

my-formatter

touches python files, does it run before or after black)? 2. Can a shell command delete files as part of

fmt

and

fix

? How does that work? 3. Maybe there can be opt-in or opt-out for invocations to run in parallel on subsets of the input? 1. How does the target distinguish between configuration file that needs to be in all invocations, and an input file should be in exactly one invocation? 2. Are outputs just naively merged (i.e. if one outputs

a.txt

and another outputs

b.txt

, the final output has both)? What happens if two invocations generate the same path (with different or identical content)? 4. Maybe lumping all goals into one command isn't quite right, since they may have very different options? 5. Is it easy/possible to express dependencies like

**/*.py

(to be able to run a custom linter on all Python files in descendent directories, without writing out the path to their individual targets)? 6. How do custom commands get installed? Some possibilities: • #17277 (or similar) for downloadable tools • a docker image environment • in-repo executables (#17405 may be relevant) Describe alternatives you've considered We are: • Actively using: shell scripts outside pants, with some sort of orchestrator for them (e.g. GitHub Actions YAML, internal documentation to copy-paste from for us; potentially Make or similar too) • Considering: custom plugins Additional context N/A pantsbuild/pants

#17733 Confusing 'DifferingFamiliesError: Expected AddressMaps to share the same parent directory' exception for use of file glob in dependencies New issue created by huonw Describe the bug I tried using a directory glob for files in a dependency array of a target, and got exceptions like

09:55:59.81 [ERROR] 1 Exception encountered:

  DifferingFamiliesError: Expected AddressMaps to share the same parent directory 'dir/*', but received: 'dir/sub/BUILD'

Reproducer: https://gist.github.com/huonw/1d2e20a961111eba39a20ac55606cdd7 (directory structure created in

make.sh

, since one cannot have directories in a gist)

git clone <mailto:git@gist.github.com|git@gist.github.com>:1d2e20a961111eba39a20ac55606cdd7.git
cd 1d2e20a961111eba39a20ac55606cdd7

bash ./make.sh

./pants version 
# 2.14.0

./pants dependencies dir:single 
# DifferingFamiliesError: Expected AddressMaps to share the same parent directory 'dir/*', but received: 'dir/sub/BUILD'
./pants dependencies dir:recursive
# DifferingFamiliesError: Expected AddressMaps to share the same parent directory 'dir/**', but received: 'dir/BUILD'

I'm suspect I'm doing something wrong here, but, if I am, the error message isn't making it obvious what that is, or how to fix it. For example, I'm not sure what

AddressMaps

is, and am confused by the mention of the

BUILD

file when it is talking about a parent directory? Pants version I tried: 2.14.0, 2.15.0a1, 2.16.0.dev0. The latter two had additional traceback, e.g.

Engine traceback:
  in `dependencies` goal
  in Resolve direct dependencies of target - dir:recursive
  in Search for addresses in BUILD files - dir/**

OS macOS Additional info https://gist.github.com/huonw/1d2e20a961111eba39a20ac55606cdd7 pantsbuild/pants

#17737 Dynamic UI is used and results in poor output within TERM=dumb contexts like emacs compilation-mode Issue created by huonw Describe the bug When I run pants within a

compilation-mode

buffer in Emacs 28, I find I have to set

PANTS_DYNAMIC_UI=false

, because the dynamic UI ends up with each progress update on its own line. This means: • it's very hard to find the useful output: error messages are hidden • the overhead of rendering all those lines seem to be enough to make the build slower Video shows

M-x compile

./pants fmt ::

in the pants repo, generating thousands of lines of output (I cut it off at ~1.3k, but it does eventually succeed), while a normal run is more like 100: Untitled.mov I'd be expecting pants to either: • have the dynamic UI work 'properly', like it does in other terminals (i.e. all the lines with spinners are transient/overwritten) • disable the dynamic UI automatically A TQDM progress bar like the following works fine, with normal dynamic behaviour:

M-x compile

python pbar.py

# pbar.py
from tqdm import tqdm # tqdm==4.64.1
import time

for i in tqdm(range(1000)):
    time.sleep(0.01)

Untitled.2.mov Similarly for another random spinner library I found:

import halo # halo==0.0.31
import time
halo.Halo(text="whatever", spinner="dots").start()
time.sleep(10)

Pants version 2.14.0 in our repo, current head (bca1b5a) of pants repo running against itself OS GNU Emacs 28.1 (build 1, aarch64-apple-darwin21.1.0, NS appkit-2113.00 Version 12.0.1 (Build 21A559)) of 2022-04-05 (macOS) Additional info N/A pantsbuild/pants

#2004 Support for generating non-hermetic console scripts in `venv` Issue created by danxmoran The

venv

command rewrites console script shebangs in the generated virtualenv here, adding

-sE

in the process. Those added flags make it more difficult to use scripts that rely on

PYTHONPATH

(i.e.

pylint

trying to load in-repo plugins). It would be nice if

venv

supported an option to disable the behavior. pantsbuild/pex

#2005 Specifying `--platform`in `lock create` impedes referring to sdists Issue created by thejcannon Both: •

pex3 lock create --platform manylinux_2_31_x86_64-cp-38-cp38 timeout-decorator~=0.4

•

pex3 lock create --style sources --platform manylinux_2_31_x86_64-cp-38-cp38 timeout-decorator~=0.4

result in an error because

timeout-decorator

is an

sdist

-only package. pantsbuild/pex

#17739 We can/should partition more tools by config (black, flake8, etc...) Issue created by mpcusack-color Is your feature request related to a problem? Please describe. Originally discussed on slack: https://pantsbuild.slack.com/archives/C046T6T9U/p1670281834539229 A large (and knotted) portion of my monorepo is a django project. We would like start using pants to do mypy type checking on it and use mypy_django_plugin. The problem is mypy config plugin installation applies everywhere and installing this plugin requires that we add our entire django project as a mypy dependency, even when checking a file that is not part of the django code. Having the django plugin is not absolutely required to run mypy on this code, but it makes the user experience significantly better and would likely speed up fixing all the current errors. Mypy doesn’t have support for only enabling a plugin for a subset files/roots. There is only one configuration file ($ROOT/mypy.ini) and it doesn’t support hierarchical config like pytest’s conftest.py. To enable the django plugin in mypy you need to make the project settings module and all transitive deps (the entire django app) a mypy tool resolve dependency. The plugin throws an error if the project can’t be

django.setup()

. Describe the solution you'd like What I need is a way in pants to specify more than one mypy config file and then specify that certain targets should use the non default one. Apparently scalafmt already somewhat does this? Related: #17328 pantsbuild/pants

#17740 coverage broken with pytest 7.2.0 Issue created by mpcusack-color Describe the bug We recently upgraded to

pytest==7.2.0

and noticed we can no longer use

test --use-coverage

.

pytest.PytestDeprecationWarning: The hookimpl CovPlugin.pytest_configure_node uses old-style configuration options (marks or attributes).
Please use the pytest.hookimpl(optionalhook=True) decorator instead
 to configure the hooks.
 See <https://docs.pytest.org/en/latest/deprecations.html#configuring-hook-specs-impls-using-markers>
 at /home/mpcusack/.cache/pants/named_caches/pex_root/venvs/s/4ea0b489/venv/lib/python3.8/site-packages/pytest_cov/plugin.py:256

https://docs.pytest.org/en/7.2.x/changelog.html#pytest-7-2-0-2022-10-23 deprecated a hook style in pytest-dev/pytest#4562. This wasn't adopted in

pytest-cov

until

4.0.0

https://pytest-cov.readthedocs.io/en/latest/changelog.html#id1 Pants hardcodes the

pytest-cov

version less than `3.1`: pants/src/python/pants/backend/python/subsystems/pytest.py Line 79 in </pantsbuild/pants/commit/bca1b5a4ae311511fe87ff0dd3ccecc32955d024|bca1b5a> Pants version 2.15.0a1 OS Linux pantsbuild/pants

#17441 go: support passing arbitrary assembler flags Issue created by tdyas

go

supports passing arbitrary assembler flags to

go tool asm

invocations via a

-asmflags

argument. Pants should support this feature. Model as fields on

go_binary

and

go_package

(for tests) that set additional assembler flags to use with defaults coming from option(s). pantsbuild/pants

#17446 go: use target spec in output path for coverage Issue created by tdyas Use target spec as part of output path for coverage results instead of import path with

/

->`_` conversion. Would match better with how other output is generated under

dist/

. pantsbuild/pants

#17746 command not found New discussion created by jgarte `Hi, when following this guide I get the following error: `` ./pants /home/jgart/pants/build-support/pants_venv: line 20: -c: command not found Bootstrapping pants_deps with requirements: /home/jgart/pants/3rdparty/python/requirements.txt /home/jgart/pants/build-support/pants_venv: line 20: -c: command not found /home/jgart/pants/build-support/pants_venv: line 20: -c: command not found /home/jgart/pants/build-support/pants_venv: line 33: : command not found Failed to create venv.

How should I set PY environment variable? I think that might be the issue.

pantsbuild/pants

#17747 Deploy pex files to Databricks jobs/pipelines using dbx Issue created by ericfeunekes Is your feature request related to a problem? Please describe. I would like to use pants to create pex files for each databricks job that I create (a python script) and then deploy those files to the applicable job clusters. Describe the solution you'd like The dbx plugin would look at the dbx

conf

files to determine which files relate to each job, it would build each of those scripts into a pex file and then deploy those to databricks to be installed when the job starts (likely using an init script) Describe alternatives you've considered Currently doing this with the dbx integration with Poetry. dbx builds a single

whl

file and deploys that to be installed on all the job clusters that use the package. The problem with this approach is that the build gets bigger and bigger as new jobs are added and the number of dependencies expands, even if those dependencies only relate to a single job. This means install times and therefore job startup times get longer. pantsbuild/pants

#2006 Overwriting sys.argv[0] leaves no remaining methods to identify the actual sys.argv[0] Issue created by jamesmeador As of the 2.1.91 release, sys.argv[0] is no longer a reliable option for identifying the executed script as called. For example, creating a symlink to a pex, then calling that symlink yields a sys.argv[0] that does not point to the symlink. The PEX environment variable also does not point to the symlink. Are there any known remaining ways to obtain argv[0] as provided by the calling binary? pantsbuild/pex

#17748 How to merge user configuration files with Pants configuration files New discussion created by sureshjoshi From Slack: https://pantsbuild.slack.com/archives/C0D7TNJHL/p1669737065712459 What's the pattern for when we want to merge user configuration with Pants configuration? I can imagine a few styles: • Allow for a "template"-style configuration with keywords (ala PyOxidizer config, where we just inject some parameters) • Accept a config and overwrite parts as we see fit (e.g. venv paths) • Accept a config and only add parts that aren't there (non-destructive, but possibly confusing to a user unless they know to leave certain configs empty to let Pants fill) • Some combination/permutation of above • Probably something else involving options/targets I'm going to work on pyright's config, and for some paths/venvs, I'll want to put Pants-generated locations, and ignore everything else. BUT, I can see a use case where a user may want to specify a local development venv for testing (for example) Josh

Do what works and makes the most sense to the user? In that order?

Benjy

IIRC there is some precedent, in some of the linters, I forget which ones, where we override user config by reading JSON/TOML, adding to it, and writing it back out

pantsbuild/pants

#2008 Release 2.1.119 Issue created by jsirois On the docket: ☑︎ The PEX_TOOLS venv --compile option should ignore compile errors. #2001 ☑︎ Add PEX_DISABLE_VARIABLES to lock down a PEX run. #2014 pantsbuild/pex

#17749 Ability to run different mypy partitions-by-config in different resolves Issue created by mpcusack-color An extension of #17739. The full problem is laid out here. We would like to use mypy_django_plugin for a subset of our repo. The first step of this is the ability to partition the

check

goal by different

mypy

configuration so we can only optionally enable the plugin. Even after this is accomplished the problem remains that in order to function the plugin requires that the entire django project be included in the

mypy

resolve. We are willing to take this

check

performance hit for the django code, but not for the rest of our codebase (the django project is large and pulls in ~150 3rdparty deps). I would like to be able to have multiple mypy resolves with different

extra_requirements

and specify which to use for the different mypy configs. pantsbuild/pants

#17754 Hardlinking feature appears to break CI Issue created by chrisjrn Changes introduced by f8df117 appear to break CI. #17751 shows that reverting the commit fixes CI. #17752 shows that increasing timeouts does not fix CI. @thejcannon suggested disabling the feature in code rather than reverting the relevant code. pantsbuild/pants

#16188 Remote cache read timeout should be applied below the concurrency limit Issue created by stuhood Our gRPC clients use a concurrency limit at the service level. But we apply a cache read timeout above that concurrency limit here, which means that rather than only applying to the network portion of the cache lookup (as might be expected), it is also applied to queueing time waiting to send the request. pantsbuild/pants

#17758 `Not a directory`/`Is a directory` error from `./pants package` after switching `pex_binary` layout with existing `dist/` entry Issue created by huonw Describe the bug It seems like

./pants package

(and particularly packaging

pex_binary

) doesn't handle updating

dist/

when export has changed from a file to a directory, or vice versa, and the old form still exists in

dist/

. This can occur if

./pants package

has been run to export a

pex_binary(layout="zipapp")

to

dist/

in the past, and then the target is updated to

pex_binary(layout="packed")

and packaged again, without clearing

dist/

. Preferably pants would overwrite the existing

dist/

entry harder, so the switch is transparent. Reproducer: https://gist.github.com/huonw/5cc8fcc78bfbaad05209bfb2fd55ac2a (Output marked by

#>

) (_edit_: see #17758 (comment) below for a less fiddly version using

export-codegen

and

experimental_shell_command

)

git clone <mailto:git@gist.github.com|git@gist.github.com>:5cc8fcc78bfbaad05209bfb2fd55ac2a.git
cd 5cc8fcc78bfbaad05209bfb2fd55ac2a

# first: file replaced by a directory
./pants package ::
file dist/pex.pex
#> dist/pex.pex: Zip archive data, ...

# switch to packed layout
sed -i '' 's/# layout/layout/' BUILD

# BUG:
./pants package ::
#> Exception: Error opening file .../dist/pex.pex/.bootstrap for writing: Os { code: 20, kind: NotADirectory, message: "Not a directory" }

# second: directory replaced by a file
rm -rf dist/
./pants package :: 
file dist/pex.pex
#> dist/pex.pex: directory

# switch back
sed -i '' 's/layout/# layout/' BUILD

# BUG:
./pants package ::
#> Exception: Error opening file .../dist/pex.pex for writing: Os { code: 21, kind: IsADirectory, message: "Is a directory" }

Pants version 2.14.0, 2.16.0.dev0 OS macOS Additional info https://gist.github.com/huonw/5cc8fcc78bfbaad05209bfb2fd55ac2a pantsbuild/pants

#17759 Visibility: cannot match `python_requirements` specified at the root level Issue created by jake-sigtech Describe the bug Using the new visibility feature

__dependencies_rules__

I cannot target a requirements file at the root project level. Example:

//:requirements#numpy

A workaround for now is to use a glob:

//**/:requirements#numpy

. I think the issue is related to this line here. I'm not sure it should remove the

//

prefix. I think the intention is to remove

./././

and

/././.

prefixes. Pants version 2.16.0.dev1 OS MacOS. Additional info My repo layout is:

core/BUILD
pants
pants.toml
<http://requirements.in|requirements.in>
BUILD

In `BUILD`:

python_requirements

are specified as

python_requirements(name="requirements", source="<http://requirements.in|requirements.in>")

In `core/BUILD`: Visibility is defined as: This doesn't work:

__dependencies_rules__(
      (
        "*",
        "/**",
        "//:requirements#numpy",
        "!*"
      ),
)

This does work:

__dependencies_rules__(
      (
        "*",
        "/**",
        "//**/:requirements#numpy",
        "!*"
      ),
)

pantsbuild/pants

#17345 `experimental_shell_command`'s sandbox and invalidation includes transitive dependencies of `experimental_shell_command` dependencies, not just outputs Issue created by huonw Describe the bug When a

experimental_shell_command

(

:second

) depends on another

experimental_shell_command

(

:first

), the sandbox for

:second

include the dependencies of

:first

, rather than just the output of

:first

. https://gist.github.com/huonw/c55d0b387ed6030cda611898ee2d0361 provides a reproducer, where

:first

generates

output.txt

by 'consuming'

input.txt

(but does nothing with it), and

:second

includes a

sleep

to demonstrate when it is running. The

archive

package includes all the

.txt

files that ended in the sandbox for `:second`: I'd expect it to only be the direct output of

:first

(

output.txt

), not the

input.txt

dependency of

:first

. BUILD file from that gist for convenience:

file(name="input", source="input.txt")
experimental_shell_command(
    name="first",
    command="""
    echo "doesn't affect output"
    echo contents > output.txt
    """,
    tools=["echo"],
    dependencies=[":input"],
    outputs=["output.txt"]
)

experimental_shell_command(
    name="second",
    command="""
    sleep 3 # make 'actually running' obvious
    """,
    dependencies=[":first"],
    tools=["sleep"],
    outputs=["*.txt"],
)
archive(name="archive", files=[":second"], format="zip")

git clone <mailto:git@gist.github.com|git@gist.github.com>:c55d0b387ed6030cda611898ee2d0361.git
cd c55d0b387ed6030cda611898ee2d0361
./pants version # 2.13.0

# initial build (takes about 3 seconds to run :second, due to sleep)
./pants package ::

# check package output:
unzip -l dist/archive.zip # two files: input.txt output.txt

# validating the cache works as expected (~instantaneous)
./pants package ::

# BUG: change to :first dependencies, that doesn't affect output... 
echo 'new contents' > input.txt
# ... reruns :second, and thus takes ~3 seconds
./pants package ::

# EXPECTED: change to :first script that doesn't affect output...
sed -i '' 's/affect output/affect output still/' BUILD
# .... doesn't rerun :second, only :first
./pants package ::

AFAICT, there's no way to break `:second`'s importing of

:input

, e.g. adjusting to

dependencies=[":first", "!:input"]

doesn't do anything: the behaviour is the same (and using

!!:input

isn't supported). The

:input

being

file

is just for convenience. This appears to apply to all other target types, e.g. a

pex_binary

or yet another

experimental_shell_command

. I haven't checked how this behaves when depending on explicit codegen targets, only these adhoc

experimental_shell_command

ones. Pants version 2.13.0 OS macOS Additional info Background: how are we using

experimental_shell_command

to hit this? We're using

experimental_shell_command

to try to bridge the gap between Pants supported code and unsupported code (JS/TS), as well as for adhoc codegen tasks (to avoid avoid having to write and maintain a plugin). This can result in 'long' chains of `experimental_shell_command`s that depend on each other (and other resources), e.g.:

# some app:
python_sources()
pex_binary(name="app", ...)

# codegen the schema itself:
experimental_shell_command(name="schema", dependencies=[":app"], outputs=["schema.json"], command="./app.pex export-schema > schema.json", tools=["python3.9"])

# set up the NPM dependencies:
experimental_shell_command(name="node_modules", outputs=["node_modules/**"], command="npm ci", ...)

# use the schema to do codegen or generate docs or whatever:
experimental_shell_command(name="codegen-from-schema", dependencies=[":node_modules", ":schema"], outputs=["codegen/**"], command="npm run do-codegen < schema.json", ...)
experimental_shell_command(name="docs-from-schema", dependencies=[":node_modules", ":schema"], outputs=["docs/**"], command="npm run generate-docs < schema.json", ...)

# continues with targets using :codegen-from-schema and :docs-from-schema... (`archive`, `experimental_shell_command` and `experimental_run_shell_command`)

The Python code that goes into

pex_binary

changes regularly and thus the PEX itself changes too, but the exported

schema.json

doesn't change so much (i.e. we're often refactoring/adding features/fixing bugs without changing the schema). In theory, if

:app

changes but the schema doesn't, `:schema`'s dependees (

:codegen-from-schema

,

:docs-from-schema

) don't need to run, but those dependees pull in the

app.pex

file from

:app

, and thus do rerun. Rerunning can take significant time. (See also: https://pantsbuild.slack.com/archives/C046T6T9U/p1666315386420799) pantsbuild/pants

#17762 Working with `conftest.py` fixtures in a Python monorepo New discussion created by alexey-tereshenkov-aiven In the layout of a monorepo

project1
  tests/
    conftest.py (with project1 specific fixtures)
    test_module.py
project2
  tests/
    conftest.py (with project2 specific fixtures)
    test_module.py
conftest.py (with generic fixtures)

when running tests in

project1/tests

directory,

pytest

visits all ancestor directories until reaching the root collecting all

conftest.py

so that fixtures are available. From the docs:

Pants will also infer dependencies on any confest.py files in the current directory and any ancestor directories, which mirrors how Pytest behaves.

This means that when making changes to the root

conftest.py

, all tests will need to be run because every test depends on that root

conftest.py

. It's possible to disable this behavior:

You can turn off this feature by setting conftests = false in the [python-infer] scope.

However, it seems fair that all subprojects may want to use a generic fixture from the root

conftest.py

. But this negatively affects the dependency management within tests as all tests (some of them maybe slow!) will need to be run every time someone touches the root

conftest.py

(since you don't know whether the fixtures a test uses are actually modified). Importing individual fixtures (so that you would split the root

conftest.py

into individual modules) seems to be discouraged:

pytest

docs suggest you define all your fixtures within one single

conftest.py

file. There is an explicit import approach, but I am not sure what are the cons of this approach?

# tests/conftest.py
import pytest
from fixtures.add import add

I see usage of

pytest_plugins = (
   "fixtures.fixture_1",
   "fixtures.fixture_2",
)

in various repositories, but this doesn't work well with Pants dependency inference (as there are no

import

statements). What kind of approach have you decided to go with? pantsbuild/pants

#17764 `export` Python virtualenvs with non-hermetic console scripts Issue created by danxmoran Is your feature request related to a problem? Please describe. The virtualenvs generated by

./pants export

currently follow Pex's default of generating "hermetic" console scripts, by rewriting all the shebangs of scripts under

bin/

to pass

-sE

to

python

. This prevents the use of exported scripts with a custom

PYTHONPATH

, which breaks use-cases like running

pylint

with in-repo plugins and a custom source root. Describe the solution you'd like Pex 2.1.118 added a

--non-hermetic-scripts

option to the

venv

tool, which disables the addition of

-sE

to script shebangs. Pants could pass that flag when running the export logic. This should definitely be safe to do when exporting a non-symlinked venv. I'm not sure if it would be safe to do in the symlink case, since the virtualenv might also be used for other processes within Pants. Maybe this should be an option on the

export

goal-subsystem, with the caveat that if you enable it you will get worse caching performance out of the symlink mode. Describe alternatives you've considered I thought about writing a wrapper script around

./pants export

that re-re-writes the shebangs to remove

-sE

. I worry about the safety of doing this when using symlinked exports, since I'd be munging files within the Pants cache. To avoid munging data in the pants cache, the wrapper script would need to do something like: 1. Run

./pants export

with symlinking 2. Delete the top-level symlink under

dist/export

3. Recreate the directory structure of a virtualenv under

dist/export

in place of the deleted link 4. Within the virtualenv skeleton, link back to everything in the Pants-cache venv except for

bin/

5. Copy all the files from Pants-cache

bin/

to the virtualenv under

dist/export

, rewriting shebangs along the way I'm confident I could write such a script, but don't really want to if it turns out it's safe for Pants to always run with

--non-hermetic-scripts

😅 Additional context The

pylint

example might seem unrealistic since we'd typically want people to run

./pants lint

, but it is important for IDE integrations. We have other custom tools that do wacky things with

importlib

that are broken in the same way. pantsbuild/pants

#17772 docs: goal associated with multiple backends only lists one Issue created by huonw Describe the bug The

export-codegen

goal is associated with/created by many backends, like protobuf, docker, and several others. However, the docs page for it (https://www.pantsbuild.org/docs/reference-export-codegen) only lists one:

pants.backend.docker

. Pants version 2.14 OS N/A Additional info N/A pantsbuild/pants

#17776 Unifying tool lockfiles and user lockfiles? New discussion created by benjyw Kicking off a thread to discuss how Python tool lockfiles are different from user lockfiles, and why, and whether we can simplify and unify these mechanisms. This is particularly in relation to

export

, which currently requires a lot of jumping through hoops in order to support the

--resolve=

flag uniformly for user and tool lockfiles. Basically, for a user lockfile it is easy to go from resolve name to lockfile content. But for a tool lockfile, we have to go via ExportPythonToolSentinel -> ExportPythonTool -> PexRequest As far as I can tell, this is so that tools can tinker with the interpreter constraints in various ways. E.g., pants/src/python/pants/backend/python/lint/black/subsystem.py Line 151 in </pantsbuild/pants/commit/7251de9bb8aef692b4ffcfc62c36943983a51145|7251de9> . But I'm wondering if this is strictly necessary? There are already interpreter constraints baked into the lockfile, and if the repo has custom interpreter constraints on the tool's subsystem then they must regenerate the lockfile anyway. So a lot of complexity seems to exist only to support the feature of automatically adopting python3.8 interpreter constraints for black and mypy (and maybe there are other examples?) without forcing you to regenerate the 3.7+ lockfile we ship with. And I'm wondering if it's worth it. We could, for example, ship a python3.8+ tool lockfile, and if you must run on 3.7 then you must generate a tool lockfile? But, I'm also wondering if I'm fully understanding this, and if I am missing more reasons why this complexity is needed. I have a feeling I'm not seeing the full picture. pantsbuild/pants

#17778 IncompleteJSONError when running any pants commands Issue created by jcrumb-u21 Describe the bug I'm adding pants to an existing repo with go code. My initial

pants tailor ::

worked and generated BUILD files, but now running any commands including

pants check ::

will generate the following error:

IncompleteJSONError: parse error: trailing garbage
          mod", 	"GoVersion": "1.19" } { 	"Path": "<http://cloud.google.com/go|cloud.google.com/go>
                     (right here) ------^

Pants version 2.14.0 OS MacOS (Ventura) Additional info My local python version used by pants is

3.9.13

and I'm building my project on Go

1.19

. Here's a gist which contains the full output from running with

ldebug

and `print-stacktrace`: https://gist.github.com/jcrumb-u21/d3b575ae2b7a8ac2060c2409a0b73171 Here's a gist containing my project's `go.mod`: https://gist.github.com/jcrumb-u21/8ef02939c226bd9eca27314ecf16f99f And finally, here's a gist of the JSON output that

go list

is generating, pulled from `pants.backend.go.util_rules.third_party_pkg.analyze_module_dependencies`: https://gist.github.com/jcrumb-u21/9c4f36108689722921dfd179e2013bd8 pantsbuild/pants