#22692 Pants "subcmd" running use case notes New discussion created by cburroughs # Pants "subcmd" running use case notes The typical Pants cli invocation looks like:

pants [options] [goals] [inputs]

This allows one to express multiple actions on the same set of targets (ex:

pants fix test src::

and this is among the most common use cases. However, it makes it difficult to express optionally that may be specific to only some "targets" (or not even apply to targets). This document outlines 3 use cases that are challenging with the existing pattern, the first two of which are likely more tractable. I have some idea how to implement the "meat" of the change, and how I would do it with click, but not how to hook it up to the existing goals/options system. The intent is to provide concrete examples to illustrate the current limitations to inform possible paths forward for implementation. (There is not an obvious way to do these today with the existing goal system.) But not to completely flesh out a design for all of them at once. ## Cases ### cache tooling (no target) #11167 Pants maintains several different caches, including ones both internal to Pants (lmdb) and cases where pants Pants is orchestrating "named" caches for other tools. Users have reasonable requests like "tell me how big my cache is" and "prune it so it doesn't grow without bounds". The size of these caches vary greatly so some of these are more valuable than others. For example on my workstation

named_caches/pex_root

is 350 GiB,

lmdb_store/

is 36 GiB, and third place is 3 GiB. Some of the tools -- notably Pex -- have their own tools for pruning or otherwise managing their cache. Unlike goals, these commands would act on the specific caches, but not targets. #### reference: pex --help

$ pex3 cache --help
usage: pex cache [-h] {dir,info,purge,prune} ...

Interact with the Pex cache.

options:
  -h, --help            show this help message and exit

subcommands:
  Interact with the Pex cache via the following subcommands.

  {dir,info,purge,prune}
    dir                 Print the current Pex cache directory path.
    info                Present information about Pex cache status.
    purge               Purge the Pex cache safely.
    prune               Prune the Pex cache safely.

$ pex3 cache prune --help
usage: pex cache prune [-h] [-H] [--older-than CUTOFF] [-n] [-o PATH] [--emit-warnings] [--pex-root PEX_ROOT] [--disable-cache]
                       [--cache-dir CACHE_DIR] [--tmpdir TMPDIR] [--rcfile RC_FILE]

options:
  -h, --help            show this help message and exit
  -H, --human-readable, --bytes, --kB, --MB, --GB, --TB, --PB
                        How to display disk usage amounts; defaults to bytes. The -H / --human-readable options display amounts in
                        a human readable way and each of the other unit options displays amounts with that specified unit.
  --older-than, --last-access, --last-access-before CUTOFF
                        Prune zipapp and venv caches (amongst others) last accessed before the specified time. If the dependencies
                        of the selected zipapps and venvs (e.g.: installed wheels) are unused by other zipapps and venvs, those
                        dependencies are pruned as well. The cutoff time can be specified as a date in the format `<day
                        number>/<month number>/<4 digit year>` or as a relative time in the format `<amount>
                        [second(s)|minute(s)|hour(s)|day(s)|week(s)]`.
  -n, --dry-run         Don't actually purge cache entries; instead, perform a dry run that just prints out what actions would be
                        taken
  -o, --output PATH     A file to output the Pex purge results to; STDOUT by default or when `-` is specified.

Global options:
  --emit-warnings, --no-emit-warnings
                        Emit runtime UserWarnings on stderr. If false, only emit them when PEX_VERBOSE is set.
  --pex-root PEX_ROOT   Specify the pex root used in this invocation of pex (if unspecified, uses /home/ecsb/.cache/pex).
  --disable-cache       Disable caching in the pex tool entirely.
  --cache-dir CACHE_DIR
                        DEPRECATED: Use --pex-root instead. The local cache directory to use for speeding up requirement lookups.
  --tmpdir TMPDIR       Specify the temporary directory Pex and its subprocesses should use.
  --rcfile RC_FILE      An additional path to a pexrc file to read during configuration parsing, in addition to reading
                        `/etc/pexrc` and `~/.pexrc`. If `PEX_IGNORE_RCFILES=true`, then all rc files will be ignored.

#### example cli (assume these launch as

experimental-cache

) • (A)

pants cache named pex prune

• (B)

pants cache pex prune

• (C)

pants cache prune pex,lmdb,mypy

• (D)

pants cache named pex -- prune

(raw cli passthru) • (E)

pants cache pex -- prune

(raw cli passthru) #### Questions • Is having a top level distinction

named

vs

lmdb

helpful future proofing? • What about

pants cache tool ...

vs

pants cache future-all-cache-cmds

? • Do we actually gain much at this point with a bunch of "wrapping" like in (A,B)? Why not just do the thinnest possible layer like (D,E)? • As a practical matter, have there been any requests for cache control features besides lmdb and pex? Future Questions: • It might be nice in the future to have some sort of "prune everything older than X in all supported cache types" command. This may be more difficult in a pure passthru implementation. • Does "online GC" make any sense for named caches, or only lmdb? ### lockfile manipulation (usually ~one resolve, or resolve ecosystem) #15704 #15568 #12880 There are a lot of different things people would like to do with lockfiles besides generating the whole thing from scratch: • Change the version of N package. • Try to update version X in the existing range • Remove N packages from the lockfile. • Point the entire lockfile at a different index while minimizing changes. The details of all of these are pretty specific to the given ecosystems. For example, it would not make sense to say "update pytest to x.y.z" to both java and Python resolves. It might make sense to say "update pytest to x.y.z" to multiple Python resolves. But, I wearly of trying to support identical operations similar but different ecoystehsms by creating semenatics to "update pytest to x.y.z" that apply to Pex, Poetry, and UV, but are different from each. #### example cli • (A)

pants lock update pex [resolve-name] --pex-specific-flags

• (B)

pants lock update pex --resolve=foo --resolve=bar --pex-specific-flags

• (C)

pants lock update --resolve=foo --dynamic-flags-based-on-type-of-resolve

(

--resolve

is mandatory?) • (D)

pants generate-lockfiles --not-actually-generate-operation=update --resolve=foo --dynamic-flags-based-on-operation

(today; not a real proposal) • (E)

pants lock direct-op --tool=pex --resolve=foo -- raw-pass-thru

REMINDER: This is illustrative of things that are hard to address with the goal system today. Pex/Poetry/UV have a lot of prior art here on the details. Unlike the cache case, these operate on things existing Pants goals know about (resolves), but are not targets. #### Questions • It is not obvious from the output of

pants help lint

and

pants help generate-lockfiles

that one operates on targets and the other does not. Should we have a name for this that shows up in

help

?

TargetGoal

? • Are any of the "subcmd" example patterns easier to implement? ### targeting #15012 For example, consider the case where a repository has multiple "types" of tests and has tagged all the unit tests with

unit

one can do `… pantsbuild/pants