https://pantsbuild.org/ logo
#general
Title
# general
a

acceptable-football-32760

03/14/2022, 12:15 PM
Good day! Is that expected that
Copy code
./pants package ::
reruns the whole built when invoked from a new terminal window? I'm trying to make it reuse the existing runs cache etc. No changes was made to the
[GLOBAL]
section apart from the following settings - with bunch of o
Copy code
[GLOBAL]
dynamic_ui = false
pants_version = "2.10.0rc1"
backend_packages = [
    "pants.backend.plugin_development",
    "pants.backend.python.lint.black",
    "pants.backend.python.lint.isort",
    "pants.backend.python.typecheck.mypy",
    "pants.backend.python",
    "deploy_package",
    "torch_requirement",
    "mypy_requirement",
]
build_patterns = ["BUILD.pants"]
pythonpath = ["%(buildroot)s/tools/pants/pants-plugins"]
# NOTE: this is needed after pantsbuild v2.11
use_deprecated_python_macros = false
Then I run
./pants package ::
from one tmux pane, then create another pane and run it there. In both cases it rebuilds everything from scratch. Maybe a more generalized question - what environment goes into the definition of "cache hit" (sorry for potentially sloppy terminology!) -- this knowledge is useful to me in case I want to invoke pants from somewhere else.
f

fast-nail-55400

03/14/2022, 1:19 PM
Could you run the experiment again but this time with
--stats-log
? Then share the
local_cache_*
counters here? That will provide knowledge of how precisely much cache usage there was.
a

acceptable-football-32760

03/14/2022, 1:34 PM
@fast-nail-55400 thanks, here it is. First run in the tmux pane `A`:
Copy code
local_cache_read_errors: 0
  local_cache_requests: 14
  local_cache_requests_cached: 0
  local_cache_requests_uncached: 14
  local_cache_total_time_saved_ms: 0
  local_cache_write_errors: 0
Then, tmux pane `B`:
Copy code
local_cache_read_errors: 0
  local_cache_requests: 14
  local_cache_requests_cached: 0
  local_cache_requests_uncached: 14
  local_cache_total_time_saved_ms: 0
  local_cache_write_errors: 0
And for control, running back from pane `A`:
Copy code
local_cache_read_errors: 0
  local_cache_requests: 0
  local_cache_requests_cached: 0
  local_cache_requests_uncached: 0
  local_cache_total_time_saved_ms: 0
  local_cache_write_errors: 0
f

fast-nail-55400

03/14/2022, 1:54 PM
The cache key for build actions is formed from, among a few other items, the command-line being run. If you run with
-ldebug
for debug logging, the log should print out the commands being run. We can look at those to see if there is some difference that is preventing caching.
👍 1
a

acceptable-football-32760

03/14/2022, 2:04 PM
Aha, thanks for that one, can it be that it is because of
$PATH
? It seems I have bunch of
/tmp/fnm_multishells/XXXX_YYYY
where
X
and
Y
are changing
👍 1
f

fast-nail-55400

03/14/2022, 2:08 PM
it can if PATH is passed into the execution sandbox that Pants uses
going to make an educated guess (since you are using Python) and guess it is the
--python-bootstrap-search-path
option which defaults to looking at
PATH
as one of the ways to find the Python interpreter
./pants help-advanced python-bootstrap
Setting it (and any other option that consults PATH) to a non-changing list of paths should fix the caching issue.
a

acceptable-football-32760

03/14/2022, 2:12 PM
Checked that. Indeed path was changing due to
fnm env
in one of the `rc`'s and then having some every-time-new tmp symlink in
path
. Removing it did the trick. But is there a flag similar to bazel's
incompatible_strict_action_env
to avoid slight changes in
PATH
trigger rebuilds? I mean, there are things like
fnm
which pollute path...
👀 1
f

fast-nail-55400

03/14/2022, 2:12 PM
Obviously in this case, set
--python-bootstrap-search-path
to the paths where Python may be found.
a

acceptable-football-32760

03/14/2022, 2:13 PM
Thanks!!
f

fast-nail-55400

03/14/2022, 2:15 PM
The equivalent of Bazel’s
incompatible_strict_action_env
is probably
--subprocess-environment-env-vars
You can set a
PATH
to be used in all execution sandboxes invoked by Pants.
./pants help-advanced subprocess-environment
That might be the easier way to fix the issue instead of tracking down every use of
PATH
in other Pants config options.
(But it only affects execution sandboxes, so I don’t know if it would fix uses of
PATH
in config options that are consulted for purposes not related to execution sandboxes.)
a

acceptable-football-32760

03/14/2022, 3:56 PM
I seem to be getting
TypeError: type object got multiple values for keyword argument 'PATH'
when setting
Copy code
[subprocess-environment]
env_vars = [
    "LANG",
    "LC_ALL",
    "LC_CTYPE",
    "PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin",
]
f

fast-nail-55400

03/14/2022, 4:04 PM
Do you have a full trace? (Add
--print-stacktrace
to command-line to ensure full trace is printed.)
h

hundreds-father-404

03/14/2022, 5:15 PM
You can set a PATH to be used in all execution sandboxes invoked by Pants.
I don't think that will work, and we should have been validating eagerly that you don't set this. It's because Pants sets the
PATH
for you differently depending on the context, like there is
[shell-setup].executable_search_path
. We went with the approach of multiple more granular options so that you can optimize for better cache hits; Python might need to look at more than Shell, for example. But overall we could do a better job I think documenting these options. @acceptable-football-32760 would you want to open a feature request for something like
incompatible_strict_action_env
? Mention your use case and what you'd like to see
a

acceptable-football-32760

03/15/2022, 8:46 PM
Hi @hundreds-father-404, thanks, do I proceed via GH issues?
h

hundreds-father-404

03/15/2022, 8:46 PM
Yeah that'd be great