I have a caching question. I'm using "pants.backend.python.providers.experimental.python_build_standalone" backend to provide hermetic pythons. Calling

pants lint ::

while I'm inside an activated conda environment calculates for a bit, and then running it again returns the result instantly from cache as expected. If I deactivate the conda environment, the same call starts calculating everything from the beginning. What changed for

pants

between these two runs to invalidate the cache between them if the python itself is hermetic and nothing else changed in the shell (PATH changed, but does that matter?) ?

Thanks for waiting. Can you share your

pants.toml

? For this sort of question, I'd be particularly looking for any environment variables that are intentionally propagated into sandboxes. Alternatively/additionally, run with

pants --keep-sandboxes=always lint ::

and then look at the

__run.sh

files in the sandbox. They should have a

env -i SOME_ENV=... ...

command within them that indicates explicitly what the env vars used are, and thus they can be compared.

Thanks for the debugging tips! I don't see anything in the

pants.toml

that looks like something affecting environment variables. Looking at the sandboxes, the lint execution itself doesn't have any env variables that could cause it but... sandbox

for Find interpreter for constraints: CPython==3.12.*

and sandbox for building the pylint_runner itself have

env -i CPPFLAGS='' LANG=en_US.UTF-8 LDFLAGS='' PATH=$

with my whole PATH variable after it (which changes when you go in and out of virtualenvs of course). Is this expected or some configuration I have is causing the PATH to be included there?

Hmm, if that process runs, then including PATH is expected (since it's doing a search to find possible python interpreters, using said PATH)... But it's definitely weird that that search runs if you're using a Python provider: it should be being skipped 🤔 The relevant code is https://github.com/pantsbuild/pants/blob/140c72af809d4926c324898f1cc91d73fb07f297/src/python/pants/backend/python/util_rules/pex.py#L338-L408... note the

if python_providers: ...

branch that avoids the

Get(..., PexCliProcess(...))

search below. Can you share your

pants.toml

?

Oh sorry about that.. I was using a branch without the hermetic python when running the above. You are right that

Find interpreter for constraints: CPython==3.12.*

disappears when using python build standalone, but both

Building 6 requirements for pylint.pex

and

Determine distributions found in pylint.pex

still use

PATH

in it's run.sh Our

pants.toml

is pretty big so I can try to give you relevant sections, but if i'm missing something tell me:

[GLOBAL]
pants_version = "2.25.0"

# <https://www.pantsbuild.org/stable/docs/getting-started/initial-configuration#enable-backends>
backend_packages = [
  "pants.backend.python",
  "pants.backend.python.providers.experimental.python_build_standalone",
  "pants.backend.plugin_development",
  "pants.backend.python.lint.isort",
  "pants.backend.python.lint.black",
  "pants.backend.python.lint.pylint",
  "pants.backend.python.lint.pyupgrade",
  "pants.backend.python.lint.flake8",
  "pants.backend.awslambda.python",
  "pants.backend.build_files.fmt.black",
  "pants.backend.shell",
  "pants.backend.experimental.python",
  "pants.backend.experimental.tools.yamllint",
  "pants.backend.experimental.adhoc",
  "progressive_mypy",
]

pythonpath = ["%(buildroot)s/pants_plugins"]

[DEFAULT]
CONST_PYTHON_VERSION = "3.12.*"  # Should be aligned with the version in `.python-version`
CONST_PANTS_PYTHON_VERSION = "3.11.*"  # Should be aligned with the version in macros PANTS_PLUGIN_INTERPRETER_CONSTRAINTS

[python]
interpreter_constraints = ["==%(CONST_PYTHON_VERSION)s"]
enable_resolves = true
default_resolve = "python-default"

[python.resolves]
pants-plugins = "toolchain/lockfiles/pants-plugins.lockfile"
python-default = "toolchain/lockfiles/python-default.lockfile"

[python.resolves_to_interpreter_constraints]
python-default = ["==%(CONST_PYTHON_VERSION)s"]
pants-plugins = ["==%(CONST_PANTS_PYTHON_VERSION)s"]

[pylint]
config = "toolchain/config/.pylintrc"
install_from_resolve = "python-default"
requirements = [
    "//ml_framework/ml_framework/core/requirements:reqs-ml#pylint",
]
source_plugins = [
    "pylint_plugins:pylint_plugins",
]

progressive_mypy

is a custom mypy plugin that is unrelated to

lint

goal

Ah, cool. Thanks for the reproducer. I think this may be controlled by https://www.pantsbuild.org/2.24/reference/subsystems/pex#executable_search_paths, which (potentially unhelpfully) defaults to

<PATH>

. If I clear it explicitly, the repo seems to be behave better:

[pex]
executable_search_paths = []

adding that in

example-python

now yields for me:

env -i CPPFLAGS='' LANG=en_US.UTF-8 LDFLAGS='' PATH='' PEX_IGNORE_RCFILES=true PEX_PYTHON=/Users/kris.tashkov/Library/Caches/nce/8fb836cadd1e1f50180fea66b6acab1f23108e9a74f158744e706c6da41a534b/bindings/venvs/2.25.0rc0/bin/python3.11 PEX_ROOT=.cache/pex_root /Users/kris.tashkov/Library/Caches/nce/8fb836cadd1e1f50180fea66b6acab1f23108e9a74f158744e706c6da41a534b/bindings/venvs/2.25.0rc0/bin/python3.11 ./pex --tmpdir .tmp --jobs 12 --no-emit-warnings --pip-version 24.2 --python-path 'path is here again' --output-file flake8.pex --venv prepend --seed verbose --venv-site-packages-copies --python /Users/kris.tashkov/.cache/pants/named_caches/python_build_standalone/3.9.21/bin/python3 --console-script flake8 --no-pre-install-wheels $'--sources-directory=source_files' --lock flake8.lock --no-pypi $'--index=<https://pypi.org/simple/>' --manylinux manylinux2014 --layout packed

with

--python-path

also containing entries from my current path including the current conda env 😭

Ah, I think https://www.pantsbuild.org/stable/reference/subsystems/python-bootstrap#search_path might also benefit from tweaking in that case

nice! I think this finally did the trick! Thanks a lot for your help 🙂 Should this be automatically set up when using hermetic pythons? It's weird that all these interpreter searching paths are active while having the hermetic python backend. Tangentially.. now that the cache is actually consistent it's a bit glaring that

mypy

's result is never cached and it's always ran from scratch. Is this normal or something else I have misconfigured?

Yeah, it does seem like an easy-to-miss sharp edge. Could you file a ticket? At the very least, we can improve documentation (but preferably behaviour, of course). Re mypy: does it run from scratch even if you make no code changes, eg

pants check :: && pants check ::

Yeah, it does seem like an easy-to-miss sharp edge. Could you file a ticket? At the very least, we can improve documentation (but preferably behaviour, of course).

will do

Re mypy: does it run from scratch even if you make no code changes, eg

pants check :: && pants check ::

yep, always reruns. Since it's one big batch of 3000+ files I always thought it's something related to size, but now I'm starting to question it.

yep, always reruns. Since it's one big batch of 3000+ files I always thought it's something related to size, but now I'm starting to question it.

Hm, strange. Mypy is a bit special in that it needs to see the whole "world", and thus changing any file in a given resolve will have pants re-run the process (theoretically with mypy using its own internal cache)... but changing no files at all should still be cached. Are you observing this re-run with your custom

progressive_mypy

backend? If so, there might be some inconsistencies in that backend (e.g. randomly ordered CLI args, including a timestamp, ...)?

it happens even without it, first thing I tested when I was wondering 😅 The backend also works after mypy is completed (it filters some of the errors)

Re

LC_CTYPE

and `LANG`: https://www.pantsbuild.org/stable/reference/subsystems/subprocess-environment#env_vars ... so many random env var-controlling options!

Okay.. new problem with caching with the above settings 😓 While

pants lint ::

works,

pants package

now can't find a python since I guess it doesn't use the python provider backend (and I've removed all the PATH variables so it can't find the system one)

Ah yeah, that sounds like https://github.com/pantsbuild/pants/issues/21048 😢 There's a start of a fix at https://github.com/pantsbuild/pants/pull/21948... maybe I'll try to brush it off soon.

That does seem to be the problem. I guess for the time being I gotta settle with local configuration with specific paths to python binaries (not entire path) but can't merge it for the whole team. Thanks again with the investigation, I will open a ticket to save our findings and have a place to track the progress in the future 🙂