I just upgraded Python to 3.11 (previously 3.8) while keeping the pants version the same as before. All my tests run locally, but randomly fail on GitHub Actions. I am using

actions/setup-python@v5

and running

python -V

gives the correct version:

3.11.7

. The problem seems not to be with the tests but finding the correct interpreter. One can see that some tests succeed and then randomly this error occurs:

11:34:39.73 [INFO] Completed: Run Pytest - (environment:local, services/weasyprint_pdf/tests/test_server_responding.py) - succeeded.
11:35:02.36 [INFO] Completed: Run Pytest - (django/dismantling/tests/queries/test_dismantling_project_cost.py:tests and 1 other files, environment:local) - succeeded.
Partition: RUN_TESTS_TOGETHER_TAG
11:35:04.64 [INFO] Completed: Run Pytest - (django/dismantling/tests/classes/test_material.py:tests and 25 other files, environment:local) - succeeded.
Partition: RUN_TESTS_TOGETHER_TAG
11:35:34.77 [INFO] Completed: Run Pytest - (django/spatial/tests/mutations/test_cluster_delete.py:tests and 15 other files, environment:local) - succeeded.
Partition: RUN_TESTS_TOGETHER_TAG
11:35:46.44 [INFO] Completed: Run Pytest - (django/dismantling/tests/queries/test_dismantling_projects.py:tests and 32 other files, environment:local) - succeeded.
Partition: RUN_TESTS_TOGETHER_TAG
11:35:50.45 [INFO] Starting: Building pytest_runner.pex
11:35:50.57 [ERROR] 1 Exception encountered:
Engine traceback:
  in `test` goal
ProcessExecutionFailure: Process 'Find interpreter for constraints: CPython<3.12,>=3.11' failed with exit code 102.
stdout:

stderr:
Could not find a compatible interpreter.

Examined the following interpreters:

1.) /usr/bin/python3.8 CPython==3.8.10



No interpreter compatible with the requested constraints was found:


  Version matches CPython<3.12,>=3.11

Note: I have disabled remote caching. Ideas what to look out for?

are you using GH provided runners or self hosted? Just thinking if there’s a possibility there’s a stray runner still that’s not been upgraded…

In addition to Andreas' question, can you share your

pants.toml

?

pants.toml

[GLOBAL]
# info: change version to update pants
pants_version = "2.16.0"
colors = true
# increasing memory usage for for better in-memory caching and fewer restarts
pantsd_max_memory_usage = "4GiB"
# storing pants log and pants file in one place (persistent because of container volume mount)
pants_workdir = ".cache_and_logs/.pants.d"
pants_subprocessdir = ".cache_and_logs/.pids"
local_store_dir = ".cache_and_logs/.pants_cache"
unmatched_build_file_globs = "error"

# <https://github.com/pantsbuild/pants/issues/17851>
named_caches_dir = "/tmp/pantsbuild/named_caches"
local_execution_root_dir = "/tmp/pantsbuild/"

build_file_prelude_globs = ["pants_plugins/macros.py"]

# also ignoring pgadmin because no read permissions for pgadmin/config
pants_ignore = [
  "/.cache_and_logs/",
  ".*/", 
  "/dist/", 
  "/cli/third/md-click",
  "/Dockerfile",
  "!/django/locale/**",
]
# uncomment to do not use gitignore by default for file visibility
# pants_ignore_use_gitignore = false

backend_packages = [
  'pants.backend.python',
  "pants.backend.awslambda.python",
  'pants.backend.docker',
  'pants.backend.docker.lint.hadolint',
  'pants.backend.python.typecheck.mypy',
  'pants.backend.experimental.python.lint.ruff',
  'pants.backend.python.lint.yapf',
]

[tailor]
ignore_adding_targets = [
  "django:manage",
  "infra/apps/resources/django:docker",
  "infra/apps/resources/services:docker",
  "infra/apps/resources/replibyte:docker",
]
ignore_paths = ["airflow/dags/**"]

[python]
interpreter_constraints = ["CPython>=%(env.PYTHON_MAJOR_VERSION)s,<3.12"]
enable_resolves = true
default_resolve = "default"
pip_version = "22.3"

[python.resolves_to_constraints_file]
default = "constraints/airflow-2.6.0-py-%(env.PYTHON_MAJOR_VERSION)s.txt"

[python-infer]
#inits = true
init_files = "always"
string_imports = true
string_imports_min_dots = 0
# This setting will ommit warnings about dependency inference as there is a modele `api` in Django
# as well as one with the same name in alle service directories.
ambiguity_resolution = "by_source_root"

[python.resolves]
default = "lockfiles/default.lock"
tools = "lockfiles/tools.lock"

[source]
# source code locations not in use by now
root_patterns = [
  "/services",
  "/libs/*",
  "/django",
  "/airflow",
  "/pants_plugins",
  "/cli",
  "/",
  "/jobs",
]

[docker]
build_args = [
  "VARIANT",
  "PYTHON_MAJOR_VERSION",
  "ENVIRONMENT",
  "IMAGE_TAG",
  "DOWNSCALING_ECR_REPO_NAME",
  "ELEVATION_ECR_REPO_NAME",
  "GEOPACKAGE_ECR_REPO_NAME",
  "GLOBALSOLARATLAS_ECR_REPO_NAME",
  "HIGHCHARTS_ECR_REPO_NAME",
  "INTERPOLATION_ECR_REPO_NAME",
  "MVA_ECR_REPO_NAME",
  "NEWA_ECR_REPO_NAME",
  "QGIS_ECR_REPO_NAME",
  "QGIS_ERROR_ECR_REPO_NAME",
  "ROUGHNESS_ECR_REPO_NAME",
  "WEASYPRINT_ECR_REPO_NAME",
  "WEIBULL_ECR_REPO_NAME",
  "SPATIAL_IMPORT_ECR_REPO_NAME",
  "SENTINEL_IMAGE_LAYER_ECR_REPO_NAME",
]
run_args.add = ["--rm"]
env_vars = [
  "DOCKER_BUILDKIT=0"
]
[docker.registries.ecr]
address = "localhost:4510"
default = true

[pytest]
install_from_resolve = "tools"
requirements = [
  "//:tools#pytest",
  "//:tools#pytest-icdiff",
  "//:tools#mixer",
  "//:tools#pytest-xdist",
  "//:tools#pytest-cov",
]
xdist_enabled = true
config_discovery = true
# db id env var set by pants for using multiple databases for parallel tests
execution_slot_var = "PANTS_EXECUTION_SLOT"

[test]
# set batch size smaller, so that caching is more efficient
batch_size=32

extra_env_vars = ["USE_LOCALSTACK=false"]

[environments-preview.names]
qgis_docker = "//services/qgis_file:local_qgis"
local = "//:local-environment"

[yapf]
config = "pyproject.toml"
install_from_resolve = "tools"
requirements = ["//:tools#yapf"]

[mypy]
config = "pyproject.toml"
install_from_resolve = "tools"
requirements = ["//:tools#mypy"]

[ruff]
config = "pyproject.toml"
install_from_resolve = "tools"
requirements = ["//:tools#ruff"]

[anonymous-telemetry]
# do not send telemetry data
enabled = false

pants.ci.toml

[GLOBAL]
colors = true
# plugins = ["hdrhistogram"]
remote_cache_read = true
remote_cache_write = true
remote_store_address = "<grpcs://remote.buildbuddy.io>"
remote_store_headers = "{'user-agent': 'pants', 'x-buildbuddy-api-key': '%(env.BUILDBUDDY_API_KEY)s'}"
remote_instance_name = "buildbuddy-io/buildbuddy/local" 
named_caches_dir = "/tmp/named_caches"
local_execution_root_dir = "/tmp"
rule_threads_core = 4

[docker]
build_args.add = [
  "DJANGO_ECR_REPO_NAME",
  "GEO_ECR_REPO_NAME",
  "AIRFLOW_ECR_REPO_NAME",
  "SNAPLET_ECR_REPO_NAME",
]

[stats]
# log = true

[test]
use_coverage = true

[coverage-py]
report = ["xml"]
global_report = true

[pytest]
args = ["-vv", "--no-header"]


[docker.registries.ecr]
address = "%(env.AWS_ACCOUNT_ID)s.dkr.ecr.%(env.AWS_REGION)<http://s.amazonaws.com|s.amazonaws.com>"
default = true

[docker.registries.ghcr]
address = "<http://ghcr.io/nefino|ghcr.io/nefino>"

[environments-preview.names]
local = "//:ci-environment"
qgis_docker = "//services/qgis_file:local_qgis"

We are using GHA hosted runners

The pants set-up doesn't depend on the system interpreter (it bundles its own) nor on the particular repository (or its contents) other than the Pants version. So yeah, that's expected

It looks like

named-caches

will never be cached:

Cache not found for input keys: pants-named-caches-Linux-cache0-py3.11-4bec308c3a7cd09c0f6fe2eef84a7cd5d52831f2dbbf9efff4fa68d47a125001-df1f334def1ba179517b10ee3b211097147652919e6d3cc82b815bdc74aa6741, pants-named-caches-Linux-cache0-py3.11-4bec308c3a7cd09c0f6fe2eef84a7cd5d52831f2dbbf9efff4fa68d47a125001-, pants-named-caches-Linux-cache0-py3.11

Something wrong with the settings?

I’ve no good ideas atm, just noting that the config perhaps should be to the directory rather than the binary:

search_path = ["/opt/hostedtoolcache/Python/3.11.7/x64/bin"]

Hmm, so should /usr/bin/python3.8 exist on the runner at all? Is that some residual python that exists there regardless of the version you provide to setup-python? Because as far as I can tell Pants has actually run interpreter selection (which is memoized but not cached) and found that interpreter, and correctly induced that it cannot use it. So assuming that python should exist, that part is all fine. The question is why it isn't finding the 3.11 one. Next step would be to constrain the search_path correctly as Andreas suggests and see if that helps. And extra weird is that some tests do pass. Maybe those were cached? (although it says "succeeded" which implies that they ran)

@curved-television-6568 I tried, but nothing changed

huh.. strange. sorry but I have not much experience with GH actions/runners so feel my guess is as good as (or worse than) yours here

@happy-kitchen-89482 I also tried busting the cache

Yeah I don't think this is a cache issue, if you completely busted the GHA cache, and you're getting a clean runner every time.

It is a GHA hosted runner. Not sure how to check if it is in lingering state?!

Well, I mean it's not a persistent dedicated runner, it's an on-demand one?

Right, only another job is running on a 4core group.

This happens consistently, it sounds like?

I have finally figured out what the problem is: One test is running in a

docker_environment

which uses an image where Python 3.8 is installed. This was a tough one as there was no hint to look out for other environments 😄

Ithought the used environment was presented in the log when running the test

It is not clear from which tests this error is raised as far as I can see