trying to run pants in my CI (tests work fine locally) and got stuck when CI is building stuff:

Building 47 requirements for requirements.pex from the thirdparty/python/default.lock resolve: alembic==1.4.3

is taking forever. any docs/tips on ci integrations? All I am running is

./pants test ::

Are you using lockfiles? In other words is CI stuck on resolving the deps, or is it using the same resolution as you used locally and is somehow stuck building the pex.

yeah I generated latest lock files thirdparty/python/default.lock and it seems to be using it

There is https://www.pantsbuild.org/docs/using-pants-in-ci But that is geared towards "kinda slow" problems, not "it seems to take forever"

What are your interpreter constraints?

pants_version = "2.15.0rc1"

[python]
enable_resolves = true
interpreter_constraints = ["CPython==3.8.*"]
tailor_pex_binary_targets = false

Bootstrapping Pants using /usr/bin/python3.8
Creating the virtualenv PEX.
Downloading the Pex PEX.
SHA256 fingerprint of <https://github.com/pantsbuild/pex/releases/download/v2.1.103/pex> verified.
Scrubbing PYTHONPATH
Installing pantsbuild.pants==2.15.0rc1 into a virtual environment at /root/.cache/pants/setup/bootstrap-Linux-x86_64/2.15.0rc1_py38
Scrubbing PYTHONPATH
New virtual environment successfully created at /root/.cache/pants/setup/bootstrap-Linux-x86_64/2.15.0rc1_py38.
19:08:52.43 [INFO] Initializing scheduler...
19:08:53.30 [INFO] Scheduler initialized.
...
19:25:14.28 [INFO] Completed: Building pytest_runner.pex
19:25:16.77 [INFO] Completed: Building 53 requirements for requirements.pex from the thirdparty/python/default.lock resolve: alembic==1.4.3, boto3==1.24.59, boto==2.49, botocore==1.27.59, cachetools==4.2.2, confluent-kafka==1.9.2,... (973 characters truncated)
19:25:17.37 [INFO] Completed: Building pytest_runner.pex
19:25:19.17 [INFO] Completed: Building pytest_runner.pex

so took 17 minutes to build.

What is in those

...

between

19:08:53.30 [INFO] Scheduler initialized.
...
19:25:14.28 [INFO] Completed: Building pytest_runner.pex

?

19:13:31.98 [INFO] Long running tasks:
  61.20s	Building 47 requirements for requirements.pex from the thirdparty/python/default.lock resolve: alembic==1.4.3, boto3==1.24.59, boto==2.49, botocore==1.27.59, cachetools==4.2.2, confluent-kafka==1.9.2, datefinder==0.7.3, dnspython==2.2.1, elasticsearch==7.12, factory-boy==3.1, fsspec==2022.8.2, grpcio-health-checking==1.51.1, grpcio==1.51.1, hl7==0.4.5, hvac==0.11.2, jsonschema==3.2, lxml-stubs==0.4, lxml==4.9.2, moto==4.0.6, networkx==2.7.1, nltk==3.7, numpy==1.22.4, opentelemetry-instrumentation-boto==0.36b0, pandas==1.4.4, protobuf==4.21.6, psycopg2-binary==2.9.5, py-grpc-prometheus==0.6, pypdf2==2.11.1, pytest-grpc==0.8, pytest==7.1.2, python-dateutil==2.8.2, python-dotenv==0.15, python-jose==3.2, python-json-logger==2.0.4, pytz==2021.1, reportlab==3.6.9, requests-aws4auth==1.1.2, requests==2.26, sentry-sdk==1.3.1, six==1.16, smart-open==5.2.1, sqlalchemy-utils==0.38.2, sqlalchemy==1.4.35, sqlparse==0.4.2, structlog==22.1, typing-extensions==3.10.0.2, xhtml2pdf==0.2.8
  79.50s	Building 42 requirements for requirements.pex from the thirdparty/python/default.lock resolve: boto3==1.24.59, boto==2.49, botocore==1.27.59, cachetools==4.2.2, confluent-kafka==1.9.2, datefinder==0.7.3, dnspython==2.2.1, elasticsearch==7.12, grpcio-health-checking==1.51.1, grpcio==1.51.1, hl7==0.4.5, hvac==0.11.2, jsonschema==3.2, lxml-stubs==0.4, lxml==4.9.2, networkx==2.7.1, nltk==3.7, numpy==1.22.4, opentelemetry-instrumentation-boto==0.36b0, pandas==1.4.4, protobuf==4.21.6, psycopg2-binary==2.9.5, py-grpc-prometheus==0.6, pypdf2==2.11.1, pytest-grpc==0.8, python-dateutil==2.8.2, python-dotenv==0.15, python-jose==3.2, python-json-logger==2.0.4, pytz==2021.1, reportlab==3.6.9, requests-aws4auth==1.1.2, requests==2.26, sentry-sdk==1.3.1, six==1.16, smart-open==5.2.1, sqlalchemy-utils==0.38.2, sqlalchemy==1.4.35, sqlparse==0.4.2, structlog==22.1, typing-extensions==3.10.0.2, xhtml2pdf==0.2.8

94.60s	Building 3 requirements for requirements.pex from the thirdparty/python/default.lock resolve: boto3==1.24.59, hvac==0.11.2, requests==2.26
  94.60s	Building pytest.pex from pytest_default.lock
19:11:25.28 [INFO] Completed: Building 1 requirement for requirements.pex from the thirdparty/python/default.lock resolve: sqlalchemy==1.4.35
19:11:30.17 [INFO] Starting: Building 12 requirements for requirements.pex from the thirdparty/python/default.lock resolve: boto3==1.24.59, elasticsearch==7.12, grpcio-health-checking==1.51.1, grpcio==1.51.1, protobuf==4.21.6, ps... (151 characters truncated)

111.91s	Building 5 requirements for requirements.pex from the thirdparty/python/default.lock resolve: elasticsearch==7.12, grpcio-health-checking==1.51.1, grpcio==1.51.1, python-json-logger==2.0.4, pytz==2021.1
  112.11s	Building 5 requirements for requirements.pex from the thirdparty/python/default.lock resolve: elasticsearch==7.12, grpcio-health-checking==1.51.1, grpcio==1.51.1, python-json-logger==2.0.4, sqlalchemy==1.4.35
  126.20s	Building 3 requirements for requirements.pex from the thirdparty/python/default.lock resolve: boto3==1.24.59, hvac==0.11.2, requests==2.26
  126.21s	Building pytest.pex from pytest_default.lock
  180.21s	Resolve transitive targets

200.10s	Building 48 requirements for requirements.pex from the thirdparty/python/default.lock resolve: alembic==1.4.3, boto3==1.24.59, boto==2.49, botocore==1.27.59, cachetools==4.2.2, confluent-kafka==1.9.2, datefinder==0.7.3, dnspython==2.2.1, elasticsearch==7.12, factory-boy==3.1, fsspec==2022.8.2, grpcio-health-checking==1.51.1, grpcio==1.51.1, hl7==0.4.5, hvac==0.11.2, jsonschema==3.2, lxml-stubs==0.4, lxml==4.9.2, moto==4.0.6, networkx==2.7.1, nltk==3.7, numpy==1.22.4, opentelemetry-instrumentation-boto==0.36b0, pandas==1.4.4, protobuf==4.21.6, psycopg2-binary==2.9.5, py-grpc-prometheus==0.6, pypdf2==2.11.1, pytest-grpc==0.8, pytest==7.1.2, python-dateutil==2.8.2, python-dotenv==0.15, python-jose==3.2, python-json-logger==2.0.4, pytz==2021.1, reportlab==3.6.9, requests-aws4auth==1.1.2, requests==2.26, s3fs==2022.8.2, sentry-sdk==1.3.1, six==1.16, smart-open==5.2.1, sqlalchemy-utils==0.38.2, sqlalchemy==1.4.35, sqlparse==0.4.2, structlog==22.1, typing-extensions==3.10.0.2, xhtml2pdf==0.2.8
  200.20s	Building 46 requirements for requirements.pex from the thirdparty/python/default.lock resolve: alembic==1.4.3, boto3==1.24.59, boto==2.49, botocore==1.27.59, cachetools==4.2.2, confluent-kafka==1.9.2, datefinder==0.7.3, dnspython==2.2.1, elasticsearch==7.12, factory-boy==3.1, grpcio-health-checking==1.51.1, grpcio==1.51.1, hl7==0.4.5, hvac==0.11.2, jsonschema==3.2, lxml-stubs==0.4, lxml==4.9.2, moto==4.0.6, networkx==2.7.1, nltk==3.7, numpy==1.22.4, opentelemetry-instrumentation-boto==0.36b0, pandas==1.4.4, protobuf==4.21.6, psycopg2-binary==2.9.5, py-grpc-prometheus==0.6, pypdf2==2.11.1, pytest-grpc==0.8, pytest==7.1.2, python-dateutil==2.8.2, python-dotenv==0.15, python-jose==3.2, python-json-logger==2.0.4, pytz==2021.1, reportlab==3.6.9, requests-aws4auth==1.1.2, requests==2.26, sentry-sdk==1.3.1, six==1.16, smart-open==5.2.1, sqlalchemy-utils==0.38.2, sqlalchemy==1.4.35, sqlparse==0.4.2, structlog==22.1, typing-extensions==3.10.0.2, xhtml2pdf==0.2.8

244.70s	Building 50 requirements for requirements.pex from the thirdparty/python/default.lock resolve: aiobotocore==2.4.2, aiohttp==3.8.3, alembic==1.4.3, boto3==1.24.59, boto==2.49, botocore==1.27.59, cachetools==4.2.2, confluent-kafka==1.9.2, datefinder==0.7.3, dnspython==2.2.1, elasticsearch==7.12, factory-boy==3.1, fsspec==2022.8.2, grpcio-health-checking==1.51.1, grpcio==1.51.1, hl7==0.4.5, hvac==0.11.2, jsonschema==3.2, lxml-stubs==0.4, lxml==4.9.2, moto==4.0.6, networkx==2.7.1, nltk==3.7, numpy==1.22.4, opentelemetry-instrumentation-boto==0.36b0, pandas==1.4.4, protobuf==4.21.6, psycopg2-binary==2.9.5, py-grpc-prometheus==0.6, pypdf2==2.11.1, pytest-grpc==0.8, pytest==7.1.2, python-dateutil==2.8.2, python-dotenv==0.15, python-jose==3.2, python-json-logger==2.0.4, pytz==2021.1, reportlab==3.6.9, requests-aws4auth==1.1.2, requests==2.26, s3fs==2022.8.2, sentry-sdk==1.3.1, six==1.16, smart-open==5.2.1, sqlalchemy-utils==0.38.2, sqlalchemy==1.4.35, sqlparse==0.4.2, structlog==22.1, typing-extensions==3.10.0.2, xhtml2pdf==0.2.8
  252.70s	Building 47 requirements for requirements.pex from the thirdparty/python/default.lock resolve: alembic==1.4.3, boto3==1.24.59, boto==2.49, botocore==1.27.59, cachetools==4.2.2, confluent-kafka==1.9.2, datefinder==0.7.3, dnspython==2.2.1, elasticsearch==7.12, factory-boy==3.1, grpcio-health-checking==1.51.1, grpcio==1.51.1, hl7==0.4.5, hvac==0.11.2, jsonschema==3.2, lxml-stubs==0.4, lxml==4.9.2, moto==4.0.6, networkx==2.7.1, nltk==3.7, numpy==1.22.4, opentelemetry-instrumentation-boto==0.36b0, pandas==1.4.4, protobuf==4.21.6, psycopg2-binary==2.9.5, py-grpc-prometheus==0.6, pypdf2==2.11.1, pytest-grpc==0.8, pytest==7.1.2, python-dateutil==2.8.2, python-dotenv==0.15, python-jose==3.2, python-json-logger==2.0.4, pytz==2021.1, ratelimiter==1.2, reportlab==3.6.9, requests-aws4auth==1.1.2, requests==2.26, sentry-sdk==1.3.1, six==1.16, smart-open==5.2.1, sqlalchemy-utils==0.38.2, sqlalchemy==1.4.35, sqlparse==0.4.2, structlog==22.1, typing-extensions==3.10.0.2, xhtml2pdf==0.2.8

470.80s	Building 47 requirements for requirements.pex from the thirdparty/python/default.lock resolve: alembic==1.4.3, boto3==1.24.59, boto==2.49, botocore==1.27.59, cachetools==4.2.2, confluent-kafka==1.9.2, datefinder==0.7.3, dnspython==2.2.1, elasticsearch==7.12, factory-boy==3.1, grpcio-health-checking==1.51.1, grpcio==1.51.1, hl7==0.4.5, hvac==0.11.2, jsonschema==3.2, jwt==1.3.1, lxml-stubs==0.4, lxml==4.9.2, moto==4.0.6, networkx==2.7.1, nltk==3.7, numpy==1.22.4, opentelemetry-instrumentation-boto==0.36b0, pandas==1.4.4, protobuf==4.21.6, psycopg2-binary==2.9.5, py-grpc-prometheus==0.6, pypdf2==2.11.1, pytest-grpc==0.8, pytest==7.1.2, python-dateutil==2.8.2, python-dotenv==0.15, python-jose==3.2, python-json-logger==2.0.4, pytz==2021.1, reportlab==3.6.9, requests-aws4auth==1.1.2, requests==2.26, sentry-sdk==1.3.1, six==1.16, smart-open==5.2.1, sqlalchemy-utils==0.38.2, sqlalchemy==1.4.35, sqlparse==0.4.2, structlog==22.1, typing-extensions==3.10.0.2, xhtml2pdf==0.2.8
  471.00s	Building 47 requirements for requirements.pex from the thirdparty/python/default.lock resolve: alembic==1.4.3, boto3==1.24.59, boto==2.49, botocore==1.27.59, cachetools==4.2.2, confluent-kafka==1.9.2, datefinder==0.7.3, dnspython==2.2.1, elasticsearch==7.12, factory-boy==3.1, grpcio-health-checking==1.51.1, grpcio==1.51.1, hl7==0.4.5, hvac==0.11.2, jsonschema==3.2, lxml-stubs==0.4, lxml==4.9.2, moto==4.0.6, networkx==2.7.1, nltk==3.7, numpy==1.22.4, opentelemetry-instrumentation-boto==0.36b0, pandas==1.4.4, prometheus-client==0.14.1, protobuf==4.21.6, psycopg2-binary==2.9.5, py-grpc-prometheus==0.6, pypdf2==2.11.1, pytest-grpc==0.8, pytest==7.1.2, python-dateutil==2.8.2, python-dotenv==0.15, python-jose==3.2, python-json-logger==2.0.4, pytz==2021.1, reportlab==3.6.9, requests-aws4auth==1.1.2, requests==2.26, sentry-sdk==1.3.1, six==1.16, smart-open==5.2.1, sqlalchemy-utils==0.38.2, sqlalchemy==1.4.35, sqlparse==0.4.2, structlog==22.1, typing-extensions==3.10.0.2, xhtml2pdf==0.2.8

So a lot of subsetting, it looks like