https://pantsbuild.org/ logo
#general
Title
# general
p

plain-night-51324

01/27/2023, 7:27 PM
trying to run pants in my CI (tests work fine locally) and got stuck when CI is building stuff:
Copy code
Building 47 requirements for requirements.pex from the thirdparty/python/default.lock resolve: alembic==1.4.3
is taking forever. any docs/tips on ci integrations? All I am running is
./pants test ::
c

curved-manchester-66006

01/27/2023, 7:46 PM
Are you using lockfiles? In other words is CI stuck on resolving the deps, or is it using the same resolution as you used locally and is somehow stuck building the pex.
p

plain-night-51324

01/27/2023, 7:47 PM
yeah I generated latest lock files thirdparty/python/default.lock and it seems to be using it
c

curved-manchester-66006

01/27/2023, 7:50 PM
There is https://www.pantsbuild.org/docs/using-pants-in-ci But that is geared towards "kinda slow" problems, not "it seems to take forever"
h

happy-kitchen-89482

01/27/2023, 7:59 PM
What are your interpreter constraints?
And which version of Pants is this?
p

plain-night-51324

01/27/2023, 8:21 PM
Copy code
pants_version = "2.15.0rc1"

[python]
enable_resolves = true
interpreter_constraints = ["CPython==3.8.*"]
tailor_pex_binary_targets = false
Copy code
Bootstrapping Pants using /usr/bin/python3.8
Creating the virtualenv PEX.
Downloading the Pex PEX.
SHA256 fingerprint of <https://github.com/pantsbuild/pex/releases/download/v2.1.103/pex> verified.
Scrubbing PYTHONPATH
Installing pantsbuild.pants==2.15.0rc1 into a virtual environment at /root/.cache/pants/setup/bootstrap-Linux-x86_64/2.15.0rc1_py38
Scrubbing PYTHONPATH
New virtual environment successfully created at /root/.cache/pants/setup/bootstrap-Linux-x86_64/2.15.0rc1_py38.
19:08:52.43 [INFO] Initializing scheduler...
19:08:53.30 [INFO] Scheduler initialized.
...
19:25:14.28 [INFO] Completed: Building pytest_runner.pex
19:25:16.77 [INFO] Completed: Building 53 requirements for requirements.pex from the thirdparty/python/default.lock resolve: alembic==1.4.3, boto3==1.24.59, boto==2.49, botocore==1.27.59, cachetools==4.2.2, confluent-kafka==1.9.2,... (973 characters truncated)
19:25:17.37 [INFO] Completed: Building pytest_runner.pex
19:25:19.17 [INFO] Completed: Building pytest_runner.pex
so took 17 minutes to build.
h

happy-kitchen-89482

01/27/2023, 10:28 PM
What is in those
...
between
Copy code
19:08:53.30 [INFO] Scheduler initialized.
...
19:25:14.28 [INFO] Completed: Building pytest_runner.pex
?
p

plain-night-51324

01/27/2023, 11:33 PM
Copy code
19:13:31.98 [INFO] Long running tasks:
  61.20s	Building 47 requirements for requirements.pex from the thirdparty/python/default.lock resolve: alembic==1.4.3, boto3==1.24.59, boto==2.49, botocore==1.27.59, cachetools==4.2.2, confluent-kafka==1.9.2, datefinder==0.7.3, dnspython==2.2.1, elasticsearch==7.12, factory-boy==3.1, fsspec==2022.8.2, grpcio-health-checking==1.51.1, grpcio==1.51.1, hl7==0.4.5, hvac==0.11.2, jsonschema==3.2, lxml-stubs==0.4, lxml==4.9.2, moto==4.0.6, networkx==2.7.1, nltk==3.7, numpy==1.22.4, opentelemetry-instrumentation-boto==0.36b0, pandas==1.4.4, protobuf==4.21.6, psycopg2-binary==2.9.5, py-grpc-prometheus==0.6, pypdf2==2.11.1, pytest-grpc==0.8, pytest==7.1.2, python-dateutil==2.8.2, python-dotenv==0.15, python-jose==3.2, python-json-logger==2.0.4, pytz==2021.1, reportlab==3.6.9, requests-aws4auth==1.1.2, requests==2.26, sentry-sdk==1.3.1, six==1.16, smart-open==5.2.1, sqlalchemy-utils==0.38.2, sqlalchemy==1.4.35, sqlparse==0.4.2, structlog==22.1, typing-extensions==3.10.0.2, xhtml2pdf==0.2.8
  79.50s	Building 42 requirements for requirements.pex from the thirdparty/python/default.lock resolve: boto3==1.24.59, boto==2.49, botocore==1.27.59, cachetools==4.2.2, confluent-kafka==1.9.2, datefinder==0.7.3, dnspython==2.2.1, elasticsearch==7.12, grpcio-health-checking==1.51.1, grpcio==1.51.1, hl7==0.4.5, hvac==0.11.2, jsonschema==3.2, lxml-stubs==0.4, lxml==4.9.2, networkx==2.7.1, nltk==3.7, numpy==1.22.4, opentelemetry-instrumentation-boto==0.36b0, pandas==1.4.4, protobuf==4.21.6, psycopg2-binary==2.9.5, py-grpc-prometheus==0.6, pypdf2==2.11.1, pytest-grpc==0.8, python-dateutil==2.8.2, python-dotenv==0.15, python-jose==3.2, python-json-logger==2.0.4, pytz==2021.1, reportlab==3.6.9, requests-aws4auth==1.1.2, requests==2.26, sentry-sdk==1.3.1, six==1.16, smart-open==5.2.1, sqlalchemy-utils==0.38.2, sqlalchemy==1.4.35, sqlparse==0.4.2, structlog==22.1, typing-extensions==3.10.0.2, xhtml2pdf==0.2.8
Copy code
94.60s	Building 3 requirements for requirements.pex from the thirdparty/python/default.lock resolve: boto3==1.24.59, hvac==0.11.2, requests==2.26
  94.60s	Building pytest.pex from pytest_default.lock
19:11:25.28 [INFO] Completed: Building 1 requirement for requirements.pex from the thirdparty/python/default.lock resolve: sqlalchemy==1.4.35
19:11:30.17 [INFO] Starting: Building 12 requirements for requirements.pex from the thirdparty/python/default.lock resolve: boto3==1.24.59, elasticsearch==7.12, grpcio-health-checking==1.51.1, grpcio==1.51.1, protobuf==4.21.6, ps... (151 characters truncated)
Copy code
111.91s	Building 5 requirements for requirements.pex from the thirdparty/python/default.lock resolve: elasticsearch==7.12, grpcio-health-checking==1.51.1, grpcio==1.51.1, python-json-logger==2.0.4, pytz==2021.1
  112.11s	Building 5 requirements for requirements.pex from the thirdparty/python/default.lock resolve: elasticsearch==7.12, grpcio-health-checking==1.51.1, grpcio==1.51.1, python-json-logger==2.0.4, sqlalchemy==1.4.35
  126.20s	Building 3 requirements for requirements.pex from the thirdparty/python/default.lock resolve: boto3==1.24.59, hvac==0.11.2, requests==2.26
  126.21s	Building pytest.pex from pytest_default.lock
  180.21s	Resolve transitive targets
Copy code
200.10s	Building 48 requirements for requirements.pex from the thirdparty/python/default.lock resolve: alembic==1.4.3, boto3==1.24.59, boto==2.49, botocore==1.27.59, cachetools==4.2.2, confluent-kafka==1.9.2, datefinder==0.7.3, dnspython==2.2.1, elasticsearch==7.12, factory-boy==3.1, fsspec==2022.8.2, grpcio-health-checking==1.51.1, grpcio==1.51.1, hl7==0.4.5, hvac==0.11.2, jsonschema==3.2, lxml-stubs==0.4, lxml==4.9.2, moto==4.0.6, networkx==2.7.1, nltk==3.7, numpy==1.22.4, opentelemetry-instrumentation-boto==0.36b0, pandas==1.4.4, protobuf==4.21.6, psycopg2-binary==2.9.5, py-grpc-prometheus==0.6, pypdf2==2.11.1, pytest-grpc==0.8, pytest==7.1.2, python-dateutil==2.8.2, python-dotenv==0.15, python-jose==3.2, python-json-logger==2.0.4, pytz==2021.1, reportlab==3.6.9, requests-aws4auth==1.1.2, requests==2.26, s3fs==2022.8.2, sentry-sdk==1.3.1, six==1.16, smart-open==5.2.1, sqlalchemy-utils==0.38.2, sqlalchemy==1.4.35, sqlparse==0.4.2, structlog==22.1, typing-extensions==3.10.0.2, xhtml2pdf==0.2.8
  200.20s	Building 46 requirements for requirements.pex from the thirdparty/python/default.lock resolve: alembic==1.4.3, boto3==1.24.59, boto==2.49, botocore==1.27.59, cachetools==4.2.2, confluent-kafka==1.9.2, datefinder==0.7.3, dnspython==2.2.1, elasticsearch==7.12, factory-boy==3.1, grpcio-health-checking==1.51.1, grpcio==1.51.1, hl7==0.4.5, hvac==0.11.2, jsonschema==3.2, lxml-stubs==0.4, lxml==4.9.2, moto==4.0.6, networkx==2.7.1, nltk==3.7, numpy==1.22.4, opentelemetry-instrumentation-boto==0.36b0, pandas==1.4.4, protobuf==4.21.6, psycopg2-binary==2.9.5, py-grpc-prometheus==0.6, pypdf2==2.11.1, pytest-grpc==0.8, pytest==7.1.2, python-dateutil==2.8.2, python-dotenv==0.15, python-jose==3.2, python-json-logger==2.0.4, pytz==2021.1, reportlab==3.6.9, requests-aws4auth==1.1.2, requests==2.26, sentry-sdk==1.3.1, six==1.16, smart-open==5.2.1, sqlalchemy-utils==0.38.2, sqlalchemy==1.4.35, sqlparse==0.4.2, structlog==22.1, typing-extensions==3.10.0.2, xhtml2pdf==0.2.8
Copy code
244.70s	Building 50 requirements for requirements.pex from the thirdparty/python/default.lock resolve: aiobotocore==2.4.2, aiohttp==3.8.3, alembic==1.4.3, boto3==1.24.59, boto==2.49, botocore==1.27.59, cachetools==4.2.2, confluent-kafka==1.9.2, datefinder==0.7.3, dnspython==2.2.1, elasticsearch==7.12, factory-boy==3.1, fsspec==2022.8.2, grpcio-health-checking==1.51.1, grpcio==1.51.1, hl7==0.4.5, hvac==0.11.2, jsonschema==3.2, lxml-stubs==0.4, lxml==4.9.2, moto==4.0.6, networkx==2.7.1, nltk==3.7, numpy==1.22.4, opentelemetry-instrumentation-boto==0.36b0, pandas==1.4.4, protobuf==4.21.6, psycopg2-binary==2.9.5, py-grpc-prometheus==0.6, pypdf2==2.11.1, pytest-grpc==0.8, pytest==7.1.2, python-dateutil==2.8.2, python-dotenv==0.15, python-jose==3.2, python-json-logger==2.0.4, pytz==2021.1, reportlab==3.6.9, requests-aws4auth==1.1.2, requests==2.26, s3fs==2022.8.2, sentry-sdk==1.3.1, six==1.16, smart-open==5.2.1, sqlalchemy-utils==0.38.2, sqlalchemy==1.4.35, sqlparse==0.4.2, structlog==22.1, typing-extensions==3.10.0.2, xhtml2pdf==0.2.8
  252.70s	Building 47 requirements for requirements.pex from the thirdparty/python/default.lock resolve: alembic==1.4.3, boto3==1.24.59, boto==2.49, botocore==1.27.59, cachetools==4.2.2, confluent-kafka==1.9.2, datefinder==0.7.3, dnspython==2.2.1, elasticsearch==7.12, factory-boy==3.1, grpcio-health-checking==1.51.1, grpcio==1.51.1, hl7==0.4.5, hvac==0.11.2, jsonschema==3.2, lxml-stubs==0.4, lxml==4.9.2, moto==4.0.6, networkx==2.7.1, nltk==3.7, numpy==1.22.4, opentelemetry-instrumentation-boto==0.36b0, pandas==1.4.4, protobuf==4.21.6, psycopg2-binary==2.9.5, py-grpc-prometheus==0.6, pypdf2==2.11.1, pytest-grpc==0.8, pytest==7.1.2, python-dateutil==2.8.2, python-dotenv==0.15, python-jose==3.2, python-json-logger==2.0.4, pytz==2021.1, ratelimiter==1.2, reportlab==3.6.9, requests-aws4auth==1.1.2, requests==2.26, sentry-sdk==1.3.1, six==1.16, smart-open==5.2.1, sqlalchemy-utils==0.38.2, sqlalchemy==1.4.35, sqlparse==0.4.2, structlog==22.1, typing-extensions==3.10.0.2, xhtml2pdf==0.2.8
Copy code
470.80s	Building 47 requirements for requirements.pex from the thirdparty/python/default.lock resolve: alembic==1.4.3, boto3==1.24.59, boto==2.49, botocore==1.27.59, cachetools==4.2.2, confluent-kafka==1.9.2, datefinder==0.7.3, dnspython==2.2.1, elasticsearch==7.12, factory-boy==3.1, grpcio-health-checking==1.51.1, grpcio==1.51.1, hl7==0.4.5, hvac==0.11.2, jsonschema==3.2, jwt==1.3.1, lxml-stubs==0.4, lxml==4.9.2, moto==4.0.6, networkx==2.7.1, nltk==3.7, numpy==1.22.4, opentelemetry-instrumentation-boto==0.36b0, pandas==1.4.4, protobuf==4.21.6, psycopg2-binary==2.9.5, py-grpc-prometheus==0.6, pypdf2==2.11.1, pytest-grpc==0.8, pytest==7.1.2, python-dateutil==2.8.2, python-dotenv==0.15, python-jose==3.2, python-json-logger==2.0.4, pytz==2021.1, reportlab==3.6.9, requests-aws4auth==1.1.2, requests==2.26, sentry-sdk==1.3.1, six==1.16, smart-open==5.2.1, sqlalchemy-utils==0.38.2, sqlalchemy==1.4.35, sqlparse==0.4.2, structlog==22.1, typing-extensions==3.10.0.2, xhtml2pdf==0.2.8
  471.00s	Building 47 requirements for requirements.pex from the thirdparty/python/default.lock resolve: alembic==1.4.3, boto3==1.24.59, boto==2.49, botocore==1.27.59, cachetools==4.2.2, confluent-kafka==1.9.2, datefinder==0.7.3, dnspython==2.2.1, elasticsearch==7.12, factory-boy==3.1, grpcio-health-checking==1.51.1, grpcio==1.51.1, hl7==0.4.5, hvac==0.11.2, jsonschema==3.2, lxml-stubs==0.4, lxml==4.9.2, moto==4.0.6, networkx==2.7.1, nltk==3.7, numpy==1.22.4, opentelemetry-instrumentation-boto==0.36b0, pandas==1.4.4, prometheus-client==0.14.1, protobuf==4.21.6, psycopg2-binary==2.9.5, py-grpc-prometheus==0.6, pypdf2==2.11.1, pytest-grpc==0.8, pytest==7.1.2, python-dateutil==2.8.2, python-dotenv==0.15, python-jose==3.2, python-json-logger==2.0.4, pytz==2021.1, reportlab==3.6.9, requests-aws4auth==1.1.2, requests==2.26, sentry-sdk==1.3.1, six==1.16, smart-open==5.2.1, sqlalchemy-utils==0.38.2, sqlalchemy==1.4.35, sqlparse==0.4.2, structlog==22.1, typing-extensions==3.10.0.2, xhtml2pdf==0.2.8
h

happy-kitchen-89482

01/29/2023, 4:27 PM
So a lot of subsetting, it looks like
Well, not a lot, but that seems to be taking time
How many cores on the CI machine? And what does your pants.toml and pants.ci.toml (if you have one) look like?
9 Views