17:02:39.11 [DEBUG] spawned local process as Some(42909) for Process { argv ["/Users/joao.albuquerque/Library/Caches/nce/b53192ff83dd22c11f093268f09503786fcdeb806b34ecf4d2e519c915578288/bindings/venvs/2.21.0/bin/python3.9", "./pex", "lock", [...]

From what I understand, pip inside the locking process (example above) was unable to resolve the group of dependencies described. My question is: How can I identify which libs are causing the problem?

I'm using pants

"2.21.0"

version, and this interpreter constraint for this resolve

chatbot = ["CPython>=3.10,<3.11"]

Testing some possibilities, I realized that: - That removing the

chainlit

library makes it possible to successfully perform the lock. However, my team has communicated that this library will be mandatory. - I also realized that if I perform the lock without

chainlit

and create a virtual environment using this lockfile and install it manually with pip

pip install chainlit

, the installation in the virtual environment works normally and it finds version

1.3.2

.

Full error and the lib list below:

18:23:17.35 [ERROR] 1 Exception encountered:

Engine traceback:
  in `generate-lockfiles` goal

ProcessExecutionFailure: Process 'Generate lockfile for chatbot' failed with exit code 1.
stdout:

stderr:
pid 42941 -> /Users/joao.albuquerque/.cache/pants/named_caches/pex_root/venvs/0deb89b23adbbad7a5b9aac06d2362acf7572cfa/6a1ac64ea92c14ecf33eb04b55a3e4438ed94d17/bin/python -sE /Users/joao.albuquerque/.cache/pants/named_caches/pex_root/venvs/0deb89b23adbbad7a5b9aac06d2362acf7572cfa/6a1ac64ea92c14ecf33eb04b55a3e4438ed94d17/pex --disable-pip-version-check --no-python-version-warning --exists-action a --no-input --isolated -q --cache-dir /Users/joao.albuquerque/.cache/pants/named_caches/pex_root/pip/24.0/pip_cache --log /private/var/folders/5h/cth2ppkj7zs3rmyydmt02jdr0000gq/T/pants-sandbox-tbdK8q/.tmp/pex-pip-log.hb5ijiy0/pip.log download --dest /private/var/folders/5h/cth2ppkj7zs3rmyydmt02jdr0000gq/T/pants-sandbox-tbdK8q/.tmp/tmpnq_f7yk2/usr.local.Cellar.python@3.10.3.10.14_1.Frameworks.Python.framework.Versions.3.10.bin.python3.10 GitPython>=3.0.0 IPython Pillow>=8.4.0 PyYAML>=5.4.1 acsylla==0.2.0 awswrangler boto3>=1.17.5 cassandra-driver chainlit confluent-kafka>=1.9.2 cookiecutter databricks-sdk fastapi<1.0.0 h3<4.0 httpx joblib>=1.0.1 jsonschema langchain langchain-openai langgraph matplotlib matplotlib-venn numpy>=1.21.4 optuna pandas<2.0,>=1.4.0 parameters-validation>=1.2.0 plotly==5.5.0 prometheus-client pyarrow!=12.0.1,>=5.0.0 pydantic!=2.6.0,>=2.5.0 pydot<2.0.0,>=1.4.2 pyspark==3.2.1; python_version == "3.8" pyspark==3.4.1; python_version == "3.10" pytest pytest-asyncio pytest-spark python-slugify qdrant-client requests rich sagemaker scikit-learn>=1.1.0 scipy seaborn sentry-sdk[fastapi] setuptools shap statsmodels tabulate typer typing_extensions uvicorn[standard]<1.0.0 watchfiles --index-url <https://pypi.org/simple/> --find-links <https://wheels.pantsbuild.org/simple> --retries 5 --timeout 15 exited with 2 and STDERR:
  WARNING: Retrying (Retry(total=4, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ProtocolError('Connection aborted.', RemoteDisconnected('Remote end closed connection without response'))': /packages/e7/8c/efaf717cb62a6065c1720abe20b7150ae5d8ee1b5b7df0008fc567a3a2dd/sagemaker-2.222.1-py3-none-any.whl.metadata
ERROR: Exception:
Traceback (most recent call last):
  File "/Users/joao.albuquerque/.cache/pants/named_caches/pex_root/venvs/0deb89b23adbbad7a5b9aac06d2362acf7572cfa/6a1ac64ea92c14ecf33eb04b55a3e4438ed94d17/lib/python3.10/site-packages/pip/_internal/cli/base_command.py", line 180, in exc_logging_wrapper
    status = run_func(*args)
  File "/Users/joao.albuquerque/.cache/pants/named_caches/pex_root/venvs/0deb89b23adbbad7a5b9aac06d2362acf7572cfa/6a1ac64ea92c14ecf33eb04b55a3e4438ed94d17/lib/python3.10/site-packages/pip/_internal/cli/req_command.py", line 245, in wrapper
    return func(self, options, args)
  File "/Users/joao.albuquerque/.cache/pants/named_caches/pex_root/venvs/0deb89b23adbbad7a5b9aac06d2362acf7572cfa/6a1ac64ea92c14ecf33eb04b55a3e4438ed94d17/lib/python3.10/site-packages/pip/_internal/commands/download.py", line 132, in run
    requirement_set = resolver.resolve(reqs, check_supported_wheels=True)
  File "/Users/joao.albuquerque/.cache/pants/named_caches/pex_root/venvs/0deb89b23adbbad7a5b9aac06d2362acf7572cfa/6a1ac64ea92c14ecf33eb04b55a3e4438ed94d17/lib/python3.10/site-packages/pip/_internal/resolution/resolvelib/resolver.py", line 95, in resolve
    result = self._result = resolver.resolve(
  File "/Users/joao.albuquerque/.cache/pants/named_caches/pex_root/venvs/0deb89b23adbbad7a5b9aac06d2362acf7572cfa/6a1ac64ea92c14ecf33eb04b55a3e4438ed94d17/lib/python3.10/site-packages/pip/_vendor/resolvelib/resolvers.py", line 546, in resolve
    state = resolution.resolve(requirements, max_rounds=max_rounds)
  File "/Users/joao.albuquerque/.cache/pants/named_caches/pex_root/venvs/0deb89b23adbbad7a5b9aac06d2362acf7572cfa/6a1ac64ea92c14ecf33eb04b55a3e4438ed94d17/lib/python3.10/site-packages/pip/_vendor/resolvelib/resolvers.py", line 457, in resolve
    raise ResolutionTooDeep(max_rounds)
pip._vendor.resolvelib.resolvers.ResolutionTooDeep: 200000

Group of libs on requirements.txt (the lock process consider them both):

chatbot.txt

chainlit
langgraph
langchain-openai
langchain
langchain_core

base.txt

GitPython>=3.0.0
Pillow>=8.4.0
PyYAML>=5.4.1
awswrangler
boto3>=1.17.5
cookiecutter
databricks-sdk
fastapi<1.0.0
h3<4.0
joblib>=1.0.1
jsonschema
matplotlib
matplotlib-venn
numpy>=1.21.4
optuna
pandas>=1.4.0,<2.0
parameters-validation>=1.2.0
plotly==5.5.0
prometheus-client
pyarrow>=5.0.0,!=12.0.1
pydantic>=2.5.0,!=2.6.0 # <https://github.com/pydantic/pydantic/issues/8670>
pydot<2.0.0,>=1.4.2
pyspark==3.2.1; python_version == "3.8"
pyspark==3.4.1; python_version == "3.10"
pytest
pytest-spark
rich
sagemaker
scikit-learn>=1.1.0
scipy
seaborn
sentry-sdk[fastapi]
setuptools
shap
statsmodels
tabulate
typer
typing_extensions
uvicorn[standard]<1.0.0
watchfiles
confluent-kafka>=1.9.2
python-slugify
cassandra-driver
qdrant-client
IPython
requests
acsylla==0.2.0

Unfortunately it is still not working. At this moment the resolve has been running for ~45min. I tried using

CPython==3.10.14

Had the same error 😞

OK, so next thing to try is to run with

--keep-sandboxes=always

and identify the sandbox directory that Pants runs

pex

in. In that sandbox will be a

__run.sh

containing the pex command line

Then we can see if we can reproduce the problem entirely outside of pants