https://pantsbuild.org/ logo
Join Slack
Powered by
I have a super strange error that I can't seem to ...
# development
g
I have a super strange error that I can't seem to track down. I'm using pants v2.21.0
Copy code
16:24:09.06 [INFO] Completed: Generate lockfile for altana-ml
16:24:09.07 [ERROR] 1 Exception encountered:

Engine traceback:
  in root
    ..
  in pants.core.goals.generate_lockfiles.generate_lockfiles_goal
    `generate-lockfiles` goal

Traceback (most recent call last):
  File "/Users/jasonbarnett/Library/Caches/nce/f46d2c12132bad9c27e0dd509186c190371abfe1add7e5ca42245466f35bea81/bindings/venvs/2.21.0/lib/python3.9/site-packages/pants/core/goals/generate_lockfiles.py", line 635, in generate_lockfiles_goal
    results = await MultiGet(
  File "/Users/jasonbarnett/Library/Caches/nce/f46d2c12132bad9c27e0dd509186c190371abfe1add7e5ca42245466f35bea81/bindings/venvs/2.21.0/lib/python3.9/site-packages/pants/engine/internals/selectors.py", line 376, in MultiGet
    return await _MultiGet(tuple(__arg0))
  File "/Users/jasonbarnett/Library/Caches/nce/f46d2c12132bad9c27e0dd509186c190371abfe1add7e5ca42245466f35bea81/bindings/venvs/2.21.0/lib/python3.9/site-packages/pants/engine/internals/selectors.py", line 174, in __await__
    result = yield self.gets
  File "/Users/jasonbarnett/Library/Caches/nce/f46d2c12132bad9c27e0dd509186c190371abfe1add7e5ca42245466f35bea81/bindings/venvs/2.21.0/lib/python3.9/site-packages/pants/backend/python/goals/lockfile.py", line 104, in generate_lockfile
    result = await Get(
  File "/Users/jasonbarnett/Library/Caches/nce/f46d2c12132bad9c27e0dd509186c190371abfe1add7e5ca42245466f35bea81/bindings/venvs/2.21.0/lib/python3.9/site-packages/pants/engine/internals/selectors.py", line 124, in __await__
    result = yield self
  File "/Users/jasonbarnett/Library/Caches/nce/f46d2c12132bad9c27e0dd509186c190371abfe1add7e5ca42245466f35bea81/bindings/venvs/2.21.0/lib/python3.9/site-packages/pants/engine/process.py", line 309, in fallible_to_exec_result_or_raise
    raise ProcessExecutionFailure(
pants.engine.process.ProcessExecutionFailure: Process 'Generate lockfile for altana-ml' failed with exit code 1.
stdout:

stderr:
.* suffix can only be used with `==` or `!=` operators
    scikit-learn (>=1.0.*) ; extra == 'pipelines'
f
You specified 
scikit-learn (>=1.0.*)
g
How do I found the actual root. Is it an issue with a 3rd party dep wheel or what?
I did not specify that. I specified:
Copy code
scikit-learn = "^1.3.0"
f
g
I don't see the relationship to the error.
f
Something is specifying >=1.0.* for scikit-learn. .* is the suffix of 1.0.x
and pip does not support using a .* version suffix with anything other than == and != version operators
g
I'm 100% confident none of my explicit deps are using that. Is it possible one of the wheels that is published to pypi.org is?
I'm wondering how the hell I find the source of the problem 🙂
f
right, you'll need to check the transitive dependencies. That https://github.com/tox-dev/pipdeptree tool from the Medium post looks interesting. It prints out pip dependency trees
g
will the tree have the deprecated notation listed for deps?
f
I haven't used it. I just dd a quick search for tools.
✅ 1
I would check your ML-related dependencies first.
I assume one of them pulls in scikit-learn.
(since it would be silly for a non-ML package to do so)
g
Thanks
I just wish the error from PEX was better.
Like it's crazy it doesn't say WHAT package when clearly it knows lol
f
pex depends on pip for dependency resolution
g
well pips error messages suck then 😛
💯 1
f
are you able to share your requirements.txt?
g
sure, but it's a cluster of 3 pyproject.toml files.
f
or just the package/version constraints from them
g
sure
here is the first
Copy code
[tool.poetry.dev-dependencies]
artifacts-keyring = "*"
black = "~23.12"
codecov = "^2.1"
coverage = "^6.4.1"
flask_testing = "^0.8"
freezegun = "^1.2.1"
MarkupSafe = "^2.0"
moto = { version = "^3.0", extras = ["s3", "sqs"] }
nose-htmloutput = "^0.6"
nose = "^1.3"
pluggy = ">=0.13"
pre-commit = "~3.6"
pylint = "^2.5"
py = "^1.10"
pytest-cov = "^3"
pytest = "^7.2"
randomize = "^0.14"
requests-mock = "^1.8"
safety = "^2.2"
setuptools = "*"
testcontainers = "^3.5"
testresources = "2.0"
tox = "^4"
tox-current-env = "*"
twine = "*"
wheel = "*"
pytest-alembic = "^0"
second
Copy code
[tool.poetry.dependencies]
python = ">=3.9,<3.10"
boto3 = "^1.34.108"
pandas = "1.4.2"           # pinned to match Databricks LTS 12.2
scikit-learn = "^1.3.0"
aiohttp = "^3.8.1"
kaleido = "0.2.1"
mlflow = "1.27.0"
nbformat = "5.4.0"
nltk = "^3.8.1"
seaborn = "^0.13.2"
shap = "0.41.0"
tqdm = "4.64.0"
requests = ">=2.28"
umap-learn = "0.5.3"
shapwrap = "0.1.0"
plotly = "^5.9"
pydantic = "^1.10.2"
tensorflow-hub = "^0.12.0"
oauthlib = "^3.2.1"
joblib = "^1.2.0"
protobuf = "^4.21.6"
xgboost = "^2"
what a total shit show 🤦‍♂️
f
I see lots of ^ specifiers there. You could try selecting more exact versions and then, if the failure repeats, you would at least know what version of the dependency to look at to find the one doing the scikit-learn >= bit
g
I will probably write something to pull all of the packages that match the constraint and search their deps. my goodness this is pathetic.
f
and PyPi does not show the transitive deps apparently
g
oof. I have no idea how to get to root. I can't believe this was put into code without a clearer error message.
f
whereas in the Rust world, crates.io does. e.g., https://crates.io/crates/tokio/1.38.0/dependencies
👍 1
g
I used poetry to show the dep tree
Copy code
poetry show --tree
going to hunt down the problem child now
@fast-nail-55400 I ended up running within the sandbox and added 
PEX_VERBOSE=9
Untitled
Untitled
I posted an issue on the packaging repo: https://github.com/pypa/packaging/issues/811
I modified the vendored source in hopes of finding my problem child...
I just hacked the source to bypass the exception and then get the name of the package. Gosh this is frustrating.
ok, so after digging I actually think it's not a packaging library issue but how pex is using it. I think I know how to make this error message better.
I found the damn culprit:
Copy code
$ pip download mlflow==1.27 --no-deps
Collecting mlflow==1.27
  Downloading mlflow-1.27.0-py3-none-any.whl.metadata (10 kB)
WARNING: Ignoring version 1.27.0 of mlflow since it has invalid metadata:
Requested mlflow==1.27 from <https://files.pythonhosted.org/packages/12/49/1c6f1535bb8b9f35463b043c08a902531a367ed42b0fe4afb3882bb28f8a/mlflow-1.27.0-py3-none-any.whl> has invalid metadata: .* suffix can only be used with `==` or `!=` operators
    scikit-learn (>=1.0.*) ; extra == 'pipelines'
                  ~~~~~~^
Please use pip<24.1 if you need to use this version.
ERROR: Ignored the following yanked versions: 2.0.0, 2.12.0
ERROR: Could not find a version that satisfies the requirement mlflow==1.27 (from versions: 0.0.1, 0.1.0, 0.2.0, 0.2.1, 0.3.0, 0.4.0, 0.4.1, 0.4.2, 0.5.0, 0.5.1, 0.5.2, 0.6.0, 0.7.0, 0.8.0, 0.8.1, 0.8.2, 0.9.0, 0.9.0.1, 0.9.1, 1.0.0, 1.1.0, 1.1.1.dev0, 1.2.0, 1.3.0, 1.4.0, 1.5.0, 1.6.0, 1.7.0, 1.7.1, 1.7.2, 1.8.0, 1.9.0, 1.9.1, 1.10.0, 1.11.0, 1.12.0, 1.12.1, 1.13, 1.13.1, 1.14.0, 1.14.1, 1.15.0, 1.16.0, 1.17.0, 1.18.0, 1.19.0, 1.20.0, 1.20.1, 1.20.2, 1.21.0, 1.22.0, 1.23.0, 1.23.1, 1.24.0, 1.25.0, 1.25.1, 1.26.0, 1.26.1, 1.27.0, 1.28.0, 1.29.0, 1.30.0, 1.30.1, 2.0.0rc0, 2.0.1, 2.1.0, 2.1.1, 2.2.0, 2.2.1, 2.2.2, 2.3.0, 2.3.1, 2.3.2, 2.4.0, 2.4.1, 2.4.2, 2.5.0, 2.6.0, 2.7.0, 2.7.1, 2.8.0, 2.8.1, 2.9.0, 2.9.1, 2.9.2, 2.10.0, 2.10.1, 2.10.2, 2.11.0, 2.11.1, 2.11.2, 2.11.3, 2.11.4, 2.12.1, 2.12.2, 2.13.0, 2.13.1, 2.13.2, 2.14.0rc0, 2.14.0, 2.14.1)
ERROR: No matching distribution found for mlflow==1.27
OK, this is definitely a pex bug. The information is there, the exception is just not being handled as well as it could be.
I'll submit an issue to pex.
f
Isn't it also an issue with the 
mlflow
library? https://github.com/mlflow/mlflow/blob/f3199e2c9354b49f30dc9325c144d028ac170e03/setup.py#L169 seems to be where the invalid version specifier is. Also appears to be a 2.year old version.
g
Yes that is the root cause of this issue but pex could have displayed an error that mentioned both the library and the version where the bad specification was made
Opened a stubbed out PR to discuss: https://github.com/pex-tool/pex/pull/2440
25 Views
Open in Slack
PreviousNext
Powered by