https://pantsbuild.org/ logo
Join Slack
Powered by
I'm trying to setup pants on a pre-existing python...
# general
c
I'm trying to setup pants on a pre-existing python repo and am unsure how to debug 
generate-lockfiles
failing with
Copy code
ERROR: Could not find a version that satisfies the requirement awswrangler==2.16.1
ERROR: No matching distribution found for awswrangler==2.16.1
Since that version clearly exists https://pypi.org/project/awswrangler/2.16.1/#history
✅ 1
Fuller command
Copy code
$ ./pants  --keep-sandboxes=on_failure generate-lockfiles ::
11:12:09.30 [INFO] Initialization options changed: reinitializing scheduler...
11:12:09.68 [INFO] Scheduler initialized.
11:12:16.25 [INFO] Preserving local process execution dir /tmp/pants-sandbox-dBTWdX for Generate lockfile for python-default
11:12:16.25 [INFO] Completed: Generate lockfile for python-default
11:12:16.25 [ERROR] 1 Exception encountered:

  ProcessExecutionFailure: Process 'Generate lockfile for python-default' failed with exit code 1.
stdout:

stderr:
pid 20310 -> /home/ecsb/.cache/pants/named_caches/pex_root/venvs/c62b9c899ed243adbed19764c68013faab4e0d7b/ddab8011daaee380698ac2fb9701af18c90c03f6/bin/python -sE /home/ecsb/.cache/pants/named_caches/pex_root/venvs/c62b9c899ed243adbed19764c68013faab4e0d7b/ddab8011daaee380698ac2fb9701af18c90c03f6/pex --disable-pip-version-check --no-python-version-warning --exists-action a --no-input --isolated -q --cache-dir /home/ecsb/.cache/pants/named_caches/pex_root/pip_cache --log /tmp/pants-sandbox-dBTWdX/.tmp/pex-pip-log.wtscry14/pip.log download --dest /tmp/pants-sandbox-dBTWdX/.tmp/tmpl2w5f4ht/usr.bin.python3.8 apache-airflow==2.2.2 awswrangler==2.16.1 backports.zoneinfo==0.2.1; python_version < "3.9" beautifulsoup4==4.10.0 boto3~=1.21.21 botocore~=1.24.46 click click-logging coolname~=2.0.0 dask-snowflake==0.1.0 dask==2022.1.1 docker gql==2.0.0 great-expectations==0.15.27 holidays~=0.16 html5lib==1.1 numpy==1.21.6 pandas==1.2.4 pangres==2.3 pendulum==2.1.2 pre-commit==2.20.0 prometheus-client==0.11.0 protobuf==3.20.1 psycopg2-binary==2.8.6 pytest-cov<5,<=4.0.0 pytest<8,>=7.2.0 python-dateutil==2.8.1 python-dotenv==0.20.0 python-slugify==4.0.1 python-ulid==1.0.2 pytz==2021.3 requests-pkcs12==1.13 requests==2.26.0 sagemaker==2.34.0 snowflake-connector-python==2.7.7 sortedcontainers==2.4.0 sqlalchemy==1.3.24 xlrd==2.0.1 --index-url <https://pypi.org/simple/> --retries 5 --timeout 15 exited with 1 and STDERR:
ERROR: Could not find a version that satisfies the requirement awswrangler==2.16.1
ERROR: No matching distribution found for awswrangler==2.16.1
g
👀 following. I’ve been getting the same issues on my end for specific versions of numpy/pandas
c
:-/ If I comment out awswrangler I next get a numpy error
g
Just curious, what type of CPU are you using? I’m on an M1 Mac so I already deal with a lot of conflicting packages with different availabilities for wheels
c
this usually comes down to the python interpreter constraints and machine platform being used.
you’ll likely find more detailed error message in 
/tmp/pants-sandbox-dBTWdX/.tmp/pex-pip-log.wtscry14/pip.log
c
Just curious, what type of CPU are you using? I’m on an M1 Mac so I already deal with a lot of conflicting packages with different availabilities for wheels
Linux x86_64
you’ll likely find more detailed error message in 
/tmp/pants-sandbox-dBTWdX/.tmp/pex-pip-log.wtscry14/pip.log
I hoped so too! But
Copy code
$ cat /tmp/pants-sandbox-dBTWdX/.tmp/pex-pip-log.wtscry14/pip.log
cat: /tmp/pants-sandbox-dBTWdX/.tmp/pex-pip-log.wtscry14/pip.log: No such file or directory
e
@curved-manchester-66006 what are your interpreter constraints?: https://www.pantsbuild.org/docs/python-interpreter-compatibility
IIRC the default if unconfigured is ">=3.7,<4". When creating a lock, the lock then needs to satisfy all those versions. That is usually hard! Not many project distributions support Python 3.11 for example which is in that range.
c
I've tried not having it set. And also these values 
interpreter_constraints = ['CPython>=3.7,<3.10']
interpreter_constraints = ['CPython>=3.7,<3.9']
e
That's the problem. Look here: https://pypi.org/project/awswrangler/2.16.1/
*Requires:* Python >=3.7.1, <3.11
That 
.1
kills you.
💀 1
So Pants / Pex correctly fail the lock. WIth very unhelpful messaging to be sure.
c
i'm not following why the 
.1
is the angel of death. Isn't 3.8.13 for example still 
Python >=3.7.1, <3.11
e
A lockfile must work for any machine / Python you claim you support. You claim you support 3.7.0
You don't so just bump the lower bound!
g
I actually ran into this on a separate, non-pants project, I believe the underlying culprit is numpy. I don’t remember if it was due to incompatibilities or just security warnings but supporting 
3.7.1
and above became much easier than 
>=3.7
c
🪤
e
So, Pants taught you something about your app - it doesn't in fact support all Python 3.7's. It did so as approxiamtely the worst teacher ever, or at least speaking a language no one but it understands.
c
Thank you very much!
g
Yeah this was extremely helpful for me too. I can’t wait for 3.7 to go away - especially with my M1 mac
e
You're welcome. I'm just part time and so just implementing locking in Pex took all my time and then some over the past year. It would be good to find time to circle back and improve error messages.
@gentle-painting-24549 just wipe it and install Asahi Linux - that will work fine with Python 3.7
g
Oh that’s super neat I’ve never seen that before. Currently I’m one of the only M1s in a sea of Intels running a 3.7 app, so things have been painful until we cut over to 3.9 in the coming months.
e
Aha. Well if you're in a sea of macOS you're sunk, but I try. My early years were on Apple IIs & logo and then later on a IIe learning Applesoft Basic. I had love for Apple then. I now do not. Its insane I can't run their os on their hardware in a vm if I've wiped their os and installed linux on it.
I my mind there harware is amazing, but they are developer hostile with their OS / rules.
g
Millennial developer over here 🙋 Apple is all I really know and I’ve always gotten along fine - it’s never really been a problem until now when I have a different processor than all my co-workers. I just got the M1 Max 16" and the hardware is impeccable though 😍
e
It truly is. Just watch your back unless you develop macOS / iOS native apps!
h
We should probably document a strong recommendation to pin your ICs to a single Python version (which wouldn’t have helped with that 
.1
issue in this case, but will at least reduce the number of times this happens)
e
I don't actually think that's a great rec in general. I do think we should not have default ICs though and force you to choose something if you're using lockfiles.
h
Most orgs seem to standardize their codebases on a single Python version, so recommending pinning the ICs to, e.g., 
==3.9.*
seems sensible?
And yes, forcing the question by not having defaults at all makes sense to me
c
also, a helpful error message would be much preferred imho over a recommendation that side steps the issue, at best. (nothing stops us from doing both, of course)
e
==3.7.*
is fine but would not have helped here since 
3.7.0
turned out problematic. So, in favor of a pin rec, but not to a patch version in general. That just doesn't generally apply / generally there is no need to pin a patch.
As far as the error message goes, see my above, but yeah - work needs to be done there in Pex.
Ah, I see now you acked the would not have helped here Benjy. Agreed!
c
ack, I’ve been pondering on how to tackle the error message situation here myself as well, as I’ve run into this precise issue as well before, and likely will again in the future… 😛
8 Views
Open in Slack
PreviousNext
Powered by