https://pantsbuild.org/ logo
#general
Title
# general
c

curved-manchester-66006

11/10/2022, 4:12 PM
I'm trying to setup pants on a pre-existing python repo and am unsure how to debug
generate-lockfiles
failing with
Copy code
ERROR: Could not find a version that satisfies the requirement awswrangler==2.16.1
ERROR: No matching distribution found for awswrangler==2.16.1
Since that version clearly exists https://pypi.org/project/awswrangler/2.16.1/#history
1
Fuller command
Copy code
$ ./pants  --keep-sandboxes=on_failure generate-lockfiles ::
11:12:09.30 [INFO] Initialization options changed: reinitializing scheduler...
11:12:09.68 [INFO] Scheduler initialized.
11:12:16.25 [INFO] Preserving local process execution dir /tmp/pants-sandbox-dBTWdX for Generate lockfile for python-default
11:12:16.25 [INFO] Completed: Generate lockfile for python-default
11:12:16.25 [ERROR] 1 Exception encountered:

  ProcessExecutionFailure: Process 'Generate lockfile for python-default' failed with exit code 1.
stdout:

stderr:
pid 20310 -> /home/ecsb/.cache/pants/named_caches/pex_root/venvs/c62b9c899ed243adbed19764c68013faab4e0d7b/ddab8011daaee380698ac2fb9701af18c90c03f6/bin/python -sE /home/ecsb/.cache/pants/named_caches/pex_root/venvs/c62b9c899ed243adbed19764c68013faab4e0d7b/ddab8011daaee380698ac2fb9701af18c90c03f6/pex --disable-pip-version-check --no-python-version-warning --exists-action a --no-input --isolated -q --cache-dir /home/ecsb/.cache/pants/named_caches/pex_root/pip_cache --log /tmp/pants-sandbox-dBTWdX/.tmp/pex-pip-log.wtscry14/pip.log download --dest /tmp/pants-sandbox-dBTWdX/.tmp/tmpl2w5f4ht/usr.bin.python3.8 apache-airflow==2.2.2 awswrangler==2.16.1 backports.zoneinfo==0.2.1; python_version < "3.9" beautifulsoup4==4.10.0 boto3~=1.21.21 botocore~=1.24.46 click click-logging coolname~=2.0.0 dask-snowflake==0.1.0 dask==2022.1.1 docker gql==2.0.0 great-expectations==0.15.27 holidays~=0.16 html5lib==1.1 numpy==1.21.6 pandas==1.2.4 pangres==2.3 pendulum==2.1.2 pre-commit==2.20.0 prometheus-client==0.11.0 protobuf==3.20.1 psycopg2-binary==2.8.6 pytest-cov<5,<=4.0.0 pytest<8,>=7.2.0 python-dateutil==2.8.1 python-dotenv==0.20.0 python-slugify==4.0.1 python-ulid==1.0.2 pytz==2021.3 requests-pkcs12==1.13 requests==2.26.0 sagemaker==2.34.0 snowflake-connector-python==2.7.7 sortedcontainers==2.4.0 sqlalchemy==1.3.24 xlrd==2.0.1 --index-url <https://pypi.org/simple/> --retries 5 --timeout 15 exited with 1 and STDERR:
ERROR: Could not find a version that satisfies the requirement awswrangler==2.16.1
ERROR: No matching distribution found for awswrangler==2.16.1
g

gentle-painting-24549

11/10/2022, 4:14 PM
👀 following. I’ve been getting the same issues on my end for specific versions of numpy/pandas
c

curved-manchester-66006

11/10/2022, 4:15 PM
:-/ If I comment out awswrangler I next get a numpy error
g

gentle-painting-24549

11/10/2022, 4:16 PM
Just curious, what type of CPU are you using? I’m on an M1 Mac so I already deal with a lot of conflicting packages with different availabilities for wheels
c

curved-television-6568

11/10/2022, 4:18 PM
this usually comes down to the python interpreter constraints and machine platform being used.
you’ll likely find more detailed error message in
/tmp/pants-sandbox-dBTWdX/.tmp/pex-pip-log.wtscry14/pip.log
c

curved-manchester-66006

11/10/2022, 4:20 PM
Just curious, what type of CPU are you using? I’m on an M1 Mac so I already deal with a lot of conflicting packages with different availabilities for wheels
Linux x86_64
you’ll likely find more detailed error message in
/tmp/pants-sandbox-dBTWdX/.tmp/pex-pip-log.wtscry14/pip.log
I hoped so too! But
Copy code
$ cat /tmp/pants-sandbox-dBTWdX/.tmp/pex-pip-log.wtscry14/pip.log
cat: /tmp/pants-sandbox-dBTWdX/.tmp/pex-pip-log.wtscry14/pip.log: No such file or directory
e

enough-analyst-54434

11/10/2022, 4:21 PM
@curved-manchester-66006 what are your interpreter constraints?: https://www.pantsbuild.org/docs/python-interpreter-compatibility
IIRC the default if unconfigured is ">=3.7,<4". When creating a lock, the lock then needs to satisfy all those versions. That is usually hard! Not many project distributions support Python 3.11 for example which is in that range.
c

curved-manchester-66006

11/10/2022, 4:26 PM
I've tried not having it set. And also these values
interpreter_constraints = ['CPython>=3.7,<3.10']
interpreter_constraints = ['CPython>=3.7,<3.9']
e

enough-analyst-54434

11/10/2022, 4:27 PM
That's the problem. Look here: https://pypi.org/project/awswrangler/2.16.1/
*Requires:* Python >=3.7.1, <3.11
That
.1
kills you.
💀 1
So Pants / Pex correctly fail the lock. WIth very unhelpful messaging to be sure.
c

curved-manchester-66006

11/10/2022, 4:29 PM
i'm not following why the
.1
is the angel of death. Isn't 3.8.13 for example still
Python >=3.7.1, <3.11
e

enough-analyst-54434

11/10/2022, 4:29 PM
A lockfile must work for any machine / Python you claim you support. You claim you support 3.7.0
You don't so just bump the lower bound!
g

gentle-painting-24549

11/10/2022, 4:32 PM
I actually ran into this on a separate, non-pants project, I believe the underlying culprit is numpy. I don’t remember if it was due to incompatibilities or just security warnings but supporting
3.7.1
and above became much easier than
>=3.7
c

curved-manchester-66006

11/10/2022, 4:33 PM
🪤
e

enough-analyst-54434

11/10/2022, 4:34 PM
So, Pants taught you something about your app - it doesn't in fact support all Python 3.7's. It did so as approxiamtely the worst teacher ever, or at least speaking a language no one but it understands.
c

curved-manchester-66006

11/10/2022, 4:35 PM
Thank you very much!
g

gentle-painting-24549

11/10/2022, 4:35 PM
Yeah this was extremely helpful for me too. I can’t wait for 3.7 to go away - especially with my M1 mac
e

enough-analyst-54434

11/10/2022, 4:36 PM
You're welcome. I'm just part time and so just implementing locking in Pex took all my time and then some over the past year. It would be good to find time to circle back and improve error messages.
@gentle-painting-24549 just wipe it and install Asahi Linux - that will work fine with Python 3.7
g

gentle-painting-24549

11/10/2022, 4:43 PM
Oh that’s super neat I’ve never seen that before. Currently I’m one of the only M1s in a sea of Intels running a 3.7 app, so things have been painful until we cut over to 3.9 in the coming months.
e

enough-analyst-54434

11/10/2022, 4:46 PM
Aha. Well if you're in a sea of macOS you're sunk, but I try. My early years were on Apple IIs & logo and then later on a IIe learning Applesoft Basic. I had love for Apple then. I now do not. Its insane I can't run their os on their hardware in a vm if I've wiped their os and installed linux on it.
I my mind there harware is amazing, but they are developer hostile with their OS / rules.
g

gentle-painting-24549

11/10/2022, 4:59 PM
Millennial developer over here 🙋 Apple is all I really know and I’ve always gotten along fine - it’s never really been a problem until now when I have a different processor than all my co-workers. I just got the M1 Max 16" and the hardware is impeccable though 😍
e

enough-analyst-54434

11/10/2022, 5:19 PM
It truly is. Just watch your back unless you develop macOS / iOS native apps!
h

happy-kitchen-89482

11/10/2022, 6:54 PM
We should probably document a strong recommendation to pin your ICs to a single Python version (which wouldn’t have helped with that
.1
issue in this case, but will at least reduce the number of times this happens)
e

enough-analyst-54434

11/10/2022, 6:56 PM
I don't actually think that's a great rec in general. I do think we should not have default ICs though and force you to choose something if you're using lockfiles.
h

happy-kitchen-89482

11/10/2022, 7:07 PM
Most orgs seem to standardize their codebases on a single Python version, so recommending pinning the ICs to, e.g.,
==3.9.*
seems sensible?
And yes, forcing the question by not having defaults at all makes sense to me
c

curved-television-6568

11/10/2022, 7:12 PM
also, a helpful error message would be much preferred imho over a recommendation that side steps the issue, at best. (nothing stops us from doing both, of course)
e

enough-analyst-54434

11/10/2022, 7:25 PM
==3.7.*
is fine but would not have helped here since
3.7.0
turned out problematic. So, in favor of a pin rec, but not to a patch version in general. That just doesn't generally apply / generally there is no need to pin a patch.
As far as the error message goes, see my above, but yeah - work needs to be done there in Pex.
Ah, I see now you acked the would not have helped here Benjy. Agreed!
c

curved-television-6568

11/10/2022, 7:34 PM
ack, I’ve been pondering on how to tackle the error message situation here myself as well, as I’ve run into this precise issue as well before, and likely will again in the future… 😛
8 Views