https://pantsbuild.org/ logo
Powered by
a

ambitious-student-81104

04/04/2022, 11:32 PM
Hi! I recently upgraded to 
2.11.0rc0
and switched from a custom constraints file generation script to 
./pants generate-lockfiles
(finally!!). However, for VS code users like myself, we still want to generate a constraints/requirement file, something we can run 
pip install -r
on to install those deps on our local environment. @hundreds-father-404 told me to do 
./pants export ::
then 
<venv>/bin/pip freeze
, and my 
export
step failed on a problem that has been there for a while, which is our required version of 
cryptography
no longer releases macOS distributions, but things have been working for us because our production is Linux, and our local installation didn't have a hard requirement on resolving everything (we used pip compile) like 
./pants export ::
seems to be strict about. Is there a way to work around this? We will still be using macOS for local dev for the foreseeable future so the dependency there will still exist. cc @happy-kitchen-89482
error log - you can see that we are not able to just relax the requirements because it's from 3rd party deps:
Copy code
Dependency on cryptography not satisfied, 1 incompatible candidate found:
1.) cryptography 3.4.8 does not have any compatible artifacts:
    <https://files.pythonhosted.org/packages/35/23/b7dc7a019e1a34b024498510686a6b6a400caf6ecd1c0ba102e27dd3658f/cryptography-3.4.8-pp37-pypy37_pp73-win_amd64.whl>
    requirers:
    via: SecretStorage==3.3.1 -> cryptography>=2.0
    via: apache-airflow==2.1.4 -> cryptography>=0.9.3
    via: authlib==0.15.2 -> cryptography
    via: moto==1.3.14 -> cryptography>=2.3.0
    via: paramiko==2.7.2 -> cryptography>=2.5
    via: snowflake-connector-python[secure-local-storage]==2.4.6 -> cryptography<4.0.0,>=2.5.0
    via: moto==1.3.14 -> sshpubkeys<4.0,>=3.1.0 -> cryptography>=2.1.4
    via: snowflake-connector-python[secure-local-storage]==2.4.6 -> azure-storage-blob<13.0.0,>=12.0.0 -> cryptography>=2.1.4
    via: snowflake-connector-python[secure-local-storage]==2.4.6 -> pyOpenSSL<21.0.0,>=16.2.0 -> cryptography>=3.2
h

happy-kitchen-89482

04/05/2022, 3:21 AM
If you're using PEX to generate the lockfile (
lockfile_generator = "pex"
in your pants.toml) then you can use some jq magic to extract enough information to build a venv for the IDE:
🤦‍♂️ 1
Copy code
cat path/to/lockfile.lock | grep -v "^//" | jq -r '.locked_resolves[].locked_requirements[] | "\(.project_name)==\(.version)"'
💯 1
You can manually or automatically strip 
cryptography
out of that and then run 
pip install
on it
a

ambitious-student-81104

04/06/2022, 7:45 PM
that sounds a bit messy..
h

happy-kitchen-89482

04/06/2022, 7:50 PM
How did you create this venv before without running into this problem?
Not sure I understand the cryptography issue
➕ 1
a

ambitious-student-81104

04/07/2022, 4:27 PM
we didn't create a venv, we used a custom script to use 
pip compile
to compile all the 
requirements.txt
into a 
constraints.txt
, and users can just use 
pip install -r constraints.txt
locally.
Not sure I understand the cryptography issue
it's just that some 3rd party packages we depend on have a transitive dep on a package 
cryptography
that no longer releases macOS dists.
h

hundreds-father-404

04/07/2022, 4:31 PM
What confuses me is why pants works at all on macOS. If export fails, I would expect the other goals to fail too
a

ambitious-student-81104

04/07/2022, 4:40 PM
why pants works at all on macOS
🤯
let me try 
./pants export ::
on master.
h

happy-kitchen-89482

04/07/2022, 4:45 PM
Well, presumably there is at least an sdist for cryptography? So that should build on macos?
a

ambitious-student-81104

04/07/2022, 5:02 PM
so 
./pants export ::
on master actually works. Master differs from my feature branch upgrading to pants 2.11 and using pex to resolve in a few ways: 1. master uses 2.10, my feature branch is 2.11.0rc0 2. master is still using a custom 
pip compile
script to generate the 
constraints.txt
file, my feature branch is using 
pex
and 
./pants generate-lockfiles
instead, which works, but then 
./pants export ::
breaks at the 
cryptography
error
@happy-kitchen-89482 I'll send you the diff in 
pants.toml
on cb slack
h

hundreds-father-404

04/07/2022, 5:05 PM
Horus, generating the lock works. But does using the lock actually work? Try running the test goal, with the option ––python-run-against-entire-lockfile to make sure that cryptography is installed. Also, you may want to use RC one, rather than RC0 Note that with pex, generating the lock does not actually install anything for you.
h

happy-kitchen-89482

04/07/2022, 5:07 PM
OK, so to be sure I understand, 
./pants export ::
works on 2.10 but does not work on 2.11?
Hmm looking here it seems like cryptography does have both an sdist and a macos wheel
Can you share (privately if necessary) the relevant requirements.txts?
I'd love to reproduce this
a

ambitious-student-81104

04/07/2022, 5:15 PM
yes happy to
i think there is a discrepancy between how the lockfiles are generated VS how mac actually just pip installs the dep
for example, if I just install my direct dep, it grabs me this version of 
cryptography
, 
cryptography-36.0.2-cp36-abi3-macosx_10_10_x86_64.whl
Copy code
❯ pip install -U --no-cache-dir "pyOpenSSL<21.0.0,>=16.2.0"
Requirement already satisfied: pyOpenSSL<21.0.0,>=16.2.0 in /Users/horus.wu@coinbase.com/.pyenv/versions/3.7.10/lib/python3.7/site-packages (20.0.1)
Requirement already satisfied: six>=1.5.2 in /Users/horus.wu@coinbase.com/.pyenv/versions/3.7.10/lib/python3.7/site-packages (from pyOpenSSL<21.0.0,>=16.2.0) (1.16.0)
Collecting cryptography>=3.2
  Downloading cryptography-36.0.2-cp36-abi3-macosx_10_10_x86_64.whl (2.5 MB)
     |████████████████████████████████| 2.5 MB 3.4 MB/s
Requirement already satisfied: cffi>=1.12 in /Users/horus.wu@coinbase.com/.pyenv/versions/3.7.10/lib/python3.7/site-packages (from cryptography>=3.2->pyOpenSSL<21.0.0,>=16.2.0) (1.15.0)
Requirement already satisfied: pycparser in /Users/horus.wu@coinbase.com/.pyenv/versions/3.7.10/lib/python3.7/site-packages (from cffi>=1.12->cryptography>=3.2->pyOpenSSL<21.0.0,>=16.2.0) (2.21)
Installing collected packages: cryptography
Successfully installed cryptography-36.0.2
WARNING: You are using pip version 21.2.2; however, version 22.0.4 is available.
You should consider upgrading via the '/Users/horus.wu@coinbase.com/.pyenv/versions/3.7.10/bin/python -m pip install --upgrade pip' command.
However, the version of 
cryptography
that's generated by 
./pants generate-lockfiles
is 
3.4.8
for some reason
Copy code
Dependency on cryptography not satisfied, 1 incompatible candidate found:
1.) cryptography 3.4.8 does not have any compatible artifacts:
    <https://files.pythonhosted.org/packages/35/23/b7dc7a019e1a34b024498510686a6b6a400caf6ecd1c0ba102e27dd3658f/cryptography-3.4.8-pp37-pypy37_pp73-win_amd64.whl>
    requirers:
    via: SecretStorage==3.3.1 -> cryptography>=2.0
    via: apache-airflow==2.1.4 -> cryptography>=0.9.3
    via: authlib==0.15.2 -> cryptography
    via: moto==1.3.14 -> cryptography>=2.3.0
    via: paramiko==2.7.2 -> cryptography>=2.5
    via: snowflake-connector-python[secure-local-storage]==2.4.6 -> cryptography<4.0.0,>=2.5.0
    via: moto==1.3.14 -> sshpubkeys<4.0,>=3.1.0 -> cryptography>=2.1.4
    via: snowflake-connector-python[secure-local-storage]==2.4.6 -> azure-storage-blob<13.0.0,>=12.0.0 -> cryptography>=2.1.4
    via: snowflake-connector-python[secure-local-storage]==2.4.6 -> pyOpenSSL<21.0.0,>=16.2.0 -> cryptography>=3.2
i think you just need the 
via: ...
s above to repro it?
oh why is it a 
win_amd64.whl
in the pants-generated lockfile?
h

happy-kitchen-89482

04/07/2022, 5:32 PM
The lockfile should be universal, so it should contain all the wheels for all platforms?
➕ 1
Good news so far is that I can reproduce this
The reason it takes 
cryptography
at 
3.4.8
is this line in the "via" above: 
snowflake-connector-python[secure-local-storage]==2.4.6 -> cryptography<4.0.0,>=2.5.0
When you say "if I just install my direct dep, it grabs me this [36.0.2] version of cryptography", what exactly is the direct dep you're installing? Presumably it does not include that 
snowflake-connector-python[secure-local-storage]==2.4.6
requirement, since that is forcing 
cryptography<4.0.0
So 3.4.8 seems like a good choice, but as you surmised, the lockfile only has the windows wheel for some reason, and no sdist
So that seems to be the real problem
Looks like you've found a bug! Tracking this here: https://github.com/pantsbuild/pex/issues/1711
a

ambitious-student-81104

04/07/2022, 9:57 PM
Ohhh i didn't see the 
cryptography<4.0.0
!
Yep - 
pip install -U --no-cache-dir "snowflake-connector-python[secure-local-storage]==2.4.6"
gives me
Copy code
❯ pip show cryptography
Name: cryptography
Version: 3.4.8
Summary: cryptography is a package which provides cryptographic recipes and primitives to Python developers.
Home-page: <https://github.com/pyca/cryptography>
Author: The Python Cryptographic Authority and individual contributors
Author-email: <mailto:cryptography-dev@python.org|cryptography-dev@python.org>
License: BSD or Apache License, Version 2.0
Location: /Users/horus.wu@coinbase.com/.pyenv/versions/3.7.10/lib/python3.7/site-packages
Requires: cffi
Required-by: snowflake-connector-python, pyOpenSSL, azure-storage-blob
h

happy-kitchen-89482

04/08/2022, 1:12 AM
Well, you find a real and fairly serious bug! Which John already has a fix for. So thanks for that!
❤️ 2
❗ 1
Powered by