https://pantsbuild.org/ logo
#general
Title
# general
c

curved-manchester-66006

03/22/2023, 2:56 PM
I'm getting
Failed to lock ArtifactURL
when generating a lock file, but I can't tell if it is a pex bug or some quicky PyPi problem. It happens with a fresh
~/.cache
and with pip version 22.3 & 23.0.1
Copy code
# pex3 --version
2.1.129
# pex3 lock create -r requirements.txt  -o foo.lock --interpreter-constraint "CPython==3.10.*" --resolver-version pip-2020-resolver --indent 4 --pip-version 22.3 --target-system linux --target-system mac  --style universal
Failed to lock ArtifactURL(raw_url='<https://files.pythonhosted.org/packages/69/2b/f0ee898c3270d965300ec30b0bf06e062c4cc92f35d17ae6046f429c5067/SQLAlchemy-1.4.25.tar.gz>', normalized_url='<https://files.pythonhosted.org/packages/69/2b/f0ee898c3270d965300ec30b0bf06e062c4cc92f35d17ae6046f429c5067/SQLAlchemy-1.4.25.tar.gz>', scheme='https', path='/packages/69/2b/f0ee898c3270d965300ec30b0bf06e062c4cc92f35d17ae6046f429c5067/SQLAlchemy-1.4.25.tar.gz'). Could not obtain its content for analysis.
# cat requirements.txt 
acryl-datahub[datahub-rest,feast,postgres,snowflake]==0.10.0.2
click>=8.0.0,<9
# TODO: Clarify targets per extra?
feast[aws,snowflake]==0.29.0
snowflake-connector-python[pandas]==2.7.7
typeguard<3
click>=8.0.0,<9
pandas>=1.3.4,<2
pytest>=7.2.0,<8
python-dateutil==2.8.2
pyyaml>=5.1
sortedcontainers==2.4.0
pytz==2021.3
sqlalchemy==1.3.24
๐Ÿงต 1
โž• 1
Caveman debugging:
Copy code
if not os.path.isfile(source_archive_path):
                        print(artifact_url.path)
                        print(selected_path)
                        print(self._download_dir)
                        print(source_archive_path)
                        print(os.listdir(self._download_dir))

                        raise AnalyzeError(
Results in:
Copy code
/packages/69/2b/f0ee898c3270d965300ec30b0bf06e062c4cc92f35d17ae6046f429c5067/SQLAlchemy-1.4.25.tar.gz
SQLAlchemy-1.4.25.tar.gz
/tmp/tmpgckall6h/usr.local.bin.python3.10
/tmp/tmpgckall6h/usr.local.bin.python3.10/SQLAlchemy-1.4.25.tar.gz
['websockets-10.4-cp310-cp310-musllinux_1_1_x86_64.whl', 'pycountry-22.3.5.tar.gz', ...very long list without SQLAlchemy-1.4.25.tar.gz
e

enough-analyst-54434

03/22/2023, 3:22 PM
@curved-manchester-66006 can you add
--preserve-pip-download-log
to the command line and then provide that full log file post-failure? You may just have a look and hopefully it reveals some info that is useful. I'll have to cut out for the day in ~10 minutes or so.
c

curved-manchester-66006

03/22/2023, 3:32 PM
e

enough-analyst-54434

03/22/2023, 3:42 PM
Ah, yeah - it is "obvious". N.B. Pip does not download it since it finds it in its own cache:
Copy code
2023-03-22T11:26:04,433   Found link <https://files.pythonhosted.org/packages/69/2b/f0ee898c3270d965300ec30b0bf06e062c4cc92f35d17ae6046f429c5067/SQLAlchemy-1.4.25.tar.gz> (from <https://pypi.org/simple/sqlalchemy/>) (requires-python:!=3.0.*,!=3.1.*,!=3.2.*,!=3.3.*,!=3.4.*,!=3.5.*,>=2.7), version: 1.4.25
2023-03-22T11:27:01,913   Looking up "<https://files.pythonhosted.org/packages/69/2b/f0ee898c3270d965300ec30b0bf06e062c4cc92f35d17ae6046f429c5067/SQLAlchemy-1.4.25.tar.gz>" in the cache
2023-03-22T11:27:01,919   Using cached SQLAlchemy-1.4.25.tar.gz (7.8 MB)
2023-03-22T11:27:02,518   Added SQLAlchemy[mypy]<2,>1 from <https://files.pythonhosted.org/packages/69/2b/f0ee898c3270d965300ec30b0bf06e062c4cc92f35d17ae6046f429c5067/SQLAlchemy-1.4.25.tar.gz> (from feast[aws,snowflake]==0.29.0->-r requirements.txt (line 4)) to build tracker '/home/ecsb/.pex/pip_cache/.tmp/pip-build-tracker-p_mxbiz7'
2023-03-22T11:27:02,977   Source in /home/ecsb/.pex/pip_cache/.tmp/pip-download-un_p8psm/sqlalchemy_9d7554da772d48898b7c6177326a3447 has version 1.4.25, which satisfies requirement SQLAlchemy[mypy]<2,>1 from <https://files.pythonhosted.org/packages/69/2b/f0ee898c3270d965300ec30b0bf06e062c4cc92f35d17ae6046f429c5067/SQLAlchemy-1.4.25.tar.gz> (from feast[aws,snowflake]==0.29.0->-r requirements.txt (line 4))
2023-03-22T11:27:02,977   Removed SQLAlchemy[mypy]<2,>1 from <https://files.pythonhosted.org/packages/69/2b/f0ee898c3270d965300ec30b0bf06e062c4cc92f35d17ae6046f429c5067/SQLAlchemy-1.4.25.tar.gz> (from feast[aws,snowflake]==0.29.0->-r requirements.txt (line 4)) from build tracker '/home/ecsb/.pex/pip_cache/.tmp/pip-build-tracker-p_mxbiz7'
Thanks for capturing that. This is a Pex bug. I can take a look tonight.
c

curved-manchester-66006

03/22/2023, 3:47 PM
Thanks! I was starring at
Copy code
2023-03-22T11:38:14,946   Source in /home/ecsb/.pex/pip_cache/.tmp/pip-download-0q351ksf/sqlalchemy_0789265de0c14a379112abe47ee96387 has version 1.4.25, which satisfies requirement SQLAlchemy[mypy]<2,>1 from <https://files.pythonhosted.org/packages/69/2b/f0ee898c3270d965300ec30b0bf06e062c4cc92f35d17ae6046f429c5067/SQLAlchemy-1.4.25.tar.gz#sha256=1adf3d25e2e33afbcd48cfad8076f9378793be43e7fec3e4334306cac6bec138> (from feast[aws,snowflake]==0.29.0->-r requirements.txt (line 4))
and was suspicious about the
mypy
extra With this
Copy code
# diff -u requirements.txt requirements-my.txt
--- requirements.txt	2023-03-22 14:55:53.191819806 +0000
+++ requirements-my.txt	2023-03-22 15:39:37.055615463 +0000
@@ -10,4 +10,4 @@
 pyyaml>=5.1
 sortedcontainers==2.4.0
 pytz==2021.3
-sqlalchemy==1.3.24
+sqlalchemy[mypy]==1.3.24
pex3 lock create
completes, but I'm not sure if that is incidental, another bug, or pex correctly yet obliquely telling me I had a real problem.
e

enough-analyst-54434

03/23/2023, 3:11 AM
Ok, thanks for that extra info @curved-manchester-66006 - that leads to an easy repro using:
Copy code
$ pex3 lock create --interpreter-constraint "==3.10.*" --style universal --resolver-version pip-2020-resolver --target-system linux --target-system mac "feast[aws,snowflake]==0.29.0" "sqlalchemy==1.3.24" --preserve-pip-download-log --indent 2 -o lock.json
pex: Preserving `pip download` log at /tmp/pex-pip-log.es42ebc1/pip.log
Failed to lock ArtifactURL(raw_url='<https://files.pythonhosted.org/packages/69/2b/f0ee898c3270d965300ec30b0bf06e062c4cc92f35d17ae6046f429c5067/SQLAlchemy-1.4.25.tar.gz#sha256=1adf3d25e2e33afbcd48cfad8076f9378793be43e7fec3e4334306cac6bec138>', normalized_url='<https://files.pythonhosted.org/packages/69/2b/f0ee898c3270d965300ec30b0bf06e062c4cc92f35d17ae6046f429c5067/SQLAlchemy-1.4.25.tar.gz>', scheme='https', path='/packages/69/2b/f0ee898c3270d965300ec30b0bf06e062c4cc92f35d17ae6046f429c5067/SQLAlchemy-1.4.25.tar.gz'). Could not obtain its content for analysis.
I've filed https://github.com/pantsbuild/pex/issues/2098 and will be working on a fix for that. I should have a Pex release with the fix tomorrow that you can try out. I'll report back here with the
pants.toml
edits you need to try it out.
๐Ÿ™ 1
Ok, Pex 2.1.130 is released now. You can try this out with the following
pants.toml
edit:
Copy code
[pex-cli]
version = "v2.1.130"
known_versions = [
    "v2.1.130|macos_arm64|0ffb9fe8146945031d596f4559bd1803d5d9f70fe318adb385ed8c4a44cb7dec|4082176",
    "v2.1.130|macos_x86_64|0ffb9fe8146945031d596f4559bd1803d5d9f70fe318adb385ed8c4a44cb7dec|4082176",
    "v2.1.130|linux_x86_64|0ffb9fe8146945031d596f4559bd1803d5d9f70fe318adb385ed8c4a44cb7dec|4082176",
    "v2.1.130|linux_arm64|0ffb9fe8146945031d596f4559bd1803d5d9f70fe318adb385ed8c4a44cb7dec|4082176"
]
c

curved-manchester-66006

03/27/2023, 7:43 PM
Confirmed this fixes my problem. Thanks!
2 Views