Hi does anybody know how to tell `pants` to download a parti Pants #general

Hi, does anybody know how to tell `pants` to downl...

adamant-energy-38096

10/01/2024, 1:58 PM

Hi, does anybody know how to tell

pants

to download a particular dependency from a specific index-url? More context in the 🧵

👀 1

adamant-energy-38096

10/01/2024, 1:58 PM

I have

onnxruntime-gpu==1.18.1

in my

requirements.txt file

, among many other packages, and I need it to be installed with

--index-url <https://aiinfra.pkgs.visualstudio.com/PublicPackages/_packaging/onnxruntime-cuda-12/pypi/simple/>

But the same package is in

<https://pypi.org/simple/>

too, so when I generate the lock file I get artifacts from both places in the lock file, like this:

Copy code

"artifacts": [
  {
    "algorithm": "sha256",
    "hash": "87d1884523f485f40d3a126677a9f93fc15569dd817fc753ee7fc519142a4425",
    "url": "<https://files.pythonhosted.org/packages/a3/81/c364ca9c0b2260a38c7bc026b8a74e6bb8caa9e4fcc9f20438558b942b0f/onnxruntime_gpu-1.18.1-cp312-cp312-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl>"
  },
  {
    "algorithm": "sha256",
    "hash": "4e9a52f5d43a84fe29e135da6bf10daa18836c81bed9060a5924efd6afc0d259",
    "url": "<https://aiinfra.pkgs.visualstudio.com/2692857e-05ef-43b4-ba9c-ccf1c22c437c/_packaging/9387c3aa-d9ad-4513-968c-383f6f7f53b8/pypi/download/onnxruntime-gpu/1.18.1/onnxruntime_gpu-1.18.1-cp310-cp310-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl>"
  },
  {
    "algorithm": "sha256",
    "hash": "1334f802cb1e4e2eb6ceebc4ef71ba44f3ef444d34216baafb940368a7a5d2f5",
    "url": "<https://aiinfra.pkgs.visualstudio.com/2692857e-05ef-43b4-ba9c-ccf1c22c437c/_packaging/9387c3aa-d9ad-4513-968c-383f6f7f53b8/pypi/download/onnxruntime-gpu/1.18.1/onnxruntime_gpu-1.18.1-cp311-cp311-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl>"
  },
  {
    "algorithm": "sha256",
    "hash": "bbb1a6c986b2392eebaebc43e198a1614e3f7d2c191725002dbfa0dceb24454b",
    "url": "<https://aiinfra.pkgs.visualstudio.com/2692857e-05ef-43b4-ba9c-ccf1c22c437c/_packaging/9387c3aa-d9ad-4513-968c-383f6f7f53b8/pypi/download/onnxruntime-gpu/1.18.1/onnxruntime_gpu-1.18.1-cp312-cp312-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl>"
  },
  {
    "algorithm": "sha256",
    "hash": "e9de7b8c7c975f7830d1e9323daca2090408df821f2adc10ea267b4f469b59e0",
    "url": "<https://files.pythonhosted.org/packages/4d/52/fa9ba5a69cfed82ba1765882f50664b05c26982f55fec6edd1bda700195f/onnxruntime_gpu-1.18.1-cp310-cp310-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl>"
  },
  {
    "algorithm": "sha256",
    "hash": "af2d3ee6fba72b57abf6f379b8aca30ee773959d4346271e7d92557dd5cf2901",
    "url": "<https://files.pythonhosted.org/packages/92/2e/5c6a3c94a8e4794a0ad1ba00f5b86c89673f271f43c77f537a90db4472c0/onnxruntime_gpu-1.18.1-cp311-cp311-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl>"
  }
],

The artifacts from pypi are not good and make my application fail. If I remove them from the lock file manually, everything works. How can I tell

pants

to just get the artifacts from

<https://aiinfra.pkgs.visualstudio.com>

brainy-parrot-55558

10/01/2024, 2:10 PM

The following solution works without pants, I don't know if pants will respect this when generating the lock files. option 1: At the start of the requirements.txt:

Copy code

--index-url <https://aiinfra.pkgs.visualstudio.com>
--extra-index-url <https://pypi.python.org/simple>

so your local repo will take precedence over the default one, for all packages. option 2: or you could specify for one package specifically, requirements.txt:

Copy code

pandas==1.5.1
your-package==* -i <https://aiinfra.pkgs.visualstudio.com>
next-package==2.0.0

maybe stating the obvious, but worth a try take a look at https://www.pantsbuild.org/stable/docs/python/overview/third-party-dependencies#custom-repositories

adamant-energy-38096

10/01/2024, 2:28 PM

Tried those before, but it's always good to double-check! Setting the index-url inline with the package in the requirements file makes no difference to the generated lock file. The issue is that as long as pypi is in the repos list (needed for other packages), the lock file knows about both wheels, and then the wrong one gets used I think I need to either: • Tell pants to look only in one index url for this specific package, or • Do something so that the correct wheel from the lock file is selected

brief-scientist-13682

10/01/2024, 7:40 PM

@brainy-parrot-55558 of your recs, only 1 could possibly work since

-i

is a global requirements.txt option: https://pip.pypa.io/en/stable/reference/requirements-file-format/#global-options It does not apply just to the individual line. And 1 is problematic since Pip has changed how it handles ordering over time; so the result depends critically on what value you configure for

--pip-version X

(not sure of Pants spelling for this option). @adamant-energy-38096 this comes down to a Pex question - Pants has no control here at all. Currently you cannot pin a package to an index using Pex. Pex just forwards to Pip and Pip doesn't support this. In the case of lock files, however, Pex does post-process Pip results and so I think this sort of feature could be implemented. If you need this feature, please file a feature request over in Pex: https://github.com/pex-tool/pex/issues/new I will say, though: Microsoft publishing the same project version artifact in two places and having the contents be different is horrific behavior. I can find no charitable or reasonable explanation:

Copy code

$ ls -1s pypi/ aiinfra.pkgs.visualstudio.com/
aiinfra.pkgs.visualstudio.com/:
total 350704
196760 onnxruntime_gpu-1.18.1-cp312-cp312-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl
153944 onnxruntime_gpu-1.18.1-cp312-cp312-win_amd64.whl

pypi/:
total 350076
196096 onnxruntime_gpu-1.18.1-cp312-cp312-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl
153980 onnxruntime_gpu-1.18.1-cp312-cp312-win_amd64.whl

$ sha256sum pypi/*.whl aiinfra.pkgs.visualstudio.com/*.whl
87d1884523f485f40d3a126677a9f93fc15569dd817fc753ee7fc519142a4425  pypi/onnxruntime_gpu-1.18.1-cp312-cp312-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl
3f2ab38a62350965f5007111728410b3ef25213104dd1e7d61ecc158002ea3f5  pypi/onnxruntime_gpu-1.18.1-cp312-cp312-win_amd64.whl
bbb1a6c986b2392eebaebc43e198a1614e3f7d2c191725002dbfa0dceb24454b  aiinfra.pkgs.visualstudio.com/onnxruntime_gpu-1.18.1-cp312-cp312-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl
bee352929e6eec2ff4e11e323a025ed8bd5eac24795005bc502ac740971fa7bd  aiinfra.pkgs.visualstudio.com/onnxruntime_gpu-1.18.1-cp312-cp312-win_amd64.whl

❤️ 2

adamant-energy-38096

10/02/2024, 2:57 PM

Thanks @brief-scientist-13682, that was super helpful! I found that using the wheel in the

requirements.txt

file as below works:

Copy code

onnxruntime-gpu @ <https://aiinfra.pkgs.visualstudio.com/2692857e-05ef-43b4-ba9c-ccf1c22c437c/_packaging/9387c3aa-d9ad-4513-968c-383f6f7f53b8/pypi/download/onnxruntime-gpu/1.18.1/onnxruntime_gpu-1.18.1-cp310-cp310-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl>

Not ideal, but could be useful in some use cases.

🙌 1

43 Views

Open in Slack

Previous Next