2 QQs: 1) does 2.11 support PyPI proxies for build...
# general
2 QQs: 1) does 2.11 support PyPI proxies for building lockfiles now? 2) When trying out 2.11, Pants was trying to get things like setuptools from pythonhosted.org, which I have never seen before (will post the exact log message later today), so had to revert back to 2.10 -- is that a new thing/expected? This happened w/o enabling lockfile creation by Pants (i.e., I didn't change anything in pants.toml except the version).
1 - yes if you use the "pex" lockfile generator (https://www.pantsbuild.org/docs/python-third-party-dependencies#pex-vs-poetry-for-lockfile-generation) 2 - that shouldn't be new, just new to your attention. If you go to PyPI and look at any project; say requests: https://pypi.org/project/requests/#files you can examine those file links and note they are all served up at pythonhosted.org and so that's where all standard Python ecosystem tools fetch packages from (Pip, Poetry, Pex, etc.). In a lockfile those final resolved artifact URLs are just made explicit.
And see here for how to set up those custom repositories: https://www.pantsbuild.org/docs/python-third-party-dependencies#custom-repositories
(1) Awe cool, pex lock files! Going to try this out again (couldn't do it in 2.10 because we don't have PyPI access) (2) Each user has a generated .pants.rc file in their homedir with just:
indexes = [https://$apitoken@$internal-index]
So that's why I was surprised Pants started attempting to download from pythonhosted when we only changed the version to 2.11 in
Attaching image of output from running a package goal using 2.11.0. This works fine on 2.10.0. It looks like the ~/.pants.rc file is not being read?
This is btw with
[python].lockfile_generator = "pex"
by chance, are you using
Don't see anything like that listed in my homedir.
Actually, there is
, but not
Interestingy, moving the setting into
induces different behavior -- this time it just builds setuptools.pex forever and ever (at 35,000 seconds now). Probably still the same urlopen issue, but perhaps down a different codepath that doesn't have finite retry/wait logic? I'll run w/ debug/pex verbosity.
I think it was just a fluke; get the same urllib errors in both cases. Filed a bug report here: https://github.com/pantsbuild/pants/issues/15445
Thanks for the issue report. The problem though is almost certainly earlier. Once a lock file is written down, it will be used. Presumably the issue is that the lock file is written down with pythonhosted.org URLs in the 1st place. Did you create the lock file or is it one shipped with Pants? If its the one shipped with Pants then you'll need to regenerate that lockfile as well as a bunch of others IIUC since all lock files shipped with Pants will always point to public repos like PyPI / pythonhosted.org.
These lock files are just JSON so you can peek inside if its a lock file you created.
@hundreds-father-404 if I've got this right is there any easy way to configure for please use all default locks but re-gen them to disk using my repos + indexes?
Sort of. You would do this:
Copy code
lockfile = "3rdparty/python/black.lock"

lockfile = "3rdparty/python/isort.lock"
And so on. That is, you'd use the default
from Pants, but your own lock. Then run
At that point, probably worth pinning versions though. That way, when you upgrade Pants versions, Pants won't complain that the lock is invalid because Pants bumped the default of a version
(note the lockfile paths can be wherever you want inside the repo)
OK, so you have to know all the subsystems you use.
@nice-florist-55958 you may just have to play whac-a-mole and do them one by one. You probably were blissfully unaware of all the subsystems you use.
👍 1
whack-a-mole, and/or look through https://www.pantsbuild.org/docs/reference-all-subsystems to see all the possible subsystems
I was able to make some progress w/ getting the new resolves working. But when trying to run the test or package a pex_binary, I get:
Copy code
22:51:20.28 [INFO] Completed: Building pytest.pex from 3rdparty/python/pytest.lock
22:51:20.28 [ERROR] 1 Exception encountered:

  ProcessExecutionFailure: Process 'Building pytest.pex from 3rdparty/python/pytest.lock' failed with exit code 1.

Traceback (most recent call last):
  File "/d/d1/user/taymarti/.cache/pants/named_caches/pex_root/unzipped_pexes/479026d892c9306a8907938e1e49f73e1514eabe/.bootstrap/pex/pex.py", line 504, in execute
    exit_value = self._wrap_coverage(self._wrap_profiling, self._execute)
  File "/d/d1/user/taymarti/.cache/pants/named_caches/pex_root/unzipped_pexes/479026d892c9306a8907938e1e49f73e1514eabe/.bootstrap/pex/pex.py", line 409, in _wrap_coverage
    return runner(*args)
  File "/d/d1/user/taymarti/.cache/pants/named_caches/pex_root/unzipped_pexes/479026d892c9306a8907938e1e49f73e1514eabe/.bootstrap/pex/pex.py", line 440, in _wrap_profiling
    return runner(*args)
  File "/d/d1/user/taymarti/.cache/pants/named_caches/pex_root/unzipped_pexes/479026d892c9306a8907938e1e49f73e1514eabe/.bootstrap/pex/pex.py", line 560, in _execute
    return self.execute_entry(self._pex_info.entry_point)
  File "/d/d1/user/taymarti/.cache/pants/named_caches/pex_root/unzipped_pexes/479026d892c9306a8907938e1e49f73e1514eabe/.bootstrap/pex/pex.py", line 696, in execute_entry
    return self.execute_pkg_resources(entry_point)
  File "/d/d1/user/taymarti/.cache/pants/named_caches/pex_root/unzipped_pexes/479026d892c9306a8907938e1e49f73e1514eabe/.bootstrap/pex/pex.py", line 728, in execute_pkg_resources
    return runner()
  File "/d/d1/user/taymarti/.cache/pants/named_caches/pex_root/installed_wheels/6a5463833452712ecf58a573b1b489c81d14aaed599e955c2af85cf793217a42/pex-2.1.84-py2.py3-none-any.whl/pex/bin/pex.py", line 766, in main
  File "/d/d1/user/taymarti/.cache/pants/named_caches/pex_root/installed_wheels/6a5463833452712ecf58a573b1b489c81d14aaed599e955c2af85cf793217a42/pex-2.1.84-py2.py3-none-any.whl/pex/bin/pex.py", line 786, in do_main
  File "/d/d1/user/taymarti/.cache/pants/named_caches/pex_root/installed_wheels/6a5463833452712ecf58a573b1b489c81d14aaed599e955c2af85cf793217a42/pex-2.1.84-py2.py3-none-any.whl/pex/bin/pex.py", line 648, in build_pex
  File "/d/d1/user/taymarti/.cache/pants/named_caches/pex_root/installed_wheels/6a5463833452712ecf58a573b1b489c81d14aaed599e955c2af85cf793217a42/pex-2.1.84-py2.py3-none-any.whl/pex/result.py", line 82, in try_
    raise ResultError(error=result)
pex.result.ResultError: There was 1 error downloading required artifacts:
1. coverage 6.3.3 from <https://msde-docker-prod.ms.com:4443/api/pypi/mspypi/bb/17/ed93a4c984bd45f282ac28e60f7488a6605a47f391a1111c92ee6028a235/coverage-6.3.3-cp37-cp37m-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl>
    HTTP Error 401:
I tried curl'ing the file and it worked just fine (I had to substitute my API credentials after the https:// but I assume Pants is just not printing this part). Furthermore, still getting the setuptools.lock error when pacakging a python_distribution, even after generating locks for all the backends being used. Here is the relevant sections of my pants.toml:
Copy code
backend_packages = [
    # "pants.backend.experimental.python.lint.autoflake",

lockfile = "3rdparty/python/pytest.lock"

lockfile = "3rdparty/python/coveragepy.lock"

lockfile = "3rdparty/python/mypy.lock"

lockfile = "3rdparty/python/isort.lock"

lockfile = "3rdparty/python/black.lock"
All of the lockfiles exist now as generated by pants. Am I missing an implied backend or?
Yes re 401s. That is fixed in Pex 2.1.86 which made it into Pants 2.13.0.dev2 yesterday and cherry-picked to 2.11.x (will be released in 2.11.1rc1) and 2.12.x (will be released in 2.12.0rc1).
Heh, never thought of setuptools as a backend like that. It works now, thank you 🙂
For the URL 401, can I just set [pex-cli].version = 2.1.86 w/ 2.11.0?
Nope. There were also changes to args passed to Pex.
OK sounds good, will await the update and try again then.