2 QQs: 1) does 2.11 support PyPI proxies for building lockfiles now? 2) When trying out 2.11, Pants was trying to get things like setuptools from pythonhosted.org, which I have never seen before (will post the exact log message later today), so had to revert back to 2.10 -- is that a new thing/expected? This happened w/o enabling lockfile creation by Pants (i.e., I didn't change anything in pants.toml except the version).

1 - yes if you use the "pex" lockfile generator (https://www.pantsbuild.org/docs/python-third-party-dependencies#pex-vs-poetry-for-lockfile-generation) 2 - that shouldn't be new, just new to your attention. If you go to PyPI and look at any project; say requests: https://pypi.org/project/requests/#files you can examine those file links and note they are all served up at pythonhosted.org and so that's where all standard Python ecosystem tools fetch packages from (Pip, Poetry, Pex, etc.). In a lockfile those final resolved artifact URLs are just made explicit.

And see here for how to set up those custom repositories: https://www.pantsbuild.org/docs/python-third-party-dependencies#custom-repositories

(1) Awe cool, pex lock files! Going to try this out again (couldn't do it in 2.10 because we don't have PyPI access) (2) Each user has a generated .pants.rc file in their homedir with just:

[python-repos]

indexes = [https://$apitoken@$internal-index]

So that's why I was surprised Pants started attempting to download from pythonhosted when we only changed the version to 2.11 in

pants.toml

by chance, are you using

~/.netrc

files?

Don't see anything like that listed in my homedir.

Thanks for the issue report. The problem though is almost certainly earlier. Once a lock file is written down, it will be used. Presumably the issue is that the lock file is written down with pythonhosted.org URLs in the 1st place. Did you create the lock file or is it one shipped with Pants? If its the one shipped with Pants then you'll need to regenerate that lockfile as well as a bunch of others IIUC since all lock files shipped with Pants will always point to public repos like PyPI / pythonhosted.org.

Sort of. You would do this:

[black]
lockfile = "3rdparty/python/black.lock"

[isort]
lockfile = "3rdparty/python/isort.lock"

And so on. That is, you'd use the default

version

from Pants, but your own lock. Then run

generate-lockfiles

At that point, probably worth pinning versions though. That way, when you upgrade Pants versions, Pants won't complain that the lock is invalid because Pants bumped the default of a version

OK, so you have to know all the subsystems you use.

whack-a-mole, and/or look through https://www.pantsbuild.org/docs/reference-all-subsystems to see all the possible subsystems

I was able to make some progress w/ getting the new resolves working. But when trying to run the test or package a pex_binary, I get:

22:51:20.28 [INFO] Completed: Building pytest.pex from 3rdparty/python/pytest.lock
22:51:20.28 [ERROR] 1 Exception encountered:

  ProcessExecutionFailure: Process 'Building pytest.pex from 3rdparty/python/pytest.lock' failed with exit code 1.
stdout:

stderr:
Traceback (most recent call last):
  File "/d/d1/user/taymarti/.cache/pants/named_caches/pex_root/unzipped_pexes/479026d892c9306a8907938e1e49f73e1514eabe/.bootstrap/pex/pex.py", line 504, in execute
    exit_value = self._wrap_coverage(self._wrap_profiling, self._execute)
  File "/d/d1/user/taymarti/.cache/pants/named_caches/pex_root/unzipped_pexes/479026d892c9306a8907938e1e49f73e1514eabe/.bootstrap/pex/pex.py", line 409, in _wrap_coverage
    return runner(*args)
  File "/d/d1/user/taymarti/.cache/pants/named_caches/pex_root/unzipped_pexes/479026d892c9306a8907938e1e49f73e1514eabe/.bootstrap/pex/pex.py", line 440, in _wrap_profiling
    return runner(*args)
  File "/d/d1/user/taymarti/.cache/pants/named_caches/pex_root/unzipped_pexes/479026d892c9306a8907938e1e49f73e1514eabe/.bootstrap/pex/pex.py", line 560, in _execute
    return self.execute_entry(self._pex_info.entry_point)
  File "/d/d1/user/taymarti/.cache/pants/named_caches/pex_root/unzipped_pexes/479026d892c9306a8907938e1e49f73e1514eabe/.bootstrap/pex/pex.py", line 696, in execute_entry
    return self.execute_pkg_resources(entry_point)
  File "/d/d1/user/taymarti/.cache/pants/named_caches/pex_root/unzipped_pexes/479026d892c9306a8907938e1e49f73e1514eabe/.bootstrap/pex/pex.py", line 728, in execute_pkg_resources
    return runner()
  File "/d/d1/user/taymarti/.cache/pants/named_caches/pex_root/installed_wheels/6a5463833452712ecf58a573b1b489c81d14aaed599e955c2af85cf793217a42/pex-2.1.84-py2.py3-none-any.whl/pex/bin/pex.py", line 766, in main
    env=env,
  File "/d/d1/user/taymarti/.cache/pants/named_caches/pex_root/installed_wheels/6a5463833452712ecf58a573b1b489c81d14aaed599e955c2af85cf793217a42/pex-2.1.84-py2.py3-none-any.whl/pex/bin/pex.py", line 786, in do_main
    cache=ENV.PEX_ROOT,
  File "/d/d1/user/taymarti/.cache/pants/named_caches/pex_root/installed_wheels/6a5463833452712ecf58a573b1b489c81d14aaed599e955c2af85cf793217a42/pex-2.1.84-py2.py3-none-any.whl/pex/bin/pex.py", line 648, in build_pex
    max_parallel_jobs=pip_configuration.max_jobs,
  File "/d/d1/user/taymarti/.cache/pants/named_caches/pex_root/installed_wheels/6a5463833452712ecf58a573b1b489c81d14aaed599e955c2af85cf793217a42/pex-2.1.84-py2.py3-none-any.whl/pex/result.py", line 82, in try_
    raise ResultError(error=result)
pex.result.ResultError: There was 1 error downloading required artifacts:
1. coverage 6.3.3 from <https://msde-docker-prod.ms.com:4443/api/pypi/mspypi/bb/17/ed93a4c984bd45f282ac28e60f7488a6605a47f391a1111c92ee6028a235/coverage-6.3.3-cp37-cp37m-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl>
    HTTP Error 401:

I tried curl'ing the file and it worked just fine (I had to substitute my API credentials after the https:// but I assume Pants is just not printing this part). Furthermore, still getting the setuptools.lock error when pacakging a python_distribution, even after generating locks for all the backends being used. Here is the relevant sections of my pants.toml:

backend_packages = [
    "pants.backend.python",
    # "pants.backend.experimental.python.lint.autoflake",
    "pants.backend.python.lint.isort",
    "pants.backend.python.lint.black",
    "pants.backend.python.typecheck.mypy",
    "pants.backend.shell",
    "pants.backend.shell.lint.shfmt",
    "pants.backend.experimental.go",
]


[pytest]
lockfile = "3rdparty/python/pytest.lock"


[coverage-py]
lockfile = "3rdparty/python/coveragepy.lock"


[mypy]
lockfile = "3rdparty/python/mypy.lock"


[isort]
lockfile = "3rdparty/python/isort.lock"


[black]
lockfile = "3rdparty/python/black.lock"

All of the lockfiles exist now as generated by pants. Am I missing an implied backend or?

Yes re 401s. That is fixed in Pex 2.1.86 which made it into Pants 2.13.0.dev2 yesterday and cherry-picked to 2.11.x (will be released in 2.11.1rc1) and 2.12.x (will be released in 2.12.0rc1).

Heh, never thought of setuptools as a backend like that. It works now, thank you 🙂

Nope. There were also changes to args passed to Pex.

OK sounds good, will await the update and try again then.