Hi, I have a question on using customized artifactory. My understanding is that, if I want to use different version of black and use different artifactory, I can do the following

[black]
version = 'black==19.10b0'
lockfile = 'black_lockfile.txt'
[python-repos]
indexes = "['<http://xyz.com/artifactory/api/pypi/pypi-abc/simple>', '<http://xyz.fg.rbc.com/artifactory/api/pypi/pypi/simple>']"

However, i see the following error which seems it does not go to customized artifactory.

Process 'Generate lockfile for black' failed with exit code 1.

HTTPSConnectionPool(host='<http://pypi.org|pypi.org>', port=443): Max retries exceeded with url: /pypi/19.10b0/json

The new lockfiles does not yet work with custom python-repos.

Indeed, the

generate-lockfiles

goal, unfortunately, does not yet work with

python-repos

. You can manually generate a lockfile though, as described at https://www.pantsbuild.org/v2.10/docs/python-third-party-dependencies#generate-lockfiles-goal-vs-manual-lockfile-generation And in Pants 2.11, we add the option to generate lockfiles by using Pex, rather than Poetry under-the-hood. That does work with

[python-repos]

. We'll hopefully do 2.11.0rc0 this week, which will add that option

wow. cool! Nice to know it is in the plan.

Indeed! It's been over year-long project to get this lockfiles redesign, and we're so close now 😭 fyi another benefit of the upcomign Pex lockfile support is it should be generally faster when both generating the lock & then installing the lock. See https://github.com/pantsbuild/pants/pull/14771

Btw, I have some other confusion on the constraints vs lockfiles since they are both mentioned in the Third-party dependencies section. Do they basically mean the same thing? and also i am wondering the (historical) reasons we choose petry instead of pex(pip) for

generate-lockfiles

.

It's really confusing that in Pants 2.9 and earlier we called constraints file a "lockfile". It's a poor man's version of a lockfile; it does help you to lock things, but it a) does not support

--hash

to make sure the artifacts you download are what you expect (supply chain attacks) b) does not enforce that every transitive dependency is pinned; you can leave stuff off The lockfiles in 2.10 fix those issues and are what lockfiles are actually meant to be. --

and also i am wondering the (historical) reasons we choose petry instead of pex(pip) for

Pip does not support generating lockfiles; the best it has is calling

pip freeze

, which is actually what that

generate_constraints.sh

script is doing. But that's not a true lockfile, e.g. no

--hash

. Also that constraints file might not work on other platforms, like a file generated on macOS might not work on Linux So, we tried using the popular tool

pip-compile

, which generates lockfiles using pip. It worked okay, but also had the same issue of not supporting multiple platforms. In practice, we found this caused lots of issues Which led us to evaluating pdm and Poetry as alternative lockfile generators. Poetry was the most mature for Pants's use case, but does have several limitations like the

[python-repos]

stuff. We decided to go with Poetry because it can handle most use cases, but to at the same time start a project to teach Pex how to generate lockfiles via pip. Which is where we are today, after some excellent work from John Sirois, Pex can now generate lockfiles! The Python ecosystem is huge and complex, so we're "dogfooding" it and trying to handle some new edge cases, hopefully so that Pants 2.11 can start using Pex for lockfile generation by default And then the final part of this long story is supporting VCS (git) requirements, which you can't normally lock because it's not a stable thing; the branch might change for example. https://github.com/pantsbuild/pex/issues/1556

I want to give your every emoji i can use for this detail and insightful info. I am trying to use

pip-compile

to generate the lockfiles. But i got the following error, anything I am missing?

InvalidLockfileError: Could not find a Pants metadata block in the lockfile `black_constraints.txt` for `black`. To resolve this error, you will need to regenerate the lockfile by running `./pants generate-lockfiles --resolve={tool_name}`.

Oh hm we should make that error message mention "or set

[python].invalid_lockfile_behavior = 'ignore'

, which you should do if you're manually generating lockfiles