https://pantsbuild.org/ logo
#general
Title
# general
w

wide-zoo-86070

03/16/2022, 1:59 PM
Hi, I have a question on using customized artifactory. My understanding is that, if I want to use different version of black and use different artifactory, I can do the following
Copy code
[black]
version = 'black==19.10b0'
lockfile = 'black_lockfile.txt'
[python-repos]
indexes = "['<http://xyz.com/artifactory/api/pypi/pypi-abc/simple>', '<http://xyz.fg.rbc.com/artifactory/api/pypi/pypi/simple>']"
However, i see the following error which seems it does not go to customized artifactory.
Copy code
Process 'Generate lockfile for black' failed with exit code 1.

HTTPSConnectionPool(host='<http://pypi.org|pypi.org>', port=443): Max retries exceeded with url: /pypi/19.10b0/json
c

curved-television-6568

03/16/2022, 2:06 PM
The new lockfiles does not yet work with custom python-repos.
If you’re using that… maybe I spoke too soon. It’s the new resolves feature that have a few limitations still.
I think @hundreds-father-404 will be able to give a better response when they wake up..
🙏 1
h

hundreds-father-404

03/16/2022, 4:00 PM
Indeed, the
generate-lockfiles
goal, unfortunately, does not yet work with
python-repos
. You can manually generate a lockfile though, as described at https://www.pantsbuild.org/v2.10/docs/python-third-party-dependencies#generate-lockfiles-goal-vs-manual-lockfile-generation And in Pants 2.11, we add the option to generate lockfiles by using Pex, rather than Poetry under-the-hood. That does work with
[python-repos]
. We'll hopefully do 2.11.0rc0 this week, which will add that option
w

wide-zoo-86070

03/16/2022, 4:01 PM
wow. cool! Nice to know it is in the plan.
h

hundreds-father-404

03/16/2022, 4:02 PM
Indeed! It's been over year-long project to get this lockfiles redesign, and we're so close now 😭 fyi another benefit of the upcomign Pex lockfile support is it should be generally faster when both generating the lock & then installing the lock. See https://github.com/pantsbuild/pants/pull/14771
w

wide-zoo-86070

03/16/2022, 4:06 PM
Btw, I have some other confusion on the constraints vs lockfiles since they are both mentioned in the Third-party dependencies section. Do they basically mean the same thing? and also i am wondering the (historical) reasons we choose petry instead of pex(pip) for
generate-lockfiles
.
👍 1
h

hundreds-father-404

03/16/2022, 4:15 PM
It's really confusing that in Pants 2.9 and earlier we called constraints file a "lockfile". It's a poor man's version of a lockfile; it does help you to lock things, but it a) does not support
--hash
to make sure the artifacts you download are what you expect (supply chain attacks) b) does not enforce that every transitive dependency is pinned; you can leave stuff off The lockfiles in 2.10 fix those issues and are what lockfiles are actually meant to be. --
and also i am wondering the (historical) reasons we choose petry instead of pex(pip) for
Pip does not support generating lockfiles; the best it has is calling
pip freeze
, which is actually what that
generate_constraints.sh
script is doing. But that's not a true lockfile, e.g. no
--hash
. Also that constraints file might not work on other platforms, like a file generated on macOS might not work on Linux So, we tried using the popular tool
pip-compile
, which generates lockfiles using pip. It worked okay, but also had the same issue of not supporting multiple platforms. In practice, we found this caused lots of issues Which led us to evaluating pdm and Poetry as alternative lockfile generators. Poetry was the most mature for Pants's use case, but does have several limitations like the
[python-repos]
stuff. We decided to go with Poetry because it can handle most use cases, but to at the same time start a project to teach Pex how to generate lockfiles via pip. Which is where we are today, after some excellent work from John Sirois, Pex can now generate lockfiles! The Python ecosystem is huge and complex, so we're "dogfooding" it and trying to handle some new edge cases, hopefully so that Pants 2.11 can start using Pex for lockfile generation by default And then the final part of this long story is supporting VCS (git) requirements, which you can't normally lock because it's not a stable thing; the branch might change for example. https://github.com/pantsbuild/pex/issues/1556
❤️ 1
👍 1
🙌 1
w

wide-zoo-86070

03/16/2022, 4:24 PM
I want to give your every emoji i can use for this detail and insightful info. I am trying to use
pip-compile
to generate the lockfiles. But i got the following error, anything I am missing?
Copy code
InvalidLockfileError: Could not find a Pants metadata block in the lockfile `black_constraints.txt` for `black`. To resolve this error, you will need to regenerate the lockfile by running `./pants generate-lockfiles --resolve={tool_name}`.
❤️ 1
h

hundreds-father-404

03/16/2022, 4:25 PM
Oh hm we should make that error message mention "or set
[python].invalid_lockfile_behavior = 'ignore'
, which you should do if you're manually generating lockfiles
1
Basically, Pants normally adds a header to the lockfile which it uses to keep track of when your lockfile is stale and should be regenerated. If you're manually regenerating lockfiles, Pants isn't keeping track of that for you
4 Views