ambitious-student-81104
03/16/2022, 3:44 PMhappy-kitchen-89482
03/16/2022, 3:45 PMambitious-student-81104
03/16/2022, 4:15 PMhigh-yak-85899
03/16/2022, 5:10 PMrequirements.txt
file.requirements.txt
.hundreds-father-404
03/16/2022, 5:12 PMhigh-yak-85899
03/16/2022, 5:13 PM==
constraints which will lead you right back to problem 1.pip-tools
, poetry
, etc. allow you to maintain a looser requirements.txt
while buying back repeatability and consistency easily.happy-kitchen-89482
03/16/2022, 6:01 PMlively-exabyte-12840
03/17/2022, 1:30 PMambitious-student-81104
03/24/2022, 5:54 PMhundreds-father-404
03/24/2022, 6:19 PMI've previously seen a solution in pants documentation saying just leave the conflicting dependency out of the lock file. what is the essential difference between that and having multiple lockfiles?Leaving it out of the lock (specifically constraints.txt) was a very very imperfect workaround. It meant that a) not all your deps would be pinned, which is bad for consistency and reproducibility, and b) you would have to re-install things rather than reusing the prior resolve, so slower perf. Now that multiple lockfiles is supported, that is the blessed way to handle conflicting dependencies FYI this blog post gets a little more into the theory of multiple lockfiles https://blog.pantsbuild.org/multiple-lockfiles-python/
will the lockfile generation command refresh all the lockfiles I have at the same time?
./pants generate-lockfiles
generates all, and you can do ./pants generate-lockfiles --resolve=my-resolve1
for example to instead only regenerate that "resolve" (aka name for a particular lockfile)
is there a way for us to tag targets in pants to distinguish between libraries and applications?Maybe the
tags
field? But read that blog and the docs at https://www.pantsbuild.org/docs/python-third-party-dependencies#multiple-lockfiles for more on this. Note that when an "app" wants to use "library" code, that library code must share the same resolve (lockfile), So, you'll sometimes have it set up that a particular library file(s) can work with multiple resolves. The blog and docs mention how that worksambitious-student-81104
03/24/2022, 6:23 PMbeta
and stable
, that is a little confusing; is it stable stable now?hundreds-father-404
03/24/2022, 6:24 PMambitious-student-81104
03/24/2022, 7:50 PM./pants generate-lockfiles
failed on an internal dependency we have 😞 is internal dependency supported now?happy-kitchen-89482
03/24/2022, 8:06 PMambitious-student-81104
03/24/2022, 8:09 PMa published wheel in an internal artifactory?yes
requirements.txt
Resolving dependencies...
SolverProblemError
Because pants-lockfile-generation depends on xxx(==0.2.0) which doesn't match any versions, version solving failed.
hundreds-father-404
03/24/2022, 8:22 PMgenereate-lockfiles
goal does not work with [python-repos]
. Fixed in Pants 2.11 by using Pex to generate lockfiles
In the meantime, those docs mention some methods to manually generate the lockfiles, similar to what you had with generate_constraints.txt
happy-kitchen-89482
03/24/2022, 8:25 PMhow do we let pants resolve know that some of our deps are from internal artifactory? the extra index url is already in the requirements.txtHave you configured the extra index URL in
[python-repos].indexes
?hundreds-father-404
03/24/2022, 8:26 PMgenerate-lockfiles
in 2.10ambitious-student-81104
03/24/2022, 8:27 PMhundreds-father-404
03/24/2022, 8:31 PM[python-repos]
works when installing the lockfiles, only not when generating them
2. We already did 2.11.0rc0, which means we are now focused on getting out a stable release. I would expect it to probably be at least 2-3 weeks for the stable release? Changing to Pex doing lockfile generation has a ton of benefits but will continue to require more testing because the Python ecosystem is so vast
3. I'm tempted to say yes because a) you are using [python-repos]
and b) you have "overlapping resolves" like where utils.py
should work with multiple resolves, which is much nicer in Pants 2.11. I think I'd recommend at least waiting till 2.11.0rc1 though, there were some bug reports that we've been fixing. Hopefully we'll do that tomorrowambitious-student-81104
03/24/2022, 8:32 PMconstraints.txt
file manually, and have internal artifactory dependencies.
We want to move to support conflicting dependencies.
What is the best course of action?Note thattrying to understand what this means for my users. currently since we generate theworks when installing the lockfiles, only not when generating them[python-repos]
constraints.txt
manually, people also install it manually with pip install -r constraints.txt
. How will the new installation work?hundreds-father-404
03/24/2022, 8:36 PMHow will the new installation work?Folks can still do
pip install -r constraints.txt
same as before; note that that's not necessary for Pants to do at all though. Pants installs the constraints.txt file in its own way
What is the best course of action?I think I recommend you use 2.10.0 given that it's stable. Specifically, you'll use the script at https://www.pantsbuild.org/docs/python-third-party-dependencies#manual-lockfile-generation-techniques Note the comments in the script about multiple resolves/lockfiles. Once you have multiple resolves set up, you will need to make a slight adjustment to the script so that the script generates for the correct resolve We will hopefully be able to stabilize 2.11 soon so you'll be able to get rid of the manual lockfile generation
ambitious-student-81104
03/24/2022, 8:39 PMhundreds-father-404
03/24/2022, 8:41 PMambitious-student-81104
03/24/2022, 8:43 PMpip install + freeze
rather than just using python -m piptools compile
?hundreds-father-404
03/24/2022, 8:47 PMpip-compile
is valid too. the chart at https://www.pantsbuild.org/docs/python-third-party-dependencies#manual-lockfile-generation-techniques mentions it
Feel free to modify your version of the script manually_generate_user_lockfile.sh
to use pip-compile instead, that's a good improvement. It's especially good that it would let you use --generate-hashes
, which reduces supply chain attack riskambitious-student-81104
03/24/2022, 9:03 PMhundreds-father-404
03/24/2022, 9:06 PM[python-repos]
ambitious-student-81104
03/24/2022, 9:07 PMDoes not supportif you have a custom index or repository other than PyPI.[python-repos]
hundreds-father-404
03/24/2022, 9:07 PMambitious-student-81104
03/24/2022, 9:07 PMhundreds-father-404
03/24/2022, 9:08 PM[python].lockfile_generator = 'pex'
-- only I can't guarantee the Pex lockfile generation will actually work because there may be bugs we don't yet know aboutambitious-student-81104
03/24/2022, 9:10 PM17:09:57.54 [ERROR] 1 Exception encountered:
MappingError: Failed to parse ./BUILD:
Targets in root-level BUILD files must be named explicitly.
hundreds-father-404
03/24/2022, 9:11 PM[GLOBAL].use_deprecated_python_macros
now defaults to False
in 2.10. You can for now set it to true
explicitly, at least temporarily while you're trying this outambitious-student-81104
03/24/2022, 9:42 PMNoCompatibleResolveException: The resolve chosen for the root targets was airflow-dags, but some of their dependencies are not compatible with that resolve:
when I already set [python].default_resolve = "pynest-default"
why is this the case?hundreds-father-404
03/25/2022, 12:32 AMambitious-student-81104
03/25/2022, 12:32 AMpynest-default
, since that is my default_resolve
hundreds-father-404
03/25/2022, 12:33 AMresolve=
for the root targets mentioned in the error message?ambitious-student-81104
03/25/2022, 12:36 AMhundreds-father-404
03/25/2022, 12:36 AM./pants peek path/to:bad_tgt
? Where you use the address of one of the root targets from the error messageambitious-student-81104
03/25/2022, 12:40 AMresolve="pynest-default"
into the root target now, seems to give me a different error (moving on)./pants peek path/to:bad_tgt
hundreds-father-404
03/25/2022, 12:42 AM[python].default_resolve
should definitely be kicking in, and that's a bug if it isn't. Could you please try running ./pants help-advanced python
and make sure the option is being set properly? It should print what the current value isambitious-student-81104
03/25/2022, 12:51 AMResolveError: The file or directory 'xxx' does not exist on disk in the workspace, so the address 'xxx' cannot be resolved.
how do I know which target is actually encountering this error?hundreds-father-404
03/25/2022, 12:55 AMxxx
in your BUILD files 🤷 sometime opened a ticket to improve it, which I agree we need to do[python].default_resolve
now working how you'd expect? Meaning that ./pants help-advanced python
shows the right thing, and you don't have to explicitly set the resolve
field for that one target?ambitious-student-81104
03/25/2022, 1:02 AM--python-default-resolve=<str>
PANTS_PYTHON_DEFAULT_RESOLVE
default_resolve
default: python-default
current value: pynest-default (from pants.toml)
The default value used for the `resolve` field.
The name must be defined as a resolve in `[python].resolves`.
yesResolveError
thoughname=reqs
?3rdparty/xxx
looks like rn:
python_requirements(
module_mapping={
"future": ["concurrent"],
},
overrides={
"omniduct": {
"dependencies": [
"//:setuptools",
"xxx#snowflake-sqlalchemy",
]
}
},
)
hundreds-father-404
03/25/2022, 1:04 AMpython_requirements
target generator? How did you end up approaching that from earlier in the day? What is [GLOBAL].use_deprecated_python_macros
set to?ambitious-student-81104
03/25/2022, 1:04 AMuse_deprecated_python_macros
is now unset, i'm on 2.11python_requirements
deprecated as a whole? What should I be using instead?hundreds-father-404
03/25/2022, 1:06 AM./pants update-build-files --fix-python-macros
already?ambitious-student-81104
03/25/2022, 1:06 AMuse_deprecated_python_macros = true
, and ran ./pants update-build-files --fix-python-macros
. it gave me:
MappingError: Failed to parse ./3rdparty/ml_models_common/statsmodels/BUILD:
__call__() got an unexpected keyword argument 'name'
python_requirements(name="reqs")
hundreds-father-404
03/25/2022, 1:11 AMuse_deprecated_python_macros = true
is set to true, it uses the old macro system where it's an error to have the name=
field
I imagine this is confusing doing both the macro deprecation at the same time as setting up multiple resolves. It might be helpful if you take a step back and completely do the macro change before touching multiple resolves at first. Specifically something like
1. git stash the work you've done
2. go back to the last thing that was working, like main
3. upgrade to 2.10.0 and follow the instructions exactly for the deprecation of the old macro system, including running update-build-files --fix-python-macros
& checking the logs if there are any manual changes you need to make
4. Run ./pants dependencies ::
to make sure that you are totally good to go. No ResolveError. Commit this all
Then, go back to adding multiple resolvesambitious-student-81104
03/25/2022, 1:13 AMhappy-kitchen-89482
03/25/2022, 2:52 AMhundreds-father-404
03/25/2022, 5:16 AMambitious-student-81104
03/25/2022, 2:22 PMBtw 2.10 does not require you to switch to multiple lockfileswe are hitting hard dependency conflicts, we need multi lock
pthon_sources()
has resolve=xxx
, do all the BUILD files under it also have resolve=xxx
automatically? Or do I need to manually add ``resolve=xxx`` in all the BUILD targets under a directory?./pants peek
is not telling me which resolve
this target is inRepo
- Lib 1
- App 1-1
- App 1-2
where App 1-1
and App 1-2
conflicts with each other?parametrize
to add a target to multiple resolves?hundreds-father-404
03/25/2022, 5:20 PMparametrize
like this:
python_sources(
resolve=parametrize("resolve-a", "resolve-b"),
)
If you're using Pants 2.10, then you have to manually create two targets for the same file, like this:
python_sources(
name="lib_resolve-a",
resolve="resolve-a",
)
python_sources(
name="lib_resolve-b",
resolve="resolve-b",
)
Yeah, it's generally desirable to have fewer resolvespeek
, you're on 2.11 right? Try doing ./pants peek path/to/file.py
- the resolve
field only shows up on the generated python_source
targets, not the python_sources
target generatorsambitious-student-81104
03/25/2022, 8:55 PMresolve=parametrize("resolve-a", "resolve-b"),
and it worked, however it only works on some targets, not all. What is the list of targets resolve
works on?
i'm 80% done with experimenting with a migration into the 2.11 + multi-lock world. And here are my observations so far, I want to check with you to see if they make sense, and what suggestions you have:
1. multi-lock is not a solution to python dep conflicts; it is one of the alternatives, and there are tradeoffs between alternatives, none is a magical solution. hard conflicts are hard conflicts, at the end of the day. They need to be either resolved, or left to not interfere with each other. Even Google who copies all deps into their repo needs to leave some as target-specific deps when they have multiple conflicting versions.
2. multi-lock is very costly, not just to migrate to, but to maintain on an ongoing basis, largely because as parallel universes (lmk if this analogy is not accurate), they are intrinsically expensive to maintain; this means you really can't afford to have too many resolve going at the same time, so you need to put on guardrails and prevent people from just creating a dozen of resolves. So at the end of the day, you're still gonna need to resolve dep conflicts, and multi-lock is just easing/delaying it, not magically solving it.
3. we are thinking that we can learn from Google but need to be a bit more flexible. Maybe still do single lockfile, and do something like this when one introduces a new dep that creates conflicts with the lockfile:
a. first, try to resolve your dep conflict;
b. if that's too difficult, you can copy the part of 3rdparty code you need into our monorepo (note: unlike Google, we don't default to this way of doing things, only resort to it), and we can build it and push to our artifactory and treat it as an internal dep, and we can sort out its transitive deps much better;
c. if b isn't feasible either, then leave that dep out of the lockfile, create a local requirements/constraints.txt and let the specific target that needs it depend on that requirement.
i. because this can cause repo divergence, we should detect these changes in CI, and require repo owner approval on such PRs.