https://pantsbuild.org/ logo
Join Slack
Powered by
How are we supposed to handle shared dependencies ...
# general
c
How are we supposed to handle shared dependencies with multiple 
resolve='...'
? I have a large company-wide monorepo and we share some modules, but it seems resolve is overly strict and when simply running tests it can't resolve all requirements because they don't all use the same resolve
f
You can put shared dependencies into both resolves using the 
parametrize
helper.
This basically duplicates the target for the shared dependency into multiple targets: one for each resolve.
c
interesting, does that work on regular 
python_sources
?
f
yes
h
Exactly. Better docs coming hopefully today on this We know that we need better error messages when a dependency is not in the same resolve
đź‘Ť 1
c
any examples of how I use parameterize?
f
basically set 
resolve=parametrize("a", 'b")
h
python_sources
There, the semantic meaning is that this code can work with either lock file. Note that pants can't actually validate that for you through static analysis. For example, it is possible to have one lock file that uses Django two and another that uses Django three. If you claim that a file using Django works with both lock files, you will have to make sure that you are using the APIs safely Fortunately, pants will run mypy and Pylint with every relevant resolve.
f
h
At the very end, it talks about how you can combine 
parametrize
with the overrides field. I switched tool chain last week to use multiple resolves, and found that often it is very helpful to be as precise as possible with which code needs to work with multiple lock files. It reduces the surface area of code that you must make sure is compatible
f
(
parametrize
is used for 
resolve
on JVM targets, but the same syntax is used for Python targets, so still applicable as an example)
âž• 2
c
hmm, that seems like it's going to be obnoxious, we have a 
common
module in our monorepo that's depended on by something like 15 other modules
I might be able to figure out another way to make it work
f
Are you using 15 different resolves though?
You should only need to use 
parametrize
with the actual different resolves.
c
I think we may need a resolve for each of our "entrypoints" (ie. a service or CLI utility)
f
You only really need different resolves if you have conflicting requirements.
c
hmm, I see
f
So if those entrypoints have a compatible set of requirements than they can share a single resolve.
c
we do have some, but there's also the problem where we have something like 150+ 3rd party deps and resolving all of them takes... longer than 12 hours, idk I gave up
f
maybe identify some services that can share a resolve? that would reduce the number of necessary resolves to less than 15 (but still more than 1)
âž• 1
h
It's often preferable to have fewer resolves
nd resolving all of them takes... longer than 12 hours, idk I gave up
when imstalling lockfiles from pex, pants only installs the subset of deps you need for a certain task
f
We could also consider adding a syntax to let you not have to manually list every resolve in order to use 
parametrize
.
c
I think part of the problem is (historically) we had dep-inference off because... reasons, idk
đź‘€ 1
turning that off made things happier
f
which version of Pants are you on?
c
2.2
-> 
2.11.0rc3
we turned it off because someone convinced us we wanted to manually manage our deps, but in hindsight that was a terrible idea
âť— 1
(yes, I know)
h
fwit, i used to be anti dep inference and now im a total convert. its too hard for humans to maintain, and very unlikely to be as precise. dep inference is file-level (python_source -> python_source). manual deps are usually dir -> dir (python_sources w/ an s)
c
when we were trying to make build-system decisions, the other candidates included Bazel, which almost won if it wasn't for the fact that Bazel is garbage and has truly awful Python support
I also grow more and more skeptical that monorepos are a net positive
is there a way to avoid having to run 
generate-lockfiles --resolve=python-default
? I have the default set to a resolve called "all" as if I don't define it, I get something like:
Copy code
UnrecognizedResolveNamesError: Unrecognized resolve name from the field `resolve` in the target module:target: python-default
Well this is confusing:
Copy code
âź© ./pants generate-lockfiles --resolve=all
15:24:21.34 [INFO] Completed: Generate lockfile for all
15:24:21.34 [ERROR] 1 Exception encountered:

  ProcessExecutionFailure: Process 'Generate lockfile for all' failed with exit code 1.
stdout:

stderr:
ERROR: Cannot install pantsbuild-pants==2.11.0rc0, pantsbuild-pants==2.11.0rc1, pantsbuild-pants==2.11.0rc2, pantsbuild-pants== 
The conflict is caused by:
     pantsbuild-pants 2.11.0rc4 depends on pex==2.1.80
     pantsbuild-pants 2.11.0rc3 depends on pex==2.1.79
     pantsbuild-pants 2.11.0rc2 depends on pex==2.1.78
     pantsbuild-pants 2.11.0rc1 depends on pex==2.1.75
     pantsbuild-pants 2.11.0rc0 depends on pex==2.1.72
w
@calm-ambulance-65371: do you have a pants plugin in your repository?
is there a way to avoid having to run 
generate-lockfiles --resolve=python-default
? I have the default set to a resolve called “all” as if I don’t define it, I get something like:
you don’t need to pass any arguments to `generate-lockfiles`: the default is to generate all of them. so if you have exactly one named resolve in the `resolves` option, that’s what will be built.
but it sounds like you don’t need multiple resolves: in which case you shouldn’t need to rename the resolve or even define a lockfile location, because there are defaults for that (
python-default
): you only need to define one if you need more than one.
relevant to whether you actually need multiple resolves: were you already using a lockfile via these 2.2 instructions? if so, do you know whether you had any inline requirements? if you didn’t have any inline requirements, and you were using a lockfile before, then you don’t need multiple resolves, and can just set 
[python].enable_resolves
without defining any additional resolves in 
[python].resolves
4 Views
Open in Slack
PreviousNext
Powered by