https://pantsbuild.org/ logo
#general
Title
# general
s

stocky-helmet-22655

12/02/2022, 9:14 PM
I have a monorepo with several services, several libraries, many dependencies, etc and I’m having trouble trying to integrate Pants into it. I’ve simplified down to where a repo with just two folders each with a Pipfile fails to run Pants due to
ModuleNotFoundError
on any 3rd party dependencies that are shared, even if they’re the same version. How am I supposed to handle this situation?
h

happy-kitchen-89482

12/02/2022, 10:46 PM
Sorry to hear you’re having trouble. The easiest way to debug would be if you could post a trivial repo to github that demonstrates the issue!
But I suspect that what’s happening is that you have two representations of the same 3rdparty dep, so dependency inference doesn’t know which one to pick. You might be seeing WARN logs to that effect?
It might make sense for Pants to disambiguate by closeness in the filesystem, for example
But generally we encourage having a single unified set of allowed requirements (aka “resolve”) for the entire repo, unless you truly need conflicting versions in different parts of the codebase
I have a TODO to write a blog post about this, because it is understandably confusing to new adopters
👍🏿 1
Without Pants you have to repeat requirements for every subproject/distribution/deployable thing, because that’s how the python package/deploy tooling works. But with Pants you just need a single “universe” of allowed requirements, and Pants selects the subset that you actually use in any given case. So it’s a different idiom.
s

stocky-helmet-22655

12/03/2022, 1:38 AM
@happy-kitchen-89482 thanks for being so quick to respond! (here and on the issue I posted, hehe) I’m happy to post a trivial repo to github, but first let me answer your responses I do in fact have two representations of the same 3rd party dep, and I have seen warning logs to that effect - it’s unclear to me why that’s an issue or more importantly how to resolve it though 😞
But generally we encourage having a single unified set of allowed requirements (aka “resolve”) for the entire repo, unless you truly need conflicting versions in different parts of the codebase
I would love for this to be the case, unfortunately the repo I am trying to integrate Pants into is a mess right now, with module different sets of dependencies, including a significant amount of overlap with different versions of the same 3rd party deps. Is there a way to tell Pants to treat it like different projects in the same repo? I had assumed that’s what
[source]
was for but that doesn’t seem to be the case
Without Pants you have to repeat requirements for every subproject/distribution/deployable thing, because that’s how the python package/deploy tooling works. But with Pants you just need a single “universe” of allowed requirements, and Pants selects the subset that you actually use in any given case. So it’s a different idiom.
Is it possible for this “universe” of allowed requirements to include multiple versions of dependencies? If not I may not be able to incorporate Pants into this repo, at least not until some significant cleanup has been done.. I’m hoping Pants can be a help in performing that cleanup though
h

happy-kitchen-89482

12/03/2022, 1:41 AM
That’s exactly what the “multiple resolves/lockfiles” feature is for
You can specify multiple “universes”, each of which is a set of mutually compatible transitive 3rdparty deps, represented by a lockfile
And you can assign different parts of your codebase to different universes (or even, in the case of library code, multiple universes)
The ideal number of universes, for codebase manageability, is 1
But in many cases, such as yours, that’s not feasible (or at least not yet)
So you define multiple resolves, and that gets things working, and then you can slowly consolidate them
Note however that the resolves represent “universes”, not projects. So if you have 20 different projects in your repo, but each can be assigned to one of 3 universes, you have 3 resolves, not 20.
So that is already more manageable
Unlike using raw Python tools directly, where each project needs its own list of 3rdparty deps - even if they don’t conflict! - because that is just how the tools work
s

stocky-helmet-22655

12/03/2022, 1:46 AM
Interesting….I had started to try figuring out the multiple resolves thing earlier this afternoon but wasn’t sure whether it was relevant and gave up a little too quickly. I’m putting up a sample repo rn and as soon as that’s up I’ll give the multiple universes thing another shot
Though for this example repo it sounds like it’ll still be one universe
h

happy-kitchen-89482

12/03/2022, 1:47 AM
I’m going offline soon, but here is some reading material: https://www.pantsbuild.org/docs/python-third-party-dependencies#lockfiles
Yeah, you only need multiple universes if you much have conflicting versions
it sounds like in the simple example you just need one
and what is happening is that you have two not because you need two but because you’ve merged two pre-pants projects that each had their own, because of course they did - that’s how things must work outside of Pants
I really really must write this up in a post
👍🏿 1
too many people get tripped up on this
The Pants idiom is very powerful, but takes some getting used to
s

stocky-helmet-22655

12/03/2022, 1:49 AM
Sounds like you don’t need to see the example, but in case you’re interested: https://github.com/nikwotton/Broken-Pants-Demo
I wonder if beyond a post, could it be possible to have Pants automatically detect that two universes are identical and merge them? and/or suggest in the log that the user merge them?
Blog posts are great and can be super helpful once found, but the log messages I think are what people always go to first
hmm, ok good news and bad news: Using two separate lockfiles/universes works! When I try to resolve both packages to the same universe though, it goes back to failing on the same errors as before
This should be enough to get me started at least, though for now I’ll be stuck having a bunch of universes…progress though!
thanks for the help!
h

happy-kitchen-89482

12/03/2022, 2:06 AM
When I try to resolve both packages to the same universe though, it goes back to failing on the same errors as before
Can you explain what you’re doing here to cause the errors?
What you mean by “resolve both packages to the same universe”?
s

stocky-helmet-22655

12/03/2022, 2:08 AM
I mean I am trying to do as you describe here
Note however that the resolves represent “universes”, not projects. So if you have 20 different projects in your repo, but each can be assigned to one of 3 universes, you have 3 resolves, not 20.
I’m trying to use each package as a project with one shared universe/resolve. In case showing is better than telling, here’s the code: https://github.com/nikwotton/Broken-Pants-Demo/commit/d0ef5cdde4f10306c7da2a472f59d77b33a68c2b When I run
./pants test ::
it throws errors
wait…should I use one pipfile per universe? maybe that’s the issue?
h

happy-kitchen-89482

12/03/2022, 2:10 AM
That is exactly what I was just about to write
yes
s

stocky-helmet-22655

12/03/2022, 2:10 AM
ahhhh
h

happy-kitchen-89482

12/03/2022, 2:10 AM
The two pipfiles create ambiguity
You can put the pipfile in the parent of the two projects, for example
or in some
3rdparty
subdir of it
That is idiomatic
s

stocky-helmet-22655

12/03/2022, 2:13 AM
hmm, should both BUILD files still have this block then if only one of them has a pipfile?
Copy code
pipenv_requirements(
    name="pipenv",
    resolve="packageA"
)
Ah I think I got it, I needed this:
Copy code
python_tests(
    name="tests",
    resolve="packageA"
)
I think I’m starting to get this
@happy-kitchen-89482 since you’re so amazingly helpful, any chance you can help with this issue too? 🙏 https://pantsbuild.slack.com/archives/C046T6T9U/p1670366302094639
4 Views