We're checking out pants with a view to move to a monorepo setup. Currently, we use conda for environments, and have separate environments for each project. These have different dependencies, and most use poetry for everything else. If we move these projects to a monorepo with pants, what's the recommended way to manage environments. I'm guessing at a high level, the structure might look like:

root
--projA
  |--pyproject.toml
  |--src
  |--tests
--projB
  |--pyproject.toml
  |--src
  |--tests
--projC
  |--pyproject.toml
  |--src
  |--tests
pants.toml

How would we maintain the independent environments for the three projects? Can we specify a conda env constraint for each project? Should we ditch conda for envs? We're also considering adopting uv over poetry at the same time. Would that change things?

Unless you have conflicting dependencies, the strongest recommendation is to use a single resolve. Pants will only bundle the necessary dependencies from the resolve when building etc; so the only extra cost is at lock-time... but it'll overall be much simpler to manage.

@gorgeous-winter-99296 @happy-kitchen-89482 Ah... so we don't need to worry about environments at all, and when pants runs a test, it'll figure out the appropriate dependencies of the relevant project, and it's not one big polluted environment? That sounds good. How would an IDE be able to figure out dependencies though? e.g. would VSCode be able to understand the relevant configuration for the target project for a particular test?

No, for editor workflows you can export a classic

venv

with

pants export

, and enable that in your editor using the appropriate method. So at edit time you do see all deps, but at "exec" time you get a reduced set.

Is the export per project? I'm thinking of scenarios of different projects having conflicting dependencies.

The setup, if you follow the golden path is one set of dependencies, one resolve, one lockfile, one export. If you have a conflicting set of dependencies, then you get two resolves; two lockfiles; two exports.

Yes... So for example, one project might use pyspark that has certain constraints on its dependencies, but then another project might not use it and choose to use a latter version of python as well as more recent versions of some dependency that the first project can't take. It's this setup I'm thinking of. What's the recommended approach there?

Different versions of Python within one resolve can be made to work. But the second part; more recent versions of a dependency, would require two resolves. Especially with ML or data tools that seems to be very common, even before thinking about GPU/non-GPU/Mac devs etc.

That's exactly our scenario... Data and ML projects. When you say resolves, could you please elaborate? Would that mean two / project specific venvs?

In short, it's a "domain" of dependencies, along with configuration for how to use those dependencies, etc. So it's wider than project, but narrower than "the repository". I'd read this blog-post https://www.pantsbuild.org/blog/2022/05/25/multiple-lockfiles-python, and at least skim https://www.pantsbuild.org/blog/2022/10/27/why-dependency-inference and https://www.pantsbuild.org/dev/docs/python/overview/third-party-dependencies.

Amazing, thanks. Will read through them.

“Resolve” is roughly a synonym for “lockfile” and we probably should just have always called it a lockfile…

@breezy-electrician-41537 I'm currently looking into migrating a monorepo to pants mainly for Data and ML projects, and particularly for PySpark support. We have the same situation where different Data/ML teams really want to control their own 3rd party dependencies and not have to worry that upgrading any will impact another team's code. we're trying out a structure like this one below, using multiple lockfiles (in this example, "projectAAA" has its own resolve/lockfile, as would a hypothetical "projectBBB" sibling)

subprojects/
├── projectAAA/
│   ├── 3rdparty/
│   │   └── python/
│   │       └── PANTSBUILD
│   └── src/
│       ├── python/
│       │   ├── apps/

things we're finding (1) it's annoying that teams need to always set the

resolve=

field on their targets, but it's not the worst thing (2) the really hard things is to make shared libraries that work across multiple subprojects that have their own resolves. I'd like to make some basic pyspark library that is shared by all PySpark apps, and we're trying to figure out how to automate this documented direction: > If a first-party target is compatible with multiple resolves, e.g., shared utility code, you can use the parametrize mechanism with the

resolve=

field. anyways if you have any learnings along the way, I'd love to hear them. It sounds like we're trying to solve the same problems

@busy-ram-14533 it'd be great if a build file in the root of a sub project could set the resolve for all targets under it. Or maybe a parent folder could set the resolve used for everything under it. Is this possible, @gorgeous-winter-99296 @happy-kitchen-89482?

Yeah, you can use `__defaults__`: https://www.pantsbuild.org/dev/docs/using-pants/key-concepts/targets-and-build-files#field-default-values

@breezy-electrician-41537 what you describe is pretty much how the new python backend will work, once I get back to actually working on it.