Hey I m new to pants and have never worked with a monorepo t Pants #general

Hey I’m new to pants and have never worked with a ...

curved-computer-88379

07/12/2023, 7:22 PM

Hey I’m new to pants and have never worked with a monorepo tool before. I am starting a monorepo that currently has folders

src/

and

projects/

where

src/

contains source code and

projects

contain individual projects that may or may not source from

src

. I was thinking that each project either builds an image from the Dockerfile within the project directory, or from the root directory base

Dockerfile

that will likely be compatible across multiple projects. What are some steps I should take to integrate python builds / services that are currently deployed in production using docker, but integrating pants so I can source shared code (versioned) across projects. Example: Say I update the shared code

~/src/utils.py

from version 0.1 to 0.2, but I don’t want to have to worry about maintaining project_1 once it’s been deployed (e.g. if

utils

version 0.2 breaks project_1, so project_1 requires utils version 0.1) For example

Copy code

project_1:
  run.py <- sources from /app/src/utils.py version 0.1
  Dockerfile
  requirements.txt
  ...

project_2:
  run.py <- requires /app/src/utils.py version 0.2 since utils.py was updated
  Dockerfile
  requirements.txt
  ...

high-yak-85899

07/12/2023, 7:27 PM

I would recommend maintaining a single resolve (i.e.

requirements.txt

) for the entire repo and not try to bifurcate for each project.

high-yak-85899

07/12/2023, 7:28 PM

The structure you've proposed above looks a lot like what poetry tries to encourage for structure. It's rather limiting in terms of things you can do and sharing code that have different resolves (i.e. different requirements files) becomes a challenge at the BUILD file level imo.

high-yak-85899

07/12/2023, 7:28 PM

With that said, there's no reason you can't do

src/

and

projects/

as you've described!

high-yak-85899

07/12/2023, 7:29 PM

It can be a great way to manage development of the repo; keep shared things in

src

and specialize/use in

projects

curved-computer-88379

07/12/2023, 7:29 PM

I’m not sure I understand how one requirements.txt file would work. Project_1 requires python version 3.9.15 with a set of requirements, and project_2 requires python version 3.10.12 with a new set of requirements

high-yak-85899

07/12/2023, 7:30 PM

well that's a different story

high-yak-85899

07/12/2023, 7:30 PM

requirements.txt doesn't even need to come into play for that issue

curved-computer-88379

07/12/2023, 7:30 PM

what i’ve been doing is put a Dockerfile for each individual project that deviated from the base Dockerfile

high-yak-85899

07/12/2023, 7:31 PM

I might not be familiar with enough of the settings, but I'm not sure pants lets you work on different portions of a repo with different interpreter versions.

high-yak-85899

07/12/2023, 7:31 PM

You can control this at the target level https://www.pantsbuild.org/docs/reference-python_sources#codeinterpreter_constraintscode

high-yak-85899

07/12/2023, 7:32 PM

So presumably you could put

__defaults__

in the right place at the top level of each project

high-yak-85899

07/12/2023, 7:32 PM

That's still not a deal breaker for having a single requirements file.

high-yak-85899

07/12/2023, 7:32 PM

There are packages that could be valid for both 3.9 and 3.10 use

curved-computer-88379

07/12/2023, 7:32 PM

for sure a single requirements file would break my code

curved-computer-88379

07/12/2023, 7:33 PM

i have tried it because project_1 requires conflicting packages from project_2 and project_3

high-yak-85899

07/12/2023, 7:33 PM

https://www.pantsbuild.org/docs/targets#field-default-values is what I'm referring to for defaults

high-yak-85899

07/12/2023, 7:33 PM

Then you're forced to use different resolves and your challenge will come when you try to share source code from

src

high-yak-85899

07/12/2023, 7:34 PM

You'll have to use the

parametrize

functionality to indicate that your source code can be used among one of multiple resolves.

high-yak-85899

07/12/2023, 7:34 PM

https://www.pantsbuild.org/docs/targets#parametrizing-targets has details on that

curved-computer-88379

07/12/2023, 7:35 PM

hmm ok thanks i’ll read through these

high-yak-85899

07/12/2023, 7:35 PM

I'm not sure I follow the example in the original post. It kind of sounds you're thinking about all of these things as separate entities which isn't really the point of a monorepo.

high-yak-85899

07/12/2023, 7:37 PM

One of the huge values of a monorepo is updating everything together. You're not using an older portion of your monorepo elsewhere in your monorepo. You're using what's tracked in source control all together. If you want to version portions of the monorepo, that sounds more like what a polyrepo setup is intended to enable.

high-yak-85899

07/12/2023, 7:38 PM

And if you did want to do that, there's nothing stopping you from using pants in each smaller repo in your polyrepo setup. It just happens to be a common use case for monorepo's because the problems pants solves become exacerbated in a monorepo setup.

curved-computer-88379

07/12/2023, 7:39 PM

I was exploring a monorepo primarily because of sharing code across projects. It’s a headache to maintain 5 different versions of the same class / function. Putting those in a shared

src

folder was my initial reason for diving into a monorepo.

curved-computer-88379

07/12/2023, 7:39 PM

Reason I asked about different versions is because I updated one of the function in

src

and it broke project_1. I don’t want to have to worry about maintaining 50+ different projects once they’ve been deployed to production

high-yak-85899

07/12/2023, 7:40 PM

I was imagining that all the things in

src

would be one repo,

project_1

would be another repo, and

project_2

would be another still.

curved-computer-88379

07/12/2023, 7:40 PM

you can look at it that way, i just ended up putting projects as it’s own directory

curved-computer-88379

07/12/2023, 7:40 PM

(not inside src)

high-yak-85899

07/12/2023, 7:40 PM

I don’t want to have to worry about maintaining 50+ different projects

This is why it kind of sounds like a monorepo isn't the right solution for you. This is precisely one of the things a monorepo architecture aims to do; provide immediate visibility when you break compatibility with downstream consumers.

curved-computer-88379

07/12/2023, 7:42 PM

when i was reading into it I thought you can version control the source? Something like

Copy code

python_binary(
    name='project_1',
    source='run.py',
    dependencies=[
        '//src:shared_utils==0.1',
        # Add other project-specific dependencies if any.
    ],
)

curved-computer-88379

07/12/2023, 7:43 PM

and then

Copy code

python_binary(
    name='project_2',
    source='run.py',
    dependencies=[
        '//src:shared_utils==0.2',
        # Add other project-specific dependencies if any.
    ],
)

curved-computer-88379

07/12/2023, 7:45 PM

I thought monorepo sounded like a great solution for what i’m looking for. I’m looking to consolidate my code into one repo. The separate projects will almost always source code from the root

src

directory. Once deployed to production doesn’t pants store the version of each py library sort of like git? If utils==0.1 is set then shouldn’t it source the old version of utils?

high-yak-85899

07/12/2023, 7:47 PM

I haven't seen anything like that. My experience is that you want what's currently in

src

to be compatible with what's currently in

projects/*

high-yak-85899

07/12/2023, 7:47 PM

But perhaps Pants has some capability to do what you're suggesting that I'm not familiar with.

curved-computer-88379

07/12/2023, 7:48 PM

interesting, ok so if let’s say I update

numpy

and no the function

np.bool

is no longer supported / throws an error. Say a line of code will error out in projects/project_1 because it calls

np.bool

. Will pants detect that?

curved-computer-88379

07/12/2023, 7:50 PM

or do i have to test the code myself some other way to find out that

np.bool

is no longer supported?

high-yak-85899

07/12/2023, 7:51 PM

If you have unit tests with coverage in the areas where

np.bool

is called, then your

pants test

commands would fail.

acoustic-library-86413

07/13/2023, 10:16 AM

There is a bit of documentation about managing third-party dependencies with multiple resolvers here: https://www.pantsbuild.org/docs/python-third-party-dependencies If you do end up maintaining an individual resolve file for each of your projects, then you could, atleast in theory, have some projects use a pinned version (that you push upstream on commit), and have other projects import from the codebase directly, always using the latest version. I'm not sure it would be pretty or easy to reason about, but I think it would be possible.

👀 1

curved-computer-88379

07/13/2023, 10:13 PM

interesting, thinking maybe I can try something like this for packages not pushed to pip

Copy code

dependencies = [
    utils==git+http://<utils url>.git@commit
]

2 Views

Open in Slack

Previous Next