Hi! Does anyone here have a good structure/pattern for a 

Python + Docker + GCP

 Monorepo architecture!? Curious to see how others have done this 🙂

following.

I don't think we have a public example. Those monorepos tend to be private. But it makes sense to create one.

How the build generalizes when adding more python services. For example, I'd like to create a monorepo with the structure:

monorepo/services/python/service1
monorepo/services/python/service2
monorepo/services/python/service3
monorepo/services/python/service4
monorepo/services/python/service5

Each with independent requirements, maybe some shared code (like a database client class, how would that work, where would it live, etc...) Lastly, I'd like to write tests, and dockerize them, and run a docker compose at the top level to spin up all services at once. I think if an example repo is there that achieves those goals, or at least shows the file structure, that would be a HUGE win for any new developer using pants!

have you seen https://www.pantsbuild.org/docs/source-roots#examples ?

Yes, still had a couple of questions 1. What if there are multiple projects under python? Does everything stay the same otherwise? 2. Do I just stick a docker file in each of the projects as if it was a single repo all by itself?

@shy-match-55049: re: 1) are you referring to a particular pattern from that page?

I was thinking the src/<lang> setup pattern

the relevant bit of the

src/<lang>

pattern is the

src

token, which you don’t have in your example above

One way to do this is to have a single source root (

src/python

if you think you might have other languages in the future, or just

src/

or even the repo root itself if not). Then you can have all the code live under

<srcroot>/myrootpkg/

with shared code in, say,

<srcroot>/myrootpkg/util

,

<srcroot>/myrootpkg/db

and so on.

I see! I'll tinker around and come back with questions 🙂

but you can also have a parallel test tree if you prefer

Is that not difficult with python? Even for the IDE to recognize?

Other tools push you towards a parallel tree because they don't know how to exclude the tests from the package when you package and deploy

Oh, interesting. I was trying to see if I could do multiple requirements, but that makes a lot of sense

So the

requirements.txt

represents the universe from which dependencies are chosen as needed, but only the actually needed ones are used.

Amazing

You can have multiple

requirements.txt

, even conflicting ones

In terms of building dockerfiles then, would it be better to keep them separate from the actual service roots, since they need to read the requirements.txt? How would that work in a clean, organized way?

afaik, the dockerfile doesn't need to see the requirements.txt pants builds the

.pex

file, and then you bung that in a dockerfile with

ENTRYPOINT [foo.pex]

Pants uses PEX by default, which is a self-contained python artifact (essentially a relocatable virtualenv)

Oh, interesting. How do I know where it will output? Is it just in the directory with the code itself?

in fact, the whole docker thing becomes a bit of a joke, because it's probably just

FROM python:3.8
COPY artifact.pex /bin
ENTRYPOINT ['artifact.pex']
RUN ['--run-stuff']

@shy-match-55049: it goes into `dist`: see https://www.pantsbuild.org/docs/python-package-goal

I see. Still having a little bit of trouble seeing how this generalizes to multiple docker images, for example I have:

monorepo/services/python/service1/Dockerfile
monorepo/services/python/service2/Dockerfile
monorepo/services/python/service3/Dockerfile
monorepo/services/python/service4/Dockerfile
monorepo/3rdparty/python/requirements.txt

Would each of those

Dockerfile

's just be

FROM python:3.9.6-slim
COPY service{n}.pex /bin
ENTRYPOINT ['service{n}.pex']
RUN ['--run-stuff']

correct

That's pretty neat

now you just need to write a plugin that takes

pants containers

and runs

docker build

we have a very venerable ticket for docker support: https://github.com/pantsbuild/pants/issues/2648 … @ambitious-actor-36781: it sounds like you might want something a little different: perhaps because your host environment is similar-enough to the docker environment?

Unsure. It looks like that ticket would be relevant. there's two (three?) different things. 1. a way to publish the output of

pants package

to a repository (currently working on this) 2. A way to build pants targets into a docker image (e.g.

docker_image(name='a_service', dockerfile='src/python/a_service/Dockerfile', dependencies=[':a_service_binary'])

) 3. (?) take output of #2 and run it through #1 so it pushes the images to a container registry.

yea, absolutely.

you've got to say "not our problem" at some point right? You could just say "ok... we've put a

pants

image on the Docker registry, and you can just do your entire build inside there"

true!

Found a docker integration

That'll be for 1.x

AFAIK there are no public 2.x plugins for docker , but several private ones

Specifying a base images and the executable target to run it in?!

@clean-night-52582: pants runs a lot of executables. so running only the root process (e.g. the one that creates the pex) is not likely to be sufficient: the resolve(s) would happen as a separate step. see the most recent comment on that ticket