https://pantsbuild.org/ logo
#general
Title
# general
d

dazzling-dress-95246

03/31/2023, 10:39 PM
I'm having some trouble understanding how to setup pants. Context: My org's repos have a weird setup where we have 3 (completely python) repos: 1. Foo 2. Bar 3. Baz Now, within our docker images, we mount them like so:
Copy code
/usr/src/app:Foo
/usr/src/app/bar:Bar
/usr/src/baz:Baz
and then we set the PYTHONPATH within the image to:
Copy code
/usr/src/app:/usr/src/baz
Now Foo's source root is "/". Bar gets mounted within Foo and so python modules in Foo can import python modules within Bar like so:
Copy code
import bar.something
Python modules in Foo can also import python modules within Baz like so:
Copy code
import baz.something
I'm currently keeping
pants.toml
within Foo. Questions: 1. what's the recommended setup here? a. I'm currently thinking of mounting all the above within a container just like our current env and then installing pants within it. 2. should I set the pythonpath env var within the container or given that pants uses its own python binary, should I define it in pants.toml somewhere? a. I do see this link but it refers to plugins which I don't think is right. 3. I'd like to begin by testing pants on a single file/module, let's say located at
Foo/module/submodule
(which can import from across the pythonpath). However setting the source root as "/" and then calling
pants tailor ::
adds BUILD files all over the repo. How can I limit it to only the one submodule? Feel free to have me read/grep documentation if the above is covered somewhere already.
re 2: So in my container at /usr/src/app, when I run
pants tailor ::
, it sadly misses adding BUILD files to the Baz repo so I think I need to tell pants to look at the PYTHONPATH too? re: 3 looks like I do need BUILD files everywhere!
e

enough-analyst-54434

03/31/2023, 11:35 PM
Pants does not use PYTHONPATH nor does it generally allow env variables to influence the processes it runs. As a general concept it is hermetic and tries to run the same regardless of your local env vars vs some other developers. Hopefully you've run down through: https://www.pantsbuild.org/docs/concepts From that list, you want to set up source roots to tell Pants about your 1st party code PYTHONPATH: https://www.pantsbuild.org/docs/source-roots - its the same concept, but language agnostic (CLASSPATH is roughly the mapping for the JVM for example). If you go to the
tailor
docs you'll see a few ignore options: https://www.pantsbuild.org/docs/reference-tailor You might try masking out a subtree in your repo for tailor to operate on using those.
d

dazzling-dress-95246

03/31/2023, 11:41 PM
From that list, you want to set up source roots to tell Pants about your 1st party code PYTHONPATH
so the issue is I've defined my pants.toml within Foo and so the only sources I can put are within Foo. However, Baz gets mounted outside the Foo root. Python still treats it as a local module import because of PYTHONPATH
I guess a fix could be to just mount Baz within Foo? Or to have pants.toml to live within /usr/src and define the roots as /usr/src/app and /usr/src/baz
e

enough-analyst-54434

03/31/2023, 11:42 PM
I dare say, IIUC, you are making things incredibly complicated.
d

dazzling-dress-95246

03/31/2023, 11:42 PM
haha sadly not my choice, just how the current system is 😞
e

enough-analyst-54434

03/31/2023, 11:43 PM
So, for me to have a hope at understanding I think it's est to not talk about repos. Just the final tree.
What is the tree?
d

dazzling-dress-95246

03/31/2023, 11:43 PM
sure: / • usr ◦ src ▪︎ app • ... • bar ▪︎ baz
with PYTHONPATH set to /usr/src/app and /usr/src/baz
e

enough-analyst-54434

03/31/2023, 11:45 PM
And, for whatever reason, you can't put pants.toml at any of /, /usr or /usr/src - correct?
d

dazzling-dress-95246

03/31/2023, 11:46 PM
Ah, I can probably add a step to copy pants.toml into /usr/src
e

enough-analyst-54434

03/31/2023, 11:46 PM
This is pretty crazy. Thinking...
So git submodules are not allowed? I.E. A framing git repo that uses submodules to pull in the others?
d

dazzling-dress-95246

03/31/2023, 11:47 PM
no, some of our dependencies don't like git submodules for whatever reason, hence why we mount them thru docker
e

enough-analyst-54434

03/31/2023, 11:48 PM
Man - I don't know. This is rough.
So if you copy in a pants.toml that means your Pants config is not really version controlled.
d

dazzling-dress-95246

03/31/2023, 11:49 PM
so the repo mounted at /usr/src/app (Bar) controls the dockerfiles where our images build from pants.toml can probably live next to the dockerfile and hence be version controlled
e

enough-analyst-54434

03/31/2023, 11:49 PM
But just not living where it actually needs to be; thus you need to copy it elsewhere.
d

dazzling-dress-95246

03/31/2023, 11:49 PM
yep
e

enough-analyst-54434

03/31/2023, 11:50 PM
I ... this is pretty nuts.
I am really not sure if Pants can handle this as is.
I'd have to experiment with toy repos.
Have you tried that?
d

dazzling-dress-95246

03/31/2023, 11:51 PM
haha im just toying around with pants right now so I'm willing to just use the actual repos to toy around with if it's low effort
e

enough-analyst-54434

03/31/2023, 11:51 PM
This is a totally unique setup in 13 years of Pants to my knowledge; so "low effort" is totally unlikely and unknown.
d

dazzling-dress-95246

03/31/2023, 11:51 PM
It does sound like the easiest ways are to either: 1. copy pants.toml to /usr/src so it can see everything 2. change the tree structure so everything lives within /usr/src/app
e

enough-analyst-54434

03/31/2023, 11:52 PM
s/easiest/only/
d

dazzling-dress-95246

03/31/2023, 11:52 PM
lol
e

enough-analyst-54434

03/31/2023, 11:52 PM
I can't see other ways.
It would require changes to core Pants to do it in other ways as far as I can tell.
d

dazzling-dress-95246

03/31/2023, 11:53 PM
well I guess I'll see what happens, theoretically this shouldn't be super different from "normal" setups as long as pants sees what it wants to?
needs*
e

enough-analyst-54434

03/31/2023, 11:54 PM
Yeah, as long as pants.toml is at or above the root of source files, you're good.
How you get that to happen in your current setup in a version controlled way seems to require a Rube Goldberg machine with re-locations, etc.
d

dazzling-dress-95246

03/31/2023, 11:55 PM
lol we got in this mess due to a factor of airflow and GCP tooling being super new when we adopted them
e

enough-analyst-54434

03/31/2023, 11:55 PM
One thing that will likely be a huge issue is
--changed-since
and anything else that relies on git.
d

dazzling-dress-95246

03/31/2023, 11:55 PM
and the easiest way to just get things working was this weird amalgamation of repos
e

enough-analyst-54434

03/31/2023, 11:55 PM
You'll have multiple
.git
dirs IIUC.
d

dazzling-dress-95246

03/31/2023, 11:56 PM
ah this is true... the most important repo for changed-since to work on though is /usr/src/app
e

enough-analyst-54434

03/31/2023, 11:57 PM
Yeah, well - I think Pants only looks for
.git
as a sibling of
pants.toml
- I could be remembering that wrong though.
d

dazzling-dress-95246

03/31/2023, 11:57 PM
oh oof
e

enough-analyst-54434

03/31/2023, 11:57 PM
You are just in a world of hurt here.
d

dazzling-dress-95246

03/31/2023, 11:57 PM
I guess I can always try mounting everything under /usr/src/app
so it should only see the one .git folder
though that brings into question what happens when the other repos change
e

enough-analyst-54434

03/31/2023, 11:58 PM
Yeah, exactly.
d

dazzling-dress-95246

03/31/2023, 11:58 PM
would git submodules fix this?
we can always try to see if dependencies have been fixed
e

enough-analyst-54434

03/31/2023, 11:59 PM
I honestly am not sure.
d

dazzling-dress-95246

03/31/2023, 11:59 PM
lemme go see if pants has git submodule docs
e

enough-analyst-54434

03/31/2023, 11:59 PM
It definitely does not.
You really want to be experimenting and not theorizing here.
There are alot of details that will be hard to reason about gedanken-style.
I'd definitely be laying down a few tiny example repos and trying to get it to work 1st if I were debugging this sort of setup possibility.
d

dazzling-dress-95246

04/01/2023, 12:00 AM
yeah that's fair... ill probably mess around this weekend
e

enough-analyst-54434

04/01/2023, 12:01 AM
Yeah, ok. If you can open source an example setup as you work through issues, that will be infinitely easier to support.
👍 1
d

dazzling-dress-95246

04/01/2023, 12:02 AM
are most people not using git submodules?
e.g. to share infra-as-code code with biz logic
e

enough-analyst-54434

04/01/2023, 12:03 AM
I really have no clue. We don't get Slack support questions about them very often if at all that I can recall.
Most Pants users use a monorepo; so there is no such thing as a git submodule by definition.
d

dazzling-dress-95246

04/01/2023, 12:04 AM
yeah, we have mostly a monorepo except for IAC so that permissions are managed separately and a client config repo so we can push configuration updates to production without having to go through develop and then master
hence why we have these 2 in addition 😕
e

enough-analyst-54434

04/01/2023, 12:05 AM
Ok. Sounds complicated!
d

dazzling-dress-95246

04/03/2023, 8:52 PM
FYI I did get things seemingly working by mounting everything under /usr/src/app.
haven't yet tested changed-since
currently trying to get tests to run within a single module and fixing dependencies as it spits out errors