https://pantsbuild.org/ logo
Powered by
b

brash-iron-24542

04/27/2023, 5:16 PM
ok, got a simpler question. When I run 
./pants tailor ::
, it creates 
BUILD
files in unexpected locations. If I have a python project at 
src/apps/foo
, I would expect the file to be at 
src/apps/foo/BUILD
, but instead, it puts it in my module at 
src/apps/foo/foo/BUILD
.
s

sparse-lifeguard-95737

04/27/2023, 5:18 PM
is 
src/apps/foo/foo/
an existing directory containing python source files?
b

brash-iron-24542

04/27/2023, 5:18 PM
@sparse-lifeguard-95737 yes
s

sparse-lifeguard-95737

04/27/2023, 5:19 PM
tailor
will create a 
BUILD
file in every directory with Python code. the auto-generated targets will cover just the files in that dir (not recursive)
you can instead have a 
BUILD
in the root that recursively covers all your sources, but you have to manually write it
b

brash-iron-24542

04/27/2023, 5:20 PM
ahh, ok. Thank you!
@sparse-lifeguard-95737 so I may be misunderstanding the structure here. Should I not embed my project in a subdirectory anymore?
s

sparse-lifeguard-95737

04/27/2023, 5:31 PM
the short answer is: it’s an unrelated/orthogonal choice can you say more about what you think you might be misunderstanding? I’m not 100% sure what might be worrying you, and don’t want to brain-dump about something that isn’t helpful 🙂
b

brash-iron-24542

04/27/2023, 5:35 PM
Haha, I appreciate it. So, I’m brand new to pants. I’m setting up a python monorepo that will have a couple of python projects as well as some shared libraries in it. My python projects generally follow the format 
/project-name/project_name/__init__.py
. With all my other random stuff in 
/project-name/.
TBF, I am coming from a background that doesn’t use a monorepo, so this is all new to me.
Thing is, in this scenario, I guess I’m not gonna have a Dockerfile in each project, and each project is not it’s own git repo, with .gitignore and all that jazz. This is just a new paradigm for me entirely 😆
s

sparse-lifeguard-95737

04/27/2023, 5:38 PM
you definitely could have a Dockerfile in each project, packaging just the code for that project (we have many in my $work repo)
ok will brain-dump and see if anything is helpful 😂
👍 1
pants.toml
is (as far as I can think of right now) the only file whose location matters. it would typically be in the root of your monorepo (not one per project, in the root of the project)
within 
pants.toml
you’ll define your source roots. these typically line up with the entries you’d used to set on 
PYTHONPATH
(i.e. in my repo we used to set 
PYTHONPATH=src
in our “set environment” script. now we have 
/src
as a source root in 
pants.toml
)
the location of 
BUILD
files doesn’t matter to 
pants
. any time you run 
pants
, it looks under your source roots for 
BUILD
files, then looks inside the files at the targets (and at the `source`/`sources` fields set on those targets)
the location of 
BUILD
files does matter when you’re trying to create explicit 
dependencies
between targets. the default of one-BUILD-per-dir lets you do things like:
Copy code
dependenices=[
  "//path/to/your/dir/file.py"
]
if you put a BUILD in a root dir with 
sources=["**/*"]
the explicit references can get messier, like:
Copy code
dependencies=[
  "//path/to/your/dir/file.py:../../../source-target-name"
]
on the flip side, if you have a target that covers many directories recursively, you can set an explicit dependency on that target and get all the enclosed files
that can be useful when you have something like an “assets” directory with a bunch of loose files that all travel together
b

brash-iron-24542

04/27/2023, 5:45 PM
so, the BUILD’s sources are relative to the build file, but the source roots actually define how you access them?
s

sparse-lifeguard-95737

04/27/2023, 5:45 PM
yup!
for python specifically - the 
PYTHONPATH
is affected only by source roots, not by 
BUILD
file placement
b

brash-iron-24542

04/27/2023, 5:47 PM
interesting…. so, let’s say my project requires an external third party package like flask. I would use pants to retrieve that package by creating a python requirement target, right?
s

sparse-lifeguard-95737

04/27/2023, 5:47 PM
yes
if you have `requirements.txt`s already, 
tailor
will create 
python_requirements
targets for you too
b

brash-iron-24542

04/27/2023, 5:48 PM
and then I would get rid of the requirements.txt file and use pants moving forward?
s

sparse-lifeguard-95737

04/27/2023, 5:50 PM
no - 
python_requirements
is a “generator” target (kinda like a macro) that reads your 
requirements.txt
content on-the-fly and produces individual 
python_requirement
targets
similar to how 
python_sources
generates a 
python_source
target for every Python file you have in a directory
so you would either: • manually convert your 
requirements.txt
entries into individual 
python_requirement
targets • keep your `requirements.txt`s and use 
python_requirements
I personally do the 2nd option because it’s more friendly to other tooling in the space (i.e. dependabot)
b

brash-iron-24542

04/27/2023, 5:53 PM
ok, ok, that makes sense. So, if I wanted to use one of my shared internal libraries in a project, I would not have to define it in requirements.txt (or poetry.toml, which is what I’ve used in the past). I would just add it as a 
python_requirement
in the project’s 
BUILD
instead?
s

sparse-lifeguard-95737

04/27/2023, 5:54 PM
is the shared internal library defined in the same monorepo?
b

brash-iron-24542

04/27/2023, 5:54 PM
yes
s

sparse-lifeguard-95737

04/27/2023, 5:55 PM
then - yes to not listing in requirements.txt/poetry.toml, but no to making it a 
python_requirement
. let me whip up an example
b

brash-iron-24542

04/27/2023, 5:55 PM
it uses a 
python_library
target, but that seems to be deprecated from what I can tell
s

sparse-lifeguard-95737

04/27/2023, 5:56 PM
yeah that’s older
say you have:
Copy code
/
  pyproject.toml
  projects/
    project1/
      __init__.py
      main.py
  libs/
    __init__.py
    lib1/
      __init__.py
      util.py
    lib2/
      __init__.py
      util.py
in 
pyproject.toml
you could have:
Copy code
[source]
root_patterns = [
  "/projects",
  "/libs",
]
with that in place, you could run 
pants tailor
to generate 
BUILD
files. by default it will generate: • 
projects/project1/BUILD
• 
libs/lib1/BUILD
• 
libs/lib2/BUILD
the content of each 
BUILD
file will be identical boilerplate:
Copy code
python_sources()
with all of that in place, you’ll be able to import between the various python files. i.e. you could write the following in `projects/project1/main.py`:
Copy code
import lib1.util
import lib2.util
Pants v2 includes dependency inference for python. when you 
import
a module in a file covered by 
python_sources
, Pants will compare that module path against all the other 
python_sources
in the repo and see if it can find a match. the matching runs relative to source roots (which is why 
lib1.util
would match to 
libs/lib1/util.py
, because 
/libs
is declared as a source root)
the upshot of all this is - if you have internal libraries in your monorepo, importing them from your projects/services should Just Work
b

brash-iron-24542

04/27/2023, 6:05 PM
that’s pretty cool. So, when I create a distribution of all this, or a docker container, pants then figures out the dependencies and packages them all up for the target?
s

sparse-lifeguard-95737

04/27/2023, 6:08 PM
it (unfortunately) varies depending on the type of target. dependencies between `python_source`s is implemented. 
docker_image
targets can infer dependencies on 
pex_binary
targets, but not raw `python_source`s (at least not yet)
the common pattern is: 1. 
tailor
all your 
python_source
targets 2. create a 
pex_binary
target per “entry point” in your monorepo, pointing at the relevant 
main.py
3. write a 
Dockerfile
that `COPY`s the 
pex_binary
into your image, instead of copying in all the raw sources 4. 
tailor
or manually write a 
docker_image
target covering your 
Dockerfile
here’s a more up-to-date example: https://github.com/pantsbuild/example-docker
b

brash-iron-24542

04/27/2023, 6:11 PM
hmm. So, my python newbie-ishness is gonna show here… if I have a pex binary in my docker image, can I also mount the source files into the docker container so that local changes are sync’d when in development?
would I just mount the relevant directories and override the module importing with a 
PYTHONPATH
?
s

sparse-lifeguard-95737

04/27/2023, 6:14 PM
if I’m understanding correctly - no, it’s not going to work that way
there might be an alternate approach that fits the same need
pants 2.15 introduced the concept of “environments”, which lets you execute `test`/`lint` processes within a container instead of on your local host machine: https://www.pantsbuild.org/docs/environments
you could carve the system dependencies etc. needed in both your dev and prod environments into a base image, and then use that base as both: 1. the image you use for your environment 2. the base image for any production `Dockerfile`s built by Pants
b

brash-iron-24542

04/27/2023, 6:19 PM
sorry, I may have mis-spoke. So, we have an existing stack that runs in k8s (in dev). For our main ruby project, plus some random node projects, we create a docker image, fire up the pod with that image, then mount the local host machine’s project directory into the running pod. That way you can edit code and have your changes reflected without having to rebuild your image every time, and can still run the project in the context of the dev cluster (to access the other services).
So I’m not thinking of building pants inside of the container, as much as I am thinking about the dev cycle itself.
s

sparse-lifeguard-95737

04/27/2023, 6:22 PM
🤔
I think yes, you can still have that experience, if you follow a certain pattern in the images
❤️ 1
one sec…
by default a 
pex_binary
target produces a zipapp archive. that archive extracts itself on first run to a content-addressed directory, so it can be tough to find the files in the running container
you can override the settings so the pex produces a directory instead of a zip. from there you can additionally run a command to convert the pex to a “normal” virtualenv, and have that be the thing in the container
if you did that, I’d expect you could recreate your existing dev flow since they’re “just” python files in a virtualenv
b

brash-iron-24542

04/27/2023, 6:28 PM
that’s awesome! Now, in docker containers I don’t necessarily need a virtualenv, right? At least I’ve never done that before. Is that what pants does automatically?
s

sparse-lifeguard-95737

04/27/2023, 6:30 PM
yes - in general pants and pex are all about hermeticity, so they won’t mess with your system python’s dependencies. so even though you don’t generally need a virtualenv within a container, that’s what you’ll end up getting if you follow the pattern 🙂
b

brash-iron-24542

04/27/2023, 6:32 PM
fair enough! Thank you so much for this explanation. You’ve really helped me out. I’ve been stuck for literally days on trying to figure out what to do with this. First went about it the way we always have before by creating separate repos for each project and each lib. But dependency management was becoming a mess, and getting it working in our dev setup was terrible.
s

sparse-lifeguard-95737

04/27/2023, 6:34 PM
np feel free to drop into #general with more questions as you go - it sounds like you’re trying to replicate something decently-complex so I’d expect some speed-bumps along the way, but nothing here stands out as impossible to me
b

brash-iron-24542

04/27/2023, 6:35 PM
thank you!
b

breezy-mouse-20493

04/27/2023, 9:23 PM
Just read this thread. Excellent information. @sparse-lifeguard-95737 hero! 🦸
h

happy-kitchen-89482

04/29/2023, 1:17 AM
@sparse-lifeguard-95737 ++! Thanks for the excellent explainer
Which we should turn into a post
2 Views
Powered by