https://pantsbuild.org/ logo
Join Slack
Powered by
What is the good way to structure a python monorep...
# general
c
What is the good way to structure a python monorepo so that mypy works well with test code? How should the roots be setup? How should the test utils be configured? The examples I found are for apps without tests, and I have issues with all variants that I tried. For example, if "tests" is not a package, mypy wants unique names of the files across all applications. If it is a package, then there are import problems. This is the structure I have
Copy code
apps
├── demo_app
│   ├── BUILD
│   ├── pyproject.toml
│   ├── src
│   │   └── demo_app
│   │       ├── BUILD
│   │       ├── __init__.py
│   │       ├── main.py
│   │       └── model.py
│   └── tests
│           ├── BUILD
│           ├── test_demo_app_main.py
│           └── util.py
and the roots are
Copy code
apps/demo_app/src
apps/demo_app/tests
I guess the question is related to @wide-midnight-78598’s one about more complete examples of how to setup a more complicated repo. While it's interesting to get through the process of figuring it out oneself, it is a lot of wasted time if everyone has to do that, and it might scare off newcomers.
👍 1
e
What are the import errors you see?
c
It's a bit difficult to show everything here, I'll try to put together a demo repo
w
@cold-vr-15232 If you can show the import errors from 
mypy
- as I mentioned in another thread, like 90-95% of my mypy problems were solved with init.py's or using namespace packages. Not suggesting blindly that will work here, but it's been my Pants mypy go-to
While it's interesting to get through the process of figuring it out oneself, it is a lot of wasted time if everyone has to do that, and it might scare off newcomers
I agree with this sentiment - personally, I want to figure out how to solve these kinda things so that when they inevitably crop up again, I know where to start. But, if it's something that only crops up once every few months, then 🤷 The other side is - as interesting as it may be, fiddling with a build system isn't revenue-generating...
b
FWIW the pants repo and our work repo use test files next to prod code ( E.g. 
test_foo.py
next to 
foo.py
) and we don't have any issues
w
I use both - unit tests are placed alongside code, integration/e2e code is separated
c
Ok, I think I figured it out. The problem was that either mypy or pytest could not find files or found conflicting files. And I got it fixed by using different package names under src and tests, for example "src/app1" and "tests/app1_test". Not the prettiest, but works now. I haven't tried having the test code in the same place. I suppose pants recognizes that and doesn't package test file in the binary? How would you handle test utilities?
b
Pants'dependency graph is precise. Unless you're importing test code, it'll never be packaged. Try it out! 
./pants dependencies <target or file>
c
Then maybe I will switch to using tests alongside code too. Even integration tests could live under src, I suppose. Thanks! (A downside is that the whole project is now dependent on pants 🙂 )
😈 2
w
pytest
(and I think 
mypy
and other tools) will happily run against tests next to code. That's how I've been running everything in the pre-pants years.
But, maybe some tools will need to be globbed away or something... hmm
I suppose pants recognizes that and doesn't package test file in the binary
I think I missed this - in your 
BUILD
, if you have 
python_sources
and 
python_tests
- you're fine. As Josh mentioned, unless you actively import test code (either in your 
.py
files or your 
dependencies
field in BUILD), it won't/shouldn't be packaged
8 Views
Open in Slack
PreviousNext
Powered by