https://pantsbuild.org/ logo
#general
Title
# general
e

early-jelly-10468

01/05/2023, 6:21 PM
Hi everyone, I'm investigating pants to see if it would work with my company's codebase as we move from a python polyrepo to a monorepo. I'm trying to build out a prototype using our code structure and running into some issues getting tests to run. I saw an issue @curved-manchester-66006 brought up yesterday and it led me here looking for any guidance.
Copy code
root
├── service_1
│   ├── app_code_1/
│   │   └── module_123.py
│   ├── tests/
│   │   ├── conftest.py
│   │   └── test_app_code_1
│   │       └── test_module_123.py
│   ├── conftest.py
│   └── requirements.txt
├── service_2
│   ├── app_code_1/
│   │   └── module_123.py
│   ├── tests/
│   │   ├── conftest.py
│   │   └── test_app_code_1
│   │       └── test_module_123.py
│   ├── conftest.py
│   └── requirements.txt
└── n...
Our code is laid out in this manner (omitting the pants files) with each service in its own directory and a separate directory for tests and the application code. If I run a
./pants test ::
it successfully picks up all the tests from each service, but they all fail due to missing 3rd party imports. I ran
./pants --filter-target-type=python_requirement list ::
and it looks like it is finding all the imports from the requirements.txt
Copy code
projects/service_1:reqs#pytest
projects/service_1:reqs#etc
n...
Makes me think something about how our code is organized is preventing the tests from running using the requirements file. I was also expecting that I would be able to do something like
./pants test /service_1/:
but that does not pickup the test directory contained inside the service, further making me think the organization is messing it up. Any guidance or direction would be helpful, I've been poking around the settings/configurations for about a day and haven't made any headway 🙂. Thanks for anyone's time
p

plain-night-51324

01/05/2023, 6:30 PM
mine was caused by multiple requirements.txt used in python_requirements(), not sure if it applies to you
w

witty-crayon-22786

01/05/2023, 6:42 PM
@early-jelly-10468: welcome! which version of Pants are you using? and are you seeing warnings about unambiguous or unowned dependencies imports?
e

early-jelly-10468

01/05/2023, 7:06 PM
Using 2.14.0 (happy to change if there is a different one); and yes! I actually am at the top, it was getting drowned out by the other following errors
Copy code
Seems like they all follow this general format 

The target {dir}:tests imports `{one of my functions}`, but Pants cannot safely infer a dependency because more than one target owns this module, so it is ambiguous which to use: [list of places]
Am I correct in thinking this is being caused by the same named function/import path existing in multiple of my services?
w

witty-crayon-22786

01/05/2023, 7:07 PM
that’s correct: the same import path
those warnings indicate that Pants knew about an import, but couldn’t actually add a dependency: they will almost always become actual import errors at runtime
e

early-jelly-10468

01/05/2023, 7:12 PM
So for my understanding, pants is trying to work out which function the file actually depends on but it cannot tell the difference between The functions located:
root/service_1/module_1.py/function_1
root/service_2/module_1.py/function_1
Because inside a file
root/service_2/module_2.py
the import statement is just
from module_1 import function_1
. Correct?
If I'm understanding correctly and reading the logs correctly I need to specify which dependency is allowed. Right now, I have no cross service imports, is there a way to specify not to use
service_1/
for everything in
service_2/
? The logs make it seem like I'd have to go write out the dependencies in every single build file with an ambiguous import (there are a lot 😞)
c

curved-manchester-66006

01/05/2023, 7:29 PM
At the risk of confusing the issue, what does
./pants roots
show?
e

early-jelly-10468

01/05/2023, 7:33 PM
default installation it was just
.
, i messed with it some and got it to show all the different services because I thought that might have been part of the problem. After using a marker file I got it to read:
Copy code
.
/root/service_1
/root/service_2
...n
Did not seem to impact the issue though
c

curved-manchester-66006

01/05/2023, 7:45 PM
Can you post a sanitized version of the error you are getting?
e

early-jelly-10468

01/05/2023, 7:45 PM
Up a few blocks
h

happy-kitchen-89482

01/05/2023, 7:51 PM
This comes up from time to time. I think it makes sense to (optionally) resolve ambiguity by preferring the symbol provider within the same source root.
That would not be hard to do, I think
w

witty-crayon-22786

01/05/2023, 8:04 PM
yea, i would be fine with that behind a flag.
e

early-jelly-10468

01/05/2023, 8:09 PM
So sounds like I am understanding correctly, so without that enhancement is there a current simple workaround? Or are my options: 1. Go set dependencies manually in every build file with a conflict 2. Rename functions to not clash
w

witty-crayon-22786

01/05/2023, 8:12 PM
rather than functions it would be modules, but yes: other than that that is the solution currently. for testing purposes, you could also ignore/exclude some projects from being built with Pants… but that won’t help you in production obviously
e

early-jelly-10468

01/05/2023, 8:13 PM
Got it, thanks for yalls help!
w

witty-crayon-22786

01/05/2023, 8:14 PM
@happy-kitchen-89482: maybe you could file a ticket for the feature you’re thinking of to resolve this? i think that i have a rough idea of what it would look like, but it’s possible that a contributor would be able to implement it
h

happy-kitchen-89482

01/05/2023, 8:41 PM
w

witty-crayon-22786

01/05/2023, 9:14 PM
thanks! yea, that should be fairly easy to implement
h

happy-kitchen-89482

01/05/2023, 9:47 PM
@early-jelly-10468 I think this would solve the issue for you, if I'm understanding your case correctly?
e

early-jelly-10468

01/05/2023, 9:50 PM
Yeah I think it would, based on our interpretation of my problem. Lots of our services work with the same types of real world objects so having the same name for modules is pretty common across services.
h

happy-kitchen-89482

01/06/2023, 11:12 AM
@early-jelly-10468 When you have the same name for modules that represent the same types of real world objects, do they contain the same/similar code? One of the reasons to have a monorepo might be to get rid of that sort of duplication.
e

early-jelly-10468

01/06/2023, 6:30 PM
@happy-kitchen-89482 No they do not. For example we deal a lot with "orders" and we have a pricing service that had code around pricing etc. But another service for invoicing. You end up with code that looks like this.:
Copy code
pricing
└── order
    └── api.py
invoicing
└── order
    └── api.py
For N number of things. Code is totally different but the context is all around the same business object so thats how it was organized
h

happy-kitchen-89482

01/06/2023, 7:11 PM
Makes sense
3 Views