https://pantsbuild.org/ logo
Join Slack
Powered by
I’ve been trying to set up type checking in my mon...
# general
g
I’ve been trying to set up type checking in my monorepo. Found some strange mismatched behaviour between two similar roots of code. Both are FastAPIs with their own unique 
requirements.txt
files each included in the 
BUILD
file with the 
python_requirements()
code. One is called 
api
and the other is called 
qa
. Here are their structures:
Copy code
src/api/
├── BUILD
├── Dockerfile
├── app
│   ├── __init__.py
│   └── api.py
├── main.py
└── requirements.txt
and:
Copy code
src/processors/qa/
├── BUILD
├── Dockerfile
├── app
│   ├── __init__.py
│   ├── api.py
│   └── qa.py
├── main.py
└── requirements.txt
The 
qa
one loads all of its requirements from pip great in both mypy runs and repl, but the 
api
one completely misses installing them. Both have the same 
BUILD
file with just the name swapped out:
Copy code
python_requirements()

python_library(
    name="<name here>",
    sources=["**/*.py"],
    interpreter_constraints=[">=3.6"],
)
Anyone have any ideas why one loads packages and the other doesn’t? Or a handy command I can use to help debug?
h
Hmmm, are you able to share a (maybe sanitized) version of the repo?
It looks like you're not specifying those 3rdparty dependencies explicitly, but rather allowing Pants to infer them for you, which is great.
g
I’m afraid it’s in a private codebase I don’t have permission to share.
h
No worries, we'll figure this out another way!
There is probably some overlap between those two 
requirements.txt
files?
If so that may be confusing the dep inference logic
g
Interesting that the auto dependency works perfect for one and not the other. Do you think its possible there might be some kind of name collision with me using the name 
api
alot?
h
First step is to double check that the dependencies are showing up by running 
./pants dependencies path/to/app.py
. You should see all first party files + third party requirements used directly in app.py. Are you seeing none, or only missing some?
h
I can't say for sure but I don't think that is likely to be the issue.
g
api/requirements.txt:
Copy code
celery[redis, librabbitmq]==4.4.0
fastapi==0.49.0
uvicorn==0.11.7
pydantic==1.4
qa/requirements:
Copy code
transformers[torch]==3.3.1
pytest
fastapi
@hundreds-father-404 Will check that now
h
An idiomatic way to work with requirements.txt with Pants is to have a single one for your repo (assuming there aren't conflicting versions) and let Pants infer which requirement subsets are actually needed in each case (for this to be performant you probably want a lockfile, see https://www.pantsbuild.org/docs/python-third-party-dependencies).
👍 1
h
(It is still valid to have multiple requirements.txt, but indeed often more idiomatic to have only one and let the tool handle it)
h
But putting that to one side, you can try adding explicit deps on the 
api
package's 
python_library()
, e.g., 
dependencies=['src/api:celery', 'src/api:uvicorn']
just to see if that helps.
g
api dependencies:
Copy code
src/api/app/__init__.py:../api
src/api/app/api.py:../api
src/api/main.py
src/api:uvicorn
src/redthread/redthread/__init__.py:../redthread
src/redthread/redthread/models.py:../redthread
qa dependencies:
Copy code
src/api:uvicorn
src/processors/qa/app/__init__.py:../qa
src/processors/qa/app/api.py:../qa
src/processors/qa/app/qa.py:../qa
src/processors/qa/main.py
src/processors/qa:transformers
src/redthread/redthread/__init__.py:../redthread
src/redthread/redthread/models.py:../redthread
h
interesting that 
qa
depends on 
src/api:uvicorn
g
I think that might be best. Interestingly the 
api
root did find the uvicorn
👍 1
Because this is a dockerized project, the multiple requirements might be confusing the auto discovery
h
My next question is if you need to set up a module mapping if those other reqs export modules different than their project names. See the top of those docs on third party deps
g
I think the quick fix is to set the python_library dependencies as @happy-kitchen-89482 recommended. I’ll report back
👍 1
Changed the 
api/BUILD
file to:
Copy code
python_requirements()

python_library(
    name="api",
    sources=["**/*.py"],
    interpreter_constraints=[">=3.6"],
    dependencies=[
        "src/api:celery", 
        "src/api:fastapi", 
        "src/api:uvicorn", 
        "src/api:pydantic"
    ]
)
Which leads to a successful repl environment (all libraries importable with the correct versions as laid out in 
api/requirements.txt
) 🎉
The duplication of requirements isn’t ideal but I can live with it for now.
h
Okay I put up a reproduction at https://github.com/Eric-Arellano/debug-3rdparty-dep-inference-liam. Could you please check if I’m missing any details from your setup? It’s expected that 
fastapi
won’t show up with dep inference. You’re defining the requirement in two different places, and Pants’s dependency inference can’t disambiguate which of the two you want to use. It assumes you don’t want both. For fastapi, you’d either need to consolidate around one requirements file, or a shared req file for common deps, or use explicit dependencies for that one I’m not sure why celery, pydantic, and uvicorn fail for you. I’d love to figure out what’s happening
Copy code
▶ ./pants dependencies src/api/main.py
src/api:celery
src/api:pydantic
src/api:uvicorn

▶ ./pants dependencies src/processors/qa/main.py
src/processors/qa:pytest
src/processors/qa:transformers
g
@hundreds-father-404 Got a new branch with a redacted, but import identical, copy of the code base in a new branch ready to push. Need write access to the repo. As soon as I have that all push it up right away. Sorry for the quirkiness on this, my lab is annoyingly insistent on IP protection 😅
h
Sounds good! Indeed, all that is relevant here is your BUILD files, pants.toml, requirements.txts, and the import statements. All actual code can be redacted - it's ignored by dep inference I DMed for your GitHub username and will add you to the repo
Thanks for posting that. Some notes: * 
fastapi
won’t work with dep inference due to two targets for it existing, as noted above * Both 
src/api/main.py
and 
src/processors/qa/main.py
fail to detect imports of the module 
app.api
. This is expected. Your project has two modules called app.api: 
src/processors/qa/app/api.py
and 
src/api/app/api.py
. You correctly set up the “source roots” to be 
src/processors/qa
and 
src/api
, which means those paths get stripped from the module name. But now, you have two different files with the same name, and Pants can’t figure out which you want to use. To fix, you’d either need explicit 
dependencies
, or to change how your imports and source roots are structured - would it be feasible to import 
processors.qa.app.api
and 
api.app.api
, rather than 
app.api
? You’d set the source root pattern to 
/src
. We recommend the latter, as it gives you namespacing so that there’s zero ambiguity what refers to what * Inference of 
celery
is working for 
./pants dependencies src/api/app/api.py
* I don’t see any imports of Pydantic, so Pants can’t infer it’s used. Do you expect to import it? * You import 
starlette
in 
src/api/app/api.py
, but it’s not used. Did you mean to include starlette in the requirements.txt?
g
Thanks for the recommendations with this! The redaction answers the starlette dependencies. I’ll look to clarify the import structure for the dep inference
👍 1
h
Hey @glamorous-river-64734, checking on how the above is going and if there’s anything we can help with?
n
Just a note that I came across this today while looking for a way to run tests across multiple lambda functions which each have their own 
requirements.txt
with overlapping requirements. It would be great if this was documented (esp. the gotcha around "it assumes you don't want both") Happy to raise a ticket if that's appropriate. It sounds like moving to a single requirements file will work for us in the long run.
h
Thanks for the feedback @nutritious-hair-72580 ! I'll take a look at the documentation. Where in the docs would you most expect to see this info? I should mention that we do plan to support multiple overlapping 
requirements.txt
, although of necessity that will require annotating your targets to say which "universe" of requirements they are allowed to dip into. So if you are able to have a universal set of mutually compatible requirements for your repo, that will always make your life easier, even if we do support more complicated arrangements.
Note that Pants will select only the necessary subset of those requirements in each case, so adding things to a single central requirements.txt doesn't have the implication of adding unnecessary dependencies to every lambda.
👍 1
n
Thanks @happy-kitchen-89482. I think here would be good. https://www.pantsbuild.org/docs/python-third-party-dependencies I think it might also be worth calling out that source roots need to be setup as well somewhere in the python section as well.
h
Thanks for the feedback, will take a look. In future note that there is also a Suggest Edits button on every documentation page, in case that is easier.
👍 1
BTW do your overlapping requirements conflict, or are they just repetition? Because if the latter then you could refactor them out and then include them in each requirements.txt with 
-r common-requirements.txt
n
I hadn't thought of that .. but i guess that would make the docker image a bit larger (we are using container images for lambda deployment). We would probably use a common base image for common requirements in that case.
h
Why would that make the docker images larger?
Oh I see because today not every lambda depends on everything in that common set?
👍 1
So this is where deploying pex files into your docker images is handy - Pants builds the pex with just the requirements you actually need, a subset of your total 
requirements.txt
.
So the pex, and therefore your docker image, is only as large as it needs to be.
n
Ok thanks - yes that would work.
4 Views
Open in Slack
PreviousNext
Powered by