https://pantsbuild.org/ logo
Join Slack
Powered by
:wave: Is there a way to reference `pytest` in the...
# general
g
đź‘‹ Is there a way to reference 
pytest
in the 
dependencies
parameter of a 
python_tests
target?
Copy code
python_tests(
    dependencies = [
        "tests/my/special/test:lib",
        "3rdparty/python:pytest", # <-- I have test sources that `import pytest` so I think the type checker wants to see an explicit dependency stated here…
    ],
)
p
if you have dependency inference turned on you shouldn't need to do that. try remove the explicit dependency and see if pants can see that dependency by running: 
./pants dependencies <path to python test file>
h
This was a bug last time it came up in here. Not sure which release the fix was slated for.
p
never encountered that but in our (toolchain) internal repo.... but maybe I am missing something.
h
Oh actually it was an issue when linting test files. Running them should not need it (as you’ve said).
p
we use pylint and mypy both will complain if a 3rd party dep is missing. I am 99% pants will include those deps in the env it runs those tools in.
h
@glamorous-cpu-22971 is pytest in your requirements.txt, or however you're generating requirements?
If not, that would explain the issue
g
Hey @happy-kitchen-89482 - it’s not in requirements.txt (yet) although I do see it specified in the default pytest lockfile…
I hesitated to add pytest to the requirements.txt file because that list seemed to be used for application dependencies rather than test dependencies.
It looked like there was a specific version of pytest being used as the test runner, and I didn’t know if it was important that the version of pytest my test sources imported used the same version as the runner. In practice it obviously didn’t matter, but I’m trying to learn the “right” way to do things in a pants world.
When I’ve set up projects with pip I would often have two requirements files, one for the application sources (conventionally requirements.txt) and for the test sources (conventionally requirements-dev.txt)
p
you don't need to do that when using pants. you can have a single 
requirements.txt
and when creating deploy-able targets (like wheels, pex files, etc...) pants will only include the needed deps. (because of dep inference). This is not how most stuff works in the python eco-system, since in a lot of projects you have multiple reqs files trying to manage the different things that needed to be done with each (generating docs, tests, linters, etc...) it is a PITA to manage those and Pants makes it so much easier. one reqs file for the entire repo and pants only takes the actual things that are needed from the file.
g
Nice! This makes total sense.
And it sounds like the path forward is clear, but just to confirm, even though 
pants
2.14 does seem to bundle in 
pytest
at version 7.0.1 to run the tests, it’s not possible to reference this bundled version when you’re explicitly doing 
import pytest
inside your test sources (we’re doing it to import an assertion or something)
So you have to sort of “re-declare” an explicit dependency on 
pytest
if you use it in your test sources.
p
I recently had work with this SW https://gitlab.com/BuildGrid/buildgrid and it use the multiple reqs file way/ such a PITA to manage that https://gitlab.com/BuildGrid/buildgrid/-/tree/master/requirements I got used to working with pants and forgot how painful it can be to deal with that way of ding things.
🔥 1
you can reference pytest... this is what 
import pytest
does if you need a specific version pytest (since you need specific APIs) you may need to also specify that version under the pytest options here: https://www.pantsbuild.org/docs/reference-pytest#version
there is also https://www.pantsbuild.org/docs/reference-pytest#extra_requirements where you can specify pytest plugins.. which will be available to all tests in the repo.
g
I guess my question is how to do specify that in the 
dependencies
field of my 
python_tests
target.
(see example at thread root)
p
you shouldn't need to do that.
g
The 
import
is getting resolved successfully when the tests run - my type checker is complaining, that’s the issue.
It sees an 
import
statement with no corresponding dependency in the graph (or I assume that’s whats happening)
p
that sounds like a bug. removing that dep from the python_tests target and running 
./pants dependencies
does pants see the pytest dep ?
are u using mypy ?
g
Yes
(and that dependency isn’t real, I just don’t know what to put there)
p
you can tell mypy to ignore missing imports (we do that internally, I just checked)
Screenshot 2023-01-17 at 11.47.34 AM.png,Screenshot 2023-01-17 at 11.47.43 AM.png
g
Yup, for sure - I guess I was wondering if there was a way to do that without silencing the warning.
I understood the warning to mean: “hey, you’re importing something i don’t see in the graph”
p
yea... that sounds like an actual bug in pants mypy goal implementation.
g
I’m not sure if it is a bug or not… genuinely (I’m a newbie)
It seems like the type checker is working as intended
I don’t think my test target has any kind of explicit dependency on pytest
And the type checker is just surfacing that reality.
But I’m not trying to argue any particular point here, I’m just trying to figure out how this works!
FWIW adding the explicit dependency on 
pytest
in requirements.txt fixed the type checker warning.
So 
pants
scans the Python sources, if it finds an 
import xyz
statement, it creates a child node on the dependency graph with the label 
xyz
and then it looks in its 
python_requirements
to see if there are modules that map to 
xyz
? Is that correct?
h
So stepping back, you can have a 
requirements-dev.txt
if you want (just wrap it in a python_requirements() target), but you don't need to - generally it's idiomatic to have a single requirements.txt with everything in it, since Pants will subset that for you. At test time it'll only take the dependencies needed for that test, at pex-building time it'll only bundle what is actually used into the pex, and so on.
And the way it does this is through dependency inference (plus any explicit deps)
dependency inference works exactly as you suggest: it maps 
import xyz
to 
whatever provides xyz
. That can either be an internal .py file in your repo, or an external requirement.
So if you put 
pytest
in your 
requirements.txt
, pants will notice that it now has something that provides the 
pytest
module, and do the mapping.
To add to the confusion, there is the pytest lockfile... That represents "pytest as a tool". There is also "pytest as an importable library", which is what I described above.
đź‘Ť 1
I am slowly working towards unifying the "tool lockfile" and "user lockfile" concepts, so that you don't have to specify pytest in two places.
But for now, alas, you do, once as a tool, once as a library
Does that all make sense?
And when you say "adding the explicit dependency on pytest in requirements.txt fixed the type checker warning" do you mean just adding pytest to requirements.txt, or also adding an explicit dep on it from the python_tests target? The latter should not be necessary
g
Does that all make sense?
It really does make sense. I’m coming from recent context in Bazel so these concepts map pretty well.
when you say “adding the explicit dependency on pytest in requirements.txt fixed the type checker warning” do you mean just adding pytest to requirements.txt, or also adding an explicit dep on it from the python_tests target?
It was just adding 
pytest
to the requirements.txt file, once that resolved the issue I got an inkling of how the dependency inferencing must be working.
I didn’t add anything to the 
python_tests
target.
This was a super helpful explanation @happy-kitchen-89482 and @polite-garden-50641. Thank you both for taking the time to walk me through it.
h
Cool, glad this makes some sort of sense
🙌 1
4 Views
Open in Slack
PreviousNext
Powered by