Recently I posted issue #15543 on GitHub (the slac...
# general
Recently I posted issue #15543 on GitHub (the slack link was not functioning at that time) where I indicated difficulties we have with running
pants test
on a monorepo with several independent Python subprojects. In a nutshell, some of the tests for a subproject will only succeed if the subproject is pip installed (or something equivalent) -- we use e.g.
to verify the installed version. The reply I got on the issue helped me somewhat further, but I am still stuck. The layout of the subprojects is as follows
Copy code
We use poetry-style dependencies, documented in
. When running tests manually, we simply activate the subproject's virtual envirnment, poetry install the subproject, and run pytest. I understand that pants uses a different approach, and does not by default install the subproject in a virtual environment, only the dependencies that that subproject has. I was told that to install the subproject, I would need to create a
target, and have the tests depend on that. This is where I get stuck. As far as I can see from experimenting, I need to define the distribution target in the top-level BUILD file (the one under
. But I seem to be unable to declare a dependency on that target from within the BUILD file under
. When I specify
with that target name I specified in the top-level BUILD file, I get a ResolveError that the target was not found in the my-project/test namespace. And when I prefix the target with the full
prefix, I get a ResolveError like
file 'src/python/my-project/src/python/my-project'
does not exist. Any help in setting up the dependency between the tests and the distribution target would be greatly appreciated.
you can run
./pants list $dir:
to see the names of targets in a directory
then, use that target name to add an explicit dependency to
if you’ve defined something at the root of the repository, you can see its name with
./pants list :
Thanks for the hint. This weekend is a holiday weekend here in the Netherlands, and next week I'm away. But my teammates are following this thread, so hopefully there'll be some progress when I return.
👍 2
My wild guess is you need to provide the path relative to pants in
, if the test target is below the
runtime_package_dependencies = ["src/python/my-project:my-project-dist]
This is bit of gotcha when accessing targets which aren't in sub directories.
Thanks for the help and suggestions! I managed to make the test depend on a python distribution. However the test fails to pick up some dependencies of the python distribution. The dependencies of the test file are correctly listed by
./pants dependencies --transitive
, however
./pants test ::
fails with
ProcessExecutionFailure: Process 'Building pytest_runner.pex' failed with exit code 1.
saying it Failed to resolve some of the requirements that were visible in
./pants dependencies
see for more info and for a minimal github repo reproducing the bug.
I don't think you should be using
resources It should be
python_sources I would avoid it completely and use the target generated under which would be called
and you can add it as dependency directly in dist dependencies as
Also you don't need to add explicit dependency on
Now it seems only this is the issue which is breaking the test I am not sure if this the correct way to read the version. I always struggle with this but there are some discussions around how to properly read/set version EDIT: Not sure if this is because of pants or python :/
hi @refined-addition-53644, Thanks for the feedback. The suggestion to use
as dependency indeed makes things a lot cleaner. The explicit dependency on
however is something we do need, it is something we directly got from the pants documentation (see pep 517 on ), and if I remove it the distribution doesn't get build correctly since the pyproject.toml is then not included in the chroot where the distribution is build, and hence all the essential configuration for building the project is missing. You can see the result of this in the following:
Copy code
pants_example maartenderickx$ ./pants package src/python/my-subproject:my_subproject_distribution
13:13:38.28 [INFO] Completed: Building build_backend.pex from setuptools_default_lockfile.txt
13:13:38.78 [INFO] Wrote dist/UNKNOWN-0.0.0-py3-none-any.whl
as you can see without the explicit dependency on pyproject.toml pants doesn't even know where to get the correct name from in order to generate the wheel.
The reason the test fails is probably because you removed the dependency on pyproject.toml, this causes the python my_subproject_distribution to be build incorrectly, and hence also my_subproject not to be installed when running the test. We only added the aforementioned test since some tests in our testsuite really depend on the project to be properly installed, and the above way was just a quick hacky way to test if the project is installed. I have added a test case which is less of a hack and more clear of what it should test:
Copy code
def test_installed() -> None:
    spec = importlib.util.find_spec("my_submodule")
    assert spec is not None
Of course our end goal is not necessarily to have our submodules installed during testing, however we do have some tests that at least at the moment require some of our submodules to be installed in order for them to pass. We are also looking into ways to make our tests pass without the package being installed, but in our migration process to pants we were trying to first just making pants work with our code as is, and later trying optimising so that tests don't depend on an installed package.