https://pantsbuild.org/ logo
#general
Title
# general
h

hallowed-church-39499

05/25/2022, 12:34 PM
Recently I posted issue #15543 on GitHub (the slack link was not functioning at that time) where I indicated difficulties we have with running
pants test
on a monorepo with several independent Python subprojects. In a nutshell, some of the tests for a subproject will only succeed if the subproject is pip installed (or something equivalent) -- we use e.g.
importlib.metadata.version()
to verify the installed version. The reply I got on the issue helped me somewhat further, but I am still stuck. The layout of the subprojects is as follows
Copy code
src/
  python/
    my-project/
      my_project/
        BUILD
        __init__.py
    	main.py
    	other.py
      tests/
    	BUILD
        __init__.py
    	test_my_project.py
      pyproject.toml
      BUILD
We use poetry-style dependencies, documented in
pyproject.toml
. When running tests manually, we simply activate the subproject's virtual envirnment, poetry install the subproject, and run pytest. I understand that pants uses a different approach, and does not by default install the subproject in a virtual environment, only the dependencies that that subproject has. I was told that to install the subproject, I would need to create a
python_distribution
target, and have the tests depend on that. This is where I get stuck. As far as I can see from experimenting, I need to define the distribution target in the top-level BUILD file (the one under
my-project
. But I seem to be unable to declare a dependency on that target from within the BUILD file under
tests
. When I specify
runtime_package_dependencies
with that target name I specified in the top-level BUILD file, I get a ResolveError that the target was not found in the my-project/test namespace. And when I prefix the target with the full
src/python/my-project
prefix, I get a ResolveError like
file 'src/python/my-project/src/python/my-project'
does not exist. Any help in setting up the dependency between the tests and the distribution target would be greatly appreciated.
w

witty-crayon-22786

05/25/2022, 5:49 PM
you can run
./pants list $dir:
to see the names of targets in a directory
then, use that target name to add an explicit dependency to
runtime_package_dependencies
if you’ve defined something at the root of the repository, you can see its name with
./pants list :
h

hallowed-church-39499

05/25/2022, 8:15 PM
Thanks for the hint. This weekend is a holiday weekend here in the Netherlands, and next week I'm away. But my teammates are following this thread, so hopefully there'll be some progress when I return.
👍 2
r

refined-addition-53644

05/25/2022, 8:25 PM
My wild guess is you need to provide the path relative to pants in
runtime_package_dependencies
, if the test target is below the
python_distribution
So
runtime_package_dependencies = ["src/python/my-project:my-project-dist]
This is bit of gotcha when accessing targets which aren't in sub directories.
w

worried-boots-82926

05/27/2022, 5:06 PM
Thanks for the help and suggestions! I managed to make the test depend on a python distribution. However the test fails to pick up some dependencies of the python distribution. The dependencies of the test file are correctly listed by
./pants dependencies --transitive
, however
./pants test ::
fails with
ProcessExecutionFailure: Process 'Building pytest_runner.pex' failed with exit code 1.
saying it Failed to resolve some of the requirements that were visible in
./pants dependencies
see https://github.com/pantsbuild/pants/issues/15688 for more info and https://github.com/koffie/pants_example/ for a minimal github repo reproducing the bug.
r

refined-addition-53644

05/27/2022, 5:27 PM
I don't think you should be using
resources
https://github.com/koffie/pants_example/blob/master/src/python/my-subproject/BUILD#L4= It should be
python_sources
https://www.pantsbuild.org/docs/reference-python_sources I would avoid it completely and use the target generated under https://github.com/koffie/pants_example/blob/master/src/python/my-subproject/my_subproject/BUILD which would be called
my_subproject
and you can add it as dependency directly in dist dependencies as
./my_subproject:my_subproject
Also you don't need to add explicit dependency on
pyproject.toml
.
Now it seems only this is the issue which is breaking the test https://github.com/koffie/pants_example/blob/master/src/python/my-subproject/my_subproject/__init__.py#L5= I am not sure if this the correct way to read the version. I always struggle with this but there are some discussions around how to properly read/set version https://stackoverflow.com/questions/458550/standard-way-to-embed-version-into-python-package EDIT: Not sure if this is because of pants or python :/
w

worried-boots-82926

05/30/2022, 11:22 AM
hi @refined-addition-53644, Thanks for the feedback. The suggestion to use
./my_subproject:my_subproject
as dependency indeed makes things a lot cleaner. The explicit dependency on
pyproject.toml
however is something we do need, it is something we directly got from the pants documentation (see pep 517 on https://www.pantsbuild.org/docs/python-distributions ), and if I remove it the distribution doesn't get build correctly since the pyproject.toml is then not included in the chroot where the distribution is build, and hence all the essential configuration for building the project is missing. You can see the result of this in the following:
Copy code
pants_example maartenderickx$ ./pants package src/python/my-subproject:my_subproject_distribution
13:13:38.28 [INFO] Completed: Building build_backend.pex from setuptools_default_lockfile.txt
13:13:38.78 [INFO] Wrote dist/UNKNOWN-0.0.0-py3-none-any.whl
as you can see without the explicit dependency on pyproject.toml pants doesn't even know where to get the correct name from in order to generate the wheel.
The reason the test https://github.com/koffie/pants_example/blob/master/src/python/my-subproject/my_subproject/__init__.py#L5= fails is probably because you removed the dependency on pyproject.toml, this causes the python my_subproject_distribution to be build incorrectly, and hence also my_subproject not to be installed when running the test. We only added the aforementioned test since some tests in our testsuite really depend on the project to be properly installed, and the above way was just a quick hacky way to test if the project is installed. I have added a test case which is less of a hack and more clear of what it should test:
Copy code
def test_installed() -> None:
    spec = importlib.util.find_spec("my_submodule")
    assert spec is not None
Of course our end goal is not necessarily to have our submodules installed during testing, however we do have some tests that at least at the moment require some of our submodules to be installed in order for them to pass. We are also looking into ways to make our tests pass without the package being installed, but in our migration process to pants we were trying to first just making pants work with our code as is, and later trying optimising so that tests don't depend on an installed package.
5 Views