https://pantsbuild.org/ logo
Join Slack
Powered by
So nice that there are people willing to answer qu...
# general
i
So nice that there are people willing to answer questions here. I have been struggling for days now trying to get an existing monorepo setup with pants for all the reasons the pants site suggests it can do for us. But i can't get 
pants dependencies blah/blah
to do much more than spit out hundreds of lines of sad text. And tests to be able to find libraries in the existing requirements.txt. So a 🧵if anyone is will to help fill in some gaps in my understanding
The structure is pretty reasonable and has been working well enough for 6 years now root beam/ project1 package project2/ package package/ (shared code and smaller projects) utility1.py models.py enums.py app1/ subdir/ app2/ subdir tests/ conftest.py package/ conftest.py app1tests/ app2tests/ utility1tests.py modelstests.py
so any of those apps or projects may use dependencies from ,./packages/*.py. No circular dependencies. shared code is very isolated, doesn't reference underlying apps
requirements_appname.txt are generated from requirements_appname.in using pip-compile and kept in the app folder. There is also a root requirements.txt that is only used when running locally that is a superset of requirements.in/.txt Here is where I'm failing I think
running tests, imports form the top level conftest.py fail on imports from the requirements.txt. In the BUILD file in one of the test folders I have
Copy code
python_tests(
    name="tests",
)

python_requirement(
    name="pytest-sockets",
    requirements=["pytest-socket==0.7.0", "pytest==8.3.4"],
    dependencies=["beam/requirements.txt:reqs"]
)
running the tests for an app I get
Copy code
Plugin: helpconfig, Hook: pytest_cmdline_parse
ConftestImportFailure: ModuleNotFoundError: No module named 'pytest_socket' (from beam/conftest.py)
For more information see <https://pluggy.readthedocs.io/en/stable/api_reference.html#pluggy.PluggyTeardownRaisedWarning>
  config = pluginmanager.hook.pytest_cmdline_parse(
ImportError while loading conftest 'beam/conftest.py'.
beam/conftest.py:2: in <module>
    from pytest_socket import disable_socket
E   ModuleNotFoundError: No module named 'pytest_socket'
If also tried referencing the superset requirements.txt directly in the build file
Copy code
python_requirements(
    name="reqs",
    source="requirements.txt",
)
that fails because the file isn't in the test folder it is at the root. doing 
source="/beam/requirements.txt
to reference a file above in the folder hierarchy is not allowed.
I tried copying requirements.txt to the test folder in using it directly in the source= line, same error
Trying to setup a simpler minimal project with the same structure I may have found a new path to answering this. This large monorepo has a 
setup.py
which enables that upper level 
package
folder to be referenced as say 
package,models
This was done long before my time so I have some reading up to do
h
So I think you’re on the right track, but this stanza:
Copy code
python_requirements(
    name="reqs",
    source="requirements.txt", 
)
needs to be in a BUILD file that is a sibling of the 
requirements.txt
file.
Also, you may need to pay attention to your source roots
How are they currently defined? (
pants roots
will list them)
q
hey mate, conftest need this python_test_utils build block to be able to do anything.
Copy code
python_test_utils(
    name="test_utils",
    dependencies=[
       # add your dependencies here
)
i
I have tried a number of things with the source roots. I've tried just the apps. I've tried the apps and the root of the packages. I've tried the root, the packages and the apps. So for the moment I'm working to understand the 'package', it's literal name, defined by setup.py and how it works. Within the apps, the shared code is referred to like 
package.models
or 
package.utilities
not 
../package/models
and there is no manipulation of sys.path.
I will look at 
python_test_utils
that could be very helpful, thanks
h
(the source roots should point to the roots of any first-party code you can import from. E.g., if you 
from foo.bar import baz
then the dir structure should be 
<source root>/foo/bar.py
or whatever)
to clarify, that is orthogonal to your requirements.txt issue. In that case you need an appropriate BUILD file stanza that is in or above the directory containing the requirements.txt
And you need just one provider of each symbol. Sounds like you have multiple requirements.txt, but that top-level one is for working inside the repo (e.g., running tests)? So just that one should have a BUILD target, and the others should not. If there are multiple providers of the same symbol then pants can’t always disambiguate (it can disambiguate via source root, which is why I brought that up, but it sounds like the BUILD file you want is above all the source roots anyway, so that won’t help)
i
ok, @happy-kitchen-89482 that was helpful. With a BUILD file at the root, and a single BUILD file at the top level source folder, my toy project works with 
run
, 
list
and 
test
and the 
pants dependencies
output looks clean. I'll work on the requirements_*.txt files next. I interpreted the documentation to say something different: 
Typically, there would be one BUILD file in every directory containing source code and any other resources you may want to use as part of your builds.
Also 
pants tailor ::
places many BUILD files throughout the code base, not just at my source root. After running 
pants tailer ::
again on my working setup, I only need to remove this from the abs root BUILD file
Copy code
python_sources(
    name="root"
)
13 Views
Open in Slack
PreviousNext
Powered by