I'm evaluating using Pants as we transition to a monorepo in my org. Is it possible to have a single

pyproject.toml

file at the root of my repository and ensure that these dependencies are installed when running e.g.

pants test /path/to/project

? The folder structure is similar to this:

my-repository/
    pants.toml
    pyproject.toml
    poetry.lock

    projects/
        project-a/
          README.md
          project_a/
           routers/
             ...

        project-b/
          README.md
          project-b/
           routers/
             ...

Hello! This should be possible; though I haven't tested it myself yet. Pants should be able to use a

pyproject.toml

as the source for a

python_requirements

build target: https://www.pantsbuild.org/docs/python-third-party-dependencies#pep-621-compliant-pyprojecttoml The rest of the setup should work as regular with setting up a default resolve, generating lockfiles, and running your tests.

Adding to what Tom said, it looks like you are using poetry, in that case you would need

poetry_requirements()

https://www.pantsbuild.org/docs/python-third-party-dependencies#poetry

Yes, @refined-addition-53644, you are correct. However, if I run a target such as

pants test /path/to/subproject/tests/unit

pants will give me a ModuleNotFound error. It appears as if having

poetry_requirements

at the top-level is not enough. What would I put in my

BUILD

files at the subproject?

Having it at the top level should be sufficient in most situations. If you run

pants dependencies /path/to/subproject/tests/unit/some_file.py

, does it reference any of your dependencies at all? Do you have a default-resolve setup, and/or does your

python_tests

and the

poetry_requirements

use the same resolve? Does generating lockfiles work (

pants generate-lockfiles

)?

I guess I have some reading to do before I can answer your questions accurately. What we've done in essence is install Pants, configured it to use the python backend, and ran

pants tailor ::

. It then generated

BUILD

files in all subdirectories containing python-sources. And the

BUILD

file at the top-level has the following:

poetry_requirements(
    name="root",
)

I had assumed that this would be sufficient for me to be able to run

pants test

afterwards, and have the necessary dependencies installed in the PEX. If I run

pants dependencies

on one of the files as you suggest it does contain references to the relevant projects in the form of

/path/to/project:poetry#requests

as an example.

/path/to/project:poetry#requests

sounds like you have multiple sets of requirements here - not sure if that's your intention. If you have a root pyproject.toml with your dependencies I'd expect that to say

//:root#requests

.

Yes, sorry. Assume we have the following:

repository/
    libs/
       lib-1/
         lib_1/
    projects/
       project-1/
         project_1/

In this case, we have

lib-1

installed as an editable development dependency in the top-level pyproject.toml, howver, lib-1 also has its own

pyproject.toml

which contains its own dependencies. I suspect this isn't the way to handle first-party dependencies in Pants. In this case the

BUILD

file in

lib-1

at the top level simply contains

poetry_requirements()

.

This seems fine (but as you say, probably a bit unusual). You can definitely combine dependencies from multiple sources as long as they are part of the same resolve. In-repo dependencies (path-based) in pyproject.toml should be ignored by Pants as it has its own mechanism for resolving them. As noted above though; I haven't yet migrated our repo to use PEP621 instead of requirements.txt so I'm only speaking generally. To back up a bit: is the module it complains about one of your local libraries or one of your dependencies? What is its name (in your pyproject.toml), and what is the import statement?

The module it complains about is one of our local libraries. In the pyproject.toml file it is defined as:

lib-1 = {path = "libs/lib-1", develop = true}

.

What are your source roots set as?

pants roots

should list them all

The output is simply:

libs
projects

Ah! Great. We have a winner, or a misconfiguration at least. So: pants does things slightly different from what you'd expect. It does not in any way build wheels for local code. It just looks at what you import, finds it in your source roots, and copies it into the appropriate location - recursively. So, in this case if you import

lib_foo.a

it'll check try to find

libs/lib_foo/a.py

and

projects/lib_foo/a.py

, etc. This means that if your layout is

libs/lib-1/lib_1/a.py

or some such it won't find it. The solution is to add each project as a source root. The easiest way to do this is to use a marker file, which allows you to say

pyproject.toml

denotes a root you can import from.

[source]
marker_filenames = ["pyproject.toml"]

Ah yes, that makes sense! Now I'm (almost) able to get everything running. I use a pytest plugin

pytest-env

to define environment variables directly in the

pytest.ini

file. However, it is saying that all of my environment variables required for the tests are not defined.

Sounds like a plan. (nvm the other part, pytest.ini should be found by auto).

Sorry to keep on bugging you. I'll check out the

extra_requirements

, that seems to work. I was also able to get

extra_env_vars

to work. However, now I'm getting another ModuleNotFoundError, this time for a third-party package that is present in the top-level

pyproject.toml

. Since last time I have removed

pyproject.toml

from

lib-1

, since it is now a first-party dependency and have updated my roots to include it correctly. If I run

pants dependencies

on the tests folder of the project the module it complains about is not listed here.

Is it named exactly the same? For example,

PIL

is provided by

pillow

, which Pants cannot know about easily.

Ah, indeed it is not. That should've occured to me as well to be honest.

Then

module_mapping

is your next step: https://www.pantsbuild.org/docs/reference-poetry_requirements#codemodule_mappingcode

That works! It's becoming clear to me that Poetry has handled a great deal behind the scenes with regards to consuming packages.

This likely means the file isn't explicitly imported anywhere. Only files and dependencies that are explicitly used are imported into the sandbox.

I see. That could be the issue, since this is a

patch

call inside a unit-test. The actual code is never directly imported since the test uses FastAPIs TestClient to carry out an HTTP Request that calls another function. I guess I would have to inform Pants that all of the functions in the

routers

are dependencies of the tests (?)

Yeah explicit dependencies are likely your goto if you cannot import the actual code.

Is there a way to define dependencies as "all targets of this subfolder"? E.g. my tests needs to depend on all files in the current project.

I think that'd be possible -- but can I ask why? It seems weird to depend on a ton of code you're not importing. Everything you import should be there.

I agree. However, we have the following issue:

fastapi.TestClient

is used to run an API query against

/my_endpoint

. This in turn calls

some_function()

in my code. However (AFAIC) this dependency cannot possibly be inferred by Pants, unless there is a plugin that tightly integrates with FastAPI. As a result, I have a bunch of tests that make API calls that in turn call functions in my project which aren't picked up automatically.

You probably need

pants dependencies --transitive

Yes, I am using the

--transitive

flag.

Ah, sorry, yeah, read too quickly.

Weird, it does not. Even with

--transitive

set (although this should be unnecessary).

It might be necessary to up the log level with

-ldebug

and see what shows up. I'm not too familiar with the dependency inference, especially if it doesn't print any warnings.

Would you look at that.

Pants cannot infer owners for the following imports in the target /path/to/users.py

* thefuzz.fuzz (line: 8)

I suspect this is related to the

module_mapping

somehow?

That seems likely, though if your dependency is

thefuzz>=1.2.3

f.ex. that should be fine.

Yes.. However, if I update the module mapping from:

"thefuzz": ["fuzz"]

To:

"thefuzz": ["fuzz", "thefuzz.fuzz"]

It is suddenly picked up correctly.

Oh! So the first one says that "the package called thefuzz is imported as fuzz", which isn't what you want.

Of course, my bad! That fixes the issue, thanks 🙂

I'd suggest following the guide here for setting up a .env file instead: https://www.pantsbuild.org/docs/setting-up-an-ide#first-party-sources

By site-package dependencies i mean my third-party dependencies, yes. E.g. if pyproject.toml includes

httpx

as a dependency, I would expect this to be present in the

dist/

folder somewhere.

It should be, hmm. You might have multiple exports depending on setup, but I guess you've checked all if you have many.

I tried to use the command as it is listed here for exporting third-party dependencies: https://www.pantsbuild.org/docs/setting-up-an-ide#python-third-party-dependencies-and-tools

pants export --py-resolve-format=symlinked_immutable_virtualenv --resolve=python-default
16:53:01.75 [ERROR] 1 Exception encountered:

Engine traceback:
  in `export` goal

IntrinsicError: Unmatched glob from the resolve `python-default`: "3rdparty/python/default.lock"

Hmm, have you enabled resolves?

[python] enable-resolves = true

in pants.toml. Then

pants generate-lockfiles ::

. Then you should get further. I don't think you can get a working export without resolves enabled.

You're right. I had not enabled resolves. After doing so it does try to generate a lockfile based on the

pyproject.toml

file. However, it errors out on all internal packages (published to our own internal package manager). The error is:

ERROR: Could not find a version that satisfies the requirement your-package<0.3.0,>=0.2.2
ERROR: No matching distribution found for your-package<0.3.0,>=0.2.2

This was installing fine when it created the PEX file to run tests before we enabled resolves. It also installs just fine using plain poetry and the same index-url. Using

-ldebug

I see that the lock action runs with

--extra-index-url

set correctly.

Is it compatible with the Python interpreter constraints you've set? Have you set any? Otherwise that sounds like a good start.

[python]
interpreter_constraints = [">=3.9,<3.10"]

and

[python.resolves_to_interpreter_constraints]
the-resolve = [">=3.9,<3.10"]

might both be necessary

Sorry, I forgot to update this thread. That was the issue indeed. I had set the interpreter to a strict

==3.11.*

.

I think the docs definitely need some love; yeah. I think if you follow them to the letter you'll get to a working setup, but it's spread across many pages and topics. There's a few GH issues around this too; f.ex. https://github.com/pantsbuild/pants/issues/15274. If you want to summarize your issues there or in a new issue I think that'd be very helpful - and of course, you're also welcome to contribute to the docs directly, but having an issue to relate to makes the PR easier 🙂 The CA bundle sounds interesting, that sounds like you might custom certs for your package host? Does pip just use that automatically? I started using Pants last year for my own needs, then adopted it for work early this year. In October I'll hit a year of usage. 😛

Yeah. I don't think we'd be able to get a setup working without you to be honest. Correct, we have certificates generated by our own custom CA. Pip needs to be pointed at a CA-bundle that includes the root CA used to sign these in order to work.

Does this setting solve it? https://www.pantsbuild.org/docs/restricted-internet-access#setting-up-a-certificate-authority

Yeah, but that in itself isn't enough. We also needed to set the REQUESTS_CA_BUNDLE, since pip relies heavily on

requests

internally.

So the setting alone didn't solve it? Just want to make sure you tried it; as that seems like a clear bug. Pex has

--cert

and

--client-cert

so we should probably forward the setting in that case, if we don't already. In which case that'd probably be a Pex bug...

The setting alone does not solve it. I guess it's possible that you would want to set

ca_certs_path

and not want pip to use this, but I honestly don't know.

Hmm, right. I'll make a note and dig in to what currently happens; as it seems like forwarding this setting to the pex command line should make the rest work without any env vars.

From the docs:

By default, Pants will respect and pass through the

SSL_CERT_DIR

and

SSL_CERT_FILE

environment variables.

The underlying issue here is that

requests

doesn't respect the

SSL_CERT_FILE

variable, and relies on the (much older)

REQUESTS_CA_BUNDLE

. Since we pass along

SSL_CERT_FILE

it should also be safe to pass along

REQUESTS_CA_BUNDLE

. AFAIK they share the same purpose, but some applications support both or only one or the other.

Yeah; but wouldn't it make more sense to respect the

ca_certs_path

? I've never worked in the kind of environment where this ends up being necessary, so maybe I'm seeing the wrong problems.

It makes sense if the bundle in

ca_certs_path

contains all certificates installed on the system. However, it's entirely possible to point it to a file containing a single certificate as well if you don't have access to modify the internal bundle. A change that automatically propagates

ca_certs_path

and sets the necessary environment variables might cause unintended issues in such a configuration.

👍 Makes sense, yeah.

On an entirely different note. Have you found a way to stream the output from pants directly? I'm trying to run my integration tests and the pytest command hangs forever. No output is provided, even with

pants test -ldebug --output=all -- -s

It's not something I've tried to do for tests; but Pants generally is quite sparse with streaming output since you might have a dozen things running in parallel.