Hi! Just getting started with Pants. I'm setting up a project with multiple independent jobs, each with their own

requirements.txt

. I want each job to be able to declare its own dependencies, even if the versions differ between jobs. I'm currently getting these warnings:

13:20:15.11 [WARN] Pants cannot infer owners for the following imports in the target jobs/some-job/file.py:app:

  * boto3 (line: 10)
  * botocore.exceptions.ClientError (line: 11)

My BUILD files look like this:

python_requirements(
    name="req",
    source='requirements.txt'
)

python_sources(
    name="app",
    dependencies=[":req"]
)

My pants.toml:

[GLOBAL]
pants_version = "2.25.0"
backend_packages = [
  "pants.backend.docker",
  "pants.backend.python"
]
build_file_prelude_globs = ["pants-plugins/macros.py"] # some docker macros that are not used yet

[source]
marker_filenames = [
  "requirements.txt"
]

[python]
# enable_resolves = true
interpreter_constraints = ['>=3.11']

[python-repos]
indexes = [
  "..."
]


[python-infer]
use_rust_parser = true

[dockerfile-parser]
use_rust_parser = true

Is it safe to ignore these warnings in my case?

I'm not sure about the safety of ignoring the errors or not.. But I had a similar issue so I added the libraries to my requirements.txt (even though the libraries were already being pulled as transitive dependencies of another library I had) and it got rid of the warnings

I believe the issue is that if you f.ex have numpy in two different requirements files, pants will not be able to infere which one a file is dependent on. I thought that linking the req file in dependencies would work, but be careful as it now links ALL the dependencies in that req file to ALL the python sources. It might also be that you need to look at module mapping, not sure if boto has a different name: https://www.pantsbuild.org/stable/docs/python/overview/third-party-dependencies#use-modules-and-module_mapping-when-the-module-name-is-not-standard You should really look into lockfiles though, I believe that is the intended method. It's described on the same document.

With a global lockfile, what happens when different services need different package versions? If one team needs a newer version while others work with older ones, we'd all have to move together with a global lockfile. I guess we could create custom resolves for exceptions, but that seems like extra work to manage. My bigger concern is the risk in our large, under-tested monorepo. Updating a package version globally could break multiple services at once, and without good test coverage, we wouldn't know until things fail in production.

I agree with Martin that you should look into lockfiles. Think of it this way: • "requirements" puts each requirement into your "universe of dependencies". ◦ having multiple requirements files around your repo still puts all of them into the same "universe" • a source file may only depend on a subset of requirements, and will use only those requirements for mypy, tests, etc. • your universe must be consistent: all requirements in your universe must be version compatible and only one version per package. When you use lockfiles (also called resolves), you allow yourself to have multiple "universes". • each universe must be consistent within itself, but doesn't care what is in your other universes • each requirement specifies what universe it belongs to:

resolve=<ABC>

◦ you can use

paremeterize

to put a requirement in multiple universes. • each source file specifies what universe it runs in (ie. where it searches for dependencies) ◦ again, you can use

parameterize

if it needs to run in multiple universes (such as a common library used by multiple jobs) • you can set a default resolve (in

pants.toml

), so that you can avoid needing to set

resolve=...

in whatever is your main set of targets • There are decent tools/techniques to avoid duplication for things that belong in multiple resolves.