I've got a bit of a challenge re: dependency inference in our monorepo. The main reason we're looking at Pants currently is we want file-level dependency inference to cut down on the number of test suites we have to run for any given change. Right now, whenever someone posts a PR that touches a shared library, we run every test for every project that imports that shared library, even tests that don't even have an indirect dependency on the modules being changed. One way to solve this is to break down the shared library into smaller pieces (we've already done that to some extent). But that's a lot of churn and it seems far better idea to use the power of Pants' file-level dependency mapping. One of our larger shared libraries (let's call it "foo") includes a

db

package containing over 100 submodules providing DB access to various features. The

foo/db/__init__.py

file implicitly imports all 100+ submodules (whether the consumer uses them or not). A super common pattern in the code base is to do:

from foo import db

... and then access submodules like:

db.bar.do_something()

without actually importing

db.bar

explicitly. Unfortunately, that means that if we're calculating which tests to run and execute an inquiry like

pants dependents --dependents-transitive lib-python/foo/db/some_feature.py

, any test or test script testing code that uses

foo.db

is going to be in the results list whether it uses

some_feature

or not. My first thought is that we could refactor

foo/db/__init__.py

to implement lazy loading (per PEP 562), but I believe that just creates the opposite problem (instead of creating a transitive relationship for all modules, changing the

foo.db

imports to be programmatic means Pants won't find any). So I'm left wondering whether the only real solution is to refactor every place where we do

from foo import db

and update them to explicitly import only the modules it needs. I just did a grep though and there are over 1000 (!) modules where we use this pattern of implicit import. My best idea so far is to enlist an LLM to help me write a script that can automate that refactor. But maybe there are other tricks I don't know about that could solve for this??

Re: LLM - I think you'd want to use the python AST to do the re-factor, it's surprisingly painless

Unfortunately, this seems to be a bit of a recurring theme, but I think it will be necessary to be explicit about the imports. A module should only import the packages/modules that it needs. When a module does

import db

and then

db.bar.do_something()

, this has effectively created an undocumented public interface. the calling module is essentially inspecting the internals of another object. (eg. see the Law of Demeter ). It is generally preferable to be explicit about everything dependencies, imports, etc.) even if it costs a few extra lines of code. From a pants perspective, all dependencies are on a file to file basis, so all that's happening in your case is seeing

import db

and inferring

db/__init__.py

as a dependency, then reading

db/__init__.py

, seeing lots of imports, and inferring everything else.

SJ: Thanks for the pointer to libCST! 👀

Also, what's your test command?

> A module should only import the packages/modules that it needs. @elegant-florist-94385 No disagreements here. This is an older code base (~12+ years) with some stubborn patterns that have stuck for the years. Not least because it's quite a slog to make updates across the whole repo (it's a great benefit of a monorepo to have this capability and keep things consistent, but you still have to do the work). 😉 Fortunately we've been able to keep it very modern in other respects (e.g. using Python 3.12).

I mean, like

pants test ::

?

pants --changed-since=origin/main --changed-dependents=transitive test

? https://www.pantsbuild.org/stable/docs/using-pants/advanced-target-selection

Oh, we're not actually using Pants yet, I'm still researching and clawing my way there. So far I've just been doing pre-work to our large monorepo to be able to run commands like

pants {dependencies,dependents,paths}

to familiarize with its capabilities and figure out what changes we'll need to make to take advantage.

Ahhh, gotcha gotcha. Okay, so, there are probably pants hacks to make this easier - but my personal opinion is to always cleanup the repos to whatever degree you can FIRST and then put tooling on top. If you use tooling to mask dependency problems, you'll get bitten eventually

Agree. I already landed a large PR yesterday to clean up our imports to be compatible with Pants' source root concept.

to shard tests

https://www.pantsbuild.org/stable/docs/using-pants/advanced-target-selection#sharding-the-input-targets Lots of builtins to help strip out some customization. However, as you're already doing - cleaning up whatever tech debt you can is always a good first step

I guess I will get started then. Glad to hear an AST-based refactoring should be surprisingly painless!

Yeah, there are some weird edge cases, but libcst is a godsend - even trying to do the same thing with the python built-in tools is much more difficult. If you check for

import ast

in the pants codebase, there are a few simple ones. We also have a few complicated ones that I probably wouldn't use as an example, since they're such odd, niche cases. Given what you're doing, arguably, you could use sed, awk, and grep - but they miss that little bit of context - and that's where those scripts fall over or get more complicated (e.g. now you have to account for multi line comments or whatever)

Totally. Our

foo

library used to be an absolute kitchen sink. A few years ago I started to chop it up -- we have a

foo-core

now with the lowest level facilities -- and I mostly used sed/awk/grep to update the imports as you mentioned. 🙂 (Actually I have gsed installed on my Mac because it has a better

-i

function for in-place updates.)

How long was the script? I'm guessing like 30ish lines?