https://pantsbuild.org/ logo
Join Slack
Powered by
Folks, is there a guide, documentation or suggeste...
# general
i
Folks, is there a guide, documentation or suggested folder structure that I can use to get the most out of pants caching and dependency tree? I am migrating a monorepo to Pants and at the moment any code change triggers all tests.
w
Are you writing something like 
pants test ::
?
i
I am
w
Okay, so in that case, it’s doing the thing you’re asking. It’s weird that caching/memoization isn’t helping you out there - so, while “all tests run” - the ones that didn’t change shouldn’t re-run. They should say something like “(memoized)” or similar in the run. What language are you using?
Another option is to use 
changed-since
(https://www.pantsbuild.org/2.21/docs/using-pants/advanced-target-selection#running-over-changed-files-with---changed-since), where you can limit the number of targets to run over. Depending on your code, often this option is used alongside transitive dependencies
Copy code
❯ pants \
  --changed-since=origin/main \
  --changed-dependents=transitive \
  test
i
my code is all in Python, I find my case weird because in the example-python repository from the Pants documentation it identifies the dependencies and only re-runs certain tests.
however my test structure is not the same, as in the example repository each test is in the same folder/package as the source code. Not sure if that is the reason.
w
Is the pants daemon running? Or does it get restarted as well?
i
it is, I thought this was a memory issue but running two small subsets of tests memoizes them, however a change to one piece of code that should only affect one set of tests clears the cache for both sets instead.
w
And I’m guessing the code change is unrelated whatever is in the tests? No imports, or anything? I’ve seen this kinda behaviour before - currently in a complicated situation where something similar is happening
i
the change I am doing is to add a print to one REST endpoint of one of my microservices, not changing a test file, so my expectation was that only tests that depend on that endpoint handler would have the cache cleared. But instead all my tests get cleared from cache
w
That’s strange. The caching is on a file-basis though, not function basis
So, if you had tests in multiple files, I would assume only 1 file re-runs. But, all tests in 1 file, and they will all re-run
i
No, my case is that I have: 
/src/services/service_a
and 
/src/services/service_b
then tests in 
/test/service_a
and 
/test/service_b
but any change to a file in 
/src/services/service_a
that is not listed as a dependency to a given test file in 
/test/service_b
still triggers a re-run for that test file.
w
Ah, okay, that’s interesting. It’s not an explicit dep, but can you check if it’s getting pulled in anyways, just to be sure?
f
right, there's dependency inference (Pants reads statically your import statements) and any manual overrides you may add in the BUILD files (e.g. subprocess call, reading a text file, dynamic import with 
importlib
etc). It's possible that you have weird dependency chains particularly if there are 
conftest.py
files involved. I'd suggest dumping dependency graph into adjacency list with the JSON format https://www.pantsbuild.org/2.21/reference/goals/dependencies#format
remember to pass the 
--transitive
flag because this is how most of the nasty stuff is discovered!
also, the 
paths
goal is very helpful as it will show you all the paths between two targets
Copy code
$ pants paths --from=tests/cli/test_cli.py --to=cheeseshop/repository/package.py                                                                                                                 
[
  [
    "tests/cli/test_cli.py:tests",
    "cheeseshop/cli/cli.py",
    "cheeseshop/repository/package.py"
  ],
  [
    "tests/cli/test_cli.py:tests",
    "cheeseshop/cli/cli.py",
    "cheeseshop/repository/query.py",
    "cheeseshop/repository/package.py"
  ],
  [
    "tests/cli/test_cli.py:tests",
    "cheeseshop/cli/cli.py",
    "cheeseshop/repository/repository.py",
    "cheeseshop/repository/package.py"
  ]
]
i
hmm, indeed with the transitive parameter if finds (somehow) a dependency between 
/test/service_b
test file and a file from 
/src/services/service_a
🤯
Thank you for the help @fresh-cat-90827 and @wide-midnight-78598, now I have enough info to start dismantling this dependency nightmare.
w
👍 You may be able to use visibility rules to help clean up as well: https://www.pantsbuild.org/2.21/docs/using-pants/validating-dependencies#example-visibility-rules
i
Thank you, that helps a lot! 🙂
Open in Slack
PreviousNext
Powered by