https://pantsbuild.org/ logo
Join Slack
Powered by
Hello, if there’s any friendlies around, I could u...
# general
f
Hello, if there’s any friendlies around, I could use some help. I’m trying to migrate a monorepo with a number of python services in it from a tox-per-directory approach to try out 
pants
. However, I’m tearing my hair out (or would be, if I had any) over a mystery problem in which 
pants
just refuses to find a file during tests that tox has no trouble finding. I have a structure that looks like:
Copy code
services/svcA/X.py
services/svcA/Y.py
services/svcA/Z.py
services/svcA/tests/test_X.py
services/svcA/tests/test_Y.py
services/svcA/tests/test_Z.py
Inside all three test files, they import the corresponding file, i.e. inside 
test_X.py
there is a statement 
import X
, inside 
test_Y.py
we have 
import Y
, and the same for Z. However, when I run the tests:
Copy code
./pants test services/svc::
2 out of the 3 sets of tests (
X
and 
Y
) pass, while the third fails with an import error, `No module named 
Z
. However, 
tox
passes all three tests and find 
Z
just fine.
./pants dependencies services/svcA
shows what I might expect:
Copy code
./pants dependencies services/svcA
services/svcA/X.py
services/svcA/Y.py
services/svcA/Z.py
I honestly don’t even know where to start to solve this.
w
hey @fancy-motherboard-46804!
do you see any difference between 
./pants dependencies services/svcA/tests/test_Z.py
and the other files?
f
Nope, all three look identical.
w
identical? or “correct”? you said that X imports X, Y imports Y, etc?
f
Ah, sorry, I mis-read that, thought it was 
Z
and not 
test_Z
. One sec.
w
(trying to confirm my understanding)
f
Ah, that’s weird, 
test_X
and 
test_Y
look correct (
test_X
has 
X
as a dependency and same with Y), but 
test_Z
throws an error.
h
Throws an error?
f
Copy code
./pants dependencies services/svcA/tests/test_Z.py                               ST 2   chore/ch1953 
16:56:59.97 [ERROR] Exception caught: (pants.engine.internals.scheduler.ExecutionError)
  File "/Users/winawer/.cache/pants/setup/bootstrap-Darwin-x86_64/2.3.0rc2_py38/lib/python3.8/site-packages/pants/bin/local_pants_runner.py", line 246, in run
    engine_result = self._perform_run(goals)
  File "/Users/winawer/.cache/pants/setup/bootstrap-Darwin-x86_64/2.3.0rc2_py38/lib/python3.8/site-packages/pants/bin/local_pants_runner.py", line 167, in _perform_run
    return self._perform_run_body(goals, poll=False)
  File "/Users/winawer/.cache/pants/setup/bootstrap-Darwin-x86_64/2.3.0rc2_py38/lib/python3.8/site-packages/pants/bin/local_pants_runner.py", line 184, in _perform_run_body
    return self.graph_session.run_goal_rules(
  File "/Users/winawer/.cache/pants/setup/bootstrap-Darwin-x86_64/2.3.0rc2_py38/lib/python3.8/site-packages/pants/init/engine_initializer.py", line 130, in run_goal_rules
    exit_code = self.scheduler_session.run_goal_rule(
  File "/Users/winawer/.cache/pants/setup/bootstrap-Darwin-x86_64/2.3.0rc2_py38/lib/python3.8/site-packages/pants/engine/internals/scheduler.py", line 568, in run_goal_rule
    self._raise_on_error([t for _, t in throws])
  File "/Users/winawer/.cache/pants/setup/bootstrap-Darwin-x86_64/2.3.0rc2_py38/lib/python3.8/site-packages/pants/engine/internals/scheduler.py", line 536, in _raise_on_error
    raise ExecutionError(

Exception message: 1 Exception encountered:

  Exception: Unmatched glob from file arguments: "services/svcA/tests/test_Z.py"
h
Hm, check out https://www.pantsbuild.org/v2.4/docs/troubleshooting#pants-cannot-find-a-file-in-your-project We improved this error message in 2.4 to better explain what could be going on here and link to that URL
f
Okay, I’ll give that a shot and report back shortly.
w
yea, the unmatched glob is likely to be a root cause here. i’m not 100% sure how you would have seen the test run at all if it was not matched, but…
also, if you’re able to share the structure of the BUILD file for the tests, that would help as well… if the glob was manually written to match the file it might just be mismatched
👍 1
f
No good. Tried adding both:
Copy code
[GLOBAL]
pants_ignore_use_gitignore = false
pants_ignore.add = ["!services/","!services/svcA/"]
and I’m still getting the glob error. The BUILD file is just the one that was created by `./pants tailor`:
Copy code
python_tests(
    name="tests",
)
Goddamnit, I’m an idiot. The glob error is legitimate, the file was renamed.
w
👍 it happens!
➕ 1
f
I still have the problem of the failing test just as above, though.
w
ok, now what does 
dependencies
report for that file?
f
But the dependencies check now shows that 
pants
doesn’t register 
Z
as a dependency for 
test_Z
.
h
Along with 
dependencies --transitive
w
if you manually declare the dependency, does the test pass?
f
What’s the best way to add the dependency (I’ve been using pants for a couple of hours here). Is it in the 
svcA/BUILD
or 
svcA/tests/BUILD
? And in 
python_library
?
w
you would add the dependency to the target that owns the test file (which will be a 
python_tests
target) in 
svcA/tests/BUILD
f
Aha, now the test passes!
w
ok, interesting.
now, in general, you should not need to add dependencies like that. but an exceptional case is when you have conflicting sources of a single module
are there multiple modules in your repository with the same name as 
X
?
h
f
Hmm. There’s a local dependency, namely that we have:
Copy code
/libs/libFOO
/libs/libFOO/BAR/Z.py
/services/svcA/Z.py
/services/svcA/test_Z.py
with a bunch of stuff omitted here. And in the BUILD for the 
svcA/BUILD
, I had to pull in the dependency:
Copy code
python_library(
  dependencies=["//libs/libFOO:..."],)
(in tox, I have to bring it in using an editable:)
Copy code
deps = 
  --editable=file:///{toxinidir}../../libs/libFOO
Could that be the issue?
That said, the same is true for 
X
and 
Y
(i.e. there exists 
/libs/libFOO/BAR/X.py
) and they’re not causing an issue.
Well, in the meantime, at least I can get on with migrating and I have a workaround, so thanks so much for the help! If there’s anything else I can provide to debug, let me know.
w
ok, sorry for the trouble. this case is common enough that we’re discussing adding a warning for it.
h
Hey Steven, feedback greatly appreciated on https://pantsbuild.slack.com/archives/CK79E5J2Y/p1616484095006300 Thanks for kicking this off, hopefully other folks won't have to tear their hair out in the future! The upcoming PR should help a lot too
4 Views
Open in Slack
PreviousNext
Powered by