https://pantsbuild.org/ logo
Powered by
f

fancy-motherboard-46804

03/22/2021, 4:49 PM
Hello, if there’s any friendlies around, I could use some help. I’m trying to migrate a monorepo with a number of python services in it from a tox-per-directory approach to try out 
pants
. However, I’m tearing my hair out (or would be, if I had any) over a mystery problem in which 
pants
just refuses to find a file during tests that tox has no trouble finding. I have a structure that looks like:
Copy code
services/svcA/X.py
services/svcA/Y.py
services/svcA/Z.py
services/svcA/tests/test_X.py
services/svcA/tests/test_Y.py
services/svcA/tests/test_Z.py
Inside all three test files, they import the corresponding file, i.e. inside 
test_X.py
there is a statement 
import X
, inside 
test_Y.py
we have 
import Y
, and the same for Z. However, when I run the tests:
Copy code
./pants test services/svc::
2 out of the 3 sets of tests (
X
and 
Y
) pass, while the third fails with an import error, `No module named 
Z
. However, 
tox
passes all three tests and find 
Z
just fine.
./pants dependencies services/svcA
shows what I might expect:
Copy code
./pants dependencies services/svcA
services/svcA/X.py
services/svcA/Y.py
services/svcA/Z.py
I honestly don’t even know where to start to solve this.
w

witty-crayon-22786

03/22/2021, 4:54 PM
hey @fancy-motherboard-46804!
do you see any difference between 
./pants dependencies services/svcA/tests/test_Z.py
and the other files?
f

fancy-motherboard-46804

03/22/2021, 4:55 PM
Nope, all three look identical.
w

witty-crayon-22786

03/22/2021, 4:56 PM
identical? or “correct”? you said that X imports X, Y imports Y, etc?
f

fancy-motherboard-46804

03/22/2021, 4:56 PM
Ah, sorry, I mis-read that, thought it was 
Z
and not 
test_Z
. One sec.
w

witty-crayon-22786

03/22/2021, 4:56 PM
(trying to confirm my understanding)
f

fancy-motherboard-46804

03/22/2021, 4:57 PM
Ah, that’s weird, 
test_X
and 
test_Y
look correct (
test_X
has 
X
as a dependency and same with Y), but 
test_Z
throws an error.
h

hundreds-father-404

03/22/2021, 4:58 PM
Throws an error?
f

fancy-motherboard-46804

03/22/2021, 4:58 PM
Copy code
./pants dependencies services/svcA/tests/test_Z.py                               ST 2   chore/ch1953 
16:56:59.97 [ERROR] Exception caught: (pants.engine.internals.scheduler.ExecutionError)
  File "/Users/winawer/.cache/pants/setup/bootstrap-Darwin-x86_64/2.3.0rc2_py38/lib/python3.8/site-packages/pants/bin/local_pants_runner.py", line 246, in run
    engine_result = self._perform_run(goals)
  File "/Users/winawer/.cache/pants/setup/bootstrap-Darwin-x86_64/2.3.0rc2_py38/lib/python3.8/site-packages/pants/bin/local_pants_runner.py", line 167, in _perform_run
    return self._perform_run_body(goals, poll=False)
  File "/Users/winawer/.cache/pants/setup/bootstrap-Darwin-x86_64/2.3.0rc2_py38/lib/python3.8/site-packages/pants/bin/local_pants_runner.py", line 184, in _perform_run_body
    return self.graph_session.run_goal_rules(
  File "/Users/winawer/.cache/pants/setup/bootstrap-Darwin-x86_64/2.3.0rc2_py38/lib/python3.8/site-packages/pants/init/engine_initializer.py", line 130, in run_goal_rules
    exit_code = self.scheduler_session.run_goal_rule(
  File "/Users/winawer/.cache/pants/setup/bootstrap-Darwin-x86_64/2.3.0rc2_py38/lib/python3.8/site-packages/pants/engine/internals/scheduler.py", line 568, in run_goal_rule
    self._raise_on_error([t for _, t in throws])
  File "/Users/winawer/.cache/pants/setup/bootstrap-Darwin-x86_64/2.3.0rc2_py38/lib/python3.8/site-packages/pants/engine/internals/scheduler.py", line 536, in _raise_on_error
    raise ExecutionError(

Exception message: 1 Exception encountered:

  Exception: Unmatched glob from file arguments: "services/svcA/tests/test_Z.py"
h

hundreds-father-404

03/22/2021, 4:59 PM
Hm, check out https://www.pantsbuild.org/v2.4/docs/troubleshooting#pants-cannot-find-a-file-in-your-project We improved this error message in 2.4 to better explain what could be going on here and link to that URL
f

fancy-motherboard-46804

03/22/2021, 5:01 PM
Okay, I’ll give that a shot and report back shortly.
w

witty-crayon-22786

03/22/2021, 5:01 PM
yea, the unmatched glob is likely to be a root cause here. i’m not 100% sure how you would have seen the test run at all if it was not matched, but…
also, if you’re able to share the structure of the BUILD file for the tests, that would help as well… if the glob was manually written to match the file it might just be mismatched
👍 1
f

fancy-motherboard-46804

03/22/2021, 5:11 PM
No good. Tried adding both:
Copy code
[GLOBAL]
pants_ignore_use_gitignore = false
pants_ignore.add = ["!services/","!services/svcA/"]
and I’m still getting the glob error. The BUILD file is just the one that was created by `./pants tailor`:
Copy code
python_tests(
    name="tests",
)
Goddamnit, I’m an idiot. The glob error is legitimate, the file was renamed.
w

witty-crayon-22786

03/22/2021, 5:18 PM
👍 it happens!
➕ 1
f

fancy-motherboard-46804

03/22/2021, 5:18 PM
I still have the problem of the failing test just as above, though.
w

witty-crayon-22786

03/22/2021, 5:18 PM
ok, now what does 
dependencies
report for that file?
f

fancy-motherboard-46804

03/22/2021, 5:18 PM
But the dependencies check now shows that 
pants
doesn’t register 
Z
as a dependency for 
test_Z
.
h

hundreds-father-404

03/22/2021, 5:18 PM
Along with 
dependencies --transitive
w

witty-crayon-22786

03/22/2021, 5:19 PM
if you manually declare the dependency, does the test pass?
f

fancy-motherboard-46804

03/22/2021, 5:20 PM
What’s the best way to add the dependency (I’ve been using pants for a couple of hours here). Is it in the 
svcA/BUILD
or 
svcA/tests/BUILD
? And in 
python_library
?
w

witty-crayon-22786

03/22/2021, 5:23 PM
you would add the dependency to the target that owns the test file (which will be a 
python_tests
target) in 
svcA/tests/BUILD
f

fancy-motherboard-46804

03/22/2021, 5:25 PM
Aha, now the test passes!
w

witty-crayon-22786

03/22/2021, 5:25 PM
ok, interesting.
now, in general, you should not need to add dependencies like that. but an exceptional case is when you have conflicting sources of a single module
are there multiple modules in your repository with the same name as 
X
?
h

hundreds-father-404

03/22/2021, 5:26 PM
f

fancy-motherboard-46804

03/22/2021, 5:30 PM
Hmm. There’s a local dependency, namely that we have:
Copy code
/libs/libFOO
/libs/libFOO/BAR/Z.py
/services/svcA/Z.py
/services/svcA/test_Z.py
with a bunch of stuff omitted here. And in the BUILD for the 
svcA/BUILD
, I had to pull in the dependency:
Copy code
python_library(
  dependencies=["//libs/libFOO:..."],)
(in tox, I have to bring it in using an editable:)
Copy code
deps = 
  --editable=file:///{toxinidir}../../libs/libFOO
Could that be the issue?
That said, the same is true for 
X
and 
Y
(i.e. there exists 
/libs/libFOO/BAR/X.py
) and they’re not causing an issue.
Well, in the meantime, at least I can get on with migrating and I have a workaround, so thanks so much for the help! If there’s anything else I can provide to debug, let me know.
w

witty-crayon-22786

03/22/2021, 5:58 PM
ok, sorry for the trouble. this case is common enough that we’re discussing adding a warning for it.
h

hundreds-father-404

03/23/2021, 7:23 AM
Hey Steven, feedback greatly appreciated on https://pantsbuild.slack.com/archives/CK79E5J2Y/p1616484095006300 Thanks for kicking this off, hopefully other folks won't have to tear their hair out in the future! The upcoming PR should help a lot too
4 Views
Powered by