Hello if there s any friendlies around I could use some help Pants #general

Hello, if there’s any friendlies around, I could u...

fancy-motherboard-46804

03/22/2021, 4:49 PM

Hello, if there’s any friendlies around, I could use some help. I’m trying to migrate a monorepo with a number of python services in it from a tox-per-directory approach to try out

pants

. However, I’m tearing my hair out (or would be, if I had any) over a mystery problem in which

pants

just refuses to find a file during tests that tox has no trouble finding. I have a structure that looks like:

Copy code

services/svcA/X.py
services/svcA/Y.py
services/svcA/Z.py
services/svcA/tests/test_X.py
services/svcA/tests/test_Y.py
services/svcA/tests/test_Z.py

Inside all three test files, they import the corresponding file, i.e. inside

test_X.py

there is a statement

import X

, inside

test_Y.py

we have

import Y

, and the same for Z. However, when I run the tests:

Copy code

./pants test services/svc::

2 out of the 3 sets of tests (

and

) pass, while the third fails with an import error, `No module named

. However,

tox

passes all three tests and find

just fine.

fancy-motherboard-46804

03/22/2021, 4:52 PM

./pants dependencies services/svcA

shows what I might expect:

Copy code

./pants dependencies services/svcA
services/svcA/X.py
services/svcA/Y.py
services/svcA/Z.py

I honestly don’t even know where to start to solve this.

witty-crayon-22786

03/22/2021, 4:54 PM

hey @fancy-motherboard-46804!

witty-crayon-22786

03/22/2021, 4:54 PM

do you see any difference between

./pants dependencies services/svcA/tests/test_Z.py

and the other files?

fancy-motherboard-46804

03/22/2021, 4:55 PM

Nope, all three look identical.

witty-crayon-22786

03/22/2021, 4:56 PM

identical? or “correct”? you said that X imports X, Y imports Y, etc?

witty-crayon-22786

03/22/2021, 4:56 PM

(trying to confirm my understanding)

fancy-motherboard-46804

03/22/2021, 4:56 PM

Ah, sorry, I mis-read that, thought it was

and not

test_Z

. One sec.

fancy-motherboard-46804

03/22/2021, 4:57 PM

Ah, that’s weird,

test_X

and

test_Y

look correct (

test_X

has

as a dependency and same with Y), but

test_Z

throws an error.

hundreds-father-404

03/22/2021, 4:58 PM

Throws an error?

fancy-motherboard-46804

03/22/2021, 4:58 PM

Copy code

./pants dependencies services/svcA/tests/test_Z.py                               ST 2   chore/ch1953 
16:56:59.97 [ERROR] Exception caught: (pants.engine.internals.scheduler.ExecutionError)
  File "/Users/winawer/.cache/pants/setup/bootstrap-Darwin-x86_64/2.3.0rc2_py38/lib/python3.8/site-packages/pants/bin/local_pants_runner.py", line 246, in run
    engine_result = self._perform_run(goals)
  File "/Users/winawer/.cache/pants/setup/bootstrap-Darwin-x86_64/2.3.0rc2_py38/lib/python3.8/site-packages/pants/bin/local_pants_runner.py", line 167, in _perform_run
    return self._perform_run_body(goals, poll=False)
  File "/Users/winawer/.cache/pants/setup/bootstrap-Darwin-x86_64/2.3.0rc2_py38/lib/python3.8/site-packages/pants/bin/local_pants_runner.py", line 184, in _perform_run_body
    return self.graph_session.run_goal_rules(
  File "/Users/winawer/.cache/pants/setup/bootstrap-Darwin-x86_64/2.3.0rc2_py38/lib/python3.8/site-packages/pants/init/engine_initializer.py", line 130, in run_goal_rules
    exit_code = self.scheduler_session.run_goal_rule(
  File "/Users/winawer/.cache/pants/setup/bootstrap-Darwin-x86_64/2.3.0rc2_py38/lib/python3.8/site-packages/pants/engine/internals/scheduler.py", line 568, in run_goal_rule
    self._raise_on_error([t for _, t in throws])
  File "/Users/winawer/.cache/pants/setup/bootstrap-Darwin-x86_64/2.3.0rc2_py38/lib/python3.8/site-packages/pants/engine/internals/scheduler.py", line 536, in _raise_on_error
    raise ExecutionError(

Exception message: 1 Exception encountered:

  Exception: Unmatched glob from file arguments: "services/svcA/tests/test_Z.py"

hundreds-father-404

03/22/2021, 4:59 PM

Hm, check out https://www.pantsbuild.org/v2.4/docs/troubleshooting#pants-cannot-find-a-file-in-your-project We improved this error message in 2.4 to better explain what could be going on here and link to that URL

fancy-motherboard-46804

03/22/2021, 5:01 PM

Okay, I’ll give that a shot and report back shortly.

witty-crayon-22786

03/22/2021, 5:01 PM

yea, the unmatched glob is likely to be a root cause here. i’m not 100% sure how you would have seen the test run at all if it was not matched, but…

witty-crayon-22786

03/22/2021, 5:04 PM

also, if you’re able to share the structure of the BUILD file for the tests, that would help as well… if the glob was manually written to match the file it might just be mismatched

👍 1

fancy-motherboard-46804

03/22/2021, 5:11 PM

No good. Tried adding both:

Copy code

[GLOBAL]
pants_ignore_use_gitignore = false
pants_ignore.add = ["!services/","!services/svcA/"]

and I’m still getting the glob error. The BUILD file is just the one that was created by `./pants tailor`:

Copy code

python_tests(
    name="tests",
)

fancy-motherboard-46804

03/22/2021, 5:18 PM

Goddamnit, I’m an idiot. The glob error is legitimate, the file was renamed.

witty-crayon-22786

03/22/2021, 5:18 PM

👍 it happens!

➕ 1

fancy-motherboard-46804

03/22/2021, 5:18 PM

I still have the problem of the failing test just as above, though.

witty-crayon-22786

03/22/2021, 5:18 PM

ok, now what does

dependencies

report for that file?

fancy-motherboard-46804

03/22/2021, 5:18 PM

But the dependencies check now shows that

pants

doesn’t register

as a dependency for

test_Z

hundreds-father-404

03/22/2021, 5:18 PM

Along with

dependencies --transitive

witty-crayon-22786

03/22/2021, 5:19 PM

if you manually declare the dependency, does the test pass?

fancy-motherboard-46804

03/22/2021, 5:20 PM

What’s the best way to add the dependency (I’ve been using pants for a couple of hours here). Is it in the

svcA/BUILD

svcA/tests/BUILD

? And in

python_library

witty-crayon-22786

03/22/2021, 5:23 PM

you would add the dependency to the target that owns the test file (which will be a

python_tests

target) in

svcA/tests/BUILD

fancy-motherboard-46804

03/22/2021, 5:25 PM

Aha, now the test passes!

witty-crayon-22786

03/22/2021, 5:25 PM

ok, interesting.

witty-crayon-22786

03/22/2021, 5:25 PM

now, in general, you should not need to add dependencies like that. but an exceptional case is when you have conflicting sources of a single module

witty-crayon-22786

03/22/2021, 5:26 PM

are there multiple modules in your repository with the same name as

hundreds-father-404

03/22/2021, 5:26 PM

The warning at https://www.pantsbuild.org/v2.4/docs/targets#sources

fancy-motherboard-46804

03/22/2021, 5:30 PM

Hmm. There’s a local dependency, namely that we have:

Copy code

/libs/libFOO
/libs/libFOO/BAR/Z.py
/services/svcA/Z.py
/services/svcA/test_Z.py

with a bunch of stuff omitted here. And in the BUILD for the

svcA/BUILD

, I had to pull in the dependency:

Copy code

python_library(
  dependencies=["//libs/libFOO:..."],)

(in tox, I have to bring it in using an editable:)

Copy code

deps = 
  --editable=file:///{toxinidir}../../libs/libFOO

Could that be the issue?

fancy-motherboard-46804

03/22/2021, 5:32 PM

That said, the same is true for

and

(i.e. there exists

/libs/libFOO/BAR/X.py

) and they’re not causing an issue.

fancy-motherboard-46804

03/22/2021, 5:33 PM

Well, in the meantime, at least I can get on with migrating and I have a workaround, so thanks so much for the help! If there’s anything else I can provide to debug, let me know.

witty-crayon-22786

03/22/2021, 5:58 PM

ok, sorry for the trouble. this case is common enough that we’re discussing adding a warning for it.

hundreds-father-404

03/23/2021, 7:23 AM

Hey Steven, feedback greatly appreciated on https://pantsbuild.slack.com/archives/CK79E5J2Y/p1616484095006300 Thanks for kicking this off, hopefully other folks won't have to tear their hair out in the future! The upcoming PR should help a lot too

4 Views

Open in Slack

Previous Next