https://pantsbuild.org/ logo
#general
Title
# general
l

loud-laptop-17949

04/19/2022, 6:12 PM
I am trying to turn on pants-based linting for more of our python files, but am running into an error that doesn't make sense to me, which works when using pylint directly–
src/django_apps/files/models.py:21:0: E0401: Unable to import 'util.url' (import-error)
Though pants shows them as a dependency:
Copy code
$ p dependencies src/django_apps/files/models.py
...
src/util/url.py
What am I missing here?
👀 1
h

hundreds-father-404

04/19/2022, 6:16 PM
huh.. unclear. we include all transitive deps in the sandbox. i recommed trying https://www.pantsbuild.org/docs/troubleshooting#debug-tip-inspect-the-sandbox-with---no-process-cleanup
e

eager-dress-66405

04/19/2022, 6:16 PM
Hard to replicate.
./pants lint src/django_apps/files/models.py
works
./pants lint src/django_apps/files::
works Only happens on very large pants lint runs and which files see these unable to import errors for seems to shift arond
l

loud-laptop-17949

04/19/2022, 6:18 PM
./pants filter --sep="\0" :: | xargs -0 ./pants lint
seems to always trigger it tho
e

eager-dress-66405

04/19/2022, 6:18 PM
But for the same files?
l

loud-laptop-17949

04/19/2022, 6:18 PM
yes, I think
s

sparse-lifeguard-95737

04/19/2022, 6:23 PM
./pants filter :: | wc -l
prints 7450 for me.
xargs
has a default value 5000 for
-n
(at least according to
man
), is it possible we’re splitting transitive deps across separate calls to
./pants lint
?
e

eager-dress-66405

04/19/2022, 6:24 PM
I would think pants should be able to figure that out. same that
./pants lint <one file>
works
Ok I'm able to capture a broken sandbox from this. running
./__run.sh
repeats the error. What now?
h

hundreds-father-404

04/19/2022, 6:50 PM
Great. How is it broken? The necessary file is not present for example?
e

eager-dress-66405

04/19/2022, 6:52 PM
Hmm, little surprising. The files do seem to be here
👀 1
Copy code
/tmp/process-execution9TnW2S$ ./__run.sh |grep import-error|cut -d: -f5|sort | uniq -c
      2  Unable to import 'util.async_task' (import-error)
      4  Unable to import 'util.celery' (import-error)
      4  Unable to import 'util.common' (import-error)
     17  Unable to import 'util.constants' (import-error)
      1  Unable to import 'util.date' (import-error)
      2  Unable to import 'util.iterables' (import-error)
      2  Unable to import 'util.iter' (import-error)
      2  Unable to import 'util.jira' (import-error)
      5  Unable to import 'util.phone_number' (import-error)
      1  Unable to import 'util.requests' (import-error)
      5  Unable to import 'util.s3' (import-error)
      2  Unable to import 'util.string' (import-error)
     19  Unable to import 'util.testutil' (import-error)
      2  Unable to import 'util.text_mailer' (import-error)
      3  Unable to import 'util.url' (import-error)
h

hundreds-father-404

04/19/2022, 6:53 PM
hmmmm. I wonder if maybe source routes are not being set up correctly...? Although it sounds like both share the same of
src
though
e

eager-dress-66405

04/19/2022, 6:53 PM
/tmp/process-execution9TnW2S/src/util/{module}.py
exists for all of the above
👍 1
w

witty-crayon-22786

04/19/2022, 6:53 PM
I wonder if maybe source roots are not being set up correctly...?
and/or
__init__.py
files
1
e

eager-dress-66405

04/19/2022, 6:54 PM
In this case the files being linted and the util modules that aren't found are all in the same root_pattern
/src
Adding a
__init__.py
file does not help
Actually you know i think this is a root_pattern problem
we (unfortunately) have
/
and
/src
as root patterns in our
pants.toml
h

hundreds-father-404

04/19/2022, 6:57 PM
checj __run.sh for PEX_EXTRA_SYS_PATH
e

eager-dress-66405

04/19/2022, 6:58 PM
Copy code
************* Module src.django_apps.files.models
src/django_apps/files/models.py:21:0: E0401: Unable to import 'util.url' (import-error)
the module being linted is
src.django_apps.files.models
, not
django_apps.files.models
👀 1
export PEX_EXTRA_SYS_PATH=$'build/python-packages/color-django-extensions:src:__plugins'
from the
PEX_EXTRA_SYS_PATH
it looks right
with
src
in the PATH i would think
import util.url
would work
Copy code
/tmp/process-execution9TnW2S$ ls -lah src/util/url.py 
-rw-r--r-- 1 mpcusack mpcusack 1.2K Apr 19 14:42 src/util/url.py
h

hundreds-father-404

04/19/2022, 7:05 PM
I would too. That is at least how it's worked with the other organizations who have been using Pylint, including Toolchain You could try modifying the run.SH script and invoke that
e

eager-dress-66405

04/19/2022, 7:06 PM
Well this is weird.
Copy code
/tmp/process-execution9TnW2S$ PEX_EXTRA_SYS_PATH=$'build/python-packages/color-django-extensions:src:__plugins' ./pylint_runner.pex_pex_shim.sh $'--rcfile=.pylintrc' $'--jobs=8' src/django_apps/files/models.py
Has no errors
but if i run the
__run.sh
script that passes ~280 additional files it fails
h

hundreds-father-404

04/19/2022, 7:08 PM
I wonder if those additional files all belong to the same source root?
Playing around with that folder will be the best approach to debug this I think. It's plausible there is an edge case we have not experienced before. Trying to get a minimal reproduction would be hugely helpful
e

eager-dress-66405

04/19/2022, 7:08 PM
they do all belong to the same source root
👀 1
The sandbox was created just linting
src/django_apps/::
and looking at the args that is true
with
build/python-packages/color-django-extensions
on the path, one of them is likely importing from that alternative python root
I gotta switch over to something else for the next hour or so, but will come back to this and see if I can figure out exactly what combo or files being linted triggers the error
👍 1
So narrowed it down to this reproducible failure:
Copy code
./pants lint src/django_apps/files/models.py     src/django_apps/feed/services/consult/util.py
Adding a
__init__.py
to
src/django_apps/feed/services/consult
fixes it.
Not clear to me: • why a missing
src/django_apps/feed/services/consult/__init__.py
would cause lint errors in
src/django_apps/files/models.py
? It seems like without it is using
/
at the python root instead of
/src
. • We have no other dirs named directories named
consult
Copy code
$ find . -type d -name consult
./src/django_apps/feed/services/consult
At least for pytest the init files are only required when there is a package name overlap. • When should we need init scripts?
3 Views