https://pantsbuild.org/ logo
Join Slack
Powered by
I am trying to turn on pants-based linting for mor...
# general
l
I am trying to turn on pants-based linting for more of our python files, but am running into an error that doesn't make sense to me, which works when using pylint directly– 
src/django_apps/files/models.py:21:0: E0401: Unable to import 'util.url' (import-error)
Though pants shows them as a dependency:
Copy code
$ p dependencies src/django_apps/files/models.py
...
src/util/url.py
What am I missing here?
👀 1
h
huh.. unclear. we include all transitive deps in the sandbox. i recommed trying https://www.pantsbuild.org/docs/troubleshooting#debug-tip-inspect-the-sandbox-with---no-process-cleanup
e
Hard to replicate. 
./pants lint src/django_apps/files/models.py
works 
./pants lint src/django_apps/files::
works Only happens on very large pants lint runs and which files see these unable to import errors for seems to shift arond
l
./pants filter --sep="\0" :: | xargs -0 ./pants lint
seems to always trigger it tho
e
But for the same files?
l
yes, I think
s
./pants filter :: | wc -l
prints 7450 for me. 
xargs
has a default value 5000 for 
-n
(at least according to 
man
), is it possible we’re splitting transitive deps across separate calls to 
./pants lint
?
e
I would think pants should be able to figure that out. same that 
./pants lint <one file>
works
Ok I'm able to capture a broken sandbox from this. running 
./__run.sh
repeats the error. What now?
h
Great. How is it broken? The necessary file is not present for example?
e
Hmm, little surprising. The files do seem to be here
👀 1
Copy code
/tmp/process-execution9TnW2S$ ./__run.sh |grep import-error|cut -d: -f5|sort | uniq -c
      2  Unable to import 'util.async_task' (import-error)
      4  Unable to import 'util.celery' (import-error)
      4  Unable to import 'util.common' (import-error)
     17  Unable to import 'util.constants' (import-error)
      1  Unable to import 'util.date' (import-error)
      2  Unable to import 'util.iterables' (import-error)
      2  Unable to import 'util.iter' (import-error)
      2  Unable to import 'util.jira' (import-error)
      5  Unable to import 'util.phone_number' (import-error)
      1  Unable to import 'util.requests' (import-error)
      5  Unable to import 'util.s3' (import-error)
      2  Unable to import 'util.string' (import-error)
     19  Unable to import 'util.testutil' (import-error)
      2  Unable to import 'util.text_mailer' (import-error)
      3  Unable to import 'util.url' (import-error)
h
hmmmm. I wonder if maybe source routes are not being set up correctly...? Although it sounds like both share the same of 
src
though
e
/tmp/process-execution9TnW2S/src/util/{module}.py
exists for all of the above
👍 1
w
I wonder if maybe source roots are not being set up correctly...?
and/or 
__init__.py
files
➕ 1
e
In this case the files being linted and the util modules that aren't found are all in the same root_pattern 
/src
Adding a 
__init__.py
file does not help
Actually you know i think this is a root_pattern problem
we (unfortunately) have 
/
and 
/src
as root patterns in our 
pants.toml
h
checj __run.sh for PEX_EXTRA_SYS_PATH
e
Copy code
************* Module src.django_apps.files.models
src/django_apps/files/models.py:21:0: E0401: Unable to import 'util.url' (import-error)
the module being linted is 
src.django_apps.files.models
, not 
django_apps.files.models
👀 1
export PEX_EXTRA_SYS_PATH=$'build/python-packages/color-django-extensions:src:__plugins'
from the 
PEX_EXTRA_SYS_PATH
it looks right
with 
src
in the PATH i would think 
import util.url
would work
Copy code
/tmp/process-execution9TnW2S$ ls -lah src/util/url.py 
-rw-r--r-- 1 mpcusack mpcusack 1.2K Apr 19 14:42 src/util/url.py
h
I would too. That is at least how it's worked with the other organizations who have been using Pylint, including Toolchain You could try modifying the run.SH script and invoke that
e
Well this is weird.
Copy code
/tmp/process-execution9TnW2S$ PEX_EXTRA_SYS_PATH=$'build/python-packages/color-django-extensions:src:__plugins' ./pylint_runner.pex_pex_shim.sh $'--rcfile=.pylintrc' $'--jobs=8' src/django_apps/files/models.py
Has no errors
but if i run the 
__run.sh
script that passes ~280 additional files it fails
h
I wonder if those additional files all belong to the same source root?
Playing around with that folder will be the best approach to debug this I think. It's plausible there is an edge case we have not experienced before. Trying to get a minimal reproduction would be hugely helpful
e
they do all belong to the same source root
👀 1
The sandbox was created just linting 
src/django_apps/::
and looking at the args that is true
with 
build/python-packages/color-django-extensions
on the path, one of them is likely importing from that alternative python root
I gotta switch over to something else for the next hour or so, but will come back to this and see if I can figure out exactly what combo or files being linted triggers the error
👍 1
So narrowed it down to this reproducible failure:
Copy code
./pants lint src/django_apps/files/models.py     src/django_apps/feed/services/consult/util.py
Adding a 
__init__.py
to 
src/django_apps/feed/services/consult
fixes it.
Not clear to me: • why a missing 
src/django_apps/feed/services/consult/__init__.py
would cause lint errors in 
src/django_apps/files/models.py
? It seems like without it is using 
/
at the python root instead of 
/src
. • We have no other dirs named directories named 
consult
Copy code
$ find . -type d -name consult
./src/django_apps/feed/services/consult
At least for pytest the init files are only required when there is a package name overlap. • When should we need init scripts?
6 Views
Open in Slack
PreviousNext
Powered by