https://pantsbuild.org/ logo
#general
Title
# general
l

loud-laptop-17949

05/19/2022, 4:24 PM
I have hit what I think might be a pants bug, but maybe I am doing something wrong. I have a small refactor branch, which just breaks a few files up and when I run this I get a spurious error:
Copy code
./pants lint --changed-since=master
...
src/util/celery.py:25:0: E0401: Unable to import 'raven.contrib.celery' (import-error)
But when running directly it works:
Copy code
./pants lint src/util/celery.py  
...
āœ“ black succeeded.
āœ“ isort succeeded.
āœ“ pylint succeeded.
pants seems to be able to see the correct dependency:
Copy code
$ ./pants dependencies src/util/celery.py | grep raven
3rdparty#raven
src/django_projects/settings/raven.py
and this is occurring both locally and in CI, so I don't think it is related to any sort of cache / env issue on my machine
b

bitter-ability-32190

05/19/2022, 4:33 PM
I've seen something maybe related. Run with
--no-process-cleanup
and then crack open the sandbox. From there invoke the python interpreter (there'll be a "shim" script which you run which is spiritually
python
). Then try to import the submodule. Then do it again without
--changed-since
What I saw was that
python
was able to import the submodule, but when accessing it as an attribute of the parent module it failed. ...only in some sandboxes though. SO not exactly the same issue. How I sleep at night is assuming some other code imported the submodule and got linted first. Then
sys.modules
or something equivalent was populated in such a way that the offending file could access the submodule correctly. When being linted alone, that was no longer the case.
And then I left it under the rug. (And FWIW the error came from
mypy
, which I trust reasonably well to be authentic)
l

loud-laptop-17949

05/19/2022, 4:36 PM
ok, i have the sandbox and reproduced the error
is there a way to open ipython in the context of a pex?
b

bitter-ability-32190

05/19/2022, 4:37 PM
I dunno about
ipython
but you can use the python interpreter. The
__run.sh
script uses a shell file as it's "python". sniff at that run script to find it
l

loud-laptop-17949

05/19/2022, 4:37 PM
ok
b

bitter-ability-32190

05/19/2022, 4:38 PM
So in one instance I see:
$'--python-executable=./requirements_venv.pex_bin_python_shim.sh'
Running that shell script as-is gives me a Python interpreter
l

loud-laptop-17949

05/19/2022, 4:46 PM
hmm i don't have a requirements shim
trying to figure out but coming up a bit empty:
Copy code
$ ls
__plugins				pylint.pex				pylint_runner.pex_bin_python_shim.sh	reports					src
__run.sh				pylint_runner.pex			pylint_runner.pex_pex_shim.sh		requirements.pex
b

bitter-ability-32190

05/19/2022, 4:47 PM
Right, it'll be a different shim. Check to see which shim
__run.sh
is using
Likely
pylint_runner.pex_bin_python_shim.sh
l

loud-laptop-17949

05/19/2022, 4:48 PM
run.sh uses
pylint_runner.pex_pex_shim.sh
. I can run that an get an invocation of
pylint
. trying to figure out how to tell it to give me
python
instead
b

bitter-ability-32190

05/19/2022, 4:48 PM
Ohhhh hmm yeah that makes sense
In my case it's
mypy
, which is being told a py executable
Might crack the shim open and sniff around?
w

witty-crayon-22786

05/19/2022, 4:51 PM
which version of Pants is this? and is it using transitive dependencies for
pylint
?
that was a relatively recent development, iirc
l

loud-laptop-17949

05/19/2022, 4:51 PM
2.11.0
not transitive for pylint
ok got an interpretter
b

bitter-ability-32190

05/19/2022, 4:55 PM
FWIW I think what @witty-crayon-22786 is referring to is a Pants bugfix which went into 2.10: https://github.com/pantsbuild/pants/pull/14289
l

loud-laptop-17949

05/19/2022, 5:03 PM
so I got to the point where I can run an interpretter in the env created by the pants run which fails. but doing an import there works fine
Copy code
>>> import raven.contrib.celery
>>>
is that what you wanted me to test @bitter-ability-32190?
interestingly if I try to import the source file it fails in other ways
Copy code
>>> import src.util.celery
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/private/var/folders/77/r5wzbyf101vd27rfq9__chyc0000gq/T/process-executiong1J3XY/src/util/celery.py", line 21, in <module>
    from color_django_extensions import honeycomb_utils
ModuleNotFoundError: No module named 'color_django_extensions'
b

bitter-ability-32190

05/19/2022, 5:04 PM
Yup. Wack. IMO
pylint
uses
astroid
for parsing files and handling imports and stuff, and there be lots of bugs there šŸ™ˆ
l

loud-laptop-17949

05/19/2022, 5:05 PM
fwiw pants does see the correct dependencies:
Copy code
./pants dependencies src/util/celery.py 
3rdparty#celery
3rdparty#ddtrace
3rdparty#django
3rdparty#honeycomb-beeline
3rdparty#psutil
3rdparty#raven
3rdparty/internal:clrenv
src/__init__.py
src/color_django_extensions/honeycomb_utils.py
src/django_projects/settings/raven.py
src/util/__init__.py
src/util/common.py:common
src/util/sentry.py
src/util/tasks.py
b

bitter-ability-32190

05/19/2022, 5:06 PM
ModuleNotFoundError: No module named 'color_django_extensions'
probably due to source root stripping perhaps? šŸ¤·ā€ā™‚ļø
l

loud-laptop-17949

05/19/2022, 5:06 PM
yeah perhaps
I am going to give up on this and just mark that line with
pylint disable
. but I would like to leave a paper trail that I can link to? ok if I file a github issue?
āœ… 1
šŸ™Œ 1
b

bitter-ability-32190

05/19/2022, 5:14 PM
Please šŸ™ Include everything you think is relevant
Do you use mypy?
If so, do you see this with mypy? If not I'd very much believe
astroid
is to blame šŸ˜…
l

loud-laptop-17949

05/19/2022, 5:17 PM
we aren't running mypy yet
yeah but why does it work when I run directly and fail with --changed-since?
h

happy-kitchen-89482

05/19/2022, 5:29 PM
FWIW you can force any pex to drop into an interpreter with PEX_INTERPRETER=1
šŸ‘€ 1
E.g.,
PEX_INTERPRETER=1 ./name_of_shim.sh
l

loud-laptop-17949

05/19/2022, 5:30 PM
oh nice
4 Views