I have hit what I think might be a pants bug but maybe I am Pants #general

I have hit what I think might be a pants bug, but ...

loud-laptop-17949

05/19/2022, 4:24 PM

I have hit what I think might be a pants bug, but maybe I am doing something wrong. I have a small refactor branch, which just breaks a few files up and when I run this I get a spurious error:

Copy code

./pants lint --changed-since=master
...
src/util/celery.py:25:0: E0401: Unable to import 'raven.contrib.celery' (import-error)

But when running directly it works:

Copy code

./pants lint src/util/celery.py  
...
✓ black succeeded.
✓ isort succeeded.
✓ pylint succeeded.

loud-laptop-17949

05/19/2022, 4:26 PM

pants seems to be able to see the correct dependency:

Copy code

$ ./pants dependencies src/util/celery.py | grep raven
3rdparty#raven
src/django_projects/settings/raven.py

loud-laptop-17949

05/19/2022, 4:26 PM

and this is occurring both locally and in CI, so I don't think it is related to any sort of cache / env issue on my machine

bitter-ability-32190

05/19/2022, 4:33 PM

I've seen something maybe related. Run with

--no-process-cleanup

and then crack open the sandbox. From there invoke the python interpreter (there'll be a "shim" script which you run which is spiritually

python

). Then try to import the submodule. Then do it again without

--changed-since

bitter-ability-32190

05/19/2022, 4:35 PM

What I saw was that

python

was able to import the submodule, but when accessing it as an attribute of the parent module it failed. ...only in some sandboxes though. SO not exactly the same issue. How I sleep at night is assuming some other code imported the submodule and got linted first. Then

sys.modules

or something equivalent was populated in such a way that the offending file could access the submodule correctly. When being linted alone, that was no longer the case.

bitter-ability-32190

05/19/2022, 4:36 PM

And then I left it under the rug. (And FWIW the error came from

mypy

, which I trust reasonably well to be authentic)

loud-laptop-17949

05/19/2022, 4:36 PM

ok, i have the sandbox and reproduced the error

loud-laptop-17949

05/19/2022, 4:36 PM

is there a way to open ipython in the context of a pex?

bitter-ability-32190

05/19/2022, 4:37 PM

I dunno about

ipython

but you can use the python interpreter. The

__run.sh

script uses a shell file as it's "python". sniff at that run script to find it

loud-laptop-17949

05/19/2022, 4:37 PM

bitter-ability-32190

05/19/2022, 4:38 PM

So in one instance I see:

$'--python-executable=./requirements_venv.pex_bin_python_shim.sh'

bitter-ability-32190

05/19/2022, 4:38 PM

Running that shell script as-is gives me a Python interpreter

loud-laptop-17949

05/19/2022, 4:46 PM

hmm i don't have a requirements shim

loud-laptop-17949

05/19/2022, 4:47 PM

trying to figure out but coming up a bit empty:

Copy code

$ ls
__plugins				pylint.pex				pylint_runner.pex_bin_python_shim.sh	reports					src
__run.sh				pylint_runner.pex			pylint_runner.pex_pex_shim.sh		requirements.pex

bitter-ability-32190

05/19/2022, 4:47 PM

Right, it'll be a different shim. Check to see which shim

__run.sh

is using

bitter-ability-32190

05/19/2022, 4:48 PM

Likely

pylint_runner.pex_bin_python_shim.sh

loud-laptop-17949

05/19/2022, 4:48 PM

run.sh uses

pylint_runner.pex_pex_shim.sh

. I can run that an get an invocation of

pylint

. trying to figure out how to tell it to give me

python

instead

bitter-ability-32190

05/19/2022, 4:48 PM

Ohhhh hmm yeah that makes sense

bitter-ability-32190

05/19/2022, 4:49 PM

In my case it's

mypy

, which is being told a py executable

bitter-ability-32190

05/19/2022, 4:49 PM

Might crack the shim open and sniff around?

witty-crayon-22786

05/19/2022, 4:51 PM

which version of Pants is this? and is it using transitive dependencies for

pylint

witty-crayon-22786

05/19/2022, 4:51 PM

that was a relatively recent development, iirc

loud-laptop-17949

05/19/2022, 4:51 PM

2.11.0

loud-laptop-17949

05/19/2022, 4:51 PM

not transitive for pylint

loud-laptop-17949

05/19/2022, 4:55 PM

ok got an interpretter

bitter-ability-32190

05/19/2022, 4:55 PM

FWIW I think what @witty-crayon-22786 is referring to is a Pants bugfix which went into 2.10: https://github.com/pantsbuild/pants/pull/14289

loud-laptop-17949

05/19/2022, 5:03 PM

so I got to the point where I can run an interpretter in the env created by the pants run which fails. but doing an import there works fine

Copy code

>>> import raven.contrib.celery
>>>

loud-laptop-17949

05/19/2022, 5:03 PM

is that what you wanted me to test @bitter-ability-32190?

loud-laptop-17949

05/19/2022, 5:04 PM

interestingly if I try to import the source file it fails in other ways

Copy code

>>> import src.util.celery
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/private/var/folders/77/r5wzbyf101vd27rfq9__chyc0000gq/T/process-executiong1J3XY/src/util/celery.py", line 21, in <module>
    from color_django_extensions import honeycomb_utils
ModuleNotFoundError: No module named 'color_django_extensions'

bitter-ability-32190

05/19/2022, 5:04 PM

Yup. Wack. IMO

pylint

uses

astroid

for parsing files and handling imports and stuff, and there be lots of bugs there 🙈

loud-laptop-17949

05/19/2022, 5:05 PM

fwiw pants does see the correct dependencies:

Copy code

./pants dependencies src/util/celery.py 
3rdparty#celery
3rdparty#ddtrace
3rdparty#django
3rdparty#honeycomb-beeline
3rdparty#psutil
3rdparty#raven
3rdparty/internal:clrenv
src/__init__.py
src/color_django_extensions/honeycomb_utils.py
src/django_projects/settings/raven.py
src/util/__init__.py
src/util/common.py:common
src/util/sentry.py
src/util/tasks.py

bitter-ability-32190

05/19/2022, 5:06 PM

ModuleNotFoundError: No module named 'color_django_extensions'

probably due to source root stripping perhaps? 🤷‍♂️

loud-laptop-17949

05/19/2022, 5:06 PM

yeah perhaps

loud-laptop-17949

05/19/2022, 5:13 PM

I am going to give up on this and just mark that line with

pylint disable

. but I would like to leave a paper trail that I can link to? ok if I file a github issue?

✅ 1

🙌 1

bitter-ability-32190

05/19/2022, 5:14 PM

Please 🙏 Include everything you think is relevant

bitter-ability-32190

05/19/2022, 5:14 PM

Do you use mypy?

bitter-ability-32190

05/19/2022, 5:14 PM

If so, do you see this with mypy? If not I'd very much believe

astroid

is to blame 😅

loud-laptop-17949

05/19/2022, 5:17 PM

we aren't running mypy yet

loud-laptop-17949

05/19/2022, 5:17 PM

yeah but why does it work when I run directly and fail with --changed-since?

loud-laptop-17949

05/19/2022, 5:17 PM

https://github.com/pantsbuild/pants/issues/15545

happy-kitchen-89482

05/19/2022, 5:29 PM

FWIW you can force any pex to drop into an interpreter with PEX_INTERPRETER=1

👀 1

happy-kitchen-89482

05/19/2022, 5:29 PM

E.g.,

PEX_INTERPRETER=1 ./name_of_shim.sh

loud-laptop-17949

05/19/2022, 5:30 PM

oh nice

4 Views

Open in Slack

Previous Next