So I’ve been looking into why I cannot install a particular pylint version… I’m trying to use

2.4.15

, which needs python 3.7 (which is what our repo is supposed to use). When I try to generate a lockfile it says:

ERROR: Could not find a version that satisfies the requirement pylint>=2.14.5
ERROR: No matching distribution found for pylint>=2.14.5

If I look in the logs, it says it’s using 3.7.4 to bootstrap the env, and if I check in the sandbox, the pex file is python 3.7.4. But if I run

__run.sh

, it gets changed to 3.6.8 (this was a pyenv artifact). If I use the pex from the sandbox folder to get pylint, it works. If I run

__run.sh

, it obviously stops working. There’s also nothing in the log file mentioned in the command line printed. How can I debug/fix this, except by uninstalling any

<3.6

I have?

@ancient-france-42909 can you provide the full command line + output of

./pants generate-lockfiles --resolve=pylint -ldebug

This is the same issue we solved yesterday:

"./pex", "lock", "create", ..., "--interpreter-constraint", "CPython==3.7.*"

You need to adjust your interpreter-constraint from

CPython==3.7.*

CPython>=3.7.2,<3.8

I seriously doubt that’s the case.

And its because we're creating a universal lock that should work on any machine and not just yours.

First of all, my interpreter constraint is

>=3.7.4

That is definitely not what the logs say.

Untitled.txt

Are there any other relevant PANTS_ env vars in your environment or a ~/.pants.rc file on your machine?

Okay, so. It seems that was the issue, in a BUILD file. But, that’s broken.

>=3.7.4

is more specific than

==3.7.*

.

If you want to distill that down to an issue, you may be right there. I'm trying to keep my focus small since this has been pretty hard to grok from a support end though. Does the 2.14.5 (temporary) solution lead to a pylint fix for you?

Well, ironically, I need to wait to generate my mypy lockfile

FWIW, passing more than 1 IC means OR them to Pex. Pants may or may not be incorrect in passing two ICs to Pex there.

It’s not my fault it takes 20 minutes to generate lockfiles.

The Pex docs fwiw:

--interpreter-constraint INTERPRETER_CONSTRAINT
                        Constrain the selected Python interpreter. Specify with Requirement-style syntax, e.g. "CPython>=2.7,<3" (A CPython interpreter with version >=2.7 AND version <3), ">=2.7,<3" (Any Python interpreter with version >=2.7 AND version <3) or "PyPy" (A
                        PyPy interpreter of any version). This argument may be repeated multiple times to OR the constraints. Try `PEX_TOOLS=1 /home/jsirois/.venv/pex/bin/python3 /home/jsirois/.venv/pex/bin/pex interpreter --verbose --indent 4` to find the exact
                        interpreter constraints of /home/jsirois/.venv/pex/bin/python3 and `PEX_TOOLS=1 /home/jsirois/.venv/pex/bin/pex interpreter --all --verbose --indent 4` to find out the interpreter constraints of all Python interpreters on the $PATH. (default: [])

So Pants may be doing the wrong thing here for sure.

My other question is, what’s with the arg list too long?

Not sure about that. If you agree to keep focus on the pylint, I'd like to handle that 1st,

So, I tried to run it, it took 4 minutes to complain that the bandit lockfile is out of date. 🙂

That's a bit surprising since your command line restricts to just

--resolve=pylint

Oh, sorry. I thought I sid that it could generate a lockfile after I removed the

==3.7.*

constraint. (Which, in my defense, was added last night) This is for trying to lint, to see if this actually fixes the namespaces issue

Ok. I'm still a bit lost, but let me know if the pylint issue is fixed or fails with full command line + `-ldebug`and full output.

Which pylint issue?

Sorry, I'm just aware of the namespace package pylint issue.

Yeah, the lockfile was generated with 2.4.15, I’m trying to figure out if that fixes the original issue, had to change the CI to not run typechecks since those would fail because of the outdated lockfile. Fixed bandit and isort for lint, running it both locally and on the CI to see which will finish first. My laptop (because apple) overheats after about 5 minutes of building virtualenvs and get throttled to like 30%, so, yeah

Ok.

Okay, so, it didn’t fix the issue. 😞

Ok. Yesterday I tried to setup what I though was an example repo locally to simulate your issue and I could not hit it. If you could do something similar, that would definitely be the easiest way to help you figure this out - for me. That is a good bit of work for you though.

Yeah, I couldn’t get a simple reproducible case either.

Frustrating.

The astroid ticket (and pylint one) does suggest that’s the case.

Pants / Pex / Pip understand vcs so you could try HEAD with

"pylint @ git+<https://github.com/PyCQA/pylint/commit/2fafadc8aca1831c594cafd35a46b691eaa3e357>"

as your custom pylint version.

Yeah, I’ll need to get the astroid commit, though.

You just do the same for astroid using https://www.pantsbuild.org/docs/reference-pylint#section-extra-requirements

Though, that was released in 3 versions so far it, seems.

So pylint HEAD says: https://github.com/PyCQA/pylint/blob/main/pyproject.toml#L40

https://github.com/PyCQA/astroid/commit/18483dc169987299b58effd95d65c1f6e8fc36a6 - 2.12.10 to 12

Ok, then my original suggestion should give you a fighting chance.

2.11.7

Ok, excellent. This has a glimmer of hope then using the vcs requirement.

Is that syntax supposed to work, or do I need the normal pip stuff?

Yeah, that's why I put the @ in ther. You can do it without pylint@, but you need to add

#egg=pylint

to the end of the vcs URL if so.

Anyway, will report back in 20-30 minutes if it worked.

Excellent, thanks.

It is using astroid

2.12.2

, but I still get the errors 😞 But, I’m gonna try harder to find out how to reproduce this. For example, in the sandbox, if I run pylint on all the files, I get the error. If I just run it on the ones that have an error, I don’t. So I’ll just start trying to add files and see when it errors out

Ok. Yeah, slogging through that and getting a on/off repro would be excellent.

OH

I'm not following completely, but it sounds like you're on to something.

We have a module named

audit

in another namespace, and that’s what’s causing the issue.

I would need to see the sys.path + the 2 file paths passed to pylint in your repro case to make more sense of this. But if you're on the way to figuring this out on your own, there is no need.

Okay, I guess this is a different issue,

is_namespace

from astroid returns true.

Definitley. The rule system is pretty flexible. I am not familiar with that rule set though. Let me see if there is an option for this. We do it for other rule sets I'm pretty sure.

Yeah, but we’d need to run them separately.

That can work, but there is upkeep of tagged targets and you have to run multiple Pants commands seperately, one per set of tags you want to operate on in isolation. So not ideal, but sometimes enough.

I mean, we plan to move to a system where we run every lib/app’s tests/checks in a separate container, so it won’t be a huge deal for much longer (2 weeks?)

Sounds good.

We already use tagging, but just for filtering, and I’m not 100% sure from there how we could split the specs by tags, I guess?

I'm not sure I follow. Tags do in fact filter any specs given on the CLI, so yes I think.

I mean, we can filter, but we cannot group by tags, so we’d need to get all the tags and manually iterate. At that point, we might as well just iterate over the sources roots.

Ah, gotcha. Yeah, sounds about right.

I managed to get a simple example reproducing this issue, I’ll make a ticket with pylint tomorrow. Thank you very much for the help, and sorry for my snarky remarks, most of this was my fault, but the very slow feedback loop and generally annoying debugging things did annoy me 😞

No worries George. Thanks for sticking through this and contributing back to Pylint.

I have a feeling that because I picked up this upgrade ticket, I will be doing that a bit 😛

Hrm. I'm glad you hunted down some error there, but it is not the original error you reported in another thread, which was:

audit/lib/tests/unit/test_validate_event.py:73:24: E1101: Module 'audit' has no 'lib' member (no-member)

I.E.: E1101 and not E0401 or E0611

It’s because I’m importing

audit.lib

there, vs

import audit; audit.lib

AHA! So you solved that other thread! That's excellent.

I found the cause and I’m making a PR right now. I only had like 2 hours today to do any actual work, Mondays seem to be the day everyone wants to have meetings in my company.

Ok, maybe not solved then. Ok, great. I had actually thought that might be the issue for E1101 and concocted that scenario in my example repo but could not get E1101 to trigger.