https://pantsbuild.org/ logo
Join Slack
Powered by
[Extra requirements for linting thread]
# development
w
[Extra requirements for linting thread]
we have dependencies that are 'dropped in'. to our code post deployment. Airflow and pyspark being the two primary ones
we install them as dev-dependencies in our run environment so that the linter recognizes them
but ddont want to specify them as dependencies for the target. tried specifying them as 
extra_requirements
 but that led to some resolution errors, and im not sure that's the intention. of that config property
h
What’s the error? It’s designed to allow you to add Pylint plugins, but you can use it for this same reason too. The result is that the pex to run Pylint will have the requirements
Also how come you don’t want the targets to include the deps?
w
because they are installed on the deployed hardware
we load libraries intto a distributed runtime that includes these dependencies already
Copy code
Failed to resolve compatible distributions:
1: astroid==2.4.2 requires lazy-object-proxy==1.4.* but lazy-object-proxy 1.5.0 was resolved
2: Flask-AppBuilder==1.13.1 requires marshmallow<2.20,>=2.18.0 but marshmallow 3.6.1 was resolved
which is odd to me b/c our lockfile exhibits no issus
h
astroid
is a requirement of Pylint. Does 
astroid
show up in your 
requirements.txt
or 
lockfile
already?
w
yes
h
Okay, hm, I suspect the conflict is between Pylint’s reqs vs. what you pinned
we load libraries intto a distributed runtime that includes these dependencies already
I’m not quite sure what you mean. These dependencies are already installed into the environment globally and you want to use those? If that’s the case, both 
dependencies
and 
--pylint-extra-requirements
wouldn’t help because they would re-install the deps from scratch from PyPI. How are you doing this with v1 currently?
w
we have a virtualenv managed with 
pipenv
(another enhancement i'd like to work on for pants) and run 
pylint src
that lints everything at once, but it has these dev-dependencies installed in. the env for linting purposes
Okay, hm, I suspect the conflict is between Pylint’s reqs vs. what you pinned
I'm using the same versions of everything in my lock file as in my pants.toml so i suspect its a difference b/w the pipenv and pex resolver
h
Is this still an issue? Also, I realized that Pants isn’t loading 3rd party requirements when running Pylint..which means you would be getting lots of 
E0401: Unable to import 'foo' (import-error)
Has this been an issue for you?
w
@hundreds-father-404 ah just saw this. dropped this work for time, but will circle back to it, yes we have. seen several. of these errors
i. assumed it had to do with. the 
dev-dependencies
imports described above. since they were present in several places but yeah this would be a. larger issue for sure
h
Okay, yeah I reproduced it yesterday. Will try to fix today, although I’m bummed that this means performance will be slower because now we have to resolve requirements :/ How important is that part of Pylint to you all of checking that every import is actually valid? I’m wondering if it would be worth the trouble to add a flag so that people can ignore requirements and they would disable that lint in Pylint.
Also, would it help you with the migration from 1.30 to 2.0 if we cherry-picked the fix to 1.30? The two branches have diverged a bit, but if it’s helpful, I can make it work. I’m not sure if you’ll want to have 1.30 Pylint fully working before 2.0, or if it’s fine to wait until 2.0
w
sorry eric just saw this again
given that python is interpreted and not. compiled, it will be very handy at catching the class of error where we forget to add a dependency to our targets
a flag makes perfect sense to me if that's preferable than having it always on
i. don't think there's a huge need for cherry picking
we can continue to lint outside of pants until we get to 2.0
Open in Slack
PreviousNext
Powered by