#development
Title
w
wonderful-iron-54019
06/25/2020, 4:08 PM[Extra requirements for linting thread]
we have dependencies that are 'dropped in'. to our code post deployment. Airflow and pyspark being the two primary ones
we install them as dev-dependencies in our run environment so that the linter recognizes them
but ddont want to specify them as dependencies for the target. tried specifying them as
extra_requirementsh
hundreds-father-404
06/25/2020, 4:10 PMWhat’s the error? It’s designed to allow you to add Pylint plugins, but you can use it for this same reason too. The result is that the pex to run Pylint will have the requirements
Also how come you don’t want the targets to include the deps?
w
wonderful-iron-54019
06/25/2020, 4:21 PMbecause they are installed on the deployed hardware
we load libraries intto a distributed runtime that includes these dependencies already
Copy code
Failed to resolve compatible distributions:
1: astroid==2.4.2 requires lazy-object-proxy==1.4.* but lazy-object-proxy 1.5.0 was resolved
2: Flask-AppBuilder==1.13.1 requires marshmallow<2.20,>=2.18.0 but marshmallow 3.6.1 was resolvedwhich is odd to me b/c our lockfile exhibits no issus
h
hundreds-father-404
06/25/2020, 5:20 PMastroidastroidrequirements.txtlockfilew
wonderful-iron-54019
06/25/2020, 5:20 PMyes
h
hundreds-father-404
06/25/2020, 5:22 PMOkay, hm, I suspect the conflict is between Pylint’s reqs vs. what you pinned
we load libraries intto a distributed runtime that includes these dependencies alreadyI’m not quite sure what you mean. These dependencies are already installed into the environment globally and you want to use those? If that’s the case, both
dependencies--pylint-extra-requirementsw
wonderful-iron-54019
06/25/2020, 5:23 PMwe have a virtualenv managed with
pipenvpylint srcthat lints everything at once, but it has these dev-dependencies installed in. the env for linting purposes
Okay, hm, I suspect the conflict is between Pylint’s reqs vs. what you pinnedI'm using the same versions of everything in my lock file as in my pants.toml so i suspect its a difference b/w the pipenv and pex resolver
h
hundreds-father-404
06/30/2020, 9:33 PMIs this still an issue? Also, I realized that Pants isn’t loading 3rd party requirements when running Pylint..which means you would be getting lots of
E0401: Unable to import 'foo' (import-error)w
wonderful-iron-54019
07/01/2020, 3:34 PM@hundreds-father-404 ah just saw this. dropped this work for time, but will circle back to it, yes we have. seen several. of these errors
i. assumed it had to do with. the
dev-dependenciesh
hundreds-father-404
07/01/2020, 4:24 PMOkay, yeah I reproduced it yesterday. Will try to fix today, although I’m bummed that this means performance will be slower because now we have to resolve requirements :/
How important is that part of Pylint to you all of checking that every import is actually valid? I’m wondering if it would be worth the trouble to add a flag so that people can ignore requirements and they would disable that lint in Pylint.
Also, would it help you with the migration from 1.30 to 2.0 if we cherry-picked the fix to 1.30? The two branches have diverged a bit, but if it’s helpful, I can make it work.
I’m not sure if you’ll want to have 1.30 Pylint fully working before 2.0, or if it’s fine to wait until 2.0
w
wonderful-iron-54019
07/02/2020, 1:06 PMsorry eric just saw this again
given that python is interpreted and not. compiled, it will be very handy at catching the class of error where we forget to add a dependency to our targets
a flag makes perfect sense to me if that's preferable than having it always on
i. don't think there's a huge need for cherry picking
we can continue to lint outside of pants until we get to 2.0