https://pantsbuild.org/ logo
#general
Title
# general
h

high-egg-2153

12/30/2020, 8:51 PM
Hey. Just starting to experiment with pants for a python project. I'm having an issue trying to run anything. When I run:
./pants run accounts_report/app.py
I am getting the following error:
from pkg_resources import get_distribution, parse_version
ModuleNotFoundError: No module named 'pkg_resources'
Some details:
Python 3.8.6
pip 19.3.1
Manjaro Linux
Any help would be greatly appreciated.
e

enough-analyst-54434

12/30/2020, 8:54 PM
So
pkg_resources
is provided by `setuptools`(https://pypi.org/project/setuptools/) which you probably already knew. In order to resolve that dependency, Pants requires a small bit of setup - namely you have to tell Pants the version of setuptools you want. More here: https://www.pantsbuild.org/docs/python-third-party-dependencies Before going further, let me know how much of this you already have setup / think you have setup.
h

high-egg-2153

12/30/2020, 8:57 PM
I know very little so far 🙂. So It's in the root BUILD file I need to tell pants about setup tools?
e

enough-analyst-54434

12/30/2020, 8:57 PM
And - fwiw - Pants will isolate your code from your environment as much as possible so things are reproducible. Unfortunately, we can't isolate the OS (Manjaro) nor Python interpreters (yet), but we do isolate away Pip.
h

high-egg-2153

12/30/2020, 8:58 PM
kk - good to know. I've been assuming my pip install was messed up so I've been trying to uninstall/reinstall/repair it
e

enough-analyst-54434

12/30/2020, 8:59 PM
So you need - somewhere - could be in root BUILD but need not be, a requirements.txt / BUILD pair of files (requirements.txt is not needed but its standard). That pair is the 1st example at the top of https://www.pantsbuild.org/docs/python-third-party-dependencies
In particular you need your requirements.txt to have a
setuptools==???
line. And then, if you look a little lower on that page, you need a module mapping, so - in the sibling BUILD file to requirements.txt you need:
Copy code
python_requirements(
    module_mapping={
        "setuptools": ["pkg_resources"]
    },
)
h

high-egg-2153

12/30/2020, 9:00 PM
Ah ok starting to make some sense. Trying that now
e

enough-analyst-54434

12/30/2020, 9:01 PM
This module mapping business isn't commonly needed. Unfortunately, pkg_resources is commonly used and its owning project - setuptools - is a bit of an oddball in packaging up both
setuptools
and
pkg_resources
top-level packages.
So you need to tell Pants that setuptools also houses pkg_resources.
h

high-egg-2153

12/30/2020, 9:03 PM
So now I have
python_requirements(
module_mapping={
"setuptools": ["pkg_resources"]
},
)
and
setuptools==51.1.1
These are at the root
e

enough-analyst-54434

12/30/2020, 9:04 PM
The former in BUILD and the latter in requirements.txt - right?
h

high-egg-2153

12/30/2020, 9:04 PM
Yes - still get the error though
app.py is in a folder that also has a BUILD file. Does that need to be in there as well as root?
e

enough-analyst-54434

12/30/2020, 9:05 PM
OK, a few questions: 1. What version of Pants? 2. What other relevant BUILD file content do you have? IE: Do you have a pex_binary target corresponding to app.py?
h

high-egg-2153

12/30/2020, 9:07 PM
1. I started on 2.1.0 and updated to 2.1.1 to see if that would help 2. Yes I have a pex_binary target in the BUILD in the folder that looks like
pex_binary(
sources=["app.py"],
entry_point='<http://accounts_report.app|accounts_report.app>',
dependencies=[
':accounts_report_lib',
],
)
python_library(
name="accounts_report_lib"
)
python_tests(name = 'tests')
e

enough-analyst-54434

12/30/2020, 9:08 PM
Ok - you couldn't and perhaps still can't mix sources= and entry_point=. I'm reviewing PRs to check on that, but relevant doc is here: https://www.pantsbuild.org/docs/python-package-goal#creating-a-pex-file-from-a-pex_binary-target
You likely can just kill sources= though since the sibling python_library - which you declare a dep on, will have that file.
❤️ 1
Its strange that you still need a pex_binary target at all in 2.2, but still better than before.
h

high-egg-2153

12/30/2020, 9:12 PM
So should I be using 2.2 at this point?
e

enough-analyst-54434

12/30/2020, 9:12 PM
No, 2.1 is absolutely fine, but you really need to follow along with the 2.1 docs closely.
h

high-egg-2153

12/30/2020, 9:13 PM
kk - I think I need to do some more reading. Thanks for the help!
e

enough-analyst-54434

12/30/2020, 9:15 PM
To put a fine point on it - you're using approach #3 here: https://www.pantsbuild.org/docs/python-package-goal#creating-a-pex-file-from-a-pex_binary-target But that approach requires the exclude of the main module from the python_library target.
Where the main module is app.py in your case.
h

high-egg-2153

12/30/2020, 9:16 PM
yes I just saw that.
e

enough-analyst-54434

12/30/2020, 10:03 PM
I've got to run for a bit but it would be good to hear back on progress when you get a chance.
h

hundreds-father-404

12/30/2020, 10:04 PM
I just got to our cabin and am back online, so I can help while John is out 🙂 Thank you John for helping with the above!
h

high-egg-2153

12/30/2020, 10:27 PM
Back - I have a couple more minutes to look at this now. I setup the BUILD and requirements.txt but still getting the No module named "pkg_resources"
h

hundreds-father-404

12/30/2020, 10:31 PM
K, can you run
./pants dependencies  --transitive path/to/app.py
please? Or whichever file has the
pkg_resources
import
h

high-egg-2153

12/30/2020, 10:31 PM
accounts_report:SQLAlchemy
accounts_report:dash
accounts_report:dash-table
accounts_report:pandas
accounts_report:requirements.txt
👍 1
requirements.txt:
dash>=1.18.1
dash-table>=4.11.1
pandas>=1.1.5
SQLAlchemy>=1.3.22
setuptools==51.1.1
BUILD:
python_requirements(
module_mapping={
"setuptools": ["pkg_resources"]
},
)
pex_binary(
sources=["app.py"],
entry_point='<http://accounts_report.app|accounts_report.app>',
)
python_tests(name = 'tests')
h

hundreds-father-404

12/30/2020, 10:35 PM
Hm weird that all the others still show up correctly, but not this one. For the sake of debugging, could you please add
accounts_report:setuptools
to the
dependencies
for the
pex_binary
target and rerun your test? That will confirm this is solely an issue with dependency inference not being picked up
h

high-egg-2153

12/30/2020, 10:37 PM
./pants dependencies  --transitive accounts_report/app.py
accounts_report:SQLAlchemy
accounts_report:dash
accounts_report:dash-table
accounts_report:pandas
accounts_report:requirements.txt
accounts_report:setuptools
18:37:58.11 [INFO] Completed: Building requirements.pex with 5 requirements: SQLAlchemy>=1.3.22, dash-table>=4.11.1, dash>=1.18.1, pandas>=1.1.5, setuptools==51.1.1
Traceback (most recent call last):
File "/home/greg/Work/foo/foo/.pants.d/tmpsnbrvmgu/accounts_report.pex/.bootstrap/pex/pex.py", line 487, in execute
File "/home/greg/Work/foo/foo/.pants.d/tmpsnbrvmgu/accounts_report.pex/.bootstrap/pex/pex.py", line 404, in _wrap_coverage
File "/home/greg/Work/foo/foo/.pants.d/tmpsnbrvmgu/accounts_report.pex/.bootstrap/pex/pex.py", line 435, in _wrap_profiling
File "/home/greg/Work/foo/foo/.pants.d/tmpsnbrvmgu/accounts_report.pex/.bootstrap/pex/pex.py", line 543, in _execute
File "/home/greg/Work/foo/foo/.pants.d/tmpsnbrvmgu/accounts_report.pex/.bootstrap/pex/pex.py", line 645, in execute_entry
File "/home/greg/Work/foo/foo/.pants.d/tmpsnbrvmgu/accounts_report.pex/.bootstrap/pex/pex.py", line 653, in execute_module
File "/usr/lib/python3.8/runpy.py", line 210, in run_module
return _run_code(code, {}, init_globals, run_name, mod_spec)
File "/usr/lib/python3.8/runpy.py", line 87, in _run_code
exec(code, run_globals)
File "/home/greg/Work/foo/foo/.pants.d/tmpsnbrvmgu/./accounts_report/app.py", line 12, in <module>
import dash_bootstrap_components as dbc
ModuleNotFoundError: No module named 'dash_bootstrap_components'
after running:
./pants run accounts_report
Just noticed that that is not the same error
h

hundreds-father-404

12/30/2020, 10:42 PM
For that one, should be fixed by adding
dash-bootstrap-components
to your
requirements.txt
so that the
python_requirements()
call sets that up as a top-level requirement, then dependency inference can do its thing I’m not sure what’s going on with
setuptools
, as it’s worked in many other repos, including Pants. I’m trying to think of why here might be different. Your BUILD file looks good
h

high-egg-2153

12/30/2020, 10:43 PM
Once I update requirements.txt do I need to do something to get it to see the changes?
h

hundreds-father-404

12/30/2020, 10:45 PM
You should not need to. The
python_requirements
thing is a macro that creates one distinct
python_requirement_library
per each entry in your requirements.txt, which you can confirm by running
./pants list ::
to see every target you have in your project Pants will notice requirements.txt has changed and recreate the mapping of module names to which code it comes from. You can check by running
./pants dependencies path/to/app.py
or
./pants dependencies --transitive path/to/app.py
h

high-egg-2153

12/30/2020, 10:46 PM
Weird it doesn't seem to pick up the changes right away
h

hundreds-father-404

12/30/2020, 10:48 PM
Hm did it eventually pick it up, or still no? Does
accounts_report:dash-bootstrap-components
show up in
./pants list ::
?
h

high-egg-2153

12/30/2020, 10:48 PM
yup - but it took a bunch of messing around. Now I've added more and they are not being picked up yet.
h

hundreds-father-404

12/30/2020, 10:49 PM
Hm, that’s a bug then we weren’t aware of. While iterating, you can try
./pants --no-pantsd
to avoid memoization and force the module mapping to be recreated
h

high-egg-2153

12/30/2020, 10:49 PM
Seems I have to make a change to BUILD and save it
Actually looks like I need to edit and save the app.py and then the run and package pick up the changes
👀 1
No stuck on
ModuleNotFoundError: No module named 'pymysql'
h

hundreds-father-404

12/30/2020, 10:56 PM
I didn’t see that in your requirments.txt from earlier. Do you directly import it? For every direct import of 3rd party libraries, you’ll want an entry in requirements.txt
h

high-egg-2153

12/30/2020, 10:56 PM
I've been gradually adding back all my requirement
SQLAlchemy>=1.3.22
ModuleNotFoundError: No module named 'sqlalchemy'
Is there something funny happening with case between requirements.txt and the import in python
h

hundreds-father-404

12/30/2020, 11:02 PM
Shouldn’t be. The default module name is pretty simple:
python_req.project_name.lower().replace("-", "_")
, where
python_req
is a
Requirement
from
pkg_resources
. Meaning, it would convert to
sqlalchemy
I thought SQLAlchemy was showing up in
./pants dependencies path/to/app.py
. Is that no longer the case?
h

high-egg-2153

12/30/2020, 11:04 PM
I think I got it! The requirement.txt I have is from a freeze I did on the app before pants. The modules are mixed case names. When I lower case them it seems to start working.
💯 1
h

hundreds-father-404

12/30/2020, 11:05 PM
The modules are mixed case names
What do you mean by this? Your import statements are mixed case?
h

high-egg-2153

12/30/2020, 11:05 PM
pants run tries to run it and no dependency problems. Now I get:
/usr/bin/python3.8: No module named accounts_report.__main__; 'accounts_report' is a package and cannot be directly executed
In requirements.txt I had:
PyMySQL>=0.10.1
I changed it to :
pymysql>=0.10.1
👍 1
I also added this to BUILD:
pex_binary(
sources=["app.py"],
entry_point='<http://accounts_report.app|accounts_report.app>',
dependencies=[
"accounts_report:setuptools",
":sqlalchemy",
":pymysql"
]
)
h

hundreds-father-404

12/30/2020, 11:08 PM
Yay to the deps being fixed! That new issue feels like a missing
__init__.py
file, but I’m not certain
h

high-egg-2153

12/30/2020, 11:09 PM
ok - confirmed its not the case but rather just that sqlalchemy and pymysql need to be in the dependencies in BUILD file.
I do have an
___init___.py
👍 1
I have to run. Will probably be looking at this later this evening. I really appreciate the help. Very impressed that you guys are so responsive and willing to help!
👍 1
h

hundreds-father-404

12/30/2020, 11:14 PM
Okay sounds good, hoping we can figure this out and also possibly make improvements to the project based on this so others don’t hit the same confusion Thanks! People might be a little slower to reply the next few days from New Years (I’ll be offline tonight for example), but I’ll be on tomorrow during the day
h

high-egg-2153

12/30/2020, 11:15 PM
Wonderful. Thanks again and happy New Year!
❤️ 2
h

happy-kitchen-89482

12/31/2020, 6:52 PM
Sorry for the trouble! I'm not sure why there should be case sensitivity in the requirements.txt, that seems like a bug somewhere.
h

hundreds-father-404

01/04/2021, 9:37 PM
Update! This error:
/usr/bin/python3.8: No module named accounts_report.__main__; ‘accounts_report’ is a package and cannot be directly executed
Is caused by this line:
Copy code
if __name__ == '__main__':
        my_dash_app.run_server(debug=True)
Setting
debug=False
. @happy-kitchen-89482 saw a similar issue with Django, that Django’s file watcher did not play nicely with Pex and Pants. Dash is a wrapper around Flask, and
debug=True
turns on Flask’s file watcher
h

happy-kitchen-89482

01/04/2021, 9:47 PM
Specifically, it has to do with how the debug server re-execs itself, which doesn't play nicely with
./pants run
for reasons I'm not yet clear on.
We plan to solve this by allowing
./pants run
itself to restart the process on file changes
pantsd
already watches the filesystem, there's no need for the debug server to do so as well
h

hundreds-father-404

01/04/2021, 9:48 PM
which doesn’t play nicely with ./pants run for reasons I’m not yet clear on.
I think it’s more an issue with Pex than with
./pants run
. Note that it still fails if doing
./pants package
then
dist/path/app.pex
h

high-egg-2153

01/05/2021, 5:46 PM
In my use case I am running dash (https://plotly.com/dash/) which wraps flask. When I run with:
dash_app.run_server(debug=True)
I get the error mentioned above. If I run like this:
dash_app.run_server(debug=True, use_reloader=False)
Then everything starts fine. So perhaps a little more evidence that file watching or hot reloading is what is causing the issue for me.
1
h

hundreds-father-404

01/13/2021, 4:42 PM
Hey Greg, fyi I opened https://github.com/pantsbuild/pex/issues/1172. This does indeed seem to be an issue with Pex and not with Pants. That's a good thing imo - it means the surface area is much smaller to fix
6 Views