https://pantsbuild.org/ logo
Join Slack
Powered by
Hey. Just starting to experiment with pants for a...
# general
h
Hey. Just starting to experiment with pants for a python project. I'm having an issue trying to run anything. When I run: 
./pants run accounts_report/app.py
I am getting the following error: 
from pkg_resources import get_distribution, parse_version
ModuleNotFoundError: No module named 'pkg_resources'
Some details: 
Python 3.8.6
pip 19.3.1
Manjaro Linux
Any help would be greatly appreciated.
e
So 
pkg_resources
is provided by `setuptools`(https://pypi.org/project/setuptools/) which you probably already knew. In order to resolve that dependency, Pants requires a small bit of setup - namely you have to tell Pants the version of setuptools you want. More here: https://www.pantsbuild.org/docs/python-third-party-dependencies Before going further, let me know how much of this you already have setup / think you have setup.
h
I know very little so far 🙂. So It's in the root BUILD file I need to tell pants about setup tools?
e
And - fwiw - Pants will isolate your code from your environment as much as possible so things are reproducible. Unfortunately, we can't isolate the OS (Manjaro) nor Python interpreters (yet), but we do isolate away Pip.
h
kk - good to know. I've been assuming my pip install was messed up so I've been trying to uninstall/reinstall/repair it
e
So you need - somewhere - could be in root BUILD but need not be, a requirements.txt / BUILD pair of files (requirements.txt is not needed but its standard). That pair is the 1st example at the top of https://www.pantsbuild.org/docs/python-third-party-dependencies
In particular you need your requirements.txt to have a 
setuptools==???
line. And then, if you look a little lower on that page, you need a module mapping, so - in the sibling BUILD file to requirements.txt you need:
Copy code
python_requirements(
    module_mapping={
        "setuptools": ["pkg_resources"]
    },
)
h
Ah ok starting to make some sense. Trying that now
e
This module mapping business isn't commonly needed. Unfortunately, pkg_resources is commonly used and its owning project - setuptools - is a bit of an oddball in packaging up both 
setuptools
and 
pkg_resources
top-level packages.
So you need to tell Pants that setuptools also houses pkg_resources.
h
So now I have 
python_requirements(
module_mapping={
"setuptools": ["pkg_resources"]
},
)
and 
setuptools==51.1.1
These are at the root
e
The former in BUILD and the latter in requirements.txt - right?
h
Yes - still get the error though
app.py is in a folder that also has a BUILD file. Does that need to be in there as well as root?
e
OK, a few questions: 1. What version of Pants? 2. What other relevant BUILD file content do you have? IE: Do you have a pex_binary target corresponding to app.py?
h
1. I started on 2.1.0 and updated to 2.1.1 to see if that would help 2. Yes I have a pex_binary target in the BUILD in the folder that looks like 
pex_binary(
sources=["app.py"],
entry_point='<http://accounts_report.app|accounts_report.app>',
dependencies=[
':accounts_report_lib',
],
)
python_library(
name="accounts_report_lib"
)
python_tests(name = 'tests')
e
Ok - you couldn't and perhaps still can't mix sources= and entry_point=. I'm reviewing PRs to check on that, but relevant doc is here: https://www.pantsbuild.org/docs/python-package-goal#creating-a-pex-file-from-a-pex_binary-target
You likely can just kill sources= though since the sibling python_library - which you declare a dep on, will have that file.
❤️ 1
Its strange that you still need a pex_binary target at all in 2.2, but still better than before.
h
So should I be using 2.2 at this point?
e
No, 2.1 is absolutely fine, but you really need to follow along with the 2.1 docs closely.
h
kk - I think I need to do some more reading. Thanks for the help!
e
To put a fine point on it - you're using approach #3 here: https://www.pantsbuild.org/docs/python-package-goal#creating-a-pex-file-from-a-pex_binary-target But that approach requires the exclude of the main module from the python_library target.
Where the main module is app.py in your case.
h
yes I just saw that.
e
I've got to run for a bit but it would be good to hear back on progress when you get a chance.
h
I just got to our cabin and am back online, so I can help while John is out 🙂 Thank you John for helping with the above!
h
Back - I have a couple more minutes to look at this now. I setup the BUILD and requirements.txt but still getting the No module named "pkg_resources"
h
K, can you run 
./pants dependencies  --transitive path/to/app.py
please? Or whichever file has the 
pkg_resources
import
h
accounts_report:SQLAlchemy
accounts_report:dash
accounts_report:dash-table
accounts_report:pandas
accounts_report:requirements.txt
👍 1
requirements.txt: 
dash>=1.18.1
dash-table>=4.11.1
pandas>=1.1.5
SQLAlchemy>=1.3.22
setuptools==51.1.1
BUILD: 
python_requirements(
module_mapping={
"setuptools": ["pkg_resources"]
},
)
pex_binary(
sources=["app.py"],
entry_point='<http://accounts_report.app|accounts_report.app>',
)
python_tests(name = 'tests')
h
Hm weird that all the others still show up correctly, but not this one. For the sake of debugging, could you please add 
accounts_report:setuptools
to the 
dependencies
for the 
pex_binary
target and rerun your test? That will confirm this is solely an issue with dependency inference not being picked up
h
./pants dependencies  --transitive accounts_report/app.py
accounts_report:SQLAlchemy
accounts_report:dash
accounts_report:dash-table
accounts_report:pandas
accounts_report:requirements.txt
accounts_report:setuptools
18:37:58.11 [INFO] Completed: Building requirements.pex with 5 requirements: SQLAlchemy>=1.3.22, dash-table>=4.11.1, dash>=1.18.1, pandas>=1.1.5, setuptools==51.1.1
Traceback (most recent call last):
File "/home/greg/Work/foo/foo/.pants.d/tmpsnbrvmgu/accounts_report.pex/.bootstrap/pex/pex.py", line 487, in execute
File "/home/greg/Work/foo/foo/.pants.d/tmpsnbrvmgu/accounts_report.pex/.bootstrap/pex/pex.py", line 404, in _wrap_coverage
File "/home/greg/Work/foo/foo/.pants.d/tmpsnbrvmgu/accounts_report.pex/.bootstrap/pex/pex.py", line 435, in _wrap_profiling
File "/home/greg/Work/foo/foo/.pants.d/tmpsnbrvmgu/accounts_report.pex/.bootstrap/pex/pex.py", line 543, in _execute
File "/home/greg/Work/foo/foo/.pants.d/tmpsnbrvmgu/accounts_report.pex/.bootstrap/pex/pex.py", line 645, in execute_entry
File "/home/greg/Work/foo/foo/.pants.d/tmpsnbrvmgu/accounts_report.pex/.bootstrap/pex/pex.py", line 653, in execute_module
File "/usr/lib/python3.8/runpy.py", line 210, in run_module
return _run_code(code, {}, init_globals, run_name, mod_spec)
File "/usr/lib/python3.8/runpy.py", line 87, in _run_code
exec(code, run_globals)
File "/home/greg/Work/foo/foo/.pants.d/tmpsnbrvmgu/./accounts_report/app.py", line 12, in <module>
import dash_bootstrap_components as dbc
ModuleNotFoundError: No module named 'dash_bootstrap_components'
after running: 
./pants run accounts_report
Just noticed that that is not the same error
h
For that one, should be fixed by adding 
dash-bootstrap-components
to your 
requirements.txt
so that the 
python_requirements()
call sets that up as a top-level requirement, then dependency inference can do its thing I’m not sure what’s going on with 
setuptools
, as it’s worked in many other repos, including Pants. I’m trying to think of why here might be different. Your BUILD file looks good
h
Once I update requirements.txt do I need to do something to get it to see the changes?
h
You should not need to. The 
python_requirements
thing is a macro that creates one distinct 
python_requirement_library
per each entry in your requirements.txt, which you can confirm by running 
./pants list ::
to see every target you have in your project Pants will notice requirements.txt has changed and recreate the mapping of module names to which code it comes from. You can check by running 
./pants dependencies path/to/app.py
or 
./pants dependencies --transitive path/to/app.py
h
Weird it doesn't seem to pick up the changes right away
h
Hm did it eventually pick it up, or still no? Does 
accounts_report:dash-bootstrap-components
show up in 
./pants list ::
?
h
yup - but it took a bunch of messing around. Now I've added more and they are not being picked up yet.
h
Hm, that’s a bug then we weren’t aware of. While iterating, you can try 
./pants --no-pantsd
to avoid memoization and force the module mapping to be recreated
h
Seems I have to make a change to BUILD and save it
Actually looks like I need to edit and save the app.py and then the run and package pick up the changes
👀 1
No stuck on 
ModuleNotFoundError: No module named 'pymysql'
h
I didn’t see that in your requirments.txt from earlier. Do you directly import it? For every direct import of 3rd party libraries, you’ll want an entry in requirements.txt
h
I've been gradually adding back all my requirement
SQLAlchemy>=1.3.22
ModuleNotFoundError: No module named 'sqlalchemy'
Is there something funny happening with case between requirements.txt and the import in python
h
Shouldn’t be. The default module name is pretty simple: 
python_req.project_name.lower().replace("-", "_")
, where 
python_req
is a 
Requirement
from 
pkg_resources
. Meaning, it would convert to 
sqlalchemy
I thought SQLAlchemy was showing up in 
./pants dependencies path/to/app.py
. Is that no longer the case?
h
I think I got it! The requirement.txt I have is from a freeze I did on the app before pants. The modules are mixed case names. When I lower case them it seems to start working.
💯 1
h
The modules are mixed case names
What do you mean by this? Your import statements are mixed case?
h
pants run tries to run it and no dependency problems. Now I get: 
/usr/bin/python3.8: No module named accounts_report.__main__; 'accounts_report' is a package and cannot be directly executed
In requirements.txt I had: 
PyMySQL>=0.10.1
I changed it to : 
pymysql>=0.10.1
👍 1
I also added this to BUILD: 
pex_binary(
sources=["app.py"],
entry_point='<http://accounts_report.app|accounts_report.app>',
dependencies=[
"accounts_report:setuptools",
":sqlalchemy",
":pymysql"
]
)
h
Yay to the deps being fixed! That new issue feels like a missing 
__init__.py
file, but I’m not certain
h
ok - confirmed its not the case but rather just that sqlalchemy and pymysql need to be in the dependencies in BUILD file.
I do have an 
___init___.py
👍 1
I have to run. Will probably be looking at this later this evening. I really appreciate the help. Very impressed that you guys are so responsive and willing to help!
👍 1
h
Okay sounds good, hoping we can figure this out and also possibly make improvements to the project based on this so others don’t hit the same confusion Thanks! People might be a little slower to reply the next few days from New Years (I’ll be offline tonight for example), but I’ll be on tomorrow during the day
h
Wonderful. Thanks again and happy New Year!
❤️ 2
h
Sorry for the trouble! I'm not sure why there should be case sensitivity in the requirements.txt, that seems like a bug somewhere.
h
Update! This error:
/usr/bin/python3.8: No module named accounts_report.__main__; ‘accounts_report’ is a package and cannot be directly executed
Is caused by this line:
Copy code
if __name__ == '__main__':
        my_dash_app.run_server(debug=True)
Setting 
debug=False
. @happy-kitchen-89482 saw a similar issue with Django, that Django’s file watcher did not play nicely with Pex and Pants. Dash is a wrapper around Flask, and 
debug=True
turns on Flask’s file watcher
h
Specifically, it has to do with how the debug server re-execs itself, which doesn't play nicely with 
./pants run
for reasons I'm not yet clear on.
We plan to solve this by allowing 
./pants run
itself to restart the process on file changes
pantsd
already watches the filesystem, there's no need for the debug server to do so as well
h
which doesn’t play nicely with ./pants run for reasons I’m not yet clear on.
I think it’s more an issue with Pex than with 
./pants run
. Note that it still fails if doing 
./pants package
then 
dist/path/app.pex
h
In my use case I am running dash (https://plotly.com/dash/) which wraps flask. When I run with: 
dash_app.run_server(debug=True)
I get the error mentioned above. If I run like this: 
dash_app.run_server(debug=True, use_reloader=False)
Then everything starts fine. So perhaps a little more evidence that file watching or hot reloading is what is causing the issue for me.
➕ 1
h
Hey Greg, fyi I opened https://github.com/pantsbuild/pex/issues/1172. This does indeed seem to be an issue with Pex and not with Pants. That's a good thing imo - it means the surface area is much smaller to fix
6 Views
Open in Slack
PreviousNext
Powered by