https://pantsbuild.org/ logo
Join Slack
Powered by
Hey there, I’m migrating to pants 2 rc0 and I’d lo...
# general
w
Hey there, I’m migrating to pants 2 rc0 and I’d love to take advantage of pants inferring my dependencies. Is there anything special that I have to do for my internal libs?
h
Hey! Check out https://www.pantsbuild.org/docs/how-to-upgrade-pants-2-0#dependency-inference-is-enabled-by-default-130-vs-20 tl;dr, you shouldn’t need to do anything special for first-party code. Third-party code sometimes requires setting up a 
module_mapping
, but first-party doesn’t need it (Only possible issue is when you have >1 target owning the same file, i.e. overlapping 
sources
fields; results in a no-op because we can’t disambiguate which target to use)
Also, use rc1 instead of rc0. Has some bug fixes
🙏 1
Also I recommend first safely landing the 2.0.0rc1 upgrade in your repo, and only after it lands, starting to delete from BUILD files. It’ll make for a smaller diff and make it easier to rollback if you need to for any reason
w
thankfully that won’t be an issue in my case, the monorepo isnt being used yet for prod deployment and there isnt much deployment going on in it yet.
👍 1
I’m not sure what I’m doing wrong but for some reason there is at least one dependency that it isnt there 😕
in pretty much all my 
.py
files I have a 
from rasa_sdk import …
and I’m still getting a
Copy code
ModuleNotFoundError: No module named 'rasa_sdk'
when trying to run the pex file
h
Use 
./pants dependencies path/to/foo.py
when debugging. I’m wondering if source roots are set up the right way? When creating the mapping of all modules, Pants strips source roots. So 
src/python/myorg/lib.py
becomes the module 
myorg.lib
w
interesting.. it isnt finding 
rasa_sdk
at all but my internal libs are there which is really cool 😄
h
To clarify, is 
rasa_sdk
a first-party dep or third-party dep?
w
third-party
h
Ahh, see https://www.pantsbuild.org/docs/python-third-party-dependencies#basic-setup. I suspect the dependency’s name != the module it exports, so you have to teach Pants
w
I think it is.. just checked under 
site-packages
and it’s rasa_sdk there.. and I install it with 
pip install rasa-sdk
Ha! Yeah, it is what you said, silly 
-
and 
_
I changed inside 
module_mapping={}
so it now looks like:
Copy code
"rasa-sdk": ["rasa_sdk"],
and now it found it
h
Oh, really? Pants should be normalizing 
rasa-sdk
to the module 
rasa_sdk
. That’s good to know we’re not
w
I was about to say that haha, I mean.. that probably means that all my other libs with dash instead of underscore will also have that same problem
h
This is our code: 
python_req.project_name.lower().replace("-", "_")
What is the full requirements string for 
rasa-sdk
in your 
requirements.txt
?
w
are you getting that back?
python_req.project_name = python_req.project_name.lower().replace("-", "_")
yeah, just saw it on the source code
👍 1
h
I’m not sure what you mean. I ran a quick experiment that runs the equivalent of Pants’s code:
Copy code
▶ pex setuptools
Python 3.8.3 (default, May 27 2020, 20:54:22)
[Clang 11.0.3 (clang-1103.0.32.59)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
(InteractiveConsole)
>>> from pkg_resources import Requirement
>>> Requirement.parse('rasa-sdk').project_name.lower().replace("-", "_")
'rasa_sdk'
The only thing I can think of is if the dash in 
rasa-sdk
is not a normal dash, so I’m wondering if you can copy and paste the string you use in 
requirements.txt
exactly
w
nope, still same wrong behavior. Tried changing all dashes everywhere
HA!
I don’t have to add anything to module_mapping if it’s just supposed to “work fine” right?
I clearly did not understand that 
module_mapping
before. I only have 
setuptools
now there since it is a different package naming. The rest I removed it and now it’s working fine. Sorry about that.
🎉 1
h
I don’t have to add anything to module_mapping if it’s just supposed to “work fine” right?
That’s correct. It’s only when you need to override the default
Yay to it working! To clarify, do you know what the issue was?
w
well.. it is finding the packages but the 
setuptools
now is causing errors 😞
getting
Copy code
ModuleNotFoundError: No module named 'pkg_resources'
but it is being defined inside 
python_requirements
like this:
Copy code
python_requirements(
    module_mapping={
        "setuptools": ["pkg_resources"],
    }
)
Apparently 
setuptools
is a dependency of one of my third-party dependencies…
Copy code
File "/Users/lzfnyy/.pyenv/versions/3.7.8/lib/python3.7/runpy.py", line 208, in run_module
    return _run_code(code, {}, init_globals, run_name, mod_spec)
  File "/Users/lzfnyy/.pyenv/versions/3.7.8/lib/python3.7/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/Users/lzfnyy/.pex/code/80df3353e9fd3bf4154fb48cf8e6aa7c612cfd56/main.py", line 5, in <module>
    import rasa_sdk.endpoint
  File "/Users/lzfnyy/.pex/installed_wheels/8ad79b9c644dff8f1b2cc9bc005597d50a98f3db/rasa_sdk-1.10.2-py3-none-any.whl/rasa_sdk/endpoint.py", line 10, in <module>
    from sanic_cors import CORS
  File "/Users/lzfnyy/.pex/installed_wheels/a4b580262eea6ce0fd763f8434011e5343a2ec82/Sanic_Cors-0.10.0.post3-py2.py3-none-any.whl/sanic_cors/__init__.py", line 11, in <module>
    from .decorator import cross_origin
  File "/Users/lzfnyy/.pex/installed_wheels/a4b580262eea6ce0fd763f8434011e5343a2ec82/Sanic_Cors-0.10.0.post3-py2.py3-none-any.whl/sanic_cors/decorator.py", line 13, in <module>
    from spf import SanicPluginsFramework
  File "/Users/lzfnyy/.pex/installed_wheels/e62f1a08fd4247125f0fec6326f5e3026e22db98/Sanic_Plugins_Framework-0.9.4-py2.py3-none-any.whl/spf/__init__.py", line 3, in <module>
    from .framework import SanicPluginsFramework
  File "/Users/lzfnyy/.pex/installed_wheels/e62f1a08fd4247125f0fec6326f5e3026e22db98/Sanic_Plugins_Framework-0.9.4-py2.py3-none-any.whl/spf/framework.py", line 15, in <module>
    from spf.config import load_config_file
  File "/Users/lzfnyy/.pex/installed_wheels/e62f1a08fd4247125f0fec6326f5e3026e22db98/Sanic_Plugins_Framework-0.9.4-py2.py3-none-any.whl/spf/config.py", line 8, in <module>
    import pkg_resources
ModuleNotFoundError: No module named 'pkg_resources'
h
Apparently setuptools is a dependency of one of my third-party dependencies…
That happens all the time 😞 many users globally install 
setuptools
, so library authors forget to include it in their requirements Unfortunately, dep inference can’t help here. Your 
module_mapping
is 100% valid. But there is no import statement for Pants to analyze On the plus side, your BUILD file will now be very declarative in calling attention to the problem:
Copy code
python_library(
  dependencies=[
    # `foo` didn't declare its dep on `setuptools`
   '3rdparty/python:setuptools',
  ]
)
👍 1
w
This rc1 is so stable and marvelous, thank y’all for the hard work
🔥 2
❤️ 2
everything is just.. working! 🎉
🎉 1
😍 1
h
Most repos are very happy to PRs to fix 
setuptools
, and it’s a tiny change. See https://github.com/pytest-dev/pytest-rerunfailures/pull/98 for an example
❤️ 1
w
just found it, and yeah, it isnt there as a requirement
💯 1
6 Views
Open in Slack
PreviousNext
Powered by