https://pantsbuild.org/ logo
Join Slack
Powered by
Hello all! I'm a Pants noob trying it as a way to ...
# general
m
Hello all! I'm a Pants noob trying it as a way to get my multiple Django projects under control and have run in to an roadblock onboarding it. My goal is to slowly introduce the build system by initially putting all of my projects under a single 
src
folder, and then slowly breaking out the shared modules/apps to get rid of all the duplication. I've moved the first repo under the 
src
folder, set my 
PYTHONPATH=src
env var, and in the 
pants.toml
file I've set the source root as 
root_patterns = ["/src"]
. However, this doesn't seem to be working. When I try to run the Django dev server using the command 
pants run src/djangoproj1/manage.py -- runserver
I get the error 
ModuleNotFoundError: No module named 'djangoproj1.settings'
. Fully aware I could be holding it wrong and missed a configuration setting while following the docs, but I would appreciate any help. I've setup a minimum reproduction using the 
example-django
repo from the Pants org trying to best emulate what I did in my private repo, here's the relevant commit with all the changes and a comment containing the traceback -- https://github.com/joshuadavidthomas/example-django/commit/6e8436eb26e15964e7ef1f6366404adfc06d4dff
šŸ‘‹ 1
āœ… 1
h
This isn't hurting you, but FYI you shouldn't need 
PYTHONPATH=src
the 
root_patterns
are the key thing here
A hypothesis on what's going on is this: Pants infers dependencies automatically by looking at 
import
statements. But Django is funky, it loads some deps at runtime by naming conventions or reading env vars. So Pants can't infer those deps, and you have to add them manually via 
dependencies
fields on your targets. (although I am noodling at a Django plugin that will automate some of this).
Oh, but the issue might be a lot simpler: Try replacing 
/src
with 
src
in your root patterns...
e
h
in this context 
/
is a special symbol that means "the repo root is a source root"
e
There is no djangoproj1 where you claim.
h
At least in your modified example-django, that 
/src
looks suspicious
m
Ah no, there wouldn't be. That was just a dummy stand-in name for my actual project
Hm, removing the forward slash is still giving me this error in the 
example-django
repo:
Copy code
pants-example-django on ļ˜ src-source-root ī˜† v3.11.1
āžœ pants roots
09:18:00.45 [INFO] Initializing scheduler...
09:18:01.20 [INFO] Scheduler initialized.
src

pants-example-django on ļ˜ src-source-root [!] ī˜† v3.11.1 took 4s
āžœ pants run src/helloworld/service/admin/manage.py -- runserver
09:18:56.77 [INFO] Completed: Building 5 requirements for lib.pex from the lockfiles/python-default.lock resolve: django-stubs>=1.10.0, django<4,>=3.2.13, gunicorn>=20.1.0, requests>=2.25.1, types-requests>=2.25.1
Traceback (most recent call last):
  File "/home/josh/.cache/pants/named_caches/pex_root/venvs/s/295d9e5d/venv/lib/python3.9/site-packages/django/core/management/base.py", line 354, in run_from_argv
    self.execute(*args, **cmd_options)
  File "/home/josh/.cache/pants/named_caches/pex_root/venvs/s/295d9e5d/venv/lib/python3.9/site-packages/django/core/management/commands/runserver.py", line 61, in execute
    super().execute(*args, **options)
  File "/home/josh/.cache/pants/named_caches/pex_root/venvs/s/295d9e5d/venv/lib/python3.9/site-packages/django/core/management/base.py", line 398, in execute
    output = self.handle(*args, **options)
  File "/home/josh/.cache/pants/named_caches/pex_root/venvs/s/295d9e5d/venv/lib/python3.9/site-packages/django/core/management/commands/runserver.py", line 68, in handle
    if not settings.DEBUG and not settings.ALLOWED_HOSTS:
  File "/home/josh/.cache/pants/named_caches/pex_root/venvs/s/295d9e5d/venv/lib/python3.9/site-packages/django/conf/__init__.py", line 82, in __getattr__
    self._setup(name)
  File "/home/josh/.cache/pants/named_caches/pex_root/venvs/s/295d9e5d/venv/lib/python3.9/site-packages/django/conf/__init__.py", line 69, in _setup
    self._wrapped = Settings(settings_module)
  File "/home/josh/.cache/pants/named_caches/pex_root/venvs/s/295d9e5d/venv/lib/python3.9/site-packages/django/conf/__init__.py", line 170, in __init__
    mod = importlib.import_module(self.SETTINGS_MODULE)
  File "/home/josh/.pyenv/versions/3.9.15/lib/python3.9/importlib/__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1030, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1007, in _find_and_load
  File "<frozen importlib._bootstrap>", line 984, in _find_and_load_unlocked
ModuleNotFoundError: No module named 'helloworld.service.admin.settings'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/tmp/pants-sandbox-8eU8st/./.cache/pex_root/venvs/34b83b0c74ebbfa316d0b4e9693282495ce26847/881dadbb13bd88f9751379a568a51e037d0e6994/pex", line 243, in <module>
    runpy.run_module(module_name, run_name="__main__", alter_sys=True)
  File "/home/josh/.pyenv/versions/3.9.15/lib/python3.9/runpy.py", line 225, in run_module
    return _run_module_code(code, init_globals, run_name, mod_spec)
  File "/home/josh/.pyenv/versions/3.9.15/lib/python3.9/runpy.py", line 97, in _run_module_code
    _run_code(code, mod_globals, init_globals,
  File "/home/josh/.pyenv/versions/3.9.15/lib/python3.9/runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "/tmp/pants-sandbox-8eU8st/src/helloworld/service/admin/manage.py", line 7, in <module>
    Service(__file__).run_manage()
  File "/tmp/pants-sandbox-8eU8st/src/helloworld/util/service.py", line 46, in run_manage
    execute_from_command_line(args)
  File "/home/josh/.cache/pants/named_caches/pex_root/venvs/s/295d9e5d/venv/lib/python3.9/site-packages/django/core/management/__init__.py", line 419, in execute_from_command_line
    utility.execute()
  File "/home/josh/.cache/pants/named_caches/pex_root/venvs/s/295d9e5d/venv/lib/python3.9/site-packages/django/core/management/__init__.py", line 413, in execute
    self.fetch_command(subcommand).run_from_argv(self.argv)
  File "/home/josh/.cache/pants/named_caches/pex_root/venvs/s/295d9e5d/venv/lib/python3.9/site-packages/django/core/management/base.py", line 367, in run_from_argv
    connections.close_all()
  File "/home/josh/.cache/pants/named_caches/pex_root/venvs/s/295d9e5d/venv/lib/python3.9/site-packages/django/db/utils.py", line 208, in close_all
    for alias in self:
  File "/home/josh/.cache/pants/named_caches/pex_root/venvs/s/295d9e5d/venv/lib/python3.9/site-packages/django/utils/connection.py", line 73, in __iter__
    return iter(self.settings)
  File "/home/josh/.cache/pants/named_caches/pex_root/venvs/s/295d9e5d/venv/lib/python3.9/site-packages/django/utils/functional.py", line 48, in __get__
    res = instance.__dict__[self.name] = self.func(instance)
  File "/home/josh/.cache/pants/named_caches/pex_root/venvs/s/295d9e5d/venv/lib/python3.9/site-packages/django/utils/connection.py", line 45, in settings
    self._settings = self.configure_settings(self._settings)
  File "/home/josh/.cache/pants/named_caches/pex_root/venvs/s/295d9e5d/venv/lib/python3.9/site-packages/django/db/utils.py", line 144, in configure_settings
    databases = super().configure_settings(databases)
  File "/home/josh/.cache/pants/named_caches/pex_root/venvs/s/295d9e5d/venv/lib/python3.9/site-packages/django/utils/connection.py", line 50, in configure_settings
    settings = getattr(django_settings, self.settings_name)
  File "/home/josh/.cache/pants/named_caches/pex_root/venvs/s/295d9e5d/venv/lib/python3.9/site-packages/django/conf/__init__.py", line 82, in __getattr__
    self._setup(name)
  File "/home/josh/.cache/pants/named_caches/pex_root/venvs/s/295d9e5d/venv/lib/python3.9/site-packages/django/conf/__init__.py", line 69, in _setup
    self._wrapped = Settings(settings_module)
  File "/home/josh/.cache/pants/named_caches/pex_root/venvs/s/295d9e5d/venv/lib/python3.9/site-packages/django/conf/__init__.py", line 170, in __init__
    mod = importlib.import_module(self.SETTINGS_MODULE)
  File "/home/josh/.pyenv/versions/3.9.15/lib/python3.9/importlib/__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1030, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1007, in _find_and_load
  File "<frozen importlib._bootstrap>", line 984, in _find_and_load_unlocked
ModuleNotFoundError: No module named 'helloworld.service.admin.settings'
Aha, if I add 
src/helloworld/service/admin:lib
to the 
dependencies
, it works, e.g. the 
BUILD
file looks like this now:
Copy code
# src/helloworld/service/admin/BUILD
# Copyright 2021 Pants project contributors.
# Licensed under the Apache License, Version 2.0 (see LICENSE).

python_sources(
    name="lib",
    dependencies=[
        "src/helloworld/service/admin:lib",
        "src/helloworld/greet",
        "src/helloworld/person",
        "src/helloworld/translate",
    ],
)

pex_binary(
    name="gunicorn",
    entry_point="gunicorn.py",
    dependencies=[
        ":lib",
    ],
    restartable=True,
)

pex_binary(
    name="manage",
    entry_point="manage.py",
    dependencies=[
        ":lib",
    ],
    restartable=True,
)
Seems odd that I would essentially need to add itself as a dependency of itself? Am I doing this right?
e
Aha, I think you found a hermeticity leak in the pre-refactor. PYTHONPATH=. was implicit then and there was no src directory to screw that up.
The way you have things now is correct I think both pre and post src dir insertion.
IOW you may have found a silent bug in the example-django repo.
Ah, no - you mean the 1st line. That is odd. Something else is going on here.
m
Yep, the first item in the 
dependencies
list.
FWIW, adding that to my actual project fixes the 
ModuleNotFoundError
e
Yeah, that line is totally whacky and a sign nothing is understood at all yet. On the face of it that line should be rejected by Pants and is probably only allowed due to support for circular deps added for totally different reasons.
Definitely super weird:
Copy code
jsirois@Gill-Windows:~/dev/joshuadavidthomas/example-django (src-source-root *) $ pants --keep-sandboxes=on_failure run src/helloworld/service/admin/manage.py -- runserver
Traceback (most recent call last):
  File "/home/jsirois/.cache/pants/named_caches/pex_root/venvs/s/065716be/venv/lib/python3.9/site-packages/django/core/management/base.py", line 354, in run_from_argv
    self.execute(*args, **cmd_options)
  File "/home/jsirois/.cache/pants/named_caches/pex_root/venvs/s/065716be/venv/lib/python3.9/site-packages/django/core/management/commands/runserver.py", line 61, in execute
    super().execute(*args, **options)
  File "/home/jsirois/.cache/pants/named_caches/pex_root/venvs/s/065716be/venv/lib/python3.9/site-packages/django/core/management/base.py", line 398, in execute
    output = self.handle(*args, **options)
  File "/home/jsirois/.cache/pants/named_caches/pex_root/venvs/s/065716be/venv/lib/python3.9/site-packages/django/core/management/commands/runserver.py", line 68, in handle
    if not settings.DEBUG and not settings.ALLOWED_HOSTS:
  File "/home/jsirois/.cache/pants/named_caches/pex_root/venvs/s/065716be/venv/lib/python3.9/site-packages/django/conf/__init__.py", line 82, in __getattr__
    self._setup(name)
  File "/home/jsirois/.cache/pants/named_caches/pex_root/venvs/s/065716be/venv/lib/python3.9/site-packages/django/conf/__init__.py", line 69, in _setup
    self._wrapped = Settings(settings_module)
  File "/home/jsirois/.cache/pants/named_caches/pex_root/venvs/s/065716be/venv/lib/python3.9/site-packages/django/conf/__init__.py", line 170, in __init__
    mod = importlib.import_module(self.SETTINGS_MODULE)
  File "/usr/lib/python3.9/importlib/__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1030, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1007, in _find_and_load
  File "<frozen importlib._bootstrap>", line 984, in _find_and_load_unlocked
ModuleNotFoundError: No module named 'helloworld.service.admin.settings'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/tmp/pants-sandbox-uFvszq/./.cache/pex_root/venvs/34b83b0c74ebbfa316d0b4e9693282495ce26847/108a3ddc84230ab282ea6312e06cb68f51008ce5/pex", line 243, in <module>
    runpy.run_module(module_name, run_name="__main__", alter_sys=True)
  File "/usr/lib/python3.9/runpy.py", line 225, in run_module
    return _run_module_code(code, init_globals, run_name, mod_spec)
  File "/usr/lib/python3.9/runpy.py", line 97, in _run_module_code
    _run_code(code, mod_globals, init_globals,
  File "/usr/lib/python3.9/runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "/tmp/pants-sandbox-uFvszq/src/helloworld/service/admin/manage.py", line 7, in <module>
    Service(__file__).run_manage()
  File "/tmp/pants-sandbox-uFvszq/src/helloworld/util/service.py", line 46, in run_manage
    execute_from_command_line(args)
  File "/home/jsirois/.cache/pants/named_caches/pex_root/venvs/s/065716be/venv/lib/python3.9/site-packages/django/core/management/__init__.py", line 419, in execute_from_command_line
    utility.execute()
  File "/home/jsirois/.cache/pants/named_caches/pex_root/venvs/s/065716be/venv/lib/python3.9/site-packages/django/core/management/__init__.py", line 413, in execute
    self.fetch_command(subcommand).run_from_argv(self.argv)
  File "/home/jsirois/.cache/pants/named_caches/pex_root/venvs/s/065716be/venv/lib/python3.9/site-packages/django/core/management/base.py", line 367, in run_from_argv
    connections.close_all()
  File "/home/jsirois/.cache/pants/named_caches/pex_root/venvs/s/065716be/venv/lib/python3.9/site-packages/django/db/utils.py", line 208, in close_all
    for alias in self:
  File "/home/jsirois/.cache/pants/named_caches/pex_root/venvs/s/065716be/venv/lib/python3.9/site-packages/django/utils/connection.py", line 73, in __iter__
    return iter(self.settings)
  File "/home/jsirois/.cache/pants/named_caches/pex_root/venvs/s/065716be/venv/lib/python3.9/site-packages/django/utils/functional.py", line 48, in __get__
    res = instance.__dict__[self.name] = self.func(instance)
  File "/home/jsirois/.cache/pants/named_caches/pex_root/venvs/s/065716be/venv/lib/python3.9/site-packages/django/utils/connection.py", line 45, in settings
    self._settings = self.configure_settings(self._settings)
  File "/home/jsirois/.cache/pants/named_caches/pex_root/venvs/s/065716be/venv/lib/python3.9/site-packages/django/db/utils.py", line 144, in configure_settings
    databases = super().configure_settings(databases)
  File "/home/jsirois/.cache/pants/named_caches/pex_root/venvs/s/065716be/venv/lib/python3.9/site-packages/django/utils/connection.py", line 50, in configure_settings
    settings = getattr(django_settings, self.settings_name)
  File "/home/jsirois/.cache/pants/named_caches/pex_root/venvs/s/065716be/venv/lib/python3.9/site-packages/django/conf/__init__.py", line 82, in __getattr__
    self._setup(name)
  File "/home/jsirois/.cache/pants/named_caches/pex_root/venvs/s/065716be/venv/lib/python3.9/site-packages/django/conf/__init__.py", line 69, in _setup
    self._wrapped = Settings(settings_module)
  File "/home/jsirois/.cache/pants/named_caches/pex_root/venvs/s/065716be/venv/lib/python3.9/site-packages/django/conf/__init__.py", line 170, in __init__
    mod = importlib.import_module(self.SETTINGS_MODULE)
  File "/usr/lib/python3.9/importlib/__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1030, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1007, in _find_and_load
  File "<frozen importlib._bootstrap>", line 984, in _find_and_load_unlocked
ModuleNotFoundError: No module named 'helloworld.service.admin.settings'
08:15:40.61 [INFO] Preserving local process execution dir /tmp/pants-sandbox-uFvszq for interactive process
jsirois@Gill-Windows:~/dev/joshuadavidthomas/example-django (src-source-root *) $ ls -l /tmp/pants-sandbox-uFvszq/src/helloworld/service/admin/
total 8
-rw-r--r-- 1 jsirois jsirois    0 Mar 22 08:15 __init__.py
-rwxr-xr-x 1 jsirois jsirois  217 Mar 22 08:15 manage.py
drwxr-xr-x 2 jsirois jsirois 4096 Mar 22 08:15 __pycache__
jsirois@Gill-Windows:~/dev/joshuadavidthomas/example-django (src-source-root *) $ git diff src/helloworld/service/admin/BUILD
diff --git a/src/helloworld/service/admin/BUILD b/src/helloworld/service/admin/BUILD
index c4eb766..404f2b7 100644
--- a/src/helloworld/service/admin/BUILD
+++ b/src/helloworld/service/admin/BUILD
@@ -24,6 +24,7 @@ pex_binary(
     entry_point="manage.py",
     dependencies=[
         ":lib",
+        "src/helloworld/service/admin/settings.py:lib",
     ],
     restartable=True,
 )
@happy-kitchen-89482 hopefully you've been working on this too. I'm out of ideas for the short time I have left at the keyboard this morning.
Aha, yeah - this works fine: 
pants run src/helloworld/service/admin:manage -- runserver
, so this is the newish 
pants run path/to/file.py
stuff failing.
So, to be clear - @most-advantage-18294 your migration commit works fine IFF you run the 
pex_binary
target address 
src/helloworld/service/admin:manage
. It fails when running the source file directly, probably because settings.py is not imported from that file.
I think it used to be the case running a source.py would use the pex_binary target if owned by one. Newish work has changed that though for better or worse.
m
Gotcha, that makes sense. So it was a little bit of me holding it wrong, I figured šŸ˜„ -- thanks for your help!
e
Well, yeah - this is new behavior IIUC, maybe since 2.14, and its totally confusing to me as well and I supposedly know what I'm doing somewhat in this space. I'm not sure whether this is considered expected behavior or is a bug.
m
Should the 
example-django
README be updated then? https://github.com/pantsbuild/example-django#to-view-the-frontend
Copy code
In three terminals run:

pants --concurrent run helloworld/service/frontend/manage.py -- runserver
pants --concurrent run helloworld/service/user/manage.py -- runserver
pants --concurrent run helloworld/service/welcome/manage.py -- runserver

And visit this URL in a browser: <http://127.0.0.1:8000/?person=sherlock&lang=es> . You will have to first set up a database and run migrations, of course.
Also, thank you so much for your help @enough-analyst-54434 and @happy-kitchen-89482, I really appreciated you both taking your time to help me through this issue! šŸ˜
h
I'll take another look when I'm back in front of a keyboard to be sure I understand what's going on here
OK, so this should be sufficient (this is in the unmodified example-django repo):
Copy code
python_sources(
    name="lib",
    dependencies=[
        "helloworld/ui",
    ],
    overrides={
        "manage.py": {"dependencies": ["helloworld/service/frontend/settings.py:lib"]}
    }
)
Your coarser dep effectively said "everything in this lib depends on everything else in this lib", but all that is actually needed is for 
manage.py
to depend on 
settings.py
(since this is not inferrable)
Thanks for the heads up on the bug in the example repo!
šŸŽ‰ 1
m
Awesome, thanks for the help!
3 Views
Open in Slack
PreviousNext
Powered by