https://pantsbuild.org/ logo
m

most-advantage-18294

03/22/2023, 2:10 PM
Hello all! I'm a Pants noob trying it as a way to get my multiple Django projects under control and have run in to an roadblock onboarding it. My goal is to slowly introduce the build system by initially putting all of my projects under a single
src
folder, and then slowly breaking out the shared modules/apps to get rid of all the duplication. I've moved the first repo under the
src
folder, set my
PYTHONPATH=src
env var, and in the
pants.toml
file I've set the source root as
root_patterns = ["/src"]
. However, this doesn't seem to be working. When I try to run the Django dev server using the command
pants run src/djangoproj1/manage.py -- runserver
I get the error
ModuleNotFoundError: No module named 'djangoproj1.settings'
. Fully aware I could be holding it wrong and missed a configuration setting while following the docs, but I would appreciate any help. I've setup a minimum reproduction using the
example-django
repo from the Pants org trying to best emulate what I did in my private repo, here's the relevant commit with all the changes and a comment containing the traceback -- https://github.com/joshuadavidthomas/example-django/commit/6e8436eb26e15964e7ef1f6366404adfc06d4dff
šŸ‘‹ 1
āœ… 1
h

happy-kitchen-89482

03/22/2023, 2:14 PM
This isn't hurting you, but FYI you shouldn't need
PYTHONPATH=src
the
root_patterns
are the key thing here
A hypothesis on what's going on is this: Pants infers dependencies automatically by looking at
import
statements. But Django is funky, it loads some deps at runtime by naming conventions or reading env vars. So Pants can't infer those deps, and you have to add them manually via
dependencies
fields on your targets. (although I am noodling at a Django plugin that will automate some of this).
Oh, but the issue might be a lot simpler: Try replacing
/src
with
src
in your root patterns...
e

enough-analyst-54434

03/22/2023, 2:17 PM
h

happy-kitchen-89482

03/22/2023, 2:18 PM
in this context
/
is a special symbol that means "the repo root is a source root"
e

enough-analyst-54434

03/22/2023, 2:18 PM
There is no djangoproj1 where you claim.
h

happy-kitchen-89482

03/22/2023, 2:19 PM
At least in your modified example-django, that
/src
looks suspicious
m

most-advantage-18294

03/22/2023, 2:19 PM
Ah no, there wouldn't be. That was just a dummy stand-in name for my actual project
Hm, removing the forward slash is still giving me this error in the
example-django
repo:
Copy code
pants-example-django on ļ˜ src-source-root ī˜† v3.11.1
āžœ pants roots
09:18:00.45 [INFO] Initializing scheduler...
09:18:01.20 [INFO] Scheduler initialized.
src

pants-example-django on ļ˜ src-source-root [!] ī˜† v3.11.1 took 4s
āžœ pants run src/helloworld/service/admin/manage.py -- runserver
09:18:56.77 [INFO] Completed: Building 5 requirements for lib.pex from the lockfiles/python-default.lock resolve: django-stubs>=1.10.0, django<4,>=3.2.13, gunicorn>=20.1.0, requests>=2.25.1, types-requests>=2.25.1
Traceback (most recent call last):
  File "/home/josh/.cache/pants/named_caches/pex_root/venvs/s/295d9e5d/venv/lib/python3.9/site-packages/django/core/management/base.py", line 354, in run_from_argv
    self.execute(*args, **cmd_options)
  File "/home/josh/.cache/pants/named_caches/pex_root/venvs/s/295d9e5d/venv/lib/python3.9/site-packages/django/core/management/commands/runserver.py", line 61, in execute
    super().execute(*args, **options)
  File "/home/josh/.cache/pants/named_caches/pex_root/venvs/s/295d9e5d/venv/lib/python3.9/site-packages/django/core/management/base.py", line 398, in execute
    output = self.handle(*args, **options)
  File "/home/josh/.cache/pants/named_caches/pex_root/venvs/s/295d9e5d/venv/lib/python3.9/site-packages/django/core/management/commands/runserver.py", line 68, in handle
    if not settings.DEBUG and not settings.ALLOWED_HOSTS:
  File "/home/josh/.cache/pants/named_caches/pex_root/venvs/s/295d9e5d/venv/lib/python3.9/site-packages/django/conf/__init__.py", line 82, in __getattr__
    self._setup(name)
  File "/home/josh/.cache/pants/named_caches/pex_root/venvs/s/295d9e5d/venv/lib/python3.9/site-packages/django/conf/__init__.py", line 69, in _setup
    self._wrapped = Settings(settings_module)
  File "/home/josh/.cache/pants/named_caches/pex_root/venvs/s/295d9e5d/venv/lib/python3.9/site-packages/django/conf/__init__.py", line 170, in __init__
    mod = importlib.import_module(self.SETTINGS_MODULE)
  File "/home/josh/.pyenv/versions/3.9.15/lib/python3.9/importlib/__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1030, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1007, in _find_and_load
  File "<frozen importlib._bootstrap>", line 984, in _find_and_load_unlocked
ModuleNotFoundError: No module named 'helloworld.service.admin.settings'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/tmp/pants-sandbox-8eU8st/./.cache/pex_root/venvs/34b83b0c74ebbfa316d0b4e9693282495ce26847/881dadbb13bd88f9751379a568a51e037d0e6994/pex", line 243, in <module>
    runpy.run_module(module_name, run_name="__main__", alter_sys=True)
  File "/home/josh/.pyenv/versions/3.9.15/lib/python3.9/runpy.py", line 225, in run_module
    return _run_module_code(code, init_globals, run_name, mod_spec)
  File "/home/josh/.pyenv/versions/3.9.15/lib/python3.9/runpy.py", line 97, in _run_module_code
    _run_code(code, mod_globals, init_globals,
  File "/home/josh/.pyenv/versions/3.9.15/lib/python3.9/runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "/tmp/pants-sandbox-8eU8st/src/helloworld/service/admin/manage.py", line 7, in <module>
    Service(__file__).run_manage()
  File "/tmp/pants-sandbox-8eU8st/src/helloworld/util/service.py", line 46, in run_manage
    execute_from_command_line(args)
  File "/home/josh/.cache/pants/named_caches/pex_root/venvs/s/295d9e5d/venv/lib/python3.9/site-packages/django/core/management/__init__.py", line 419, in execute_from_command_line
    utility.execute()
  File "/home/josh/.cache/pants/named_caches/pex_root/venvs/s/295d9e5d/venv/lib/python3.9/site-packages/django/core/management/__init__.py", line 413, in execute
    self.fetch_command(subcommand).run_from_argv(self.argv)
  File "/home/josh/.cache/pants/named_caches/pex_root/venvs/s/295d9e5d/venv/lib/python3.9/site-packages/django/core/management/base.py", line 367, in run_from_argv
    connections.close_all()
  File "/home/josh/.cache/pants/named_caches/pex_root/venvs/s/295d9e5d/venv/lib/python3.9/site-packages/django/db/utils.py", line 208, in close_all
    for alias in self:
  File "/home/josh/.cache/pants/named_caches/pex_root/venvs/s/295d9e5d/venv/lib/python3.9/site-packages/django/utils/connection.py", line 73, in __iter__
    return iter(self.settings)
  File "/home/josh/.cache/pants/named_caches/pex_root/venvs/s/295d9e5d/venv/lib/python3.9/site-packages/django/utils/functional.py", line 48, in __get__
    res = instance.__dict__[self.name] = self.func(instance)
  File "/home/josh/.cache/pants/named_caches/pex_root/venvs/s/295d9e5d/venv/lib/python3.9/site-packages/django/utils/connection.py", line 45, in settings
    self._settings = self.configure_settings(self._settings)
  File "/home/josh/.cache/pants/named_caches/pex_root/venvs/s/295d9e5d/venv/lib/python3.9/site-packages/django/db/utils.py", line 144, in configure_settings
    databases = super().configure_settings(databases)
  File "/home/josh/.cache/pants/named_caches/pex_root/venvs/s/295d9e5d/venv/lib/python3.9/site-packages/django/utils/connection.py", line 50, in configure_settings
    settings = getattr(django_settings, self.settings_name)
  File "/home/josh/.cache/pants/named_caches/pex_root/venvs/s/295d9e5d/venv/lib/python3.9/site-packages/django/conf/__init__.py", line 82, in __getattr__
    self._setup(name)
  File "/home/josh/.cache/pants/named_caches/pex_root/venvs/s/295d9e5d/venv/lib/python3.9/site-packages/django/conf/__init__.py", line 69, in _setup
    self._wrapped = Settings(settings_module)
  File "/home/josh/.cache/pants/named_caches/pex_root/venvs/s/295d9e5d/venv/lib/python3.9/site-packages/django/conf/__init__.py", line 170, in __init__
    mod = importlib.import_module(self.SETTINGS_MODULE)
  File "/home/josh/.pyenv/versions/3.9.15/lib/python3.9/importlib/__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1030, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1007, in _find_and_load
  File "<frozen importlib._bootstrap>", line 984, in _find_and_load_unlocked
ModuleNotFoundError: No module named 'helloworld.service.admin.settings'
Aha, if I add
src/helloworld/service/admin:lib
to the
dependencies
, it works, e.g. the
BUILD
file looks like this now:
Copy code
# src/helloworld/service/admin/BUILD
# Copyright 2021 Pants project contributors.
# Licensed under the Apache License, Version 2.0 (see LICENSE).

python_sources(
    name="lib",
    dependencies=[
        "src/helloworld/service/admin:lib",
        "src/helloworld/greet",
        "src/helloworld/person",
        "src/helloworld/translate",
    ],
)

pex_binary(
    name="gunicorn",
    entry_point="gunicorn.py",
    dependencies=[
        ":lib",
    ],
    restartable=True,
)

pex_binary(
    name="manage",
    entry_point="manage.py",
    dependencies=[
        ":lib",
    ],
    restartable=True,
)
Seems odd that I would essentially need to add itself as a dependency of itself? Am I doing this right?
e

enough-analyst-54434

03/22/2023, 2:28 PM
Aha, I think you found a hermeticity leak in the pre-refactor. PYTHONPATH=. was implicit then and there was no src directory to screw that up.
The way you have things now is correct I think both pre and post src dir insertion.
IOW you may have found a silent bug in the example-django repo.
Ah, no - you mean the 1st line. That is odd. Something else is going on here.
m

most-advantage-18294

03/22/2023, 2:35 PM
Yep, the first item in the
dependencies
list.
FWIW, adding that to my actual project fixes the
ModuleNotFoundError
e

enough-analyst-54434

03/22/2023, 2:39 PM
Yeah, that line is totally whacky and a sign nothing is understood at all yet. On the face of it that line should be rejected by Pants and is probably only allowed due to support for circular deps added for totally different reasons.
Definitely super weird:
Copy code
jsirois@Gill-Windows:~/dev/joshuadavidthomas/example-django (src-source-root *) $ pants --keep-sandboxes=on_failure run src/helloworld/service/admin/manage.py -- runserver
Traceback (most recent call last):
  File "/home/jsirois/.cache/pants/named_caches/pex_root/venvs/s/065716be/venv/lib/python3.9/site-packages/django/core/management/base.py", line 354, in run_from_argv
    self.execute(*args, **cmd_options)
  File "/home/jsirois/.cache/pants/named_caches/pex_root/venvs/s/065716be/venv/lib/python3.9/site-packages/django/core/management/commands/runserver.py", line 61, in execute
    super().execute(*args, **options)
  File "/home/jsirois/.cache/pants/named_caches/pex_root/venvs/s/065716be/venv/lib/python3.9/site-packages/django/core/management/base.py", line 398, in execute
    output = self.handle(*args, **options)
  File "/home/jsirois/.cache/pants/named_caches/pex_root/venvs/s/065716be/venv/lib/python3.9/site-packages/django/core/management/commands/runserver.py", line 68, in handle
    if not settings.DEBUG and not settings.ALLOWED_HOSTS:
  File "/home/jsirois/.cache/pants/named_caches/pex_root/venvs/s/065716be/venv/lib/python3.9/site-packages/django/conf/__init__.py", line 82, in __getattr__
    self._setup(name)
  File "/home/jsirois/.cache/pants/named_caches/pex_root/venvs/s/065716be/venv/lib/python3.9/site-packages/django/conf/__init__.py", line 69, in _setup
    self._wrapped = Settings(settings_module)
  File "/home/jsirois/.cache/pants/named_caches/pex_root/venvs/s/065716be/venv/lib/python3.9/site-packages/django/conf/__init__.py", line 170, in __init__
    mod = importlib.import_module(self.SETTINGS_MODULE)
  File "/usr/lib/python3.9/importlib/__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1030, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1007, in _find_and_load
  File "<frozen importlib._bootstrap>", line 984, in _find_and_load_unlocked
ModuleNotFoundError: No module named 'helloworld.service.admin.settings'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/tmp/pants-sandbox-uFvszq/./.cache/pex_root/venvs/34b83b0c74ebbfa316d0b4e9693282495ce26847/108a3ddc84230ab282ea6312e06cb68f51008ce5/pex", line 243, in <module>
    runpy.run_module(module_name, run_name="__main__", alter_sys=True)
  File "/usr/lib/python3.9/runpy.py", line 225, in run_module
    return _run_module_code(code, init_globals, run_name, mod_spec)
  File "/usr/lib/python3.9/runpy.py", line 97, in _run_module_code
    _run_code(code, mod_globals, init_globals,
  File "/usr/lib/python3.9/runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "/tmp/pants-sandbox-uFvszq/src/helloworld/service/admin/manage.py", line 7, in <module>
    Service(__file__).run_manage()
  File "/tmp/pants-sandbox-uFvszq/src/helloworld/util/service.py", line 46, in run_manage
    execute_from_command_line(args)
  File "/home/jsirois/.cache/pants/named_caches/pex_root/venvs/s/065716be/venv/lib/python3.9/site-packages/django/core/management/__init__.py", line 419, in execute_from_command_line
    utility.execute()
  File "/home/jsirois/.cache/pants/named_caches/pex_root/venvs/s/065716be/venv/lib/python3.9/site-packages/django/core/management/__init__.py", line 413, in execute
    self.fetch_command(subcommand).run_from_argv(self.argv)
  File "/home/jsirois/.cache/pants/named_caches/pex_root/venvs/s/065716be/venv/lib/python3.9/site-packages/django/core/management/base.py", line 367, in run_from_argv
    connections.close_all()
  File "/home/jsirois/.cache/pants/named_caches/pex_root/venvs/s/065716be/venv/lib/python3.9/site-packages/django/db/utils.py", line 208, in close_all
    for alias in self:
  File "/home/jsirois/.cache/pants/named_caches/pex_root/venvs/s/065716be/venv/lib/python3.9/site-packages/django/utils/connection.py", line 73, in __iter__
    return iter(self.settings)
  File "/home/jsirois/.cache/pants/named_caches/pex_root/venvs/s/065716be/venv/lib/python3.9/site-packages/django/utils/functional.py", line 48, in __get__
    res = instance.__dict__[self.name] = self.func(instance)
  File "/home/jsirois/.cache/pants/named_caches/pex_root/venvs/s/065716be/venv/lib/python3.9/site-packages/django/utils/connection.py", line 45, in settings
    self._settings = self.configure_settings(self._settings)
  File "/home/jsirois/.cache/pants/named_caches/pex_root/venvs/s/065716be/venv/lib/python3.9/site-packages/django/db/utils.py", line 144, in configure_settings
    databases = super().configure_settings(databases)
  File "/home/jsirois/.cache/pants/named_caches/pex_root/venvs/s/065716be/venv/lib/python3.9/site-packages/django/utils/connection.py", line 50, in configure_settings
    settings = getattr(django_settings, self.settings_name)
  File "/home/jsirois/.cache/pants/named_caches/pex_root/venvs/s/065716be/venv/lib/python3.9/site-packages/django/conf/__init__.py", line 82, in __getattr__
    self._setup(name)
  File "/home/jsirois/.cache/pants/named_caches/pex_root/venvs/s/065716be/venv/lib/python3.9/site-packages/django/conf/__init__.py", line 69, in _setup
    self._wrapped = Settings(settings_module)
  File "/home/jsirois/.cache/pants/named_caches/pex_root/venvs/s/065716be/venv/lib/python3.9/site-packages/django/conf/__init__.py", line 170, in __init__
    mod = importlib.import_module(self.SETTINGS_MODULE)
  File "/usr/lib/python3.9/importlib/__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1030, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1007, in _find_and_load
  File "<frozen importlib._bootstrap>", line 984, in _find_and_load_unlocked
ModuleNotFoundError: No module named 'helloworld.service.admin.settings'
08:15:40.61 [INFO] Preserving local process execution dir /tmp/pants-sandbox-uFvszq for interactive process
jsirois@Gill-Windows:~/dev/joshuadavidthomas/example-django (src-source-root *) $ ls -l /tmp/pants-sandbox-uFvszq/src/helloworld/service/admin/
total 8
-rw-r--r-- 1 jsirois jsirois    0 Mar 22 08:15 __init__.py
-rwxr-xr-x 1 jsirois jsirois  217 Mar 22 08:15 manage.py
drwxr-xr-x 2 jsirois jsirois 4096 Mar 22 08:15 __pycache__
jsirois@Gill-Windows:~/dev/joshuadavidthomas/example-django (src-source-root *) $ git diff src/helloworld/service/admin/BUILD
diff --git a/src/helloworld/service/admin/BUILD b/src/helloworld/service/admin/BUILD
index c4eb766..404f2b7 100644
--- a/src/helloworld/service/admin/BUILD
+++ b/src/helloworld/service/admin/BUILD
@@ -24,6 +24,7 @@ pex_binary(
     entry_point="manage.py",
     dependencies=[
         ":lib",
+        "src/helloworld/service/admin/settings.py:lib",
     ],
     restartable=True,
 )
@happy-kitchen-89482 hopefully you've been working on this too. I'm out of ideas for the short time I have left at the keyboard this morning.
Aha, yeah - this works fine:
pants run src/helloworld/service/admin:manage -- runserver
, so this is the newish
pants run path/to/file.py
stuff failing.
So, to be clear - @most-advantage-18294 your migration commit works fine IFF you run the
pex_binary
target address
src/helloworld/service/admin:manage
. It fails when running the source file directly, probably because settings.py is not imported from that file.
I think it used to be the case running a source.py would use the pex_binary target if owned by one. Newish work has changed that though for better or worse.
m

most-advantage-18294

03/22/2023, 3:37 PM
Gotcha, that makes sense. So it was a little bit of me holding it wrong, I figured šŸ˜„ -- thanks for your help!
e

enough-analyst-54434

03/22/2023, 3:39 PM
Well, yeah - this is new behavior IIUC, maybe since 2.14, and its totally confusing to me as well and I supposedly know what I'm doing somewhat in this space. I'm not sure whether this is considered expected behavior or is a bug.
m

most-advantage-18294

03/22/2023, 3:43 PM
Should the
example-django
README be updated then? https://github.com/pantsbuild/example-django#to-view-the-frontend
Copy code
In three terminals run:

pants --concurrent run helloworld/service/frontend/manage.py -- runserver
pants --concurrent run helloworld/service/user/manage.py -- runserver
pants --concurrent run helloworld/service/welcome/manage.py -- runserver

And visit this URL in a browser: <http://127.0.0.1:8000/?person=sherlock&lang=es> . You will have to first set up a database and run migrations, of course.
Also, thank you so much for your help @enough-analyst-54434 and @happy-kitchen-89482, I really appreciated you both taking your time to help me through this issue! šŸ˜
h

happy-kitchen-89482

03/22/2023, 4:47 PM
I'll take another look when I'm back in front of a keyboard to be sure I understand what's going on here
OK, so this should be sufficient (this is in the unmodified example-django repo):
Copy code
python_sources(
    name="lib",
    dependencies=[
        "helloworld/ui",
    ],
    overrides={
        "manage.py": {"dependencies": ["helloworld/service/frontend/settings.py:lib"]}
    }
)
Your coarser dep effectively said "everything in this lib depends on everything else in this lib", but all that is actually needed is for
manage.py
to depend on
settings.py
(since this is not inferrable)
Thanks for the heads up on the bug in the example repo!
šŸŽ‰ 1
m

most-advantage-18294

03/22/2023, 11:39 PM
Awesome, thanks for the help!
3 Views