https://pantsbuild.org/ logo
Join Slack
Powered by
Good afternoon I am running django project using p...
# general
b
Good afternoon I am running django project using pants but it seems not to import the indirect dependecies I have tried to use module mapping but I still get this error
Copy code
Traceback (most recent call last):
  File "/home/kennan/.cache/pants/named_caches/pex_root/venvs/c327cad8531010afb442b743400c471a24792439/dee6656b00a07c8ae98b1c0dfcda4e61ad3e5014/pex", line 236, in <module>
    runpy.run_module(module_name, run_name="__main__", alter_sys=True)
  File "/usr/lib/python3.9/runpy.py", line 225, in run_module
    return _run_module_code(code, init_globals, run_name, mod_spec)
  File "/usr/lib/python3.9/runpy.py", line 97, in _run_module_code
    _run_code(code, mod_globals, init_globals,
  File "/usr/lib/python3.9/runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "/home/kennan/NPA/fleet-backend/fleetdatabaseFinal/manage.py", line 21, in <module>
    main()
  File "/home/kennan/NPA/fleet-backend/fleetdatabaseFinal/manage.py", line 17, in main
    execute_from_command_line(sys.argv)
  File "/home/kennan/.cache/pants/named_caches/pex_root/venvs/c327cad8531010afb442b743400c471a24792439/dee6656b00a07c8ae98b1c0dfcda4e61ad3e5014/lib/python3.9/site-packages/django/core/management/__init__.py", line 446, in execute_from_command_line
    utility.execute()
  File "/home/kennan/.cache/pants/named_caches/pex_root/venvs/c327cad8531010afb442b743400c471a24792439/dee6656b00a07c8ae98b1c0dfcda4e61ad3e5014/lib/python3.9/site-packages/django/core/management/__init__.py", line 420, in execute
    django.setup()
  File "/home/kennan/.cache/pants/named_caches/pex_root/venvs/c327cad8531010afb442b743400c471a24792439/dee6656b00a07c8ae98b1c0dfcda4e61ad3e5014/lib/python3.9/site-packages/django/__init__.py", line 24, in setup
    apps.populate(settings.INSTALLED_APPS)
  File "/home/kennan/.cache/pants/named_caches/pex_root/venvs/c327cad8531010afb442b743400c471a24792439/dee6656b00a07c8ae98b1c0dfcda4e61ad3e5014/lib/python3.9/site-packages/django/apps/registry.py", line 91, in populate
    app_config = AppConfig.create(entry)
  File "/home/kennan/.cache/pants/named_caches/pex_root/venvs/c327cad8531010afb442b743400c471a24792439/dee6656b00a07c8ae98b1c0dfcda4e61ad3e5014/lib/python3.9/site-packages/django/apps/config.py", line 193, in create
    import_module(entry)
  File "/usr/lib/python3.9/importlib/__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1030, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1007, in _find_and_load
  File "<frozen importlib._bootstrap>", line 984, in _find_and_load_unlocked
ModuleNotFoundError: No module named 'rest_framework'
this is my build file at 3rdparty/build
Copy code
python_requirement(
    name="djangorestframework",
    requirements=["djangorestframework>=3.12.2"],
    modules=["rest_framework"],
)
python_requirement(
    name="djangorestframework-simplejwt",
    requirements=["djangorestframework-simplejwt>=4.4.0"],
    modules=["rest_framework_simplejwt"],
)


python_requirements(
    name="reqs",
    module_mapping={"djangorestframework": ["rest_framework"],
                    "djangorestframework-simplejwt": ["rest_framework_simplejwt"]
                    }
                
)
c
The module mapping is for your 
requirements.txt
file, but for some reason, you don’t have DRF in there, but in dedicated 
python_requirement
targets.
in other words, try removing those and add them to your 
requirements.txt
file instead. OR, move the module mapping to their respective targets.
b
I have added DRF and djangorestframework-simplejwt to the requirements.txt but there is no change it is giving the same error
this is by build file now
Copy code
python_requirements(
    name="reqs",
    module_mapping={"djangorestframework": ["rest_framework"],
                    "djangorestframework-simplejwt": ["rest_framework_simplejwt"]
                    }
                
)
It still gives an error of 
ModuleNotFoundError: No module named 'rest_framework'
c
hmm.. actually, these mappings are already in the defaults, so it seems pants is not picking up your requirements targets.. https://github.com/pantsbuild/pants/blob/2417cfeaf5de9b8c4bcf2a847bffc03551c2f773/[…]s/backend/python/dependency_inference/default_module_mapping.py
do you use resolves?
b
Resolves not really
May be you can show me how
They are actually used in the default mappings
c
apologies, I didn’t read your initial error message properly, it looked like a module mapping issue (also based on you providing that information), but the issue is Pants is not providing the module at all, which indicates there is no dependency from your code to djangorestframework.
do you import 
rest_framework
at all in your code?
b
Yes I did
c
and is that code part of what you are trying to run?
b
from rest_framework import serializers from .models import Access_level, Vehicle, Tyre, Fault, Service_schedule, Maintenance_type,Work_order,Maintenance_record,CustomUser #Creating general Serializers class Access_levelSerializer(serializers.ModelSerializer): class Meta: model = Access_level fields = '__all__' class VehicleSerializer(serializers.ModelSerializer): class Meta: model = Vehicle fields = '__all__'
this is my 
serialisers.py model
c
what pants command gives you the error?
b
Yes this code is part of what I am trying to run
Could the issue be with my pants version??
c
the root cause is that pants does not infer a dependency on djangorestframework for you here.. for some reason. Finding that reason will be the exercise. for this purpose, running pants with 
--keep-sandboxes=on_failure
can be instructive, as it allows you to inspect the execution sandbox used, where you can see which sources where actually used (and that usually helps clarify what is a-miss)
I would not think this will be any different based on pants version, no.
b
Kindly explain clearer on the first point made
c
All processes executed by Pants are hermetic in nature. Meaning, that they run from a temporary directory where Pants only includes the sources and 3rd party libraries that your code depends upon. So when there are files or libraries missing, it means there is no dependency from the code you run/test/package to what you are missing (may be a direct dependency, or a transitive one, as all dependencies of all dependencies are included recursively)
so you need to look at your entry point, and follow all it’s dependencies and so forth until you reach the djangorestframework in this case. There’s a pants command that shows these for you, if you run 
pants paths --from=some/target --to=some/other/target
it will list all dependency links that connects the two. If you get an empty list, it means there is none.
Django in particular may need some explicit dependencies in the BUILD files due to the “magic” involved by django how things are imported
which Pants currently are unaware of.
For instance, when you list a “Django APP” under 
INSTALLED_APPS
in 
settings.py
Pants does not recognize this, and while Django will import your app, Pants will not infer a dependency to it.
b
Oh I see so how do I sort this??
f
Most Pants targets have a 
dependencies
field in which you can add explicit dependencies.
c
what I’ve seen is to have a
Copy code
target(
  name="all_apps",
  dependencies=[
    "src/app1",
    "src/app2",
    ...
  ]
)

pex_binary(..., dependencies=[":all_apps"])
adapt as needded.
also note that you likely need to explicitly include the migrations too, as the app itself usually don’t depend on those.
f
https://github.com/pantsbuild/example-django is an example of a Pants repo with Django.
☝️ 2
c
another useful command is 
pants peek some/target
which will show you what pants think of a target
b
so I am supposed to include the dependencies in the 
pex_binary
??
c
if that is what you have.
I prefer to add the dependencies as close to the thing that requires them. So if you have a 
manage.py
for instance, I prefer to make that file depend on all the apps and stuff it requires to run, rather than the encapsulating pex, if you see what I’m getting at.
b
the is the build file at the root of the project
Copy code
python_sources(
    overrides={
        "manage.py":{
            "run_goal_use_sandbox":False,
            #"restartable":True,
        },
    },
)

pex_binary(
    name="django-admin",
    dependencies=[
        "3rdparty:reqs#Django",
    ],
    script="django-admin",
)

pex_binary(
    name="manage",
    entry_point="manage.py",
    dependencies=[
        "fleetdatabaseFinal:fleetdatabaseFinal",
        "accounts:accounts",
        "3rdparty:reqs#psycopg2-binary",
        "3rdparty:reqs#gunicorn",
    ],
    restartable=True,
    inherit_path="prefer",
)

pex_binary(
    name="production",
    dependencies=[
        "fleetdatabaseFinal:fleetdatabaseFinal",
        "accounts:accounts",
        "3rdparty:reqs#psycopg2-binary",
        "3rdparty:reqs#gunicorn",
        "3rdparty:reqs#python-dotenv",
    ],
    script="gunicorn",
    restartable=True,
    inherit_path="prefer",
)
That means I would add them to my fleetdatabaseFinal:manage target
c
here’s the thing, each of your pex targets will need to depend on all your apps and migrations etc. That’s why in the snippet I showed above, there was a generic 
target
target where you can list all “arbitrary” deps you need, and then have only a single target to add to as few places as possible, 3 in this case might be OK.
might even want to layer the generic targets, to group them, if that makes sense. I’ll share an example:
Copy code
python_sources(
    name="manage",
    sources=["manage.py"],
    dependencies=[
        "src/python/exampleinc/app:all_apps",
        "src/python/exampleinc/app:bootstrap",
    ],
)

pex_binary(
    name="app",
    entry_point="manage.py",
    dependencies=[
        # TEMP solution until django plugins are inferred/registered properly
        "!!3rdparty/python#django-axes@resolve=python-default",
        "!!3rdparty/python#django-ipware@resolve=python-default",
        "!!3rdparty/python#PyJWT@resolve=python-default",
    ],
)


pex_binary(
    name="testing",
    entry_point="manage.py:testing",
    dependencies=[
        "src/python/exampleinc/app/test_app",
    ],
)

target(
    name="bootstrap",
    dependencies=[
        "src/python/exampleinc/app/app:app",
        "src/python/exampleinc/app/locale:locale",
        "src/python/exampleinc/app/production/management/commands:commands",
        "src/python/exampleinc/app:all_apps",
        "src/python/exampleinc/app:all_migrations",
        "src/python/exampleinc/app:all_urls",
    ],
)

target(
    name="all_urls",
    dependencies=[
        "src/python/exampleinc/app/core/urls:urls",
        "src/python/exampleinc/app/dms:dms",
        "src/python/exampleinc/app/dms/urls:urls",
        "src/python/exampleinc/app/files/urls:urls",
        "src/python/exampleinc/app/oauth:oauth",
        "src/python/exampleinc/app/reports:reports",
    ],
)

target(
    name="all_migrations",
    dependencies=[
        "src/python/exampleinc/app/dms/migrations:migrations",
        "src/python/exampleinc/app/files/migrations:migrations",
        "src/python/exampleinc/app/reports/migrations:migrations",
    ],
)

target(
    name="all_apps",
    dependencies=[
        "src/python/exampleinc/app/core:core",
        "src/python/exampleinc/app/dms:dms",
        "src/python/exampleinc/app/files:files",
        "src/python/exampleinc/app/oauth:oauth",
        "src/python/exampleinc/app/reports:reports",
    ],
)

python_test_utils(
    name="conftest",
    sources=["conftest.py"],
    dependencies=[
        "src/python/exampleinc/app/test_app",
        "src/python/exampleinc/app/app/pytest_settings.py",
    ],
)
not saying this is an ideal setup, but should at least give you some inspiration
this is mostly complete, just pruned and redacted
also, look at the example django app repo that @fast-nail-55400 linked to above.
b
It seems I have to change almost everything about my BUILD file
Is there any example that uses Django Rest Framework??
If there is, I would to see how dependencies were handled
c
for third party deps, it may look like:
Copy code
python_requirements(
    module_mapping={
        "aws-sam-translator": ("samtranslator",),
        "celery-redbeat": ("redbeat",),
        "cfn-lint": ("cfnlint",),
        "django-axes": ("axes",),
        "django-ipware": ("ipware",),
        "django-jsoneditor": ("jsoneditor",),
        "django-storages": ("storages",),
        "djangorestframework-jwt": ("rest_framework_jwt",),
        "future": ("libfuturize", "libpasteurize", "past"),
        "google-auth": ("google.auth",),
        "google-pasta": ("pasta",),
        "jpype1": ("jpype",),
        "libclang": ("clang",),
        "ndg-httpsclient": ("ndg",),
        "pdfminer.six": ("pdfminer",),
        "psycopg2": [
            "psycopg2",
            # This is for dependency inference only, when we see an import for
            # `django.contrib.postgres` we should infer a dependency on `psycopg2`.
            "django.contrib.postgres",
        ],
        "pycryptodome": ("Crypto",),
        "pyrsistent": ("pvectorc",),
        "pyscss": ("scss",),
        "python3-saml": ("onelogin",),
        "websocket-client": ("websocket",),
    },
    overrides={
        (
            "aws-xray-sdk",
            "django-axes",
        ): {
            "dependencies": ("#setuptools",),
        },
        "django": {
            "dependencies": (
                # Make sure all django plugins and dependencies are listed here, any app specific
                # exceptions to this list is handled as exclude where applicable, as a temporary
                # workaround until we have a better solution for inferring/registering django
                # magics.
                "#django-axes",  # auth only
                "#django-cors-headers",
                "#django-extensions",
                "#django-filter",  # not auth
                "#django-fsm",  # not auth
                "#django-ipware",  # auth only
                "#django-jsoneditor",  # not auth
                "#django-redis",  # obsolete in Django4
                "#django-ses",
                "#django-storages",
                "#psycopg2",
            ),
        },
        "fakeredis": {
            "dependencies": ("#sortedcontainers",),
        },
    },
)
b
Okay this seems somewhat more complicated and confusing
What is the source of this, I think it would be easier understood in context
c
it’s from a private project, so can’t share the context unfortunately. I wanted to highlight what we’ve done in order to direct pants dependency inference to include the proper 3rd party libraries. It could perhaps be clearer if the number of entries were reduced to include only one or two to have fewer lines, but other than that is to get more familiar with how Pants works and then revisit here.
Spend some time on your project, and inspect the dependencies of sources and targets in the project and compare with what is expected is a good start.
b
so when we use this format, we don't need to create a requirements.txt file
c
if you squint a little, maybe it becomes less confusing. The above example, with the noise removed:
Copy code
python_requirements(
    module_mapping={
        ...
    },
    overrides={
        ...
    },
)
so no, there’s still a 
requirements.txt
file backing the target informing it which requirements we want here.
b
Oh i see let me try this and get back to you
So do I use this in the 3rdparty directory??
c
that’s where we have it, but that’s not hard rule it has to be there.
but the 
source
field for 
python_requirements
should pick up your requirements.txt file, so it is most convenient to have it in the same directory as that file.
b
Copy code
15:38:12.52 [ERROR] 1 Exception encountered:

  ResolveError: The address 3rdparty:3rdparty from the `AllTargets` rule does not exist.

The target name ':3rdparty' is not defined in the directory 3rdparty. Did you mean one of these target names?

  * :reqs
I am getting this error and I seem not to trace its source because I do not have no target name 
3rdparty
, I only have a dependency directory named that
I have changed the build file to this
Copy code
python_requirements(
    name="reqs",
    module_mapping={
        "djangorestframework": ("rest_framework",),
        "djangorestframework-simplejwt": ("rest_framework_simplejwt",)
    },
    overrides={
        "django": {
            "dependencies":(
                "#djangorestframework",
                "#djangorestframework-simplejwt",
            )
        }
    }
                
)
f
and that target is defined in 
3rdparty/BUILD
?
b
Yes it is
f
And how did you refer to the target in the other location?
(as defined, that target's address is 
3rdparty:reqs
. If the reference was just 
3rdparty
, then Pants will assume the target's name is the same as the directory and the address becomes 
3rdparty:3rdparty
)
Hence the error message: 
The target name ':3rdparty' is not defined in the directory 3rdparty.
because the target defined in 3rdparty/BUILD was 
req
and not 
3rdparty
b
Oh so I am supposed to change the target name to 3rdparty??
this is how I referenced it in the other location
Copy code
python_sources(
    overrides={
        "manage.py":{
            "run_goal_use_sandbox":False,
            #"restartable":True,
        },
    },
)

pex_binary(
    name="django-admin",
    dependencies=[
        "3rdparty:reqs#Django",
    ],
    script="django-admin",
)

pex_binary(
    name="manage",
    entry_point="manage.py",
    dependencies=[
        "fleetdatabaseFinal:fleetdatabaseFinal",
        "accounts:accounts",
        "3rdparty:reqs#psycopg2-binary",
        "3rdparty:reqs#gunicorn",
        "3rdparty:reqs#djangorestframework",
        "3rdparty:reqs#djangorestframework-simplejwt",
    ],
    restartable=True,
    inherit_path="prefer",
)
So does pants assume the target name even if I have given the target a name??
f
There are two default rules: • When you define a target and omit 
name
, then Pants will use the name of the directory as the target's name. As you might guess, only one target in a directory can have 
name
omitted. • For a target's address (i.e., a reference to the target), if you omit the 
NAME
component from the address 
path/to/directory/with/a/build/file:NAME
, then Pants will assume that 
NAME
is the name of the directory. (More specifically, the last path component of the path. So a reference to 
src/python/foo/bar
is equivalent to 
src/python/foo/bar:bar
b
But I named the target 
reqs
but it seems pants doesn't read it
Actually my referencing format of the target and dependency is as below 
"3rdparty:reqs#djangorestframework",
But pants still shows it can't read it
So may be @happy-kitchen-89482 can offer some more help and insight here
h
I think we've reached the limit of what we can usefully debug via Slack conversation. The next step would be to create a small example github repo that reproduces the problem and a README that explains what to run in that repo to demonstrate the problem.
b
https://github.com/132kibsdanger/Open-fleet-backend.git here is a link to an open source part of the project tat runs the error....you can kindly check it out and get back to me please
👀 1
so the command I use to run migrations is 
./pants run fleetdatabaseFinal:manage -- makemigrations accounts
Then to run it is 
./pants run fleetdatabaseFinal:manage -- runserver --noreload
So that is it @happy-kitchen-89482
h
Unrelated to any of this, but just FYI: you don't need the 
./pants
script any more, you can just run the 
pants
launcher binary (see here for how to install it).
Also, your command 
pants run fleetdatabaseFinal:manage -- makemigrations accounts
does not run migrations, it creates migration files by comparing changes to your models to the existing migrations, and then you check those resulting migrations files in.
And your 
pants run fleetdatabaseFinal:manage -- runserver --noreload
fails for me because I don't have a database running...
b
So what is the better way to do the migrations?? Also for the database I was using the docker postgres image and running it as a container
h
Migrations need to be generated once and then checked in to the repo. This isn't a pants thing, this is a Django thing. I'm just pointing out that you may be misunderstanding an aspect of how Django works.
You run the checked-in migrations using 
manage.py migrate
But that is unrelated to the Pants issue you're having
just pointing it out
b
And the issue of importing djangorestframework and djangorestframework-simplejwt ?? Because even when they are included in requirements.txt it seems not to import 
rest_framework
and 
rest_framework
rest_framework_simplejwt
h
In your repo, I see:
Copy code
$ waldorf:[~/src/Open-fleet-backend][main]$ pants paths --from=fleetdatabaseFinal:manage --to=3rdparty:reqs#djangorestframework
[
  [
    "fleetdatabaseFinal:manage",
    "3rdparty:reqs#djangorestframework"
  ],
  [
    "fleetdatabaseFinal:manage",
    "accounts/serializers.py",
    "3rdparty:reqs#djangorestframework"
  ]
]
So there is a dependency path (in fact, two of them) from manage.py to djangorestframework
So that part is fine
b
howbeit when I run the that command I get
Copy code
20:23:43.43 [ERROR] 1 Exception encountered:

  ResolveError: The address 3rdparty:3rdparty from the `dependencies` field of the target 3rdparty:reqs#Django does not exist.

The target name ':3rdparty' is not defined in the directory 3rdparty. Did you mean one of these target names?

  * :reqs
What could be the root cause??
This is the build file under the 3rdparty directory. Could it be the cause??
Copy code
python_requirements(
    name="reqs",
    module_mapping={
        "djangorestframework": ("rest_framework",),
        "djangorestframework-simplejwt": ("rest_framework_simplejwt",)
    },
    overrides={
        "django": {
            "dependencies":(
                "#djangorestframework",
                "#djangorestframework-simplejwt",
            )
        }
    }
                
)
h
When you run which command?
Now that I have postgres set up I am able to run your dev server in your repo without any error
b
Copy code
pants paths --from=fleetdatabaseFinal:manage --to=3rdparty:reqs#djangorestframework
h
So I think we need step-by-step instructions, in a README in that repo, on how to reproduce the error on a blank machine (including database setup instructions)
And for you to test that those steps do reproduce the problem
That is weird, 
pants paths --from=fleetdatabaseFinal:manage --to=3rdparty:reqs#djangorestframework
works for me in your repo
So we may not be looking at the same exact code
Since this is slippery, I suggest creating a fully hermetic reproduction of the problem, with all steps clearly laid out for how to trigger it
And you should test that if you clone the reproduction repo yourself and follow its steps, you see the problem
Right now we're going in circles because we're not talking about the same exact state.
b
Sorry about this but this grants you access to the actual repo 
<https://github.com/132kibsdanger/Open-fleetbackend.git>
you can try it with this because it is the same exact code I have
@happy-kitchen-89482 have you tried it yet. this right here is the actual code producing the error
h
What is the command you are running that exposes the error?
b
running 
pants paths --from=fleetdatabaseFinal:manage --to=3rdparty:reqs#djangorestframework
was giving me the error
Copy code
20:23:43.43 [ERROR] 1 Exception encountered:

  ResolveError: The address 3rdparty:3rdparty from the `dependencies` field of the target 3rdparty:reqs#Django does not exist.

The target name ':3rdparty' is not defined in the directory 3rdparty. Did you mean one of these target names?

  * :reqs
this error
h
The error message is telling you the problem: You have written dependencies in that file incorrectly. It should be
Copy code
overrides={
        "django": {
            "dependencies":(
                ":reqs#djangorestframework",
                ":reqs#djangorestframework-simplejwt",
            )
        }
    }
When pants sees 
"#djangorestframework"
it assumes that this belongs to a target with the same name as the directory, but you don't have such a target. Your target's name is 
reqs
, so the dep should be written 
":reqs#djangorestframework"
b
Okay thank you for the insight let me correct it and get back to you
It still gives the error 
ModuleNotFoundError: No module named 'rest_framework_simplejwt
that's what I get when I run 
./pants run fleetdatabaseFinal:manage -- runserver --noreload
@happy-kitchen-89482 have you found any issues with the code....Is it reproducing the error??
h
Look at that error more closely
ModuleNotFoundError: No module named 'rest_framework_simplejwt '
Do you see the issue? There is an extra space there at the end
That is a typo in your settings.py
So far the solution to the issues has been in the error messages, so I recommend paying close attention to those while debugging
b
Okay sir thank you so much
h
You're welcome! Let us know here if you hit more issues that you can't figure out from the error messages. Hopefully they are informative enough to help you make progress, but if not that is good for us to know.
🦸 1
p
Came here after a long rabbithole search to find that psycopg2 -> Django contrib postgres snippet I will try as soon as I come back to my computer. Thanks!!!
👍 1
Open in Slack
PreviousNext
Powered by