hi good people, I am running into an issue when running pants test for a poc. I am getting module not founderror here is the output from pants:

[ERROR] Completed: Run Pytest - shared/tests/common/base_lambdas/api_lambda_application_test.py:tests failed (exit code 4).                                                                                                                             
ImportError while loading conftest '/tmp/pants-sandbox-8uhzoL/shared/tests/common/base_lambdas/conftest.py'.                                                                                                                                                        
shared/tests/common/base_lambdas/conftest.py:3: in <module>                                                                                                                                                                                                         
    from shared.common.base_lambdas.api_lambda_application import ApiLambdaApplication                                                                                                                                                                              
shared/shared/common/base_lambdas/api_lambda_application.py:3: in <module>                                                                                                                                                                                          
    from spine_aws_common import LambdaApplication                                                                                                                                                                                                                  
/home/user/.cache/pants/named_caches/pex_root/venvs/s/876c00e6/venv/lib/python3.9/site-packages/spine_aws_common/__init__.py:4: in <module>                                                                                                                      
    from spine_aws_common.lambda_application import LambdaApplication                                                                                                                                                                                               
/home/user/.cache/pants/named_caches/pex_root/venvs/s/876c00e6/venv/lib/python3.9/site-packages/spine_aws_common/lambda_application.py:9: in <module>                                                                                                            
    from aws_lambda_powertools.utilities import parameters                                                                                                                                                                                                          
/home/user/.cache/pants/named_caches/pex_root/venvs/s/876c00e6/venv/lib/python3.9/site-packages/aws_lambda_powertools/__init__.py:7: in <module>
    from .logging import Logger
/home/user/.cache/pants/named_caches/pex_root/venvs/s/876c00e6/venv/lib/python3.9/site-packages/aws_lambda_powertools/logging/__init__.py:3: in <module>
    from .logger import Logger
/home/user/.cache/pants/named_caches/pex_root/venvs/s/876c00e6/venv/lib/python3.9/site-packages/aws_lambda_powertools/logging/logger.py:26: in <module>
    import jmespath
E   ModuleNotFoundError: No module named 'jmespath'

here is output from pants.toml

[GLOBAL]
pants_version = "2.14.0"
backend_packages = [
    "pants.backend.python",
    "pants.backend.awslambda.python",
    "pants.backend.python.lint.docformatter",
    "pants.backend.python.lint.black",
    #    "pants.backend.python.lint.flake8",
    "pants.backend.python.lint.isort",
    "pants.backend.python.typecheck.mypy",
]
pants_ignore.add = [
    "/tox.ini",
    "backend/",
    "bootstrap/",
    "scripts/",
    "terraform/",
]

[source]
root_patterns = [
    "/lambdas/*",
    "/shared",
    # for conftest
    "/",
]

[anonymous-telemetry]
enabled = false

[python]
interpreter_constraints = ["==3.9.*"]
enable_resolves = true
default_resolve = "common"

[python.resolves]
common = "3rdparty/python/common.lock"

and pyproject.toml

[tool.poetry]
name = "rver-multi-lambda-nextjs-python"
version = "0.1.0"
description = "Projct for Pants. Python Lambdas, Terraform, NextJS"
authors = ["berna <thewest1989@gmail.com>"]
readme = "README.md"
packages = [{include = "rver_multi_lambda_nextjs_python"}]

[tool.poetry.dependencies]
python = "^3.9"
spine-aws-common = {url = "<https://github.com/NHSDigital/spine-core-aws-common/releases/download/v0.2.10/spine_aws_common-0.2.10-py3-none-any.whl>"}
requests = "^2.28.1"
aws-lambda-powertools = "^2.4.0"
importlib-metadata = "^5.0.0"
boto3 = "^1.26.22"
redis-py-cluster = "^2.1.3"
oic = "^1.5.0"
simplejson = "^3.18.1"
dpath = "^2.1.4"
phonenumbers = "^8.13.4"
notifications-python-client = "^6.4.1"
jmespath = "^1.0.1"
arrow = "^1.2.3"
python-Levenshtein = "^0.20.8"
chevron = "^0.14.0"

[tool.poetry.group.dev.dependencies]
packaging = "^20.9"
colorama = "^0.4.4"
coverage = "^6.1.2"
PyYAML = "5.4.1"
awsume = "^4.5.3"

[tool.poetry.group.test.dependencies]
pytest = "^7.2.0"
mock = "^4.0.3"
moto = "^2.0.7"
#pylint = "^2.11.1"
pytest-html = "^3.1.1"
fakeredis = "^2.5.0"
parameterized = "^0.8.1"
freezegun = "^1.2.2"
pytest-cov = "^4.0.0"
pytest-custom-report = "^1.0.1"
pytest-deadfixtures = "^2.2.1"
python-dotenv = "^0.21.0"
pytest-env = "^0.8.1"
pytest-xdist = "^3.1.0"
pytest-rerunfailures = "^10.3"
types-mock = "^4.0.15"
types-python-dateutil = "^2.8.19"
types-requests = "^2.28.11"
types-setuptools = "^65.6.0"
types-simplejson = "^3.18.0"
types-toml = "^0.10.8"


[tool.poetry.group.localstack.dependencies]
awscli-local = "^0.20"
awscli = "^1.27.22"


[build-system]
requires = ["poetry-core"]
build-backend = "poetry.core.masonry.api"

the strange thing is when I run the test using

pytest

directly they pass

I've had the same error. I think that the newest version of

aws-labmda-powertools

does not declare its dependencies correctly. I solved by downgrading to

aws_lambda_powertools==1.25.7

. One more option would be to "fix" this package manually. See this.

fix manually how

didn't try it myself, but this paragraph (requirements-with-undeclared-dependencies) seems to suggest a solution.

so downgrading the version defo worked

Is

jmespath

added explictly to the

pyproject.toml

? If not then you need to add it to pyproject.toml as a dependency. What’s happening currently is that pants can’t find it.

yes it is. if you look at pyproject.toml output its defined as

jmespath = "^1.0.1"

and my root

BUILD

located where the

pyproject.toml

looks like this

poetry_requirements(
    name="root",
    overrides={
            "aws-lambda-powertools": {"dependencies": [":root#jmespath"]},
        },
)

#python_requirement(
#    name="jmespath",
#    requirements=["jmespath"],
#)

#python_requirements(
#    name="reqs",
#    overrides={
#        "aws-lambda-powertools": {"dependencies": [":root#jmespath"]},
#    },
#)

python_test_utils(
    name="test_utils0",
)

have also run pants generate-lockfiles as well

Where do you set the resolve? i.e.

poetry_requirements(
    name="root",
    overrides={
            "aws-lambda-powertools": {"dependencies": [":root#jmespath"]},
        },
     resolve="common"
)

the resolve is set in the

pants.toml

with

default_resolve = "common"

And the same resolve then should be set up for

python_sources

and

python_tests

Yeah that’s just defining your resolve. You need to then feed it to the targets. One way to set it for everything by default is writing inside your BUILD file using

___defaults___

https://www.pantsbuild.org/docs/targets#field-default-values

__defaults__(all=dict(resolve="common"))

yeah thought so too. so am a bit lost what else its missing

pants should have complained about it when generating lockfile then. So I don’t think that’s the case

like suggested downgrading to the version specified @alert-psychiatrist-14102 worked. but want to use "lates powertools"

Pants only picks up the required dependencies. So most probably when building the pex for test, it doesn’t see it as a required dependencies. Just because all the dependencies are installed inside a venv, it doesn’t mean pants will use all of them.

makes sense. so is the above the normal way the correct way to address such issues (if it is an issue"

By the way I was able to run just with but I only tried with handful of dependencies.

poetry_requirements(
    name="root",
    overrides={
            "aws-lambda-powertools": {"dependencies": [":root#jmespath"]},
        },
)

So it does seem like some other dependency is messing it up as you see with

spine-aws-common

thanks for your help again. . will see what els I manage to break as I go further

But I have never come across such case. It’s still bit weird that even with explicit dependency pants failed to figure it out.

at this point it could be the underlying spine-aws-common even due I thought adding the override to

aws-lambda-powertools

would fix it. as removing the override and downgrading powertools in my

pyproject.toml

seems to work

I am not sure. You can give it a try.

this is the output

InvalidLockfileError: You are using the lockfile at 3rdparty/python/common.lock to install the resolve `common` (from `[python].resolves`). However, it is not compatible with the current targets because:

- The targets depend on requirements that are not in the lockfile: ['aws-lambda-powertools==1.25.7']

This most often happens when adding a new requirement to your project, or bumping requirement versions. You can fix this by regenerating the lockfile with `generate-lockfiles`.

To regenerate your lockfile, run `./pants generate-lockfiles --resolve=common`.

and the output of lockfile generation

The conflict is caused by:
     The user requested aws-lambda-powertools<3.0.0 and >=2.3.1
     The user requested aws-lambda-powertools==1.25.7

 To fix this you could try to:
 1. loosen the range of package versions you've specified
 2. remove package versions to allow pip attempt to solve the dependency conflict