https://pantsbuild.org/ logo
#general
Title
# general
a

abundant-analyst-12845

02/05/2023, 2:14 PM
hi good people, I am running into an issue when running pants test for a poc. I am getting module not founderror here is the output from pants:
Copy code
[ERROR] Completed: Run Pytest - shared/tests/common/base_lambdas/api_lambda_application_test.py:tests failed (exit code 4).                                                                                                                             
ImportError while loading conftest '/tmp/pants-sandbox-8uhzoL/shared/tests/common/base_lambdas/conftest.py'.                                                                                                                                                        
shared/tests/common/base_lambdas/conftest.py:3: in <module>                                                                                                                                                                                                         
    from shared.common.base_lambdas.api_lambda_application import ApiLambdaApplication                                                                                                                                                                              
shared/shared/common/base_lambdas/api_lambda_application.py:3: in <module>                                                                                                                                                                                          
    from spine_aws_common import LambdaApplication                                                                                                                                                                                                                  
/home/user/.cache/pants/named_caches/pex_root/venvs/s/876c00e6/venv/lib/python3.9/site-packages/spine_aws_common/__init__.py:4: in <module>                                                                                                                      
    from spine_aws_common.lambda_application import LambdaApplication                                                                                                                                                                                               
/home/user/.cache/pants/named_caches/pex_root/venvs/s/876c00e6/venv/lib/python3.9/site-packages/spine_aws_common/lambda_application.py:9: in <module>                                                                                                            
    from aws_lambda_powertools.utilities import parameters                                                                                                                                                                                                          
/home/user/.cache/pants/named_caches/pex_root/venvs/s/876c00e6/venv/lib/python3.9/site-packages/aws_lambda_powertools/__init__.py:7: in <module>
    from .logging import Logger
/home/user/.cache/pants/named_caches/pex_root/venvs/s/876c00e6/venv/lib/python3.9/site-packages/aws_lambda_powertools/logging/__init__.py:3: in <module>
    from .logger import Logger
/home/user/.cache/pants/named_caches/pex_root/venvs/s/876c00e6/venv/lib/python3.9/site-packages/aws_lambda_powertools/logging/logger.py:26: in <module>
    import jmespath
E   ModuleNotFoundError: No module named 'jmespath'
here is output from pants.toml
Copy code
[GLOBAL]
pants_version = "2.14.0"
backend_packages = [
    "pants.backend.python",
    "pants.backend.awslambda.python",
    "pants.backend.python.lint.docformatter",
    "pants.backend.python.lint.black",
    #    "pants.backend.python.lint.flake8",
    "pants.backend.python.lint.isort",
    "pants.backend.python.typecheck.mypy",
]
pants_ignore.add = [
    "/tox.ini",
    "backend/",
    "bootstrap/",
    "scripts/",
    "terraform/",
]

[source]
root_patterns = [
    "/lambdas/*",
    "/shared",
    # for conftest
    "/",
]

[anonymous-telemetry]
enabled = false

[python]
interpreter_constraints = ["==3.9.*"]
enable_resolves = true
default_resolve = "common"

[python.resolves]
common = "3rdparty/python/common.lock"
and pyproject.toml
Copy code
[tool.poetry]
name = "rver-multi-lambda-nextjs-python"
version = "0.1.0"
description = "Projct for Pants. Python Lambdas, Terraform, NextJS"
authors = ["berna <thewest1989@gmail.com>"]
readme = "README.md"
packages = [{include = "rver_multi_lambda_nextjs_python"}]

[tool.poetry.dependencies]
python = "^3.9"
spine-aws-common = {url = "<https://github.com/NHSDigital/spine-core-aws-common/releases/download/v0.2.10/spine_aws_common-0.2.10-py3-none-any.whl>"}
requests = "^2.28.1"
aws-lambda-powertools = "^2.4.0"
importlib-metadata = "^5.0.0"
boto3 = "^1.26.22"
redis-py-cluster = "^2.1.3"
oic = "^1.5.0"
simplejson = "^3.18.1"
dpath = "^2.1.4"
phonenumbers = "^8.13.4"
notifications-python-client = "^6.4.1"
jmespath = "^1.0.1"
arrow = "^1.2.3"
python-Levenshtein = "^0.20.8"
chevron = "^0.14.0"

[tool.poetry.group.dev.dependencies]
packaging = "^20.9"
colorama = "^0.4.4"
coverage = "^6.1.2"
PyYAML = "5.4.1"
awsume = "^4.5.3"

[tool.poetry.group.test.dependencies]
pytest = "^7.2.0"
mock = "^4.0.3"
moto = "^2.0.7"
#pylint = "^2.11.1"
pytest-html = "^3.1.1"
fakeredis = "^2.5.0"
parameterized = "^0.8.1"
freezegun = "^1.2.2"
pytest-cov = "^4.0.0"
pytest-custom-report = "^1.0.1"
pytest-deadfixtures = "^2.2.1"
python-dotenv = "^0.21.0"
pytest-env = "^0.8.1"
pytest-xdist = "^3.1.0"
pytest-rerunfailures = "^10.3"
types-mock = "^4.0.15"
types-python-dateutil = "^2.8.19"
types-requests = "^2.28.11"
types-setuptools = "^65.6.0"
types-simplejson = "^3.18.0"
types-toml = "^0.10.8"


[tool.poetry.group.localstack.dependencies]
awscli-local = "^0.20"
awscli = "^1.27.22"


[build-system]
requires = ["poetry-core"]
build-backend = "poetry.core.masonry.api"
the strange thing is when I run the test using
pytest
directly they pass
a

alert-psychiatrist-14102

02/05/2023, 2:19 PM
I've had the same error. I think that the newest version of
aws-labmda-powertools
does not declare its dependencies correctly. I solved by downgrading to
aws_lambda_powertools==1.25.7
. One more option would be to "fix" this package manually. See this.
a

abundant-analyst-12845

02/05/2023, 2:21 PM
fix manually how
but let me downgrade for now to see if that gets me going
a

alert-psychiatrist-14102

02/05/2023, 2:24 PM
didn't try it myself, but this paragraph (requirements-with-undeclared-dependencies) seems to suggest a solution.
a

abundant-analyst-12845

02/05/2023, 2:26 PM
so downgrading the version defo worked
will try and make sense in ther link you specified
tried this:
Copy code
poetry_requirements(
    name="root",
    overrides={
            "aws-lambda-powertools": {"dependencies": [":root#jmespath"]},
        },
)
didn't work so could be am doing it wrong perhaps
r

refined-addition-53644

02/05/2023, 2:55 PM
Is
jmespath
added explictly to the
pyproject.toml
? If not then you need to add it to pyproject.toml as a dependency. What’s happening currently is that pants can’t find it.
a

abundant-analyst-12845

02/05/2023, 2:58 PM
yes it is. if you look at pyproject.toml output its defined as
jmespath = "^1.0.1"
and my root
BUILD
located where the
pyproject.toml
looks like this
Copy code
poetry_requirements(
    name="root",
    overrides={
            "aws-lambda-powertools": {"dependencies": [":root#jmespath"]},
        },
)

#python_requirement(
#    name="jmespath",
#    requirements=["jmespath"],
#)

#python_requirements(
#    name="reqs",
#    overrides={
#        "aws-lambda-powertools": {"dependencies": [":root#jmespath"]},
#    },
#)

python_test_utils(
    name="test_utils0",
)
have also run pants generate-lockfiles as well
👀 1
r

refined-addition-53644

02/05/2023, 3:06 PM
Where do you set the resolve? i.e.
Copy code
poetry_requirements(
    name="root",
    overrides={
            "aws-lambda-powertools": {"dependencies": [":root#jmespath"]},
        },
     resolve="common"
)
Unless you have set it somewhere else by default, I think you aren’t using the resolve.
a

abundant-analyst-12845

02/05/2023, 3:08 PM
the resolve is set in the
pants.toml
with
default_resolve = "common"
r

refined-addition-53644

02/05/2023, 3:10 PM
And the same resolve then should be set up for
python_sources
and
python_tests
Yeah that’s just defining your resolve. You need to then feed it to the targets. One way to set it for everything by default is writing inside your BUILD file using
___defaults___
https://www.pantsbuild.org/docs/targets#field-default-values
Copy code
__defaults__(all=dict(resolve="common"))
Sorry you have set it as a
default_resolve
already. So that should have worked 😞
a

abundant-analyst-12845

02/05/2023, 3:13 PM
yeah thought so too. so am a bit lost what else its missing
unless the versiosn are too far apart?
r

refined-addition-53644

02/05/2023, 3:14 PM
pants should have complained about it when generating lockfile then. So I don’t think that’s the case
a

abundant-analyst-12845

02/05/2023, 3:14 PM
like suggested downgrading to the version specified @alert-psychiatrist-14102 worked. but want to use "lates powertools"
so doing the following worked
Copy code
poetry_requirements(
    name="root",
    overrides={
#            "aws-lambda-powertools": {"dependencies": [":root#jmespath"]},
            "spine-aws-common": {"dependencies": [
                ":root#jmespath",
                ":root#boto3",
                ]},
        },
)
which makes me thing that the
spine-aws-common
dependency which relied on
aws-lambda-powertools
might have caused. sadly when adding the
jmespath
it complained about missing
boto3
so had to add it
still don't understand why it works from i.e a venv perspective running
pytest
but not from pants
r

refined-addition-53644

02/05/2023, 3:38 PM
Pants only picks up the required dependencies. So most probably when building the pex for test, it doesn’t see it as a required dependencies. Just because all the dependencies are installed inside a venv, it doesn’t mean pants will use all of them.
a

abundant-analyst-12845

02/05/2023, 3:39 PM
makes sense. so is the above the normal way the correct way to address such issues (if it is an issue"
r

refined-addition-53644

02/05/2023, 3:40 PM
By the way I was able to run just with but I only tried with handful of dependencies.
Copy code
poetry_requirements(
    name="root",
    overrides={
            "aws-lambda-powertools": {"dependencies": [":root#jmespath"]},
        },
)
So it does seem like some other dependency is messing it up as you see with
spine-aws-common
yeah in case of pants not being able to figure out transitive dependency because of some issue, this is the way to go about it.
a

abundant-analyst-12845

02/05/2023, 3:42 PM
thanks for your help again. . will see what els I manage to break as I go further
r

refined-addition-53644

02/05/2023, 3:43 PM
But I have never come across such case. It’s still bit weird that even with explicit dependency pants failed to figure it out.
a

abundant-analyst-12845

02/05/2023, 3:45 PM
at this point it could be the underlying spine-aws-common even due I thought adding the override to
aws-lambda-powertools
would fix it. as removing the override and downgrading powertools in my
pyproject.toml
seems to work
i guess I could maybe create a
python_requirements
block with
requirement
to the downgraded version and override
spine-aws-common
to sue that. would that cause ambiguous error?
r

refined-addition-53644

02/05/2023, 3:49 PM
I am not sure. You can give it a try.
a

abundant-analyst-12845

02/05/2023, 3:59 PM
this is the output
Copy code
InvalidLockfileError: You are using the lockfile at 3rdparty/python/common.lock to install the resolve `common` (from `[python].resolves`). However, it is not compatible with the current targets because:

- The targets depend on requirements that are not in the lockfile: ['aws-lambda-powertools==1.25.7']

This most often happens when adding a new requirement to your project, or bumping requirement versions. You can fix this by regenerating the lockfile with `generate-lockfiles`.

To regenerate your lockfile, run `./pants generate-lockfiles --resolve=common`.
and the output of lockfile generation
Copy code
The conflict is caused by:
     The user requested aws-lambda-powertools<3.0.0 and >=2.3.1
     The user requested aws-lambda-powertools==1.25.7

 To fix this you could try to:
 1. loosen the range of package versions you've specified
 2. remove package versions to allow pip attempt to solve the dependency conflict
8 Views