https://pantsbuild.org/ logo
#general
Title
# general
c

chilly-holiday-77415

12/22/2022, 12:32 PM
I think I’m having a module mapping and transitive dependency issue: in a standard .venv, when I do
pip install aws-cdk-lib
, it pulls in
aws_cdk
, as well as some transitive dependencies that are submodules -
aws_cdk.asset_awscli_v1
for example - for clarity, as
.venv/lib/python3.9/site-packages/aws_cdk/asset_awscli_v1
. In my
poetry_requirements(name="poetry", module_mapping={"aws-cdk-lib": ["aws_cdk"]})
top level build file I map the different name In my cdk project I declare the transitive deps not being inferred
Copy code
python_sources(
    name="cdklib",
    dependencies=[
        "//:poetry0#aws-cdk-asset-awscli-v1",
        "//:poetry0#jsii",
    ],
)

pex_binary(
    name="cdk_infra",
    entry_point="app.py",
)
and this gets me really close, but still fails with an import error - if I unzip the pex, I can see it’s pulling
aws-cdk-asset-awscli-v1
separately into it’s own
aws_cdk/assert_awscli_v1
, rather than into the initial
aws_cdk
install which is what I think I need. Hopefully that makes sense - I can cut an issue if that’s helpful! I tried mapping
“aws-cdk-asset-awscli-v1”: [“aws_cdk.asset_awscli_v1"]
but that didn’t seem to help.
e

enough-analyst-54434

12/22/2022, 3:23 PM
@chilly-holiday-77415 this should be fixed in 2.16.x: https://github.com/pantsbuild/pants/commit/077ea7132b4fd74dde10324bf0e97a43655728d0. If you try 2.16.0.dev3 - the latest 2.16.x, you'll just need to go back to trying
"aws-cdk-asset-awscli-v1": ["aws_cdk.asset_awscli_v1"]
, which was the right idea.
h

happy-kitchen-89482

12/22/2022, 3:26 PM
I think this might be a deeper problem. Even without Pants, pex fails here:
Copy code
$ pex aws-cdk-lib 
>>> import aws_cdk
Traceback (most recent call last):
  File "/Users/benjyw/.pyenv/versions/3.9.13/lib/python3.9/code.py", line 90, in runcode
    exec(code, self.locals)
  File "<console>", line 1, in <module>
  File "/Users/benjyw/.pex/installed_wheels/fae279efcfc4a6e276759a0deed6c0ee6751e27727538540010ad9081d409e5b/aws_cdk_lib-2.56.0-py3-none-any.whl/aws_cdk/__init__.py", line 1282, in <module>
    from ._jsii import *
  File "/Users/benjyw/.pex/installed_wheels/fae279efcfc4a6e276759a0deed6c0ee6751e27727538540010ad9081d409e5b/aws_cdk_lib-2.56.0-py3-none-any.whl/aws_cdk/_jsii/__init__.py", line 13, in <module>
    import aws_cdk.asset_awscli_v1._jsii
ModuleNotFoundError: No module named 'aws_cdk.asset_awscli_v1'
The issue seems to be that
aws_cdk/__init__.py
exists (so the package is not an implicit namespace package) but does not contain namespace package incantations. So
aws_cdk
is not a namespace package, and importing
aws_cdk.asset_awscli_v1
fails due to it not being nested in the same package tree.
e

enough-analyst-54434

12/22/2022, 3:28 PM
Ah, let me look. Pants still needs your dep inference fix though right?
h

happy-kitchen-89482

12/22/2022, 3:29 PM
(it's in its own compartment in .deps)
I don't think so?
aws-cdk-asset-awscli-v1
is a dep of
aws_cdk
, which tries to import it and fails.
So
aws_cdk
itself fails, even if the end user is not importing from
aws_cdk.asset_awscli_v1
e

enough-analyst-54434

12/22/2022, 3:31 PM
I'm trying to get you to focus on Daves problem, not your backtrace. There was a bug in Pants dep inference that would still be hit once this import issue is fixed IIUC. The dep inference used to stop at the root package. You fixed that.
h

happy-kitchen-89482

12/22/2022, 3:31 PM
So dep inference isn't even in play here
If I understand correctly
Right, but Dave is not importing from
aws_cdk.asset_awscli_v1
, Dave is importing from
aws_cdk
(and that dep is correctly inferred), and then that is importing from
aws_cdk.asset_awscli_v1
and failing.
So I don't think the dep inference bug is relevant here?
e

enough-analyst-54434

12/22/2022, 3:32 PM
Dave mentions 0 backtraces is my only point. I'm too slow and you've moved on to a new thing fwict.
h

happy-kitchen-89482

12/22/2022, 3:34 PM
maybe I'm misunderstanding something, but I think I'm talking about the original thing.
That backtrace is the same import error the OP reported, I am assuming. Dave, can you confirm?
e

enough-analyst-54434

12/22/2022, 3:36 PM
Ok, so the real venv gets along furthest. These packages are a bit crazy, so 1st step - actually use a full real venv:
Copy code
$ pex aws-cdk-lib --venv-site-packages-copies --venv -- -c 'import aws_cdk; print(aws_cdk.__file__)'
Traceback (most recent call last):
  File "/home/jsirois/.pex/venvs/e0d8e2d1c3cc3e1315a92d099c3e7d5a3e928ae6/5985ed09b49a653d6596b0e14d134c5456cf1a9f/pex", line 237, in <module>
    exec(ast, globals_map, locals_map)
  File "-c <cmd>", line 1, in <module>
    #!/home/jsirois/.pex/venvs/s/bbcea5a4/venv/bin/python3.10 -sE
  File "/home/jsirois/.pex/venvs/e0d8e2d1c3cc3e1315a92d099c3e7d5a3e928ae6/5985ed09b49a653d6596b0e14d134c5456cf1a9f/lib/python3.10/site-packages/aws_cdk/__init__.py", line 1282, in <module>
    from ._jsii import *
  File "/home/jsirois/.pex/venvs/e0d8e2d1c3cc3e1315a92d099c3e7d5a3e928ae6/5985ed09b49a653d6596b0e14d134c5456cf1a9f/lib/python3.10/site-packages/aws_cdk/_jsii/__init__.py", line 13, in <module>
    import aws_cdk.asset_awscli_v1._jsii
  File "/home/jsirois/.pex/venvs/e0d8e2d1c3cc3e1315a92d099c3e7d5a3e928ae6/5985ed09b49a653d6596b0e14d134c5456cf1a9f/lib/python3.10/site-packages/aws_cdk/asset_awscli_v1/_jsii/__init__.py", line 13, in <module>
    __jsii_assembly__ = jsii.JSIIAssembly.load(
  File "/home/jsirois/.pex/venvs/e0d8e2d1c3cc3e1315a92d099c3e7d5a3e928ae6/5985ed09b49a653d6596b0e14d134c5456cf1a9f/lib/python3.10/site-packages/jsii/_runtime.py", line 54, in load
    _kernel.load(assembly.name, assembly.version, os.fspath(assembly_path))
  File "/home/jsirois/.pex/venvs/e0d8e2d1c3cc3e1315a92d099c3e7d5a3e928ae6/5985ed09b49a653d6596b0e14d134c5456cf1a9f/lib/python3.10/site-packages/jsii/_kernel/__init__.py", line 301, in load
    self.provider.load(LoadRequest(name=name, version=version, tarball=tarball))
  File "/home/jsirois/.pex/venvs/e0d8e2d1c3cc3e1315a92d099c3e7d5a3e928ae6/5985ed09b49a653d6596b0e14d134c5456cf1a9f/lib/python3.10/site-packages/jsii/_kernel/providers/process.py", line 352, in load
    return self._process.send(request, LoadResponse)
  File "/home/jsirois/.pex/venvs/e0d8e2d1c3cc3e1315a92d099c3e7d5a3e928ae6/5985ed09b49a653d6596b0e14d134c5456cf1a9f/lib/python3.10/site-packages/jsii/_utils.py", line 24, in wrapped
    stored.append(fgetter(self))
  File "/home/jsirois/.pex/venvs/e0d8e2d1c3cc3e1315a92d099c3e7d5a3e928ae6/5985ed09b49a653d6596b0e14d134c5456cf1a9f/lib/python3.10/site-packages/jsii/_kernel/providers/process.py", line 347, in _process
    process.start()
  File "/home/jsirois/.pex/venvs/e0d8e2d1c3cc3e1315a92d099c3e7d5a3e928ae6/5985ed09b49a653d6596b0e14d134c5456cf1a9f/lib/python3.10/site-packages/jsii/_kernel/providers/process.py", line 260, in start
    self._process = subprocess.Popen(
  File "/usr/lib/python3.10/subprocess.py", line 969, in __init__
    self._execute_child(args, executable, preexec_fn, close_fds,
  File "/usr/lib/python3.10/subprocess.py", line 1845, in _execute_child
    raise child_exception_type(errno_num, err_msg, err_filename)
FileNotFoundError: [Errno 2] No such file or directory: 'node'
Exception ignored in: <function _NodeProcess.__del__ at 0x7f167692f1c0>
Traceback (most recent call last):
  File "/home/jsirois/.pex/venvs/e0d8e2d1c3cc3e1315a92d099c3e7d5a3e928ae6/5985ed09b49a653d6596b0e14d134c5456cf1a9f/lib/python3.10/site-packages/jsii/_kernel/providers/process.py", line 233, in __del__
    self.stop()
  File "/home/jsirois/.pex/venvs/e0d8e2d1c3cc3e1315a92d099c3e7d5a3e928ae6/5985ed09b49a653d6596b0e14d134c5456cf1a9f/lib/python3.10/site-packages/jsii/_kernel/providers/process.py", line 291, in stop
    assert self._process.stdin is not None
AttributeError: '_NodeProcess' object has no attribute '_process'
The
--venv-site-packages-copies
and
--venv
are critical.
Ah, ok - you need node installed for this to work. TIL jsii - my god.
😅 1
After installing node now I get:
Copy code
$ pex aws-cdk-lib --venv-site-packages-copies --venv -- -c 'import aws_cdk; print(aws_cdk.__file__)'
b'!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!\n'
b'!!                                                                            !!\n'
b'!!  Node 12 has reached end-of-life on 2022-04-30 and is not supported.       !!\n'
b'!!  Please upgrade to a supported node version as soon as possible.           !!\n'
b'!!                                                                            !!\n'
b'!!  This software is currently running on node v12.22.9.                      !!\n'
b'!!  As of the current release of this software, supported node releases are:  !!\n'
b'!!  - ^18.0.0 (Planned end-of-life: 2025-04-30)                               !!\n'
b'!!  - ^16.3.0 (Planned end-of-life: 2023-09-11)                               !!\n'
b'!!  - ^19.0.0 (Planned end-of-life: 2023-06-01)                               !!\n'
b'!!  - ^14.6.0 (Planned end-of-life: 2023-04-30)                               !!\n'
b'!!                                                                            !!\n'
b'!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!\n'
And it hangs looking for input apparently?
So, wrapping up, none of this works unless you're using
pex_binary.execution_mode="venv"
if in a
pex_binary
, the rest of Pants internals uses venvs. The other bit is to use https://www.pantsbuild.org/v2.15/docs/reference-pex_binary#codevenv_site_packages_copiescode in Pants 2.15+. That though just covers
pex_binary
targets. The rest of Pants infra, tests etc, should be fine. They use
--venv
--venv-site-packages-copies
.
@happy-kitchen-89482 so things may in fact be back to module mappings. @chilly-holiday-77415 your detailed backtraces will be very helpful.
c

chilly-holiday-77415

12/22/2022, 4:00 PM
hello - apologies for not including everything in my OP, I didn’t want to overburden assumed user error 🙂
Copy code
./dist/projects.infrastructure/cdk_infra.pex 
Traceback (most recent call last):
  File "/usr/lib/python3.9/runpy.py", line 197, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/usr/lib/python3.9/runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "/root/.pex/unzipped_pexes/2d411bcd942017d96046acee712d91c2a41948ca/__main__.py", line 103, in <module>
    bootstrap_pex(__entry_point__, execute=__execute__, venv_dir=__venv_dir__)
  File "/root/.pex/unzipped_pexes/2d411bcd942017d96046acee712d91c2a41948ca/.bootstrap/pex/pex_bootstrapper.py", line 601, in bootstrap_pex
    pex.PEX(entry_point).execute()
  File "/root/.pex/unzipped_pexes/2d411bcd942017d96046acee712d91c2a41948ca/.bootstrap/pex/pex.py", line 541, in execute
    sys.exit(self._wrap_coverage(self._wrap_profiling, self._execute))
  File "/root/.pex/unzipped_pexes/2d411bcd942017d96046acee712d91c2a41948ca/.bootstrap/pex/pex.py", line 448, in _wrap_coverage
    return runner(*args)
  File "/root/.pex/unzipped_pexes/2d411bcd942017d96046acee712d91c2a41948ca/.bootstrap/pex/pex.py", line 479, in _wrap_profiling
    return runner(*args)
  File "/root/.pex/unzipped_pexes/2d411bcd942017d96046acee712d91c2a41948ca/.bootstrap/pex/pex.py", line 568, in _execute
    return self.execute_entry(
  File "/root/.pex/unzipped_pexes/2d411bcd942017d96046acee712d91c2a41948ca/.bootstrap/pex/pex.py", line 752, in execute_entry
    return self.execute_module(entry_point.module)
  File "/root/.pex/unzipped_pexes/2d411bcd942017d96046acee712d91c2a41948ca/.bootstrap/pex/pex.py", line 760, in execute_module
    runpy.run_module(module_name, run_name="__main__", alter_sys=True)
  File "/usr/lib/python3.9/runpy.py", line 210, in run_module
    return _run_module_code(code, init_globals, run_name, mod_spec)
  File "/usr/lib/python3.9/runpy.py", line 97, in _run_module_code
    _run_code(code, mod_globals, init_globals,
  File "/usr/lib/python3.9/runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "/root/.pex/unzipped_pexes/2d411bcd942017d96046acee712d91c2a41948ca/infrastructure/app.py", line 3, in <module>
    from aws_cdk import App, Environment, Tags
  File "/root/.pex/installed_wheels/041058205895fbc5b2a082f76f9bb90ae6851fe21f8c31a65cd52ee35e0ef9c1/aws_cdk_lib-2.56.0-py3-none-any.whl/aws_cdk/__init__.py", line 1282, in <module>
    from ._jsii import *
  File "/root/.pex/installed_wheels/041058205895fbc5b2a082f76f9bb90ae6851fe21f8c31a65cd52ee35e0ef9c1/aws_cdk_lib-2.56.0-py3-none-any.whl/aws_cdk/_jsii/__init__.py", line 13, in <module>
    import aws_cdk.asset_awscli_v1._jsii
ModuleNotFoundError: No module named 'aws_cdk.asset_awscli_v1'
and yeah, this is possibly just popping the lid off a whole can of worms - jsii and a node dependency were the problem I was expecting to hit hard. Happy to walk you through that problem statement too if you’re interested!
e

enough-analyst-54434

12/22/2022, 4:06 PM
No, so you are building a PEX and running it, so the work above is needed. You need: 1. Pants 2.15+ 2. On your
pex_binary
target,
execution_mode="venv"
3. On your `pex_binary`target,
venv_site_packages_copies=True
(Only available in Pants 2.15.x +)
Then re-run
./pants package ...
then try
./dist/projects.infrastructure/cdk_infra.pex
again.
@chilly-holiday-77415 let us know how that fares.
Steps 2 + 3 are done by Pants for internal operations like those supporting the `test`goal; its just
run
and `package`that need this special treatment on the external artifact produced by
pex_binary
.
c

chilly-holiday-77415

12/22/2022, 4:32 PM
success! Running the pex now behaves as running the entrypoint in a venv. The next challenge is figuring out if that’s actually useful 🙂 I’ll read through those links to build my understanding better - appreciate the detailed help, both
e

enough-analyst-54434

12/22/2022, 4:37 PM
Ok, great. Don't be afraid of spamming details. In async remote debugging, too many is always better than too few.
c

chilly-holiday-77415

12/22/2022, 4:45 PM
hugely impressed with the magic of pex just emulating a python file in a wildly alien context - my naive first attempt seems to just work:
cdk ls --app ../../dist/projects.infrastructure/cdk_infra.pex
seems functionally equivalent to
cdk ls --app app.py
- I have a little more work to do but I think Pants can fill a gaping hole in developer experience, at least in Python-land, for AWS-native apps
e

enough-analyst-54434

12/22/2022, 5:09 PM
Ok, cool. I'd expect road bumps still - this whole jsii thing is wild, but who knows.
c

chilly-holiday-77415

12/22/2022, 5:14 PM
who doesn’t love a typescript error from a python framework so you can hit a third party api and hope it works 🙂
l

lemon-yak-80782

02/18/2023, 5:01 PM
Is it possible to use python_lambdas as a dependency. I do not have access to the generated zip. Is it possible to run python_source with steps 2 and 3?
c

chilly-holiday-77415

02/20/2023, 9:45 AM
I’m not sure I follow “I do not have access to the generated zip.” - they’re created in your /dist/ folder as default and can be targeted by cdk like any other zip?
l

lemon-yak-80782

02/22/2023, 10:35 PM
Are they not only in the dist folder if I build them with
./pants package ::
? Currently I deploy as you do , but from a shell script. First I run
package
and move the zips to a folder with docker files. Then I run the cdk command, that deploys docker lambdas. This whole process does not feel quite right to me. But it works for now.
c

chilly-holiday-77415

02/23/2023, 10:36 AM
ah, yeah - I think that’s quite a specific implementation detail of cdk.Asset() though. If you can run DinD, you could use Pants to publish docker images and consume them in CDK instead which breaks the dissonance caused by CDK magic
l

lemon-yak-80782

02/24/2023, 6:56 AM
@chilly-holiday-77415 Thanks. I will give it a shot.
c

chilly-holiday-77415

02/24/2023, 9:58 AM
I’m not smart enough to figure out how to write a Kaniko plugin yet but I have it in back of mind as something to look in to that would help us use Pants what I sense would be a lot more idiomatic way, but no DinD for our CI is a constraint
6 Views