I think I’m having a module mapping and transitive dependency issue: in a standard .venv, when I do

pip install aws-cdk-lib

, it pulls in

aws_cdk

, as well as some transitive dependencies that are submodules -

aws_cdk.asset_awscli_v1

for example - for clarity, as

.venv/lib/python3.9/site-packages/aws_cdk/asset_awscli_v1

. In my

poetry_requirements(name="poetry", module_mapping={"aws-cdk-lib": ["aws_cdk"]})

top level build file I map the different name In my cdk project I declare the transitive deps not being inferred

python_sources(
    name="cdklib",
    dependencies=[
        "//:poetry0#aws-cdk-asset-awscli-v1",
        "//:poetry0#jsii",
    ],
)

pex_binary(
    name="cdk_infra",
    entry_point="app.py",
)

and this gets me really close, but still fails with an import error - if I unzip the pex, I can see it’s pulling

aws-cdk-asset-awscli-v1

separately into it’s own

aws_cdk/assert_awscli_v1

, rather than into the initial

aws_cdk

install which is what I think I need. Hopefully that makes sense - I can cut an issue if that’s helpful! I tried mapping

“aws-cdk-asset-awscli-v1”: [“aws_cdk.asset_awscli_v1"]

but that didn’t seem to help.

@chilly-holiday-77415 this should be fixed in 2.16.x: https://github.com/pantsbuild/pants/commit/077ea7132b4fd74dde10324bf0e97a43655728d0. If you try 2.16.0.dev3 - the latest 2.16.x, you'll just need to go back to trying

"aws-cdk-asset-awscli-v1": ["aws_cdk.asset_awscli_v1"]

, which was the right idea.

I think this might be a deeper problem. Even without Pants, pex fails here:

$ pex aws-cdk-lib 
>>> import aws_cdk
Traceback (most recent call last):
  File "/Users/benjyw/.pyenv/versions/3.9.13/lib/python3.9/code.py", line 90, in runcode
    exec(code, self.locals)
  File "<console>", line 1, in <module>
  File "/Users/benjyw/.pex/installed_wheels/fae279efcfc4a6e276759a0deed6c0ee6751e27727538540010ad9081d409e5b/aws_cdk_lib-2.56.0-py3-none-any.whl/aws_cdk/__init__.py", line 1282, in <module>
    from ._jsii import *
  File "/Users/benjyw/.pex/installed_wheels/fae279efcfc4a6e276759a0deed6c0ee6751e27727538540010ad9081d409e5b/aws_cdk_lib-2.56.0-py3-none-any.whl/aws_cdk/_jsii/__init__.py", line 13, in <module>
    import aws_cdk.asset_awscli_v1._jsii
ModuleNotFoundError: No module named 'aws_cdk.asset_awscli_v1'

Ah, let me look. Pants still needs your dep inference fix though right?

(it's in its own compartment in .deps)

I'm trying to get you to focus on Daves problem, not your backtrace. There was a bug in Pants dep inference that would still be hit once this import issue is fixed IIUC. The dep inference used to stop at the root package. You fixed that.

So dep inference isn't even in play here

Dave mentions 0 backtraces is my only point. I'm too slow and you've moved on to a new thing fwict.

maybe I'm misunderstanding something, but I think I'm talking about the original thing.

Ok, so the real venv gets along furthest. These packages are a bit crazy, so 1st step - actually use a full real venv:

$ pex aws-cdk-lib --venv-site-packages-copies --venv -- -c 'import aws_cdk; print(aws_cdk.__file__)'
Traceback (most recent call last):
  File "/home/jsirois/.pex/venvs/e0d8e2d1c3cc3e1315a92d099c3e7d5a3e928ae6/5985ed09b49a653d6596b0e14d134c5456cf1a9f/pex", line 237, in <module>
    exec(ast, globals_map, locals_map)
  File "-c <cmd>", line 1, in <module>
    #!/home/jsirois/.pex/venvs/s/bbcea5a4/venv/bin/python3.10 -sE
  File "/home/jsirois/.pex/venvs/e0d8e2d1c3cc3e1315a92d099c3e7d5a3e928ae6/5985ed09b49a653d6596b0e14d134c5456cf1a9f/lib/python3.10/site-packages/aws_cdk/__init__.py", line 1282, in <module>
    from ._jsii import *
  File "/home/jsirois/.pex/venvs/e0d8e2d1c3cc3e1315a92d099c3e7d5a3e928ae6/5985ed09b49a653d6596b0e14d134c5456cf1a9f/lib/python3.10/site-packages/aws_cdk/_jsii/__init__.py", line 13, in <module>
    import aws_cdk.asset_awscli_v1._jsii
  File "/home/jsirois/.pex/venvs/e0d8e2d1c3cc3e1315a92d099c3e7d5a3e928ae6/5985ed09b49a653d6596b0e14d134c5456cf1a9f/lib/python3.10/site-packages/aws_cdk/asset_awscli_v1/_jsii/__init__.py", line 13, in <module>
    __jsii_assembly__ = jsii.JSIIAssembly.load(
  File "/home/jsirois/.pex/venvs/e0d8e2d1c3cc3e1315a92d099c3e7d5a3e928ae6/5985ed09b49a653d6596b0e14d134c5456cf1a9f/lib/python3.10/site-packages/jsii/_runtime.py", line 54, in load
    _kernel.load(assembly.name, assembly.version, os.fspath(assembly_path))
  File "/home/jsirois/.pex/venvs/e0d8e2d1c3cc3e1315a92d099c3e7d5a3e928ae6/5985ed09b49a653d6596b0e14d134c5456cf1a9f/lib/python3.10/site-packages/jsii/_kernel/__init__.py", line 301, in load
    self.provider.load(LoadRequest(name=name, version=version, tarball=tarball))
  File "/home/jsirois/.pex/venvs/e0d8e2d1c3cc3e1315a92d099c3e7d5a3e928ae6/5985ed09b49a653d6596b0e14d134c5456cf1a9f/lib/python3.10/site-packages/jsii/_kernel/providers/process.py", line 352, in load
    return self._process.send(request, LoadResponse)
  File "/home/jsirois/.pex/venvs/e0d8e2d1c3cc3e1315a92d099c3e7d5a3e928ae6/5985ed09b49a653d6596b0e14d134c5456cf1a9f/lib/python3.10/site-packages/jsii/_utils.py", line 24, in wrapped
    stored.append(fgetter(self))
  File "/home/jsirois/.pex/venvs/e0d8e2d1c3cc3e1315a92d099c3e7d5a3e928ae6/5985ed09b49a653d6596b0e14d134c5456cf1a9f/lib/python3.10/site-packages/jsii/_kernel/providers/process.py", line 347, in _process
    process.start()
  File "/home/jsirois/.pex/venvs/e0d8e2d1c3cc3e1315a92d099c3e7d5a3e928ae6/5985ed09b49a653d6596b0e14d134c5456cf1a9f/lib/python3.10/site-packages/jsii/_kernel/providers/process.py", line 260, in start
    self._process = subprocess.Popen(
  File "/usr/lib/python3.10/subprocess.py", line 969, in __init__
    self._execute_child(args, executable, preexec_fn, close_fds,
  File "/usr/lib/python3.10/subprocess.py", line 1845, in _execute_child
    raise child_exception_type(errno_num, err_msg, err_filename)
FileNotFoundError: [Errno 2] No such file or directory: 'node'
Exception ignored in: <function _NodeProcess.__del__ at 0x7f167692f1c0>
Traceback (most recent call last):
  File "/home/jsirois/.pex/venvs/e0d8e2d1c3cc3e1315a92d099c3e7d5a3e928ae6/5985ed09b49a653d6596b0e14d134c5456cf1a9f/lib/python3.10/site-packages/jsii/_kernel/providers/process.py", line 233, in __del__
    self.stop()
  File "/home/jsirois/.pex/venvs/e0d8e2d1c3cc3e1315a92d099c3e7d5a3e928ae6/5985ed09b49a653d6596b0e14d134c5456cf1a9f/lib/python3.10/site-packages/jsii/_kernel/providers/process.py", line 291, in stop
    assert self._process.stdin is not None
AttributeError: '_NodeProcess' object has no attribute '_process'

hello - apologies for not including everything in my OP, I didn’t want to overburden assumed user error 🙂

No, so you are building a PEX and running it, so the work above is needed. You need: 1. Pants 2.15+ 2. On your

pex_binary

target,

execution_mode="venv"

3. On your `pex_binary`target,

venv_site_packages_copies=True

(Only available in Pants 2.15.x +)

success! Running the pex now behaves as running the entrypoint in a venv. The next challenge is figuring out if that’s actually useful 🙂 I’ll read through those links to build my understanding better - appreciate the detailed help, both

Ok, great. Don't be afraid of spamming details. In async remote debugging, too many is always better than too few.

hugely impressed with the magic of pex just emulating a python file in a wildly alien context - my naive first attempt seems to just work:

cdk ls --app ../../dist/projects.infrastructure/cdk_infra.pex

seems functionally equivalent to

cdk ls --app app.py

- I have a little more work to do but I think Pants can fill a gaping hole in developer experience, at least in Python-land, for AWS-native apps

Ok, cool. I'd expect road bumps still - this whole jsii thing is wild, but who knows.

who doesn’t love a typescript error from a python framework so you can hit a third party api and hope it works 🙂

Is it possible to use python_lambdas as a dependency. I do not have access to the generated zip. Is it possible to run python_source with steps 2 and 3?

I’m not sure I follow “I do not have access to the generated zip.” - they’re created in your /dist/ folder as default and can be targeted by cdk like any other zip?

Are they not only in the dist folder if I build them with

./pants package ::

? Currently I deploy as you do , but from a shell script. First I run

package

and move the zips to a folder with docker files. Then I run the cdk command, that deploys docker lambdas. This whole process does not feel quite right to me. But it works for now.

ah, yeah - I think that’s quite a specific implementation detail of cdk.Asset() though. If you can run DinD, you could use Pants to publish docker images and consume them in CDK instead which breaks the dissonance caused by CDK magic

@chilly-holiday-77415 Thanks. I will give it a shot.

I’m not smart enough to figure out how to write a Kaniko plugin yet but I have it in back of mind as something to look in to that would help us use Pants what I sense would be a lot more idiomatic way, but no DinD for our CI is a constraint