https://pantsbuild.org/ logo
#general
Title
# general
b

bumpy-noon-80834

10/06/2022, 10:21 AM
Did someone find a way to run Pulumi with Pants? I tried to manually set PYTHONPATH to the exported virtualenv to satisfy the "pulumi" command but it still fails. With the same trick, I tried to
import pulumi
, and it fails with:
ImportError: cannot import name 'cygrpc' from 'grpc._cython'
. However, I can
import pulumi
without any issue when I provision a shell with
./pants repl ::
. In this case, cygrpc is imported from
~/.cache/pants/named_caches/pex_root/installed_wheels/b821403907e865e8377af3eee62f0cb233ea2369ba0fcdce9505ca5bfaf4eeb3/grpcio-1.47.0-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl/grpc/_cython/cygrpc.cpython-39-x86_64-linux-gnu.so
I'll try to write a standalone Pulumi program and run it as a PEX. But that's sub-optimal as I'd loose a lot of Pulumi's tooling. Any suggestion?
Here is a working solution.
Copy code
$ tree -I __pycache__
.
├── Pulumi.develop.yaml
├── Pulumi.yaml
└── src
    └── infrastructure
        ├── BUILD
        └── __main__.py
$ cat Pulumi.yaml 
name: infrastructure
main: src/infrastructure
runtime:
  name: python
  options:
    virtualenv: ../../dist/export/python/virtualenvs/python-default/3.9.14
$ cat src/infrastructure/BUILD 
python_sources()

python_requirement(
    name = "pulumi",
    requirements = ["pulumi>=3.0.0,<4.0.0"],
)

python_requirement(
    name = "pulumi-aws",
    requirements = ["pulumi-aws>=5.0.0,<6.0.0"],
)
$ cd ../.. && ./pants export :: && cd -
$ pulumi up
Said another way: 1. Declare Pulumi as a dependency 2. Export the virtualenv 3. Reference it in Pulumi.yaml
h

happy-kitchen-89482

10/06/2022, 3:02 PM
Well, I tried to manually set PYTHONPATH to the exported virtualenv
I wouldn’t expect this to work. A virtualenv needs to be activated, you can’t manually set PYTHONPATH to it.
But you shouldn’t have to use export at all
You can have a
pex_binary
with pulumi’s entry point as its entry point, did you try that?
b

bumpy-noon-80834

10/06/2022, 3:06 PM
The
pulumi
command is not (AFAIK) installable as a PyPI package. It's a go binary, installed somewhere else on my system. It then forks and load a Python interpreter using that venv. (Pulumi supports multiple languages)
h

happy-kitchen-89482

10/06/2022, 3:09 PM
Oh, gotcha
So, remind me, what are you trying to with Pants+Pulumi? Have Pants invoke it as a tool?
What is this exported venv doing?
b

bumpy-noon-80834

10/06/2022, 3:11 PM
I use Pants to manage the whole monorepo, which contains a few libraries, Lambda functions and Pulumi deployment code. Then I use Pulumi to actually deploy the Lambda functions and a bunch of other resources.
The exported venv is used by Pulumi to load the "pulumi program" (deployment code) dependencies and pulumi's own libraries (the "pulumi" command runs the "pulumi program", which can be written in go/typescript/python/etc and depends on Pulumi's libraries, written in the same language)
Here is a Pulumi program written in TypeScript: https://github.com/pmuller/website/blob/develop/packages/deployment/src/index.ts I am basically trying to replicate that monorepo architecture in Python. I'll share some code if you're interested once I'll have something working.
h

happy-kitchen-89482

10/06/2022, 3:41 PM
I see, complicated!
So pulumi is a native binary but it has Python bindings that need to be present at its runtime, alongside your code?
b

bumpy-noon-80834

10/06/2022, 3:42 PM
Exactly!
Complicated... Yes! But if you spend years fighting with Terraform to implement complex stuff, you usually appreciate how much easier Pulumi makes implementing the same logic in real programming languages. :)
h

happy-kitchen-89482

10/06/2022, 3:46 PM
So it sounds like the idiomatic way to do this would be a plugin that invokes the pulumi binary, perhaps
but if the export thing works that’s probably sufficient too
There will be a significant speedup to venv export coming in 2.15
b

bumpy-noon-80834

10/06/2022, 3:49 PM
A plugin would be nice actually. My current solution works, but it's pretty hackish: • Need to export the virtualenv • Need to configure Pulumi to use it (and includes the Python interpreter version, so it's not generic) • Then I invoke
PYTHONPATH=$(pwd)/src pulumi pre
to make the "pulumi program" available to the Python interprer spawned by Pulumi
And I need to re-export the virtualenv everytime I change a dependency
How hard is it to write such a plugin?
h

happy-kitchen-89482

10/06/2022, 7:38 PM
Hard to say without knowing more about pulumi. There are examples of plugins that use a native binary, like helm and terraform. But they don’t also create a venv. However there are many examples of creating a venv, so… One thing would be how to model this so we can in the future support pulumi code in other languages.
b

bumpy-noon-80834

10/06/2022, 7:40 PM
I'll read the terraform backends to get a better idea of it, thanks
f

faint-businessperson-86903

10/07/2022, 5:55 PM
@bumpy-noon-80834 My team has written our own plugin for this, which took a lot of trial and error to get it to work. The secret sauce was to set the
PULUMI_PYTHON_CMD
environment variable to the path of the python executable you want to use. This can be a pex without an entrypoint (run as an interpreter), or the interpreter of a virtual environment (e.g.
env/bin/python
).
🙏 1
Happy to share our plugin code with you but it’s a bit rough around the edges
pdl-pulumi-plugin.zip
b

bumpy-noon-80834

10/07/2022, 8:41 PM
Oh that's great, I'll dig into it, thank you @faint-businessperson-86903 😄
5 Views