https://pantsbuild.org/ logo
Join Slack
Powered by
Did someone find a way to run Pulumi with Pants? ...
# general
b
Did someone find a way to run Pulumi with Pants? I tried to manually set PYTHONPATH to the exported virtualenv to satisfy the "pulumi" command but it still fails. With the same trick, I tried to 
import pulumi
, and it fails with: 
ImportError: cannot import name 'cygrpc' from 'grpc._cython'
. However, I can 
import pulumi
without any issue when I provision a shell with 
./pants repl ::
. In this case, cygrpc is imported from 
~/.cache/pants/named_caches/pex_root/installed_wheels/b821403907e865e8377af3eee62f0cb233ea2369ba0fcdce9505ca5bfaf4eeb3/grpcio-1.47.0-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl/grpc/_cython/cygrpc.cpython-39-x86_64-linux-gnu.so
I'll try to write a standalone Pulumi program and run it as a PEX. But that's sub-optimal as I'd loose a lot of Pulumi's tooling. Any suggestion?
Here is a working solution.
Copy code
$ tree -I __pycache__
.
├── Pulumi.develop.yaml
├── Pulumi.yaml
└── src
    └── infrastructure
        ├── BUILD
        └── __main__.py
$ cat Pulumi.yaml 
name: infrastructure
main: src/infrastructure
runtime:
  name: python
  options:
    virtualenv: ../../dist/export/python/virtualenvs/python-default/3.9.14
$ cat src/infrastructure/BUILD 
python_sources()

python_requirement(
    name = "pulumi",
    requirements = ["pulumi>=3.0.0,<4.0.0"],
)

python_requirement(
    name = "pulumi-aws",
    requirements = ["pulumi-aws>=5.0.0,<6.0.0"],
)
$ cd ../.. && ./pants export :: && cd -
$ pulumi up
Said another way: 1. Declare Pulumi as a dependency 2. Export the virtualenv 3. Reference it in Pulumi.yaml
h
Well, I tried to manually set PYTHONPATH to the exported virtualenv
I wouldn’t expect this to work. A virtualenv needs to be activated, you can’t manually set PYTHONPATH to it.
But you shouldn’t have to use export at all
You can have a 
pex_binary
with pulumi’s entry point as its entry point, did you try that?
b
The 
pulumi
command is not (AFAIK) installable as a PyPI package. It's a go binary, installed somewhere else on my system. It then forks and load a Python interpreter using that venv. (Pulumi supports multiple languages)
h
Oh, gotcha
So, remind me, what are you trying to with Pants+Pulumi? Have Pants invoke it as a tool?
What is this exported venv doing?
b
I use Pants to manage the whole monorepo, which contains a few libraries, Lambda functions and Pulumi deployment code. Then I use Pulumi to actually deploy the Lambda functions and a bunch of other resources.
The exported venv is used by Pulumi to load the "pulumi program" (deployment code) dependencies and pulumi's own libraries (the "pulumi" command runs the "pulumi program", which can be written in go/typescript/python/etc and depends on Pulumi's libraries, written in the same language)
Here is a Pulumi program written in TypeScript: https://github.com/pmuller/website/blob/develop/packages/deployment/src/index.ts I am basically trying to replicate that monorepo architecture in Python. I'll share some code if you're interested once I'll have something working.
h
I see, complicated!
So pulumi is a native binary but it has Python bindings that need to be present at its runtime, alongside your code?
b
Exactly!
Complicated... Yes! But if you spend years fighting with Terraform to implement complex stuff, you usually appreciate how much easier Pulumi makes implementing the same logic in real programming languages. :)
h
So it sounds like the idiomatic way to do this would be a plugin that invokes the pulumi binary, perhaps
but if the export thing works that’s probably sufficient too
There will be a significant speedup to venv export coming in 2.15
b
A plugin would be nice actually. My current solution works, but it's pretty hackish: • Need to export the virtualenv • Need to configure Pulumi to use it (and includes the Python interpreter version, so it's not generic) • Then I invoke 
PYTHONPATH=$(pwd)/src pulumi pre
to make the "pulumi program" available to the Python interprer spawned by Pulumi
And I need to re-export the virtualenv everytime I change a dependency
How hard is it to write such a plugin?
h
Hard to say without knowing more about pulumi. There are examples of plugins that use a native binary, like helm and terraform. But they don’t also create a venv. However there are many examples of creating a venv, so… One thing would be how to model this so we can in the future support pulumi code in other languages.
b
I'll read the terraform backends to get a better idea of it, thanks
f
@bumpy-noon-80834 My team has written our own plugin for this, which took a lot of trial and error to get it to work. The secret sauce was to set the 
PULUMI_PYTHON_CMD
environment variable to the path of the python executable you want to use. This can be a pex without an entrypoint (run as an interpreter), or the interpreter of a virtual environment (e.g. 
env/bin/python
).
🙏 1
Happy to share our plugin code with you but it’s a bit rough around the edges
pdl-pulumi-plugin.zip
pdl-pulumi-plugin.zip
b
Oh that's great, I'll dig into it, thank you @faint-businessperson-86903 😄
5 Views
Open in Slack
PreviousNext
Powered by