https://pantsbuild.org/ logo
#general
Title
# general
m

melodic-butcher-43231

02/10/2023, 12:38 PM
Hi everybody! 🙂 Thank you for creating pants, it looks very promising! I am trying to get the experimental terraform backend up and running, but it keeps complaining that init has not been run to download the required providers. Running init manually did not fix it for me, and I did not find anything helpful in docs or issues except for an old issue that claims to have solved that (or something similar). How should I proceed? Thanks a lot and cheers for all your efforts! 🙂
👋 1
e

enough-analyst-54434

02/10/2023, 2:48 PM
It looks like the plugin never calls "init" as you suggest:
Copy code
jsirois@Gill-Windows:~/dev/pantsbuild/jsirois-pants (main) $ git grep -A5 "TerraformProcess("
src/python/pants/backend/terraform/goals/check.py:            TerraformProcess(
src/python/pants/backend/terraform/goals/check.py-                args=("validate", directory),
src/python/pants/backend/terraform/goals/check.py-                input_digest=source_files.snapshot.digest,
src/python/pants/backend/terraform/goals/check.py-                output_files=tuple(files),
src/python/pants/backend/terraform/goals/check.py-                description=f"Run `terraform fmt` on {pluralize(len(files), 'file')}.",
src/python/pants/backend/terraform/goals/check.py-            ),
--
src/python/pants/backend/terraform/lint/tffmt/tffmt.py:        TerraformProcess(
src/python/pants/backend/terraform/lint/tffmt/tffmt.py-            args=("fmt", directory),
src/python/pants/backend/terraform/lint/tffmt/tffmt.py-            input_digest=request.snapshot.digest,
src/python/pants/backend/terraform/lint/tffmt/tffmt.py-            output_files=request.files,
src/python/pants/backend/terraform/lint/tffmt/tffmt.py-            description=f"Run `terraform fmt` on {pluralize(len(request.files), 'file')}.",
src/python/pants/backend/terraform/lint/tffmt/tffmt.py-        ),
So you'd want to start by filing an issue with your Pants version, full command line and output. If you're interested in contributing a fix we can use that ticket as the rally point and you can ask questions in #development / read up starting here: https://www.pantsbuild.org/docs/plugins-overview
except for an old issue that claims to have solved that (or something similar).
What issue is that?
h

happy-kitchen-89482

02/10/2023, 5:08 PM
@fast-nail-55400 thoughts here?
m

melodic-butcher-43231

02/10/2023, 6:35 PM
Thank you for your quick response! 😄 I may be embarrassingly wrong, but https://github.com/pantsbuild/pants/pull/12948 sounds as if there had been a problem with missing inits as well?
e

enough-analyst-54434

02/10/2023, 6:46 PM
That fixed a test of all this to use .tf files that don't require init! So it worked around the problem you see.
If you look at the diff you'll see
resource { ... }
blocks are removed from the .tf to avoid the need for init.
So, closer to embarrassing on our side. I'm not sure how this can work with any real-world terraform use at all.
I think the intent was just to plant a seed for terraform support as is the case with early experimental contribs. This one just never had anyone's time to circle back and continue the work.
m

melodic-butcher-43231

02/10/2023, 6:59 PM
Ah, alright then! I've had to do quick fixes like that. Not embarrassing - "Real Life" (TM). 😉 So I see mainly two, maybe three tasks I would like to be able to perform: terraform validate (which I agree fits perfectly under "check") terraform plan / apply (would that fit into run, package or something else entirely?) terraform fmt works just fine and demonstrates nicely why I would want to use pants over poetry and terraform separately. 🙂 How would the "init" - especially "init -upgrade" - fit into pants?
f

fast-nail-55400

02/10/2023, 9:24 PM
I think the intent was just to plant a seed for terraform support as is the case with early experimental contribs. This one just never had anyone's time to circle back and continue the work.
Indeed, that is the case.
The terraform backend only fits the Pants model for
fmt
.
plan
and
apply
are both inherently stateful and don't really fit with the "hermetic build" model of Pants.
And
init
is also stateful since technically it can and does update the lockfiles used by Terraform (
.terraform.lock.hcl
)
And Pants has not been taught how to deal with those.
And
validate
wants access to the
.terraform.lock.hcl
and so it must be present and so the Pants Terraform backend would need to be taught how to copy that into the execution sandbox.
There is less value for Pants for things like
terraform init
if it just becomes a custom Pants goal, maybe like
./pants tf-init ::
At that point, you could just run
terraform init
directly
for context, I had proposed a
tf-run
goal in https://github.com/pantsbuild/pants/pull/13023 back in 2021, but there did not seem to be much appetite to include it in Pants. That PR actually knew how to deal with the
.terraform
directory and
.terraform.lock.hcl
file.
at this point, the Terraform backend is essentially unmaintained. Not sure how we want to deal with it, but we should probably consider deprecating and then removing it.
m

melodic-butcher-43231

02/10/2023, 10:12 PM
Hmmm, that would be a pity, as uniting the actual code and the infrastructure-as-code to provision and run it on seem like a major use case for a monorepo to me. Especially with the rise of Microservices, coordinating both is one of the main arguments in favour of them, and leaving that whole feature field to others sounds... sad. 😕 OTOH, this sounds like a fundamental question on the use and scope of pants, which will likely need some more thinking about.
h

happy-kitchen-89482

02/10/2023, 10:18 PM
That would be an interesting direction to take this plugin in, but it would require some work.
It would be quite something if
./pants deploy path/to/binary
would not just package the thing but actually provision infrastructure for it and deploy it to that infrastructure
but there are a lot of missing pieces before that could happen
m

melodic-butcher-43231

02/10/2023, 10:20 PM
Okay, just read up on that old PR. Maybe Terraform really isn't in scope for pants. I will try to gather some more experience with pants & python then and see what opinions will form from that. 🙂
After all, pants is not aiming to replace GitHub Actions, is it? 🙃
And i could imagine goals "provision" and "deploy", but that might easily turn into mere runners for Terraform - which is already an abstraction layer over the actual provisioning, and does not really touch the deployment, either. Would that be good or bad?
h

happy-kitchen-89482

02/10/2023, 10:49 PM
The interesting aspect here would be if it's possible to chain the infra to the binary/image. Today you'd use Pants to build a deployable binary and embed it in a Docker image. Then your infra would pick up that image from a tag.
It also depends on how closely connected the binary is to the infra that it runs on
For example, internally we use Kubernetes, which is another layer between the two. The terraform infra describes the k8s cluster, but not what runs on it.
So, this would require a lot of Product Management work and market research to get right
m

melodic-butcher-43231

02/12/2023, 5:41 PM
I think to me, the situation looks much like between a program/app and a library it uses, with the code being in the role of the program and the infra in the role of the library. You may change the lib underneath the app or the app on top of the lib, but the true added value happens whenever you need coordinated changes. So I would ask, how much of that already handled relationship could be harnessed to the code/infra scenario? And what in that has not been covered yet?
Also, where we have code and infra, especially the three cases -Terraform, Docker, K8s - you named, what is comparable between the three, and what is analogy-breakingly different?
e

enough-analyst-54434

02/12/2023, 6:07 PM
Taking a step back; so it's clear - Pants does support interacting with the outside world (somewhat obviously) - but, at its core, it models execution as side-effect free functions. It is challenging to deal with external systems sometimes as a result. We have a built-in filesystem watcher that , in combination with process sandboxing, allows us to get control over your file edits. If the external system is truly external, like a remote service - this takes alot more care. That service may or may not be idempotent, etc. So this sort of consideration is what is in play that generally makes integrating these sorts of external system take longer and require more careful deliberation thatn adding a new linter or language vertical to Pants support.
m

melodic-butcher-43231

02/13/2023, 12:40 PM
Thank you John, I will ruminate on that. 🙂
And there currently is no "proper" backend to deploy to MS Azure, as there is for AWS and Google Cloud, correct? So... would the shell script backend (although experimental) work to rig up something for my own use (I am using a shell script to run the azure CLI for actual deployment), or are there currently still caveats, as with the terraform backend?
And yes, that very much presumes that the Azure CLI will be fine running in parallel to build and push code from distinct folders to distinct functions. I'll just trust my luck for this for now. 😉
e

enough-analyst-54434

02/13/2023, 4:12 PM
And there currently is no "proper" backend to deploy to MS Azure, as there is for AWS and Google Cloud, correct?
I am not aware of any backend that deploys to anywhere. We have support for packaging AWSLambda and Google Cloud Function zip files, but you have to deploy those yourself.
h

happy-kitchen-89482

02/13/2023, 4:54 PM
There is a
./pants experimental-deploy
goal that (AFAIK) currently publishes dists to PyPI (by running Twine), and pushes Docker images and Helm charts to registries. But these are relatively straightforward since they require very little extra metadata to know where to deploy to, and they're just pushing files to, essentially, storage. I think doing AWS and GCP might also be fairly straightforward. Terraform seems like a much bigger kettle of fish...
m

melodic-butcher-43231

02/14/2023, 10:20 AM
Great, thank you both for clarifying! 😄
This is less about a feature wish than about something I need for my current Pants use case, so I will look into that! 🙂
4 Views