Hey, we"re running Gitlab CI runners on eks in AWS. I'm now trying to deploy a helm deployment via experimental-deploy, but I keep receiving "connection refused". What is pants doing differently then executing helm directly in the pipeline? A kubectl command right before the pants deploy works fine.

Did you manage to debug anything with the insight about auth?

Unfortunately not. If I execute a "kubectl version" or even helm manually everything works fine, just when pants tries to execute helm it throws a connection refused.

$ pants --changed-dependees=transitive experimental-deploy apps/workflowprobe::
07:10:22.20 [INFO] Initializing scheduler...
07:10:23.51 [INFO] Scheduler initialized.
07:10:29.49 [INFO] Starting: Rendering Helm deployment apps/workflowprobe/deploy-eks/workflowprobe:sandbox
07:10:29.86 [INFO] Completed: Rendering Helm deployment apps/workflowprobe/deploy-eks/workflowprobe:sandbox
07:10:36.35 [INFO] Starting: Building helm_k8s_parser.pex from helm-k8s-parser_default.lock
07:11:15.36 [INFO] Completed: Building helm_k8s_parser.pex from helm-k8s-parser_default.lock
07:11:26.75 [INFO] Starting: Building helm_post_renderer.pex from helm-post-renderer_default.lock
07:11:29.86 [INFO] Completed: Building helm_post_renderer.pex from helm-post-renderer_default.lock
07:11:29.87 [INFO] Deploying targets...
WARNING: Kubernetes configuration file is group-readable. This is insecure. Location: /opt/app-root/.kube/config
WARNING: Kubernetes configuration file is world-readable. This is insecure. Location: /opt/app-root/.kube/config
Error: Kubernetes cluster unreachable: Get "<http://localhost:8080/version>": dial tcp [::1]:8080: connect: connection refused
helm.go:84: [debug] Get "<http://localhost:8080/version>": dial tcp [::1]:8080: connect: connection refused
Kubernetes cluster unreachable
<http://helm.sh/helm/v3/pkg/kube.(*Client).IsReachable|helm.sh/helm/v3/pkg/kube.(*Client).IsReachable>
	<http://helm.sh/helm/v3/pkg/kube/client.go:121|helm.sh/helm/v3/pkg/kube/client.go:121>
<http://helm.sh/helm/v3/pkg/action.(*History).Run|helm.sh/helm/v3/pkg/action.(*History).Run>
	<http://helm.sh/helm/v3/pkg/action/history.go:48|helm.sh/helm/v3/pkg/action/history.go:48>
main.newUpgradeCmd.func2
	<http://helm.sh/helm/v3/cmd/helm/upgrade.go:99|helm.sh/helm/v3/cmd/helm/upgrade.go:99>
<http://github.com/spf13/cobra.(*Command).execute|github.com/spf13/cobra.(*Command).execute>
	<http://github.com/spf13/cobra@v1.5.0/command.go:872|github.com/spf13/cobra@v1.5.0/command.go:872>
<http://github.com/spf13/cobra.(*Command).ExecuteC|github.com/spf13/cobra.(*Command).ExecuteC>
	<http://github.com/spf13/cobra@v1.5.0/command.go:990|github.com/spf13/cobra@v1.5.0/command.go:990>
<http://github.com/spf13/cobra.(*Command).Execute|github.com/spf13/cobra.(*Command).Execute>
	<http://github.com/spf13/cobra@v1.5.0/command.go:918|github.com/spf13/cobra@v1.5.0/command.go:918>
main.main
	<http://helm.sh/helm/v3/cmd/helm/helm.go:83|helm.sh/helm/v3/cmd/helm/helm.go:83>
runtime.main
	runtime/proc.go:250
runtime.goexit
	runtime/asm_amd64.s:1571
✕ apps/workflowprobe/deploy-eks/workflowprobe:sandbox failed

The mentioned kubeconfig in the code snippet is empty, which is needed by helm (if we deploy manually) to not complain and just fallback to the default configuration.

Ah, yeah. Pants executes processes in a temporary sandbox, meaning a temp dir and a limited set of environment variables (to reduce “works on my machine” syndrome) There’s various ways to mark env vars as blessed to pass them through to processes. For this particular case, it looks like the

[helm]

subsystem supports an

extra_env_vars

option that could be set in pants.toml to include those to pass them through from the ambient environment: https://www.pantsbuild.org/docs/reference-helm

I do this now:

[helm]
extra_env_vars.add = [
    "KUBERNETES_SERVICE_HOST",
    "KUBERNETES_SERVICE_PORT",
    "KUBERNETES_SERVICE_ACCOUNT_TOKEN",
    "KUBERNETES_SERVICE_ACCOUNT_CA_CERT_FILE"
]

and it works. Though it seems *_ACCOUNT_TOKEN and *_ACCOUNT_CA_CERT_FILE are not set, but might be in some cases. So for our gitlab ci pipeline *_SERVICE_HOST and *_SERVICE_PORT seem to do the trick