#20624 Inability to pass env vars to native engine breaks AWS authentication for assumed roles Issue created by serjx86 Describe the bug TL;DR There has to be a way to propagate env vars to native code. Either it exists and we did not figure out how to do it, or it needs to be added, otherwise native code is not implementing AWS authentication routine correctly (env vars should take precedence over other auth methods). * * * We were successfully using

resource

targets with

http_source

pointing to s3 URI. However, recently we were experimenting with permissions management on our CI and decided to use AWS IAM role chaining to avoid duplicating policies for different build environments that were running same builds. For the best of my knowledge, credentials for assumed role can be consumed either via env vars of updating AWS profile configs in user's home dir. On CI we naturally don't use local config profiles, so the only way to switch to assumed role is by making an API call to assume the role and setting the session credentials we get back in env vars, something to the effect of

export $(printf "AWS_ACCESS_KEY_ID=%s AWS_SECRET_ACCESS_KEY=%s AWS_SESSION_TOKEN=%s" \
$(aws sts assume-role \
--role-arn arn:aws:iam::123456789012:role/MyAssumedRole \
--role-session-name MySessionName \
--query "Credentials.[AccessKeyId,SecretAccessKey,SessionToken]" \
--output text))

(stolen form stackoverflow as an example) Our setup for CI env we were testing is as follows. EC2 instance has an IAM role profile A attached, the role A does not have s3 access. Profile role A is granted permission to assume role B. Role B has s3 access. In testing this setup we saw that after successfully assuming the role and setting env vars for role B, pants ignored the env vars and instead queried the metadate server, which gave it credentials for role A. (Which is perfectly normal, metadata server has no knowledge of locally assumed roles, as I understand it). Native engine is not the only pants component we rely on to access s3 URIs. We also have a plugin that relies on

backend.url_handlers.s3

. Looking at debug logs for

botocore

used by

backend.url_handlers.s3

we saw that env vars credentials were checked first, but then metadata server was still queried. Using this clue, we set

[subprocess-environment]
env_vars = [
    "AWS_ACCESS_KEY_ID",
    "AWS_SECRET_ACCESS_KEY",
    "AWS_SESSION_TOKEN",
]

and we saw that indeed, this fixed the issue for

backend.url_handlers.s3

and the credentials from env vars took precedence over metadata server issued credentials. However, we still could not get

resource

targets with

http_source

pointing to s3 URIs to work. Looking at the debug logs we noted that

AWS_...

variables were logged for for PEX processes that were used by our plugin, but these variables were not reported for calls used to resolve s3

resource

targets. We thus concluded that env vars are not propagated to the native code that resolves

http_source

. We are not sure if there is already a way to pass variable to the native code or if this is a bug. And we believe this is a bug (and not a feature request), because this makes native code's implementation of AWS authentication incorrect: credentials in env vars, if present, should take precedence over metadata server issued credentials. Pants version 2.19.0rc5 OS Linux (Ubuntu Focal) Additional info Not sure how to reproduce, since the problem here is something is not happening. The fix, however, should be easy to observe: an env var specifically set/forwarded to native code should be logged in debug logs as passed to the process, even if that env var is not used for anything in the code itself. pantsbuild/pants