We're using docker environments for integration tests running on Github Workflows. This is pretty nice, because we can do things like set up the database and other systems the way the integration tests expects. However, we can never test anything relying on our cloud provider because we can't put the keys in the dockerfile (with

docker run -v

) because there's just no way to customize docker running for environments. This is somewhat related to https://github.com/pantsbuild/pants/issues/18577, but in this case, we need to customize the docker run args, not the environment variable passed to the docker run command. My question is: is my understanding of the limitations correct? Am I missing some other way to do what we want? And if my understanding is correct, and there is no other way, should I open up a feature request for this?

Just clarifying: it sounds like you want to mount file(s) that contain the secrets into the container, and then container's process read from those files. Is that correct? If so, I think you're correct that there's not currently support for it. Another option that comes to mind is providing credentials via environment variables, for the test executions themselves. Those env vars will be propagated into the command run in the docker image (that is, each invocation will have the env vars set individually, rather than being pervasive within the container, as if the command was run like

docker exec --env ... some-container /path/to/the/command ...

or similar).

Yes, this is correct, we want to mount the credentials. It might also be possible to do this via environment variables, but the normal value of

GOOGLE_APPLICATION_CREDENTIALS

is expected to point to a credentials file, not the credentials themselves, so it'd be a bit hacky to take in another environment variable, write it to the file system, then point to that with

GOOGLE_APPLICATION_CREDENTIALS

.

So for docker images (not environments) what we do is use docker-compose + the pytest integration and do something like:

services:
  cl_hello_world:
    image: foo/bar-dagster-hello-world:localdev
    container_name: cl_hello_world
    restart: always
    environment:
      DAGSTER_POSTGRES_USER: "postgres"

which injects env vars. And then for integration tests that need cloud credentials we do something like:

@pytest.fixture(scope="session", autouse=True)
def aws_credentials():
    with my_corp_test_helpers.aws_credentials(
        "arn:aws:iam::123234234:role/foo-telemetry-ingest-role"
    ):
        yield

However, I don't think we have a case where we inject aws creds into a docker container. If the compose+pyest pattern works for you I think you could do something like

- type: bind
        source: '${HOME}/.aws'
        target: '/root/.aws'
        consistency: 'cached'

Thanks, it seems like that would work if you have or could compute credentials locally (inside the docker environment), which might be possible somehow, but I haven't figured out how to do it yet. I also don't really get how to use docker compose + pytest. I didn't even think Pants supported docker compose, and running tests not in an environment but in a docker seems like something only possible outside of Pants as well.

docker compose + pytest.

We use https://pypi.org/project/pytest-docker/

Interesting, I'll check it out, thanks! And you run this with

pants test?

Yep.

We're not python, but using a similar strategy for spinning up docker compose for the duration of tests (via custom plugin). Iirc there's an issue for pre/post somewhere in the pants issues

Thanks - this did work to some extent (there was a fair bit of strangeness in how

pytest-docker

brought up and down the docker container, though). I was a bit confused at first, though - this doesn't run tests in a docker container like Pants does, it just sets up Docker containers for dependencies, which is useful, but also we're using Github workflows, which already do this. So I just switched to doing the equivalent thing in workflows, which worked. This solution, though, has the advantage of tests being more standardized between developer and Github workflows.

@refined-cat-61929 I have the exact same issue as you: I want to mount secrets to a volume using run_args for running tests in a docker_environment. This mirrors how secrets are made available in google cloud run in production. I think there's an issue here https://github.com/pantsbuild/pants/issues/18994 Were you able to find a way to mount the secrets into your test docker environment?

No, unfortunately it doesn't seem possible. I did get things working, but the basic idea was that instead of running tests from docker, we run test outside of Pant's docker environment and use docker composition for all of our external needs.

Got it, hopefully this feature gets implemented. I found another potential solution using a

conftest.py

file to write the secret files when setting up the tests. For dev I'm jsut generating dummy values but I think I could use google secrets manager client for integration tests.

I think you'll have trouble with that, since you can't inject the google auth secret to your tests, which you'll need to access the secret manager client.

Yeah, exactly. And sounds like we can't inject environment variables into a docker_environment either.

Right, I think that's my recollection as well.

So you can actually pass in environment variables to tests using the following:

[test]
extra_env_vars = [
  "key=%(env.VAR)s"
]

I guess you could either pass in all the keys this way or pass in the credentials for gcloud and then set up the secret files using the gcloud client.

I don't believe that works in this situation. Try it - you should see that in the pants environment, you don't get the value you set.

So I tried it and it did work for me. I set the environment variable from where I ran pants and then printed it from within my tests that were run within docker.

Interesting, thanks for the update on that. I don't know why we saw different things; but I'm just going by what I remember, which may not be 100% accurate

Yeah, another possibility is using a file() build target that'll get copied into the docker environment and being careful not to commit that file to the repository. You could for example use the google auth github action to create a credential file and then copy this into the environment with the build target.

Interesting, yes that could work - it generally isn't good form to build your secretes directly in the docker but in this case, maybe it's reasonable.

Yeah, I'm a little concerned using file() will keep the secrets around in a cache somewhere.