https://pantsbuild.org/ logo
Join Slack
Powered by
Hi! :wave: I'm having trouble understanding why my...
# general
b
Hi! šŸ‘‹ I'm having trouble understanding why my environment variables aren't being propagated to tests running in a Docker environment. I have a small example test which expects "allow-listed" environment variables from the parent pants process. The test passes with a local environment, but not a docker environment. Would somebody mind explaining how I can propagate these? https://github.com/ConorStevenson/pants-example-docker-env
šŸ‘‹ 1
c
ah, disregard the docker part, the docker env vars are for when you build/run docker images. You are running the 
test
goal, so it is these you want: https://www.pantsbuild.org/docs/reference-test#extra_env_vars
b
Thanks, I was setting them everywhere I could find šŸ˜„. This works:
Copy code
export WHITELISTED=VALUE
pants test --test-extra-env-vars="['WHITELISTED=$WHITELISTED']" ::
But I was hoping to have the env vars propagated from the parent environment like this, which doesn't work:
Copy code
pants test --test-extra-env-vars="['WHITELISTED']" ::
Entries are strings in the form 
ENV_VAR=value
to use explicitly; or just 
ENV_VAR
to copy the value of a variable in Pants's own environment.
h
I do not reproduce this
it works for me
f
Doesn't work for me too as it seems:
Copy code
WHITELISTED=VALUE pants test --test-extra-env-vars="['WHITELISTED=$WHITELISTED']" ::
10:00:48.40 [ERROR] Completed: Run Pytest - (environment:local_docker, src/test_docker_env.py:tests) - failed (exit code 1).
============================= test session starts ==============================
platform linux -- Python 3.8.17, pytest-7.0.1, pluggy-1.0.0
rootdir: /pants-sandbox/pants-sandbox-zctlSX
plugins: forked-1.6.0, cov-3.0.0, xdist-2.5.0
collected 1 item

src/test_docker_env.py F                                                 [100%]

=================================== FAILURES ===================================
_______________________ test_env_is_propagated_to_tests ________________________

    def test_env_is_propagated_to_tests():
        assert os.environ["HARDCODED"] == "VALUE"
        assert "NOT_WHITELISTED" not in os.environ
>       assert os.environ["WHITELISTED"] == "VALUE"
E       AssertionError: assert '' == 'VALUE'
E         - VALUE

src/test_docker_env.py:7: AssertionError
- generated xml file: /pants-sandbox/pants-sandbox-zctlSX/src.test_docker_env.py.tests.xml -
=========================== short test summary info ============================
FAILED src/test_docker_env.py::test_env_is_propagated_to_tests - AssertionErr...
============================== 1 failed in 0.06s ===============================



āœ• src/test_docker_env.py:tests failed in 0.28s (ran in docker environment `local_docker`).
and if I try just the key I get
Copy code
WHITELISTED=VALUE pants test --test-extra-env-vars="['WHITELISTED']" ::
10:02:12.27 [ERROR] Completed: Run Pytest - (environment:local_docker, src/test_docker_env.py:tests) - failed (exit code 1).
============================= test session starts ==============================
platform linux -- Python 3.8.17, pytest-7.0.1, pluggy-1.0.0
rootdir: /pants-sandbox/pants-sandbox-BHydas
plugins: forked-1.6.0, cov-3.0.0, xdist-2.5.0
collected 1 item

src/test_docker_env.py F                                                 [100%]

=================================== FAILURES ===================================
_______________________ test_env_is_propagated_to_tests ________________________

    def test_env_is_propagated_to_tests():
        assert os.environ["HARDCODED"] == "VALUE"
        assert "NOT_WHITELISTED" not in os.environ
>       assert os.environ["WHITELISTED"] == "VALUE"

src/test_docker_env.py:7:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

self = environ({'HOSTNAME': 'c330ad1fe9bc', 'PYTHON_VERSION': '3.8.17', 'PWD': '/pants-sandbox/pants-sandbox-BHydas', 'PYTHON...PEX_EXTRA_SYS_PATH__': 'src', 'PYTEST_CURRENT_TEST': 'src/test_docker_env.py::test_env_is_propagated_to_tests (call)'})
key = 'WHITELISTED'

    def __getitem__(self, key):
        try:
            value = self._data[self.encodekey(key)]
        except KeyError:
            # raise KeyError with the original key value
>           raise KeyError(key) from None
E           KeyError: 'WHITELISTED'

/usr/local/lib/python3.8/os.py:675: KeyError
- generated xml file: /pants-sandbox/pants-sandbox-BHydas/src.test_docker_env.py.tests.xml -
=========================== short test summary info ============================
FAILED src/test_docker_env.py::test_env_is_propagated_to_tests - KeyError: 'W...
============================== 1 failed in 0.06s ===============================



āœ• src/test_docker_env.py:tests failed in 0.24s (ran in docker environment `local_docker`).
c
I donā€™t repro either, so this feels like a configuration issue. Could you please run
Copy code
WHITELISTED=VALUE pants test --test-extra-env-vars="['WHITELISTED']" --help
and look for the current value of the extra_env_vars.. something like this:
Copy code
--test-extra-env-vars="['<str>', '<str>', ...]"
  PANTS_TEST_EXTRA_ENV_VARS
  extra_env_vars
      default: []
      current value: [
          ...,
          "FOO=baz"
      ] (from .pants.rc, pants.toml, from command-line flag)
      Additional environment variables to include in test processes. Entries are strings in the form `ENV_VAR=value` to use explicitly; or just `ENV_VAR` to copy the value of a variable in Pants's own environment.

      Can be overriden by field `test_extra_env_vars` on `local_environment`, `docker_environment`, or `remote_environment` targets.
f
Copy code
--test-extra-env-vars="['<str>', '<str>', ...]"
  PANTS_TEST_EXTRA_ENV_VARS
  extra_env_vars
      default: []
      current value: [
          "WHITELISTED"
      ] (from command-line flag)
for me. I've copy and pasted your command above.
c
hmm.. interesting.
Copy code
āÆ FOO=baz pants test --extra-env-vars=FOO .. -- -k test_env
[...]
=================================== FAILURES ===================================
___________________________________ test_env ___________________________________

    def test_env():
>       assert os.environ.get("FOO", "<not set>") == "BAR"
E       AssertionError: assert 'baz' == 'BAR'
E         - BAR
E         + baz
also tried with your syntax (slightly changed, as Iā€™m trying this in a bigger project, so canā€™t replace the test envs entirely..)
Copy code
āÆ FOO=baz pants test --extra-env-vars="+['FOO']"
still works for me.
f
Copy code
WHITELISTED=VALUE pants test --test-extra-env-vars=WHITELISTED ::
16:47:42.17 [ERROR] Completed: Run Pytest - (environment:local_docker, src/test_docker_env.py:tests) - failed (exit code 1).
============================= test session starts ==============================
platform linux -- Python 3.8.17, pytest-7.0.1, pluggy-1.0.0
rootdir: /pants-sandbox/pants-sandbox-P2YAzC
plugins: forked-1.6.0, cov-3.0.0, xdist-2.5.0
collected 1 item

src/test_docker_env.py F                                                 [100%]

=================================== FAILURES ===================================
_______________________ test_env_is_propagated_to_tests ________________________

    def test_env_is_propagated_to_tests():
        assert os.environ["HARDCODED"] == "VALUE"
        assert "NOT_WHITELISTED" not in os.environ
>       assert os.environ["WHITELISTED"] == "VALUE"

src/test_docker_env.py:7:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

self = environ({'HOSTNAME': 'c330ad1fe9bc', 'PYTHON_VERSION': '3.8.17', 'PWD': '/pants-sandbox/pants-sandbox-P2YAzC', 'PYTHON...PEX_EXTRA_SYS_PATH__': 'src', 'PYTEST_CURRENT_TEST': 'src/test_docker_env.py::test_env_is_propagated_to_tests (call)'})
key = 'WHITELISTED'

    def __getitem__(self, key):
        try:
            value = self._data[self.encodekey(key)]
        except KeyError:
            # raise KeyError with the original key value
>           raise KeyError(key) from None
E           KeyError: 'WHITELISTED'

/usr/local/lib/python3.8/os.py:675: KeyError
- generated xml file: /pants-sandbox/pants-sandbox-P2YAzC/src.test_docker_env.py.tests.xml -
=========================== short test summary info ============================
FAILED src/test_docker_env.py::test_env_is_propagated_to_tests - KeyError: 'W...
============================== 1 failed in 0.05s ===============================



āœ• src/test_docker_env.py:tests failed in 0.26s (ran in docker environment `local_docker`).
c
this is key: `ran in docker environment `local_docker``
f
ah ok, so it's in the project then? I was thinking you try the example repo, too
c
sorry, I missed the context here
f
ok, it's macOS with docker desktop
c
heh šŸ˜›
hang onā€¦.
h
Ah, I was testing on native macos
f
so without docker?
FYI I just hooked in here out of curiousity
c
good you did, as both I and Benjy both overlooked the docker_env part of the issue šŸ˜®
f
šŸŽ‰
b
ah, that was very important context and I should have highlighted it šŸ˜„ I'm on mac using docker desktop
c
ah, I donā€™t have docker desktop, and it doesnā€™t seem like colima works with pantsā€¦
but I get the feeling that pants is not properly propagating all the environment variables properly
f
docker just consumes the environment from which it is getting started, optionally supports .env files
also optionally additional env vars can be providede via args
c
if you run with 
-ldebug
it would be interesting to see the commands invoked, and possibly if thereā€™s any sandboxes to look into.
f
Copy code
17:08:14.24 [INFO] Initialization options changed: reinitializing scheduler...
17:08:17.67 [INFO] Scheduler initialized.
17:08:20.11 [ERROR] Completed: Run Pytest - (environment:local_docker, src/test_docker_env.py:tests) - failed (exit code 1).
============================= test session starts ==============================
platform linux -- Python 3.8.17, pytest-7.0.1, pluggy-1.0.0
rootdir: /pants-sandbox/pants-sandbox-67IjmE
plugins: forked-1.6.0, cov-3.0.0, xdist-2.5.0
collected 1 item

src/test_docker_env.py F                                                 [100%]

=================================== FAILURES ===================================
_______________________ test_env_is_propagated_to_tests ________________________

    def test_env_is_propagated_to_tests():
        assert os.environ["HARDCODED"] == "VALUE"
        assert "NOT_WHITELISTED" not in os.environ
>       assert os.environ["WHITELISTED"] == "VALUE"

src/test_docker_env.py:7:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

self = environ({'HOSTNAME': '5881b149961d', 'PYTHON_VERSION': '3.8.17', 'PWD': '/pants-sandbox/pants-sandbox-67IjmE', 'PYTHON...PEX_EXTRA_SYS_PATH__': 'src', 'PYTEST_CURRENT_TEST': 'src/test_docker_env.py::test_env_is_propagated_to_tests (call)'})
key = 'WHITELISTED'

    def __getitem__(self, key):
        try:
            value = self._data[self.encodekey(key)]
        except KeyError:
            # raise KeyError with the original key value
>           raise KeyError(key) from None
E           KeyError: 'WHITELISTED'

/usr/local/lib/python3.8/os.py:675: KeyError
- generated xml file: /pants-sandbox/pants-sandbox-67IjmE/src.test_docker_env.py.tests.xml -
=========================== short test summary info ============================
FAILED src/test_docker_env.py::test_env_is_propagated_to_tests - KeyError: 'W...
============================== 1 failed in 0.07s ===============================



āœ• src/test_docker_env.py:tests failed in 0.45s (ran in docker environment `local_docker`).
ah forgot the keep sandbox
c
pants -ldebug test ā€¦
f
Copy code
WHITELISTED=VALUE pants -ldebug test --test-extra-env-vars=WHITELISTED ::
17:10:46.96 [ERROR] Completed: Run Pytest - (environment:local_docker, src/test_docker_env.py:tests) - failed (exit code 1).
============================= test session starts ==============================
platform linux -- Python 3.8.17, pytest-7.0.1, pluggy-1.0.0
rootdir: /pants-sandbox/pants-sandbox-BDxIIJ
plugins: forked-1.6.0, cov-3.0.0, xdist-2.5.0
collected 1 item

src/test_docker_env.py F                                                 [100%]

=================================== FAILURES ===================================
_______________________ test_env_is_propagated_to_tests ________________________

    def test_env_is_propagated_to_tests():
        assert os.environ["HARDCODED"] == "VALUE"
        assert "NOT_WHITELISTED" not in os.environ
>       assert os.environ["WHITELISTED"] == "VALUE"

src/test_docker_env.py:7:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

self = environ({'HOSTNAME': '5881b149961d', 'PYTHON_VERSION': '3.8.17', 'PWD': '/pants-sandbox/pants-sandbox-BDxIIJ', 'PYTHON...PEX_EXTRA_SYS_PATH__': 'src', 'PYTEST_CURRENT_TEST': 'src/test_docker_env.py::test_env_is_propagated_to_tests (call)'})
key = 'WHITELISTED'

    def __getitem__(self, key):
        try:
            value = self._data[self.encodekey(key)]
        except KeyError:
            # raise KeyError with the original key value
>           raise KeyError(key) from None
E           KeyError: 'WHITELISTED'

/usr/local/lib/python3.8/os.py:675: KeyError
- generated xml file: /pants-sandbox/pants-sandbox-BDxIIJ/src.test_docker_env.py.tests.xml -
=========================== short test summary info ============================
FAILED src/test_docker_env.py::test_env_is_propagated_to_tests - KeyError: 'W...
============================== 1 failed in 0.06s ===============================



āœ• src/test_docker_env.py:tests failed in 0.26s (ran in docker environment `local_docker`).
my bad
how can i reliably get the sandbox?
c
--keep-sandbox=always
I think it is
f
I guess that's a hint 
rootdir: /pants-sandbox/pants-sandbox-XJNMRw
- yes, already added the flag
c
keep-sandboxes
f
but from here I messed it up last time
c
that root dir would be in the container, and less interesting, Iā€™m interested in what happens going into the container
f
Copy code
l /private/var/folders/ll/98hnvg_n5l3by0224q_qybc40000gn/T/pants-sandbox-XJNMRw
Permissions Size User     Date Modified Name
drwxr-xr-x@    - elmcrest 13 Sep 17:11  .cache
.rw-r--r--@   51 elmcrest 13 Sep 17:11  .coveragerc
drwxr-xr-x@    - elmcrest 13 Sep 17:11  .pytest_cache
lrwxr-xr-x@   43 elmcrest 13 Sep 17:11  .python-build-standalone -> /pants-named-caches/python_build_standalone
.rwxr-xr-x@  368 elmcrest 13 Sep 17:11  __run.sh
drwxr-xr-x@    - elmcrest 13 Sep 17:11  extra-output
drwxr-xr-x@    - elmcrest 13 Sep 17:11  local_dists.pex
drwxr-xr-x@    - elmcrest 13 Sep 17:11  pytest.pex
drwxr-xr-x@    - elmcrest 13 Sep 17:11  pytest_runner.pex
.rwxr-xr-x@ 2.6k elmcrest 13 Sep 17:11  pytest_runner.pex_bin_python_shim.sh
.rwxr-xr-x@ 2.6k elmcrest 13 Sep 17:11  pytest_runner.pex_pex_shim.sh
drwxr-xr-x@    - elmcrest 13 Sep 17:11  requirements.pex
drwxr-xr-x@    - elmcrest 13 Sep 17:11  src
.rw-r--r--@ 1.3k elmcrest 13 Sep 17:11  src.test_docker_env.py.tests.xml
not sure if that's the correct sandbox
ok now, I am
XJNMRw
c
yea, thatā€™s the pytest sandbox, while there we can check the 
__run.sh
just to confirm the env
f
Copy code
cat __run.sh --style=plain
#!/usr/bin/env bash
# This command line should execute the same process as pants did internally.
cd /private/var/folders/ll/98hnvg_n5l3by0224q_qybc40000gn/T/pants-sandbox-XJNMRw
env -i HARDCODED=VALUE PEX_EXTRA_SYS_PATH=src ./pytest_runner.pex_pex_shim.sh $'--color=yes' $'--junit-xml=src.test_docker_env.py.tests.xml' -o $'junit_family=xunit2' src/test_docker_env.py
c
yep, expected
f
interesting
c
so we want to inspect the docker invocation.. but given how that is done, I suspect thereā€™s no sandbox for that, unfortunately
f
but HARDCODED=VALUE is coming from the project
c
yep
the issue is that docker doesnā€™t see the WHITELISTED env var..
thatā€™s present in the pants process
f
wouldn't I expect it to be present in __run.sh?
c
it should be, but I wasnā€™t expecting it currently based on the errors we see
so, what debug output from pants do you have between command invocation and the pytest output?
f
Copy code
WHITELISTED=VALUE pants -ldebug test --test-extra-env-vars=WHITELISTED :: --keep-sandboxes=always
17:20:06.03 [INFO] Preserving local process execution dir /private/var/folders/ll/98hnvg_n5l3by0224q_qybc40000gn/T/pants-sandbox-xbXhgg for Install Python for Pants usage
17:20:06.17 [INFO] Preserving local process execution dir /private/var/folders/ll/98hnvg_n5l3by0224q_qybc40000gn/T/pants-sandbox-rQOAbH for Run Pytest for src/test_docker_env.py:tests
17:20:06.41 [ERROR] Completed: Run Pytest - (environment:local_docker, src/test_docker_env.py:tests) - failed (exit code 1).
============================= test session starts ==============================
platform linux -- Python 3.8.17, pytest-7.0.1, pluggy-1.0.0
rootdir: /pants-sandbox/pants-sandbox-rQOAbH
plugins: forked-1.6.0, cov-3.0.0, xdist-2.5.0
collected 1 item

src/test_docker_env.py F                                                 [100%]

=================================== FAILURES ===================================
_______________________ test_env_is_propagated_to_tests ________________________

    def test_env_is_propagated_to_tests():
        assert os.environ["HARDCODED"] == "VALUE"
        assert "NOT_WHITELISTED" not in os.environ
>       assert os.environ["WHITELISTED"] == "VALUE"

src/test_docker_env.py:7:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

self = environ({'HOSTNAME': '7f5378804b38', 'PYTHON_VERSION': '3.8.17', 'PWD': '/pants-sandbox/pants-sandbox-rQOAbH', 'PYTHON...PEX_EXTRA_SYS_PATH__': 'src', 'PYTEST_CURRENT_TEST': 'src/test_docker_env.py::test_env_is_propagated_to_tests (call)'})
key = 'WHITELISTED'

    def __getitem__(self, key):
        try:
            value = self._data[self.encodekey(key)]
        except KeyError:
            # raise KeyError with the original key value
>           raise KeyError(key) from None
E           KeyError: 'WHITELISTED'

/usr/local/lib/python3.8/os.py:675: KeyError
- generated xml file: /pants-sandbox/pants-sandbox-rQOAbH/src.test_docker_env.py.tests.xml -
=========================== short test summary info ============================
FAILED src/test_docker_env.py::test_env_is_propagated_to_tests - KeyError: 'W...
============================== 1 failed in 0.06s ===============================



āœ• src/test_docker_env.py:tests failed in 0.24s (ran in docker environment `local_docker`).
this?
c
yes, letā€™s check this sandbox 
/private/var/folders/ll/98hnvg_n5l3by0224q_qybc40000gn/T/pants-sandbox-rQOAbH
f
run.sh?
c
nvm, itā€™s the mounted sandbox
f
ok
c
yea, as I suspected we donā€™t have much more to go on without starting to tracing/debugging the rust engine
f
so sandboxes depend on each other?
c
noā€¦? how do you mean?
pants sets a sandbox up on your host system then mounts that when it is to be used in a container
f
yeah ok, good ... would be counterintuitive given the name. no idea, that in a sequence of steps step 0,1,2 have sandboxes but still somehow depend on another
c
well, there may be outputs from running a command in one sandbox that is later used in another sandbox..
f
ok so ... sandboxish? šŸ˜„
c
šŸ™‚
f
I guess toddlers can throw sand from one box to the other
šŸ˜ 1
c
fwiw this might be a related issue https://github.com/pantsbuild/pants/issues/18544
f
at least you can hope! šŸ˜„
but yeah, sounds like chances aren't zero
b
I'm in well over my head, but it seems fixed by explicitly passing the environment through to the docker subprocess here
Copy code
env_process_result = await Get(
    ProcessResult,
    Process(
        ["env", "-0"],
        env=session_values[CompleteEnvironmentVars],  # <----- Explicitly pass env
        description=f"Extract environment variables from {description_of_env_source}",
        level=LogLevel.DEBUG,
        cache_scope=env_tgt.executable_search_path_cache_scope(),
    ),
)
f
seems like a valid solution to me ... šŸ¤· (I but I can't tell of course)
b
I'm not suggesting it as a solution yet - I'd like to understand how the environment is implicitly propagated in Andreas and Benjy's (non-docker-desktop-for-mac) setups before chucking it in explicitly šŸ˜„
c
Ah.. good find. That explains it. What weā€™re missing is a way to tell pants what env vars we want go into the container runtime. Now, youā€™d need to have a 
ENV WHITELISTED=value
in the 
Dockerfile
building your image you use for the 
docker_environment
but as we use 
python:3.8.17-slim
there is no such env var set in there. I forgot to treat the docker env as a remote execution target. Consider remote execution, then itā€™s the environment of that remote machine that will be used, and the same is with running in a docker contariner, it is the environment in that container that is used, and what the local environment running the pants client is does not affect either of them.
In short, this is by design. So you if you want to use the local env var in a test running remotely (in a container or on another machine) you need to use the 
extra-vars=NAME=VALUE
syntax and rely on your invocation to set that value accordingly.
When we ran locally, Benjy and I, we hit this case: https://github.com/pantsbuild/pants/blob/fcdda2d28732944f82c4c66944e1122387a5cdaa/src/python/pants/engine/internals/platform_rules.py#L66-L67
Copy code
if env_tgt.val:
        if env_tgt.val.has_field(DockerImageField):
            description_of_env_source = f"the Docker image {env_tgt.val[DockerImageField].value}"
        elif env_tgt.val.has_field(RemotePlatformField):
            description_of_env_source = "the remote execution environment"
        else:
            # Else, it's a local environment.
            return session_values[CompleteEnvironmentVars]  # << HERE
f
ok, makes sense. for completeness I can confirm it's working for me:
Copy code
pants test --test-extra-env-vars="['WHITELISTED=VALUE']" ::
08:02:00.58 [INFO] Completed: Run Pytest - (environment:local_docker, src/test_docker_env.py:tests) - succeeded.

āœ“ src/test_docker_env.py:tests succeeded in 0.21s (memoized for docker environment `local_docker`).
FYI it doesn't matter if I use 
--test-extra-env-vars
or 
--extra-env-vars
- both work.
h
That's because the Pants CLI has a shortcut where flags placed after a goal scope to that goal by default. So 
pants test --extra-env-vars
implicitly adds the 
test-
scope to the flag.
f
good to know, thx
2 Views
Open in Slack
PreviousNext
Powered by