https://pantsbuild.org/ logo
#general
Title
# general
t

thousands-plumber-33255

12/08/2022, 8:48 AM
Hello 🙂 I am trying to use the new environments feature to run a test in a docker container. I have this folder structure:
Copy code
pants.toml
/services
  /serviceA
    BUILD
    /tests
      BUILD
      example_test.py
In
pants.toml
I have added this entry:
Copy code
[environments-preview.names]
linux_docker = "//:local_qgis"
In `serviceA/BUILD`:
Copy code
docker_environment(
  name="local_qgis",
  image="qgis/qgis:final-3_22_9",
)
In `serviceA/tests/BUILD`:
Copy code
python_tests(
    environment="local_qgis"
)
i am getting the followring error with `./pants test services/serviceA/tests/example_test.py`:
Copy code
ResolveError: The address //:local_qgis from the option [environments-preview].names does not exist.

The target name ':local_qgis' is not defined in the directory . Did you mean one of these target names?

  * :root
Quite sure I have missed something fundamental, but don't get it. Can someone help? 🙂
r

refined-addition-53644

12/08/2022, 9:10 AM
The
name
should be
linux_docker
inside
docker_environment
and in other places.
t

thousands-plumber-33255

12/08/2022, 9:47 AM
Are you sure? It is exactly like I did it here right?
r

refined-addition-53644

12/08/2022, 10:29 AM
Yeah I think so. Worth trying. But agree it’s bit confusing because of how
local_environment
is bit different.
Not sure now
Wait, I think the
python_test
should use the
linux_docker
. The docker_environment should be as it is.
Yeah just tested it and that works. So basically the
docker_environment
generates the target using the
name
you provide and other target consumes it using the alias you set for the
name
in pants.toml
t

thousands-plumber-33255

12/08/2022, 12:10 PM
Out of sudden, the pants installation just hangs after
New virtual environment successfully created at /home/vscode/.cache/pants/setup/bootstrap-Linux-x86_64/2.15.0.dev4_py39
. all the time. Any idea? Cancelling it and install 2.14.0 works just fine
Copy code
vscode ➜ /nefino_li (feature/2087-add-tests-to-services-qgis-export-feature ✗) $ ./pants --no-pantsd --no-local-cache --print-stacktrace -ldebug --version
Bootstrapping Pants using /usr/bin/python3.9
Creating the virtualenv PEX.
Downloading the Pex PEX.
SHA256 fingerprint of <https://github.com/pantsbuild/pex/releases/download/v2.1.103/pex> verified.
Scrubbing PYTHONPATH
Installing pantsbuild.pants==2.15.0a1 into a virtual environment at /home/vscode/.cache/pants/setup/bootstrap-Linux-x86_64/2.15.0a1_py39
Scrubbing PYTHONPATH
New virtual environment successfully created at /home/vscode/.cache/pants/setup/bootstrap-Linux-x86_64/2.15.0a1_py39.
12:28:36.54 [DEBUG] File handle limit is: 1048576
12:28:36.55 [DEBUG] Using [bounded::CommandRunner { inner: SwitchedCommandRunner { .. }, .. }] for process execution.
12:28:37.18 [DEBUG] Launching 1 roots (poll=false).
Not getting any further. @curved-television-6568 Any idea?
b

busy-vase-39202

12/08/2022, 12:42 PM
cc @ancient-vegetable-10556
r

refined-addition-53644

12/08/2022, 12:56 PM
Try 2.15.0a1 I think there was some bug in dev release
t

thousands-plumber-33255

12/08/2022, 1:13 PM
I also tried that one
😔 1
c

curved-television-6568

12/08/2022, 1:24 PM
@thousands-plumber-33255 🤷 sadly no ideas 😉 (I’ve not yet tried the new environment stuff out..)
t

thousands-plumber-33255

12/08/2022, 3:53 PM
That is currently not the problem, but installing 2.15 at all 😞
w

witty-crayon-22786

12/08/2022, 5:52 PM
thanks for testing the alpha releases… is there anything useful in
.pants.d/pants.log
?
t

thousands-plumber-33255

12/09/2022, 8:49 AM
Nothing at all actually, only:
08:45:23.59 [INFO] handling request:
--pants-bin-name=./pants --pants-version=2.15.0a1 --version``
We tried on the machine of a co-worker: 1. Pants 2.14 was installed, then successfully upgrade to 2.15.0a1 2. Trying to run a tests hangs on the same line as I had above:
Copy code
./pants --print-stacktrace -ldebug test services/qgis/tests/example_test.py
10:02:26.00 [DEBUG] acquiring lock: <pants.pantsd.lock.OwnerPrintingInterProcessFileLock object at 0x7f96edb4a9d0>
10:02:26.00 [DEBUG] terminating pantsd
10:02:26.00 [DEBUG] sending signal 15 to pid 3087
10:02:27.01 [DEBUG] successfully terminated pid 3087
10:02:27.01 [DEBUG] purging metadata directory: .pants/.pids/4b81b7112ed6/pantsd
10:02:27.01 [DEBUG] Launching pantsd
10:02:27.01 [DEBUG] purging metadata directory: .pants/.pids/4b81b7112ed6/pantsd
10:02:27.01 [DEBUG] pantsd command is: PANTS_DAEMON_ENTRYPOINT=pants.pantsd.pants_daemon:launch_new_pantsd_instance PYTHONPATH=/home/vscode/.cache/pants/setup/bootstrap-Linux-x86_64/pants.4UZp49/install/bin:/nefino_li/dist/export/python/virtualenvs/default/3.8.15/lib/python3.8/site-packages:/nefino_li:/usr/lib/python39.zip:/usr/lib/python3.9:/usr/lib/python3.9/lib-dynload:/home/vscode/.cache/pants/setup/bootstrap-Linux-x86_64/2.15.0a1_py39/lib/python3.9/site-packages /home/vscode/.cache/pants/setup/bootstrap-Linux-x86_64/2.15.0a1_py39/bin/python /home/vscode/.cache/pants/setup/bootstrap-Linux-x86_64/2.15.0a1_py39/bin/pants --pants-bin-name=./pants --pants-version=2.15.0a1 --print-stacktrace -ldebug test services/qgis/tests/example_test.py
10:02:28.91 [DEBUG] pantsd is running at pid 3786, pailgun port is 45317
10:02:28.91 [DEBUG] releasing lock: <pants.pantsd.lock.OwnerPrintingInterProcessFileLock object at 0x7f96edb4a9d0>
10:02:28.91 [DEBUG] Connecting to pantsd on port 45317
10:02:28.91 [DEBUG] Connecting to pantsd on port 45317 attempt 1/3
10:02:28.92 [DEBUG] Connected to pantsd
10:02:28.93 [DEBUG] Launching 1 roots (poll=false).
3. Tried again with `--no-pantsd`:
Copy code
./pants --print-stacktrace -ldebug test --no-pantsd services/qgis/tests/example_test.py
10:04:27.84 [DEBUG] File handle limit is: 1048576
10:04:27.85 [DEBUG] Using [cache::CommandRunner { inner: bounded::CommandRunner { inner: SwitchedCommandRunner { .. }, .. }, .. }, cache::CommandRunner { inner: bounded::CommandRunner { inner: SwitchedCommandRunner { .. }, .. }, .. }] for process execution.
10:04:28.37 [DEBUG] Launching 1 roots (poll=false).
4. As still not working we were trying to use the latest successfull commit from 2.15.x branch. But now even
./pants --version
hangs again at this stage All of that is running in a devcontainer with Dockerfile
python-3.8.15-bullseye
w

witty-crayon-22786

12/09/2022, 4:20 PM
hm. that’s very suspicious. increasing the log level to
-ltrace
might help give a bit more information? but most likely the next step is going to be to try to attach a debugger like
gdb
or
lldb
to see where threads are. thanks again for testing the alphas, and sorry for the trouble!
also, between attempts, you might consider killing the
pantsd
process: it’s not necessary in general, but something strange is going on here.
t

thousands-plumber-33255

12/12/2022, 7:47 AM
Not seing any more info with
-ltrace
really. I also tried using the
2.15.0rc0
, but still no success. Can you tell me what to run to give you more debug information with e.g.
gdb
?
w

witty-crayon-22786

12/12/2022, 5:43 PM
while it’s hung, attach with
gdb
, and then run:
Copy code
thread apply all bt
if you’re able to share the trace level logs of a
--no-pantsd
run in a DM, that might be helpful.
t

thousands-plumber-33255

12/13/2022, 11:36 AM
For whatever reason, I had a few runs where it worked. But a co-worker was running into that again. I have sent you the gdb logs in a DM.
w

witty-crayon-22786

12/13/2022, 6:33 PM
thanks for the stacktrace. surprisingly, it does actually appear to be blocked in some python code. if you could 1. gather two more (from the same or different processes), to help confirm that that wasn’t a momentary blip 2. use
py-spy dump -p $pid
to gather a python stacktrace, so we can see which function it’s in
t

thousands-plumber-33255

12/14/2022, 8:32 AM
Done and sent as a DM
Okay, with a lot of trial and error I finally found the issue. I have commented out
build_file_prelude_globs = ["pants_plugins/macros.py"]
and then no issue at all. My
macros.py
looks as follows:
Copy code
def http_service_image(name, service, **kwargs):
    python_awslambda(
        name=f"{service}-lambda",
        runtime="python3.8",
        handler=f"{service}/lambda_handler.py:handler",
        # Otherwise pants cannot safely infer as dir name is the same in all services
        dependencies=[f"//services/{service}/api", f"//services/{service}/lambda_handler.py"]
    )

    docker_image(
        name=name,
        dependencies=[f":{service}-lambda", ":entrypoint"],
        image_tags=["{build_args.IMAGE_TAG}"],
        extra_build_args=[f"SERVICE={service}"],
        repository=f"{{build_args.{service.upper()}_REPO}}",
        **kwargs
    )


def env_vars_django(**kwargs):
    return [
        "ENVIRONMENT",
        "DEBUG",
        "ALLOWED_HOSTS",
        "SECRET_KEY",
        "DJANGO_LOGGING_ENABLED",
        "DJANGO_LOG_LEVEL",
        "DOMAIN",
    ]
When I comment out
http_service_image
, it also works. Any idea what is wrong with that? @curved-television-6568 helped me setting this up here. It perfectly works with
2.14.0
.
@refined-addition-53644 I have updated the environment name in the
python_tests
target as suggested, but now I am getting:
Copy code
14:37:59.83 [ERROR] 1 Exception encountered:

Engine traceback:
  in `test` goal
  in Run Pytest - (environment:linux_docker, services/qgis/tests/example_test.py)

ProcessExecutionFailure: Process 'Extract environment variables from the Docker image python:3.8-bullseye' failed with exit code 126.
stdout:
OCI runtime exec failed: exec failed: unable to start container process: chdir to cwd ("/pants-sandbox/pants-sandbox-kqW2I5") set in config.json failed: no such file or directory: unknown

stderr:
I can see that the container is being started. It should be noted that I am running pants itself in a Docker container with access to
/var/run/docker.sock
. Do I need to mount something to the pants container as well such that files are correctly mounted from the host to the test container?
r

refined-addition-53644

12/19/2022, 4:46 PM
This is beyond my expertise. So sending it to main chat in case someone else can have a look.
w

witty-crayon-22786

12/19/2022, 6:12 PM
@thousands-plumber-33255: could you file an issue containing this error: https://pantsbuild.slack.com/archives/C046T6T9U/p1671460920088449?thread_ts=1670489335.652639&amp;cid=C046T6T9U … and as much repro context as you can? this seems potentially docker-in-docker related, but i’m not able to further investigate it right now
t

thousands-plumber-33255

12/19/2022, 8:16 PM
@witty-crayon-22786 I will for sure! Any idea why it hangs while installing as shown due to the macro? This was not obvious, at least for me, and maybe we can help others avoiding this.
@witty-crayon-22786 I have added a repo to the issue https://github.com/pantsbuild/pants/issues/17852 which reproduces the error.
w

witty-crayon-22786

01/03/2023, 6:14 PM
thank you! will take a look today.
@thousands-plumber-33255: hey Chris: thanks again for making the repro repo. it looks like it is private though (or a 404 at least).
t

thousands-plumber-33255

01/04/2023, 8:15 AM
@witty-crayon-22786 Ah sorry, changed the visibility now
@witty-crayon-22786 Have you had a look already?
w

witty-crayon-22786

01/06/2023, 7:35 PM
i see what this is: and you were right that it was related to the macro. sorry for all the distraction before getting to it!
https://github.com/pantsbuild/pants/pull/17939 : will cherry pick before the end of the day.
8 Views