https://pantsbuild.org/ logo
#general
Title
# general
c

creamy-ghost-8929

07/28/2022, 5:35 PM
hi, I have the following structure: • train/ ◦ semantic_segmentation/ ▪︎ BUILD ▪︎ config/ • docker/ ◦ semantic_segmentation/ ▪︎ BUILD The folder config/ has config files distributed among subfolders. I need to have this folder in the working dir of my docker file: I saw in the docks that the .PEX file cannot included files (that we can use
open
), so inside the build file, I defined the docker file:
Copy code
docker_image(
    name="gpu",
    dependencies = ["docker:base_docker_gpu", "train/semantic_segmentation"],
    instructions = [
        "FROM base_docker_gpu",
        "COPY train.semantic_segmentation/train.pex /bin",
        "COPY train/semantic_segmentation/config /root/",
    ]
)
But the copy instruction fails (
./pants package docker/semantic_segmentation:gpu
):
COPY failed: file not found in build context or excluded by .dockerignore: stat train/semantic_segmentation/config: file does not exist
I tired to add the config folder as a target with the files target:
Copy code
files(
    name="semantic_segmentation_config",
    sources=["config/*/*", "config/*.yaml", "config/*/*.yaml"],
)
So, what would be the right way to add the config/ folder to my docker image using?
1
n

nutritious-minister-3808

07/28/2022, 5:41 PM
Add the "semantic_segmentation_config" target to the dependencies field of the docker_image target. That should ensure that the files are included in the build context.
👆 1
h

happy-kitchen-89482

07/28/2022, 5:43 PM
In some cases, such as when copying a single file, pants can infer the dependency on that file. But here you're copying the entire dir, so pants can't infer the dependency, and you have to add it manually
c

creamy-ghost-8929

07/28/2022, 5:43 PM
I have tried:
Copy code
docker_image(
    name="gpu",
    dependencies = ["//:clearml", "docker:base_docker_gpu", "train/semantic_segmentation:semantic_segmentation_config"],
    instructions = [
        "FROM base_docker_gpu",
        "COPY train.semantic_segmentation/train.pex /bin",
        "COPY train/semantic_segmentation/config /root/",
    ]
)
The image builds ok, but the /root folder is empty.
h

happy-kitchen-89482

07/28/2022, 5:45 PM
OK, so the fact that the image builds tells me that
train/semantic_segmentation/config
exists in the context, but apparently it's empty
Looking at the
files
target, I wonder if those sources are correct
n

nutritious-minister-3808

07/28/2022, 5:46 PM
```files(
name="semantic_segmentation_config",
sources=["config/*/*", "config/*.yaml", "config/*/*.yaml"],
)```
It seems like you may want
config/**/*.yaml
h

happy-kitchen-89482

07/28/2022, 5:46 PM
maybe it should just be be
sources=["config/**/*.yaml"],
coke
c

creamy-ghost-8929

07/28/2022, 5:52 PM
Thanks, it worked with
config/**/*.yaml
🎉 1
I just found out I cannot use my config folder from the working dir, It has to conserve the relative path to the entry point in my .pex file. Is there a way to get the
config
folder inside the folder in which the .pex file unpacks? In this example, it is something like:
/root/.pex/user_code/4a911aef70bdab03e901678aa2947f11c11df376/semantic_segmentation
h

happy-kitchen-89482

07/29/2022, 5:38 AM
You can bundle it as a
resource
into the pex itself
Create a
resource
target to wrap the files, and have the
pex_binary
target depend on that, instead of the
docker_image
then the files get baked into the pex
c

creamy-ghost-8929

07/29/2022, 3:48 PM
Thanks, I read in the docs that, as a resource, they cannot be normally accessed, .e.g by
open
, that holds? Or they will get unpacked with the rest of the .pex?
h

happy-kitchen-89482

07/29/2022, 4:22 PM
It depends on how the pex is built and run. If it's a venv-mode pex then it creates a regular venv, so you should be able to do whatever you could do in a venv
a zipapp-mode pex can only load files embedded inside it as resources, using
pkg_resources
mechanisms, because the files don't actually exist on disk
So in practice you should be OK
c

creamy-ghost-8929

07/29/2022, 6:55 PM
Thanks, it worked by adding the files as
resources
and setting the
.pex
execution mode to
venv
.
🔥 1
7 Views