witty-crayon-22786
02/09/2023, 9:44 PMexec
’d inside an existing container may not be able to observe all of the inputs in their sandboxes (the parent directory of which is mounted once per-container)fast-nail-55400
02/09/2023, 10:06 PMwitty-crayon-22786
02/09/2023, 10:15 PMthere is also copying files in to and out of a docker volumethis is essentially the tar-pipe idea… more complicated “copy files into a container” cases seem to suggest tar-pipes. there doesn’t appear to be a way to create a volume “from scratch” containing content.
longer term, mount an NFSv4 volume into the container and then have pantsd export a NFSv4 server hosting input roots.yea… would be generally useful too.
fast-nail-55400
02/09/2023, 10:16 PMwitty-crayon-22786
02/09/2023, 10:18 PMfast-nail-55400
02/09/2023, 10:19 PMwitty-crayon-22786
02/09/2023, 10:20 PMenough-analyst-54434
02/09/2023, 10:27 PMwitty-crayon-22786
02/09/2023, 10:34 PMIs it write to a volume mount, then serially exec into a container that can see that volume mount? I’d guess so, but I’m just guessing, etc.write to a directory that is bind-mounted into the container. on Linux the mounts are synchronous since there is no virtualization involved (uses overlay2, etc), but on macOS it has used a variety of strategies in the last few years (osxfs, grpc-fuse, virtiofs)
enough-analyst-54434
02/09/2023, 10:35 PMwitty-crayon-22786
02/09/2023, 10:35 PMvirtiofs
plus macOS’s new virtualization framework (which is still an experimental Docker feature) will resolve thisenough-analyst-54434
02/09/2023, 10:36 PMwitty-crayon-22786
02/09/2023, 10:38 PMIf you knew more about the filesystems and what they actually guaranteed you could write the files, then write a token file, then in the exec, use a shim binary that blocked on appearance of the token. If the FS was async but guaranteed order, that would work.mm, yea: this is a good thought too. a particularly annoying bit about both
osxfs
and grpc-fuse
is that they are closed source. and grpc-fuse is deprecating osxfs
, but has drastically less documentation than osxfs
did.enough-analyst-54434
02/09/2023, 10:38 PMwitty-crayon-22786
02/09/2023, 10:39 PMIs ditching Docker an option?maybe, yea. but afaict, all of the macOS-containers systems have this same issue, and have been bouncing between implementations. they seem to be trending toward virtiofs. the first half of this article is good: https://www.cncf.io/blog/2023/02/02/docker-on-macos-is-slow-and-how-to-fix-it/
Plowing on though, your bullets above mention 750ms mac, I did spend a good bit of effort not using Docker and getting 100ms starts in ~2018/2019 or so. I used crun + jqi fully expect that this is macOS doing virtualization rather than docker itself… latency is much lower on Linux, afaik
enough-analyst-54434
02/09/2023, 10:40 PMwitty-crayon-22786
02/09/2023, 10:41 PMenough-analyst-54434
02/09/2023, 10:42 PMwitty-crayon-22786
02/09/2023, 10:42 PMWell that’s future so we can’t just wait. AFAICT your send a tar by network idea - presumably you need a shim binary to coordinate? - sounds like the most sane option when not fully understanding the problem.no, no need for a shim binary: you can pipe directly into tar. the docker API allows for streaming stdin to a process
enough-analyst-54434
02/09/2023, 10:42 PMwitty-crayon-22786
02/09/2023, 10:43 PMI personally would not try until I understood the problem.see above re: grpc-fuse being closed source… i’m not sure i will actually get an answer there
enough-analyst-54434
02/09/2023, 10:43 PMi fully expect that this is macOS doing virtualization rather than docker itself… latency is much lower on Linux, afaikI was bringing - wild variablity - 500ms - 1s down to 100ms constant FWIW.
witty-crayon-22786
02/09/2023, 10:45 PMenough-analyst-54434
02/09/2023, 10:47 PMwitty-crayon-22786
02/09/2023, 10:47 PMenough-analyst-54434
02/09/2023, 10:48 PM"brand new headline feature of the release"Ok, that sounds like marketing speak to me.
witty-crayon-22786
02/09/2023, 10:48 PMenough-analyst-54434
02/09/2023, 10:48 PMwitty-crayon-22786
02/09/2023, 10:49 PMenough-analyst-54434
02/09/2023, 10:51 PMhappy-kitchen-89482
02/09/2023, 11:17 PMwitty-crayon-22786
02/10/2023, 12:40 AMcurved-television-6568
02/10/2023, 1:29 AMcc @curved-television-6568, @enough-analyst-54434: anything else jump out at you?nothing obvious, no.
witty-crayon-22786
02/13/2023, 10:48 PMmain
, and land something to disable the container cache instead. what a mess.stat
for all of the inputs.stat
shows that all input files for a task exist, including files which the process claims don’t exist: https://gist.github.com/stuhood/6c78ab4a9e511b14df2c3247962604ee__pkgs__/internal_reflectlite/__pkg__.a
is claimed missing, but stat
from within the container shows it as existing..a
files have the same permissions… which is fishy, but i don’t know how it could result in a “no such file or directory” errorenough-analyst-54434
02/15/2023, 9:33 PMwitty-crayon-22786
02/15/2023, 9:36 PMmain
(but not on 2.15.x
… sigh)enough-analyst-54434
02/15/2023, 9:38 PMwitty-crayon-22786
02/15/2023, 9:39 PMenough-analyst-54434
02/15/2023, 9:39 PMwitty-crayon-22786
02/15/2023, 9:41 PMenough-analyst-54434
02/15/2023, 9:41 PMwitty-crayon-22786
02/15/2023, 9:44 PMenough-analyst-54434
02/15/2023, 9:44 PMfast-nail-55400
02/15/2023, 9:48 PMwitty-crayon-22786
02/15/2023, 9:49 PMtar-pipe
change did.tar
, hehfast-nail-55400
02/15/2023, 9:49 PMdocker exec
itwitty-crayon-22786
02/15/2023, 9:51 PMenough-analyst-54434
02/15/2023, 9:51 PMwitty-crayon-22786
02/15/2023, 9:52 PMenough-analyst-54434
02/15/2023, 9:53 PMwitty-crayon-22786
02/15/2023, 9:53 PMenough-analyst-54434
02/15/2023, 9:54 PMfast-nail-55400
02/15/2023, 9:54 PMwitty-crayon-22786
02/15/2023, 9:54 PMenough-analyst-54434
02/15/2023, 9:55 PMwitty-crayon-22786
02/15/2023, 9:55 PMenough-analyst-54434
02/15/2023, 9:56 PMfast-nail-55400
02/15/2023, 10:12 PMProcessExecutionFailure: Process 'Link Go binary: ./package_analyzer' failed with exit code 1.
stdout:
loadinternal: cannot find runtime/cgo
stderr:
2023/02/15 22:10:13 reference to undefined builtin "runtime.duffzero" from package "runtime"
witty-crayon-22786
02/15/2023, 10:15 PM--no-local-cache
?fast-nail-55400
02/15/2023, 10:16 PM/usr/local/go/src/syscall/env_unix.go:12:2: could not import runtime (open __pkgs__/runtime/__pkg__.a: no such file or directory)
witty-crayon-22786
02/15/2023, 10:19 PMfast-nail-55400
02/15/2023, 10:20 PM./pants_from_sources --no-local-cache package race:racy_docker
)witty-crayon-22786
02/15/2023, 10:22 PMstat
vs open
was partially correct: wc -c
will report “No such file or directory” in some cases, and then eventually stabilize on agreeing that all files exist. but then go
will still fail to find some inputs.enough-analyst-54434
02/15/2023, 10:22 PMwitty-crayon-22786
02/15/2023, 10:23 PMtmp
filesystem.fast-nail-55400
02/15/2023, 10:29 PMwitty-crayon-22786
02/15/2023, 10:30 PM2.15.x
, so i don’t understand how folks were seeing this there.go
case with hardlinks disabled (by raising this limit), and no extra filesystem synchronization. so main question now is: how did someone observe this in 2.15.x, given that we hadn’t started hardlinking things at that pointnamed_caches
into a volume anyway, so might be willing to guess and check thereenough-analyst-54434
02/15/2023, 11:06 PMpex
script. The original issue has not enough info to determine what ./pex
is afaict. Is that the Pex PEX? If so, that's a PEX file materialized by the engine.witty-crayon-22786
02/15/2023, 11:07 PMenough-analyst-54434
02/15/2023, 11:08 PM./pex
is not foundwitty-crayon-22786
02/15/2023, 11:10 PMsrc/python/pants/backend/python/util_rules/pex_cli.py
now, but… we must be invoking it as python ./pex
…?enough-analyst-54434
02/15/2023, 11:10 PMwitty-crayon-22786
02/15/2023, 11:12 PM$ python3 ./does-not-exist
/Users/stuhood/.pyenv/versions/3.6.10/bin/python3: can't open file './does-not-exist': [Errno 2] No such file or directory
enough-analyst-54434
02/15/2023, 11:12 PMwitty-crayon-22786
02/15/2023, 11:14 PM2.16.0.dev5
! damnitmain
as well! so that’s it then. holy crap.enough-analyst-54434
02/16/2023, 1:36 AMwitty-crayon-22786
02/16/2023, 1:39 AMenough-analyst-54434
02/16/2023, 1:42 AM