https://pantsbuild.org/ logo
Join Slack
Powered by
I am seeing `IntrinsicError: Invalid cross-device ...
# development
f
I am seeing 
IntrinsicError: Invalid cross-device link (os error 18)
error thrown in a container in CI when using any version 
2.17.X
except 
2.17.0.dev0
which works fine. 🧵
it looks like there might have been some changes that cause this behavior in our container environment (the issue is not reproducible on host / developers' machines). This is for Linux environment only (macOS is irrelevant). https://github.com/pantsbuild/pants/commit/3236b4e2eb2b0b6ef4aa53fa4a07c372a77da6c2 disabled hardlinking of immutable inputs, there's also https://github.com/pantsbuild/pants/pull/18153 which may be relevant. FWIW there's also memory leak when attempting to run dependencies on many targets (
pantsd
goes over 15GB of RAM usage), running dependencies on a single file manifests in 
IntrinsicError: Invalid cross-device link (os error 18)
or more verbose 
IntrinsicError: Failed to create hardlink to /var/tmp/jenkins-cache/pants/lmdb_store/immutable/files/fa/faad51a6a108fba9d40b2a10e82a2646fccbaf8c3d9be47818f4bffae02d94b8 at /tmp/pants-sandbox-igPIBD/pex: Invalid cross-device link (os error 18)
depending on what version of 2.17 is used. @witty-crayon-22786 @bitter-ability-32190 do you have any suggestions on what kind of data would help troubleshoot this?
f
Copy code
[root@pf37x-8 ~]# df /tmp /var/tmp 
Filesystem     1K-blocks     Used Available Use% Mounted on
/dev/vda5      334411756 39982624 293661648  12% /tmp
/dev/vda5      334411756 39982624 293661648  12% /
same underlying filesystem, but different mount point
f
FWIW inspecting the streamed logs before the process I see
Copy code
16:00:43.41 [WARN] Executor shutdown took unexpectedly long: tasks were likely leaked!
f
we need to make a bug in GH for this I think
Copy code
[root@pf37x-8 ~]# findmnt
TARGET                                        SOURCE                                FSTYPE    OPTIONS
/                                             /dev/vda5[/root/var/lib/machines/jenkins-agent]
                                                                                    btrfs     rw,relatime,seclabel,discard=async,space_cache=v2,subvolid=256,subvol=/root
├─/tmp                                        tmpfs                                 tmpfs     rw,nosuid,nodev,seclabel,size=1632552k,nr_inodes=409600,inode64
│ └─/tmp                                      /dev/vda5[/root/var/lib/machines/jenkins-agent/var/tmp]
f
FWIW I get the same error in 
2.18.0a0
Copy code
IntrinsicError: Failed to create hardlink to /root/.cache/pants/lmdb_store/immutable/files/fa/faad51a6a108fba9d40b2a10e82a2646fccbaf8c3d9be47818f4bffae02d94b8 at /tmp/pants-sandbox-Raz1Rv/pex: Invalid cross-device link (os error 18)
w
There is an existing bug, but I didn't think that it would exhibit in this way.
‼️ 1
f
I can repro outside of pants
Copy code
[root@pf37x-8 tmp]# ls -l /var/tmp/source
-rw-r--r--. 1 root root 0 Sep 14 16:04 /var/tmp/source
[root@pf37x-8 tmp]# ln /var/tmp/source /tmp/link
ln: failed to create hard link '/tmp/link' => '/var/tmp/source': Invalid cross-device link
b
How did I miss this issue?!
f
How did I miss this issue?!
same here wow
w
It's "the same" device underneath (with two mount points), so it misses our check.
b
Hey @witty-crayon-22786 I haven't looked at the code, but what do you think of us switching to a strategy of "try and hardlink and if it fails copy"? I forgot why we didn't just go that route, because its infallible
f
it works on my dev machine, and I'm actually going across devices between cache and sandbox dir there
or maybe i'm not, because of that 
/var/tmp
override i use for the cache dir.. 🤔
f
It's "the same" device underneath (with two mount points), so it misses our check.
thanks a ton @witty-crayon-22786 for taking a look! Would this issue explain the high memory usage, i.e. it attempts to perform an operation and the resources are not returned to the system?
f
yeah it's a bind mount, I guess I didn't realize you can't hardlink across bind mounts (makes sense in retrospect)
w
No, would not explain resource usage.
@bitter-ability-32190: we could do that, but I'd rather avoid per-copy overheads and test the two roots once and memoize it
f
interesting. If I run 
pants --local-execution-root-dir=/home --no-pantsd --no-local-cache -ldebug dependencies ::
it all proceeds and the memory is the same as in previous versions, so setting 
--local-execution-root-dir
helped
b
: we could do that, but I'd rather avoid per-copy overheads and test the two roots once and memoize it
Works for me. That's still infallible. If someone files a ticket I'll dive in at some point soon 🙂
w
It shouldn't impact resource usage, but yea: ensuring that the 
local_store_dir
and 
local_execution_root_dir
are on the same mount is advisable. When we fix this, we might have the "these aren't the same mounts" check also trigger a warning, since it will impact performance.
...if we've confirmed that. @bitter-ability-32190: I didn't end up doing benchmarks for that change
i’ve unassigned https://github.com/pantsbuild/pants/issues/18757 for now, because https://github.com/pantsbuild/pants/issues/19775 is more egregious and doesn’t have a clear workaround.
but if noone beats me to it i’ll work on it.
4 Views
Open in Slack
PreviousNext
Powered by