https://pantsbuild.org/ logo
Join Slack
Powered by
Continuing to spam this channel :sweat_smile: I'm...
# plugins
a
Continuing to spam this channel 😅 I'm seeing a really weird issue where 3 file targets are having their sources copied into a digest as executable that are not supposed to be. They are not executable in the source directory. They are originally being brought in via a 
HydrateSourcesRequest
and then merged together with all of the other sources with 
MergeDigests
. I'm running with 
--no-local-cache
and I added an 
exit 1
so I can see the digest before any process is run on it. The only commonality as far as I can tell is all 3 of the files are larger, roughly 8mb each.
Only other commonality I'm seeing in trace logs that all 3 of these files are hydrated while this process is running?
Copy code
14:44:23.23 [32m[DEBUG][0m Running Find interpreter for constraints: CPython==3.11.* under semaphore with concurrency id: 4, and concurrency: 1
14:44:23.23 [32m[DEBUG][0m Completed: setup_sandbox
14:44:23.25 [32m[DEBUG][0m spawned local process as Some(771001) for Process { argv: ["/home/nickd/.cache/nce/60b513559c7b53eb2acecbd7b8aaaeb942686f3997d07fa77377b51324f58fda/bindings/venvs/2.22.0.dev3/bin/python3.9", "./pex", "--tmpdir", ".tmp", "--no-emit-warnings", "--pip-version", "24.0", "--python-path", "/home/nickd/.pyenv/versions/3.11.9/bin:/home/nickd/.pyenv/versions/3.9.19/bin:/home/nickd/.pyenv/shims:/home/nickd/.pyenv/bin:/home/nickd/.cargo/bin:/home/nickd/.vscode-server/bin/dc96b837cf6bb4af9cd736aa3af08cf8279f7685/bin/remote-cli:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/usr/lib/wsl/lib:/mnt/c/Windows/system32:/mnt/c/Windows:/mnt/c/Windows/System32/Wbem:/mnt/c/Windows/System32/WindowsPowerShell/v1.0/:/mnt/c/Windows/System32/OpenSSH/:/mnt/c/Program Files (x86)/NVIDIA Corporation/PhysX/Common:/mnt/c/WINDOWS/system32:/mnt/c/WINDOWS:/mnt/c/WINDOWS/System32/Wbem:/mnt/c/WINDOWS/System32/WindowsPowerShell/v1.0/:/mnt/c/WINDOWS/System32/OpenSSH/:/mnt/c/Program Files/dotnet/:/mnt/c/Program Files/Git/cmd:/mnt/c/Program Files/Calibre2/:/mnt/c/Program Files/PowerShell/7/:/mnt/c/Program Files/Docker/Docker/resources/bin:/mnt/c/Users/nickd/AppData/Local/Microsoft/WindowsApps:/mnt/c/Users/nickd/.dotnet/tools:/mnt/c/Program Files/JetBrains/IntelliJ IDEA Community Edition 2022.1.3/bin:/mnt/c/Users/nickd/AppData/Local/Programs/Microsoft VS Code/bin:/snap/bin:/home/nickd/.local/bin", "--interpreter-constraint", "CPython==3.11.*", "--", "-c", "import hashlib, os, sys\n\npython = os.path.realpath(sys.executable)\nprint(python)\n\nhasher = hashlib.sha256()\nwith open(python, \"rb\") as fp:\n  for chunk in iter(lambda: fp.read(8192), b\"\"):\n      hasher.update(chunk)\nprint(hasher.hexdigest())\n"], env: {"CPPFLAGS": "", "LANG": "C.UTF-8", "LDFLAGS": "", "PATH": "/home/nickd/.pyenv/shims:/home/nickd/.pyenv/bin:/home/nickd/.cargo/bin:/home/nickd/.vscode-server/bin/dc96b837cf6bb4af9cd736aa3af08cf8279f7685/bin/remote-cli:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/usr/lib/wsl/lib:/mnt/c/Windows/system32:/mnt/c/Windows:/mnt/c/Windows/System32/Wbem:/mnt/c/Windows/System32/WindowsPowerShell/v1.0/:/mnt/c/Windows/System32/OpenSSH/:/mnt/c/Program Files (x86)/NVIDIA Corporation/PhysX/Common:/mnt/c/WINDOWS/system32:/mnt/c/WINDOWS:/mnt/c/WINDOWS/System32/Wbem:/mnt/c/WINDOWS/System32/WindowsPowerShell/v1.0/:/mnt/c/WINDOWS/System32/OpenSSH/:/mnt/c/Program Files/dotnet/:/mnt/c/Program Files/Git/cmd:/mnt/c/Program Files/Calibre2/:/mnt/c/Program Files/PowerShell/7/:/mnt/c/Program Files/Docker/Docker/resources/bin:/mnt/c/Users/nickd/AppData/Local/Microsoft/WindowsApps:/mnt/c/Users/nickd/.dotnet/tools:/mnt/c/Program Files/JetBrains/IntelliJ IDEA Community Edition 2022.1.3/bin:/mnt/c/Users/nickd/AppData/Local/Programs/Microsoft VS Code/bin:/snap/bin:/home/nickd/.local/bin", "PEX_IGNORE_RCFILES": "true", "PEX_PYTHON": "/home/nickd/.cache/nce/60b513559c7b53eb2acecbd7b8aaaeb942686f3997d07fa77377b51324f58fda/bindings/venvs/2.22.0.dev3/bin/python3.9", "PEX_ROOT": ".cache/pex_root"}, working_directory: None, input_digests: InputDigests { complete: DirectoryDigest { digest: Digest { hash: Fingerprint<5b303bb6b7dce3ff6ca48ff0df5e73b78935a26409389aa3544472b4738da3f6>, size_bytes: 158 }, tree: "Some(..)" }, nailgun: DirectoryDigest { digest: Digest { hash: Fingerprint<e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855>, size_bytes: 0 }, tree: "Some(..)" }, inputs: DirectoryDigest { digest: Digest { hash: Fingerprint<5b303bb6b7dce3ff6ca48ff0df5e73b78935a26409389aa3544472b4738da3f6>, size_bytes: 158 }, tree: "Some(..)" }, immutable_inputs: {}, use_nailgun: {} }, output_files: {}, output_directories: {}, timeout: None, execution_slot_variable: None, concurrency_available: 0, description: "Find interpreter for constraints: CPython==3.11.*", level: Debug, append_only_caches: {CacheName("pex_root"): RelativePath(".cache/pex_root"), CacheName("python_build_standalone"): RelativePath(".python-build-standalone")}, jdk_home: None, cache_scope: PerRestartSuccessful, execution_environment: ProcessExecutionEnvironment { name: None, platform: Linux_x86_64, strategy: Local }, remote_cache_speculation_delay: 0ns, attempt: 0 }
14:44:23.39 [35m[TRACE][0m Completed: Fingerprinting: datapants.duckdb
14:44:23.39 [35m[TRACE][0m Completed: Snapshotting: datapants.duckdb
14:44:23.39 [35m[TRACE][0m Completed: pants.core.target_types.hydrate_file_source
14:44:23.39 [35m[TRACE][0m Completed: Hydrate the `sources` field - //:db
14:44:23.39 [35m[TRACE][0m Completed: Fingerprinting: seeds/raw_items.csv
14:44:23.39 [35m[TRACE][0m Completed: Snapshotting: seeds/raw_items.csv
14:44:23.39 [35m[TRACE][0m Completed: pants.core.target_types.hydrate_file_source
14:44:23.39 [35m[TRACE][0m Completed: Hydrate the `sources` field - //:root#seeds/raw_items.csv
14:44:23.43 [35m[TRACE][0m Completed: Fingerprinting: seeds/raw_orders.csv
14:44:23.43 [35m[TRACE][0m Completed: Snapshotting: seeds/raw_orders.csv
14:44:23.43 [35m[TRACE][0m Completed: pants.core.target_types.hydrate_file_source
14:44:23.43 [35m[TRACE][0m Completed: Hydrate the `sources` field - //:root#seeds/raw_orders.csv
14:44:23.50 [DEBUG] Launching 1 roots (poll=true).
14:44:24.00 [DEBUG] Launching 1 roots (poll=true).
14:44:24.50 [DEBUG] Launching 1 roots (poll=true).
14:44:24.91 [32m[DEBUG][0m Completed: Find interpreter for constraints: CPython==3.11.*
Also for additional context, this is problematic because I need write permissions for 
datapants.duckdb
in particular.
I pushed the code to Github in case anyone wants to check it out and see if they can reproduce the issue. The failing command is 
pants export-codegen ::
from within the 
sample-project
directory (which is a separate pants project). https://github.com/ndellosa95/data-pants/tree/wip
b
Is there any file larger than 512KB that does NOT behave like these problematic ones? Just working out if it might be connected to the “FSDB” cache
a
Nope, these are the only files of with size greater than 512KB.
Any reason you can think of to just not change this to 777 permissions?
Or would it be preferable to preserve permissions from the original file?
b
Preserving permission seems sensible. Maybe @bitter-ability-32190 can remember why we did that overriding?
b
Because it's one base file for all copies, and the engine doesn't use perms in the xache
So if we copied perms it'd be "first one wins" which isn't great
r
Because it's one base file for all copies, and the engine doesn't use perms in the xache
By this do you mean the FSDB is implemented as a single file that doesn’t store permissions?
b
Its implemented as a single file on disk, hardlinked to by all digests which have a file with the same sha256.
It can't be writeable, because then you'd be changing the file for all references to it (it's content addressed)
You'll have to 
cp
the file somewhere in the sandbox and then you can change the perms
b
Aaah of course. FSDB hardlinking means the file caches shouldn't be written and the permissions are used to reduce the chance of errors. I have a vague recollection that there might somehow be option to materialize a digest as mutable? And so the sandbox set-up will do a copy rather than hard link: https://github.com/pantsbuild/pants/blob/2395041570ea2f45d9cbc32a7a54cf8d2a0e4e84/src/rust/engine/fs/store/src/lib.rs#L1320-L1343 I don't know how to pass the 
can_be_immutable
flag down from rule code, though
a
Ooh, so if I include the offending file as an output file it comes through as writable
So I think the only change I'd like to make is just to document that behavior.
b
Yeah in that case we don't do the hardlink, since presumably it's an output file because you want to write to it. Put another way, a safe assumption is that all input-only files are readable and all outputs are writable
a
Okay so annoyingly...this only works if you include the file as part of 
output_files
- if it is encompassed as part of 
output_directories
it does not work.
That's an actual bug I think, no?
b
Yeah thats a bug I think
a
cool I'll create an issue for it and see if I can figure out a fix for it
13 Views
Open in Slack
PreviousNext
Powered by