#2155 Non-deterministic output due to non-deterministic os.walk and unix file permissions of original files Issue created by ento Context I'm using Pants to package up a GitHub Actions action I started writing in Python. The packaged pex file is checked in to the repo, so that when the action gets run, all dependencies get pulled down as part of the action itself, similar to how actions/typescript-action uses ncc. Problem and current workaround It appeared to work fine until I added a CI check that builds a pex file in the CI environment and verifies the checked in pex file is identical to the one that just got built: even though none of the source files nor dependencies changed, the check didn't pass. I tracked down the causes and was able to use a forked pex repo with a few changes to make the check pass: main...ento:pex:deterministic-bootstrap 1.

os.walk

was yielding directories in a non-deterministic order, apparently. Fixed by applying

.sort()

in a few places. • I wrote a few tests that reproduces this issue: main...ento:pex:deterministic-os-walk 2. My local environment and the CI environment had different umask values, and because pex preserves filesystem permissions, the resulting zip archive ended up being different too. Fixed by applying something like a umask when adding entries to the zip archive. • Although this worked in my case, I don't think the way I did it really guarantees reproducibility: umask at the OS level works fine when it comes to reproducibility because the default permission is something constant when a new filesystem entry is created (although I don't know if that's the same across all (POSIX-compliant?) OS'es), but in the case of pex, there's no default permission per se and it uses the permission of the entry on disk as the starting point. Questions I can work on opening a PR or two to address these sources of non-determinism if that's something good to incorporate into pex. 1. For

os.walk

, I'm thinking of adding a wrapper around

os.walk

like

deterministic_os_walk

and using it at least in the two places I needed to patch. Would it be a good idea to use it everywhere, or should it be limited to just these two places? Or is this something out of scope of guarantees that pex is willing to provide? 2. Is providing deterministic builds when it comes to unix file permissions something within scope of pex? If so, what would be a good approach? • I did think of specifying the umask in the CI environment and not handling it within pex when I was trying to come up with a fix, but enforcing the umask value in local dev environments is going to be hard, unless all contributors' machines are set up the same way, as the value would need to be set before you clone the GitHub actions repo. • I noticed there's a

--use-system-time

flag for allowing non-deterministic timestamps. With file permissions, we could similarly have a non-deterministic mode and deterministic mode, where deterministic mode creates new zipfile entries with

u=rw,g=rw,o=rw

for files and

u=rwx,g=rwx,o=rwx

for directories and with a configurable umask applied, but.. that will mean files that were originally executable will lose that bit. The determinisitc mode could, instead, limit what it controls to

r

and

w

permissions and let

x

pass through, which might work good enough in practice. pantsbuild/pex