For anyone more into Rust than me, what's the fastest way (lib or no-lib) to calculate the SHA256 of a file in Rust? As a POC, in Python I was using

hashlib.file_digest

and then in Rust, using the

sha2

library, it took 2x as long. I swapped over to

ring

as per https://rust-lang-nursery.github.io/rust-cookbook/cryptography/hashing.html and that ended up being roughly on par with hashlib, as I would have expected. Ran this over 230k files and took the total run time as my comparison point

I'm not a Rust expert, but according to my experience with ripgrep, maybe you need some SIMD feature flag settings?

Can you post your actual code? It could be something like: wrapping the file in a buffered reader would help?

Yeah, so for the

sha2

lib, I enabled some of the feature flags that bought it roughly inline with

ring

(like, close enough that I didn't care) - otherwise, the code I used for the hashing was pulled straight from https://rust-lang-nursery.github.io/rust-cookbook/cryptography/hashing.html#calculate-the-sha-256-digest-of-a-file - and I just played with the buffer size

Bear in mind that a default build doesn't enable CPU-specific flags... Try throwing in

RUSTFLAGS="-C target-cpu=native"

Yeah, I had tried a run with native enabled - small difference I think, nothing crazy

--release

+ native CPU + buffering IO are the only things that jump out to me 🙂

Yep! Thanks! All 3 covered 🙂

Late to the party, but you probably want to be able to do I/O and hashing in parallel and figure out if you're I/O bound or CPU bound. SHA256 is also probably not the best algorithm for these sorts of things unless you have to conform to some existing protocol.

Yeah, so after some more work - it’s pretty clear that the time due to reading in the data is substantially more than hashing or anything else. This is also intentionally not wholly optimized, as I was doing a 1:1 between a Python and Rust equivalent to see which language to write the tool in. e.g. once I ran it through rayon, the time dropped substantially, but again, not really the point of the exercise

I am not an expert at I/O performance, but you will probably get performance benefits from parallelizing your reads, especially if you have a lot of small files

Yep, thanks! The "actual" way I do this will be more clever. In the test I'm running, I'm intentionally hitting multi-gig files, small K files, symlinks, hardlinks, etc. Basically a worst-case folder design vs what I will actually run into in the future