Tech

Dropbox Releases Code for New DivANS Compression Algorithm

If you thought that 7-Zip provided excellent compression ratios, then you might be in for a treat since Dropbox released their new DivANS compression algorithm as an open-source project. Developers claim that DivANS can compress data more densely than 7-Zip, Brotli or even Zstandard.

The code makes extensive use of the Rust systems programming language, which is sponsored by Mozilla. Rust was designed to be safe and concurrent, which makes it an excellent language for developing this sort of software in.

DivANS uses new vector intrinsic technology that’s provided by the Rust platform. Since it’s multi-threaded, it shouldn’t be at all slow either.

Another newer technology called intermediate representation gives developers the freedom to force different compression algorithms together and therefore build better optimizers over time.

Needless to say, the project is always looking for more community involvement from those who would like to volunteer their time toward producing a better compression algorithm.

DivANS got released under the Apache License, which should help to spread the code since the license is quite permissive. That being said, derivative compression code based on DivANS may not technically have to be distributed with the same license. This is somewhat unlike the copyleft licenses that accompany GPL-based software releases.

One of the most interesting aspects of the algorithm is related to its compilation. DivANS was written in Rust, but it got compiled against WebAssembly (WASM). This might help to explain how there’s already a browser demo available.

WASM code was designed to speed up execution of online subroutines that JavaScript and other interpretative languages would have struggled with. While it’s not intended to replace JavaScript, this looks like a situation where WASM could really be beneficial since writing a compression algorithm in JavaScript would potentially doom it to rather slow execution.

Considering the immense amount of data that Dropbox stores on behalf of clients, it could be easy to see the benefits of a new compression method from their point of view. Even a reduction by 1 or 2 percent of the total amount of data Dropbox stores would provide theoretically massive savings. It could also reduce the amount of time sent transmitting data over a network back to users.

John Rendace


John is a GNU/Linux expert with a hobbyist's background in C/C++, Web development, storage and file system technologies. In his free time, he maintains custom and vintage PC hardware. He's been compiling his own software from source since the DOS days and still prefers using the command line all these years later.
Close
Close