Post a Comment On: cbloom rants

A continuing series reporting on the performance of recently released Oodle Kraken First some general notes about Oodle before the big dump of numbers. (skip to the bottom for charts) Oodle is not intended to solve every problem in data compression. Oodle Data Compression is mainly designed for compress-once load-many usage patterns, where the compressed size and decode speed is important, but encode speed is not as important. Things like distribution, packaging, when you bake content into a compressed archive and then serve it to many people. I do consider it a requirement to keep the encoders faster than 1 MB/s on my ancient laptop, because less than that is just too slow even for content baking. Like most data compressors, Oodle has various compression level options that trade off encoder speed for compressed size. So there are faster and slower encoders; some of the fast modes are good tradeoffs for space-speed. Kraken turns out to be pretty good at getting high compression even in the fast modes. I'll do a post that goes into this in the future. Oodle is mainly intended for packing binary data. Because Oodle is built for games, we focus on the types of data that games ship, such as textures, models, animations, levels, and other binary structured data that often contains various types of packed numeric data. Oodle is good on text but is not really built for text. In general if you are focusing on a specific type of data (eg. text/html/xml) you will get the best results with domain-specific preprocessors, such as xwrt or wbpe. Oodle+wbpe is an excellent text compressor. Oodle Kraken is intended for high compression use cases, where you care about size. There's a whole family of compressors now that live in the "just below lzma (7zip)" compression ratio domain, such LZHAM, Brotli, Oodle BitKnit, and ZStd. In general the goal of these is to get as close to lzma compression ratio as possible while providing much better speed. This is where Kraken achieves far more than anyone has before, far more than we thought possible. Today I will be focusing on high compression modes, looking at decode speed vs. compression ratio. All compressors will generally be run in their max compression mode, without going into the "ridiculously slow" below 1 MB/s range. (for Oodle this means running -zl6 Optimal2 but not -zl7 or higher). Okay, on to lots of numbers. Rather than post an average on a test set, which can be misleading due to the selection of the test set or the way the results are averaged (total compressed size or average ratio?), I'll post results on a selection of individual files : ------------------------------------------------------- "silesia_mozilla" by ratio: lzma : 3.88:1 , 2.0 enc mbps , 63.7 dec mbps lzham : 3.56:1 , 1.5 enc mbps , 186.4 dec mbps Oodle Kraken: 3.51:1 , 1.2 enc mbps , 928.0 dec mbps zstdmax : 3.24:1 , 2.8 enc mbps , 401.0 dec mbps zlib9 : 2.51:1 , 12.4 enc mbps , 291.5 dec mbps lz4hc : 2.32:1 , 36.4 enc mbps , 2351.6 dec mbps by encode speed: lz4hc : 2.32:1 , 36.4 enc mbps , 2351.6 dec mbps zlib9 : 2.51:1 , 12.4 enc mbps , 291.5 dec mbps zstdmax : 3.24:1 , 2.8 enc mbps , 401.0 dec mbps lzma : 3.88:1 , 2.0 enc mbps , 63.7 dec mbps lzham : 3.56:1 , 1.5 enc mbps , 186.4 dec mbps Oodle Kraken: 3.51:1 , 1.2 enc mbps , 928.0 dec mbps by decode speed: lz4hc : 2.32:1 , 36.4 enc mbps , 2351.6 dec mbps Oodle Kraken: 3.51:1 , 1.2 enc mbps , 928.0 dec mbps zstdmax : 3.24:1 , 2.8 enc mbps , 401.0 dec mbps zlib9 : 2.51:1 , 12.4 enc mbps , 291.5 dec mbps lzham : 3.56:1 , 1.5 enc mbps , 186.4 dec mbps lzma : 3.88:1 , 2.0 enc mbps , 63.7 dec mbps ------------------------------------------------------- "lzt99" by ratio: lzma : 2.65:1 , 3.1 enc mbps , 42.3 dec mbps Oodle Kraken: 2.46:1 , 2.3 enc mbps , 957.1 dec mbps lzham : 2.44:1 , 1.9 enc mbps , 166.0 dec mbps zstdmax : 2.27:1 , 3.8 enc mbps , 482.3 dec mbps zlib9 : 1.77:1 , 13.3 enc mbps , 286.2 dec mbps lz4hc : 1.67:1 , 30.3 enc mbps , 2737.4 dec mbps by encode speed: lz4hc : 1.67:1 , 30.3 enc mbps , 2737.4 dec mbps zlib9 : 1.77:1 , 13.3 enc mbps , 286.2 dec mbps zstdmax : 2.27:1 , 3.8 enc mbps , 482.3 dec mbps lzma : 2.65:1 , 3.1 enc mbps , 42.3 dec mbps Oodle Kraken: 2.46:1 , 2.3 enc mbps , 957.1 dec mbps lzham : 2.44:1 , 1.9 enc mbps , 166.0 dec mbps by decode speed: lz4hc : 1.67:1 , 30.3 enc mbps , 2737.4 dec mbps Oodle Kraken: 2.46:1 , 2.3 enc mbps , 957.1 dec mbps zstdmax : 2.27:1 , 3.8 enc mbps , 482.3 dec mbps zlib9 : 1.77:1 , 13.3 enc mbps , 286.2 dec mbps lzham : 2.44:1 , 1.9 enc mbps , 166.0 dec mbps lzma : 2.65:1 , 3.1 enc mbps , 42.3 dec mbps ------------------------------------------------------- "all_dds" by ratio: lzma : 2.37:1 , 2.1 enc mbps , 40.8 dec mbps Oodle Kraken: 2.18:1 , 1.0 enc mbps , 684.6 dec mbps lzham : 2.17:1 , 1.3 enc mbps , 127.7 dec mbps zstdmax : 2.02:1 , 3.3 enc mbps , 289.4 dec mbps zlib9 : 1.83:1 , 13.3 enc mbps , 242.9 dec mbps lz4hc : 1.67:1 , 20.4 enc mbps , 2226.9 dec mbps by encode speed: lz4hc : 1.67:1 , 20.4 enc mbps , 2226.9 dec mbps zlib9 : 1.83:1 , 13.3 enc mbps , 242.9 dec mbps zstdmax : 2.02:1 , 3.3 enc mbps , 289.4 dec mbps lzma : 2.37:1 , 2.1 enc mbps , 40.8 dec mbps lzham : 2.17:1 , 1.3 enc mbps , 127.7 dec mbps Oodle Kraken: 2.18:1 , 1.0 enc mbps , 684.6 dec mbps by decode speed: lz4hc : 1.67:1 , 20.4 enc mbps , 2226.9 dec mbps Oodle Kraken: 2.18:1 , 1.0 enc mbps , 684.6 dec mbps zstdmax : 2.02:1 , 3.3 enc mbps , 289.4 dec mbps zlib9 : 1.83:1 , 13.3 enc mbps , 242.9 dec mbps lzham : 2.17:1 , 1.3 enc mbps , 127.7 dec mbps lzma : 2.37:1 , 2.1 enc mbps , 40.8 dec mbps ------------------------------------------------------- "baby_robot_shell.gr2" by ratio: lzma : 4.35:1 , 3.1 enc mbps , 59.3 dec mbps Oodle Kraken: 3.77:1 , 1.5 enc mbps , 878.3 dec mbps lzham : 3.77:1 , 1.6 enc mbps , 162.5 dec mbps zstdmax : 2.77:1 , 5.7 enc mbps , 405.7 dec mbps zlib9 : 2.19:1 , 13.9 enc mbps , 332.9 dec mbps lz4hc : 1.78:1 , 40.1 enc mbps , 2364.4 dec mbps by encode speed: lz4hc : 1.78:1 , 40.1 enc mbps , 2364.4 dec mbps zlib9 : 2.19:1 , 13.9 enc mbps , 332.9 dec mbps zstdmax : 2.77:1 , 5.7 enc mbps , 405.7 dec mbps lzma : 4.35:1 , 3.1 enc mbps , 59.3 dec mbps lzham : 3.77:1 , 1.6 enc mbps , 162.5 dec mbps Oodle Kraken: 3.77:1 , 1.5 enc mbps , 878.3 dec mbps by decode speed: lz4hc : 1.78:1 , 40.1 enc mbps , 2364.4 dec mbps Oodle Kraken: 3.77:1 , 1.5 enc mbps , 878.3 dec mbps zstdmax : 2.77:1 , 5.7 enc mbps , 405.7 dec mbps zlib9 : 2.19:1 , 13.9 enc mbps , 332.9 dec mbps lzham : 3.77:1 , 1.6 enc mbps , 162.5 dec mbps lzma : 4.35:1 , 3.1 enc mbps , 59.3 dec mbps ------------------------------------------------------- "win81" by ratio: lzma : 2.95:1 , 2.5 enc mbps , 51.9 dec mbps lzham : 2.77:1 , 1.6 enc mbps , 177.6 dec mbps Oodle Kraken: 2.70:1 , 1.0 enc mbps , 877.0 dec mbps zstdmax : 2.64:1 , 3.5 enc mbps , 417.8 dec mbps zlib9 : 2.07:1 , 16.8 enc mbps , 269.6 dec mbps lz4hc : 1.91:1 , 28.8 enc mbps , 2297.6 dec mbps by encode speed: lz4hc : 1.91:1 , 28.8 enc mbps , 2297.6 dec mbps zlib9 : 2.07:1 , 16.8 enc mbps , 269.6 dec mbps zstdmax : 2.64:1 , 3.5 enc mbps , 417.8 dec mbps lzma : 2.95:1 , 2.5 enc mbps , 51.9 dec mbps lzham : 2.77:1 , 1.6 enc mbps , 177.6 dec mbps Oodle Kraken: 2.70:1 , 1.0 enc mbps , 877.0 dec mbps by decode speed: lz4hc : 1.91:1 , 28.8 enc mbps , 2297.6 dec mbps Oodle Kraken: 2.70:1 , 1.0 enc mbps , 877.0 dec mbps zstdmax : 2.64:1 , 3.5 enc mbps , 417.8 dec mbps zlib9 : 2.07:1 , 16.8 enc mbps , 269.6 dec mbps lzham : 2.77:1 , 1.6 enc mbps , 177.6 dec mbps lzma : 2.95:1 , 2.5 enc mbps , 51.9 dec mbps ------------------------------------------------------- "enwik7" by ratio: lzma : 3.64:1 , 1.8 enc mbps , 79.5 dec mbps lzham : 3.60:1 , 1.4 enc mbps , 196.5 dec mbps zstdmax : 3.56:1 , 2.2 enc mbps , 394.6 dec mbps Oodle Kraken: 3.49:1 , 1.5 enc mbps , 789.7 dec mbps zlib9 : 2.38:1 , 22.2 enc mbps , 234.3 dec mbps lz4hc : 2.35:1 , 27.5 enc mbps , 2059.6 dec mbps by encode speed: lz4hc : 2.35:1 , 27.5 enc mbps , 2059.6 dec mbps zlib9 : 2.38:1 , 22.2 enc mbps , 234.3 dec mbps zstdmax : 3.56:1 , 2.2 enc mbps , 394.6 dec mbps lzma : 3.64:1 , 1.8 enc mbps , 79.5 dec mbps Oodle Kraken: 3.49:1 , 1.5 enc mbps , 789.7 dec mbps lzham : 3.60:1 , 1.4 enc mbps , 196.5 dec mbps by decode speed: lz4hc : 2.35:1 , 27.5 enc mbps , 2059.6 dec mbps Oodle Kraken: 3.49:1 , 1.5 enc mbps , 789.7 dec mbps zstdmax : 3.56:1 , 2.2 enc mbps , 394.6 dec mbps zlib9 : 2.38:1 , 22.2 enc mbps , 234.3 dec mbps lzham : 3.60:1 , 1.4 enc mbps , 196.5 dec mbps lzma : 3.64:1 , 1.8 enc mbps , 79.5 dec mbps ------------------------------------------------------- In chart form : [Image] (lz4 decode speed is off the top of the chart) [Image] I'm not including the other Oodle compressors here just to keep things as simple as possible. If you do want more compression than Kraken, and care about decode speed, then Oodle LZNA or BitKnit are much faster to decode than lzma (7zip) at comparable or better compression ratios. Visit radgametools.com to learn more about Kraken and Oodle.