Can you work out some sort of deal where you give the Oodle paging software away for free for non-commercial/indie use, and then charge $ for commercial games making a certain amount of money? That would give people a chance to see how much better their game and dev pipeline ran with it, and also serve as an initial experience using a RAD product for many programmers.
"Unfortunately, while compression is very interesting theoretically, make a compressor that's 5% better than an alternative is just not that compelling in terms of the end result that it has."
Just sharing experience here. I'm frequently requested to create some "customized" compression algorithms for specific data sets.
As a matter of fact, such specialized algorithm are way better than "10% more". I'm typically something like "better than LZ4 HC ratio at LZ4 Fast speed", or something along these lines. And, most of the time, it's possible.
The key thing here is the word "specialized". Such algorithm will fare badly on any other data set, because assumptions are hardwired. On the other hand knowing what to expect, on the other hand, gives a great advantage, which translates in speed, memory usage, and efficiency, which cannot be reached by "general usage" algorithms.
This is the kind of need for which I expect there is still a market for compression developers.
December 14, 2013 at 5:04 AM
When I started work on Oodle, I specifically didn't want to do a compression library. For one thing, I had done a lot of compression
and don't like to repeat the same work, I need new topics and things to learning. For another, I doubted that we could sell a compression
library; selling compression is notoriously hard, because there's so much good free stuff out there, and even if you do something amazing
it will only be 10% better than the free stuff due to the diminishing returns asymptote; customers also don't understand things like space-speed
tradeoffs. But most of all I just didn't think that a compression library solved important problems in games. Any time I do work I don't
like to go off into esoteric perfectionism that isn't really helping much, I like to actually attack the important problems.
That's why Oodle was originally a paging / packaging / streaming / data management product. I thought that we had some good work on that at
Oddworld and it seemed natural to take those concepts and clean them up and sell that to the masses. It also attacks what I consider to be
very important problems in games.
Unfortunately it became pretty clear that nobody would buy a paging product. Game companies are convinced that they "already have" that, or
that they can roll it themselves easily (in fact they don't have that, and it's not easy). On the other hand we increasingly saw interest in
a compression library, so that's the direction we went.
(I don't mean to imply that the clients are entirely at fault for wanting the wrong thing; it's sort of just inherently problematic to sell a
paging library, because it's too fundamental to the game engine. It's something you want to write yourself and have full control over.
Really the conception of Oodle was problematic from the beginning, because the ideal RAD product is a very narrow API that can be added at the
last second and does something that is not too tied to the fundamental operation of the game, and also that game coders don't want to deal
with writing themselves)
The two big problems that I wanted to address with Original Oodle was -
1. Ridiculous game level load times.
2. Ridiculous artist process ; long bake times ; no real previews, etc.
These are very different problems - one is the game runtime, one is in the dev build and tools, but they can actually be solved at the same
time by the same system.
Oodle was designed to offer per-resource paging; async IO and loading, background decompression. Resources could be grouped into bundles;
the same resource might go into several bundles to minimize loads and seeks. Resources could be stored in various levels of "optimization"
and the system would try to load the most-optimized. Oodle would store hashes and mod times so that old/incompatible data wouldn't be loaded.
By checking times and hashes you can do a minimal incremental rebuild of only the bundles that need to be changed.
The same paging system can be used for hot-loading, you just page out the old version and page in the new one - boom, hot loaded content.
The same system can provide fast in-game previews. You just load an existing compiled/optimized level, and then page in a replacement of
the individual resource that the artist is working on.
The standard way to use such a system is that you still have a nightly content build that makes the super-optimized
bundles of the latest content, but then throughout the day you can make instant changes to any of the content, and
the newer versions are automatically loaded instead of the nightly version. It means that you're still loading optimized
bakes for 90% of the content (thus load times and such aren't badly affected) but you get the latest all day long.
And if the nightly bake ever fails, you don't stop the studio, people just keep working and still see all the latest,
it just isn't all fully baked.
These are important problems, and I still get passionate about them (aka enraged) when I see how awful the resource pipeline is at most game companies.
(I kept trying to add features to the paging product to make it something that people would buy; I would talk to devs and say "what does it
need to do for you to license it", and everybody would say something different (and even if it had that feature they wouldn't actually license it).
That was a bad road to go down; it would have led to huge feature bloat, been impossible to maintain, and made a product that wasn't as lean
and focused as it should be; customers don't know what they want, don't listen to them!)
Unfortunately, while compression is very interesting theoretically, make a compressor that's 5% better than an alternative is just not that
compelling in terms of the end result that it has.
"11-25-13 - Oodle and the real problems in games"
2 Comments -
Can you work out some sort of deal where you give the Oodle paging software away for free for non-commercial/indie use, and then charge $ for commercial games making a certain amount of money? That would give people a chance to see how much better their game and dev pipeline ran with it, and also serve as an initial experience using a RAD product for many programmers.
December 12, 2013 at 2:52 PM
"Unfortunately, while compression is very interesting theoretically, make a compressor that's 5% better than an alternative is just not that compelling in terms of the end result that it has."
Just sharing experience here.
I'm frequently requested to create some "customized" compression algorithms for specific data sets.
As a matter of fact, such specialized algorithm are way better than "10% more". I'm typically something like "better than LZ4 HC ratio at LZ4 Fast speed", or something along these lines. And, most of the time, it's possible.
The key thing here is the word "specialized". Such algorithm will fare badly on any other data set, because assumptions are hardwired. On the other hand knowing what to expect, on the other hand, gives a great advantage, which translates in speed, memory usage, and efficiency, which cannot be reached by "general usage" algorithms.
This is the kind of need for which I expect there is still a market for compression developers.
December 14, 2013 at 5:04 AM