Speaker A
Hello, it's great to be here. Right, let's get started. So, I'm going to talk about OpenVDB and some of the changes that we've made to the file format.
Overview of OpenVDB file format evolution, new IO codec framework, and performance improvements for future-proof volumetric data handling.
Ask about this video. Answers come from its transcript only — with the timestamp, so you can check them.
Generated from the transcript and can be wrong — check the timestamp.
Key Takeaways
- OpenVDB file format needed modernization to keep up with advances in hardware and storage.
- Introducing a new IO codec framework enables better compression and faster data access.
- Zstandard is the preferred compression method for the updated OpenVDB format.
- Understanding storage tiers (cold, warm, hot) is crucial for optimizing volumetric data IO.
- Future development will focus on integration with NanoVDB and enhanced tooling.
What the video covers
- OpenVDB is a sparse hierarchical volumetric data structure with a rich toolset and IO handling, originally open sourced by DreamWorks in 2012.
- The file format has remained mostly unchanged since 2014, prompting a major update to keep pace with modern hardware and storage technologies.
- The presentation discusses three storage tiers: cold, warm, and hot, and their impact on IO performance and data loading times.
- Key design considerations include OS paging, memory mapping, IO-bound vs compute-bound operations, and trade-offs in compression and decompression.
- Comparison with other open source formats like OpenEXR, Alembic, and USD highlights different approaches to compression and parallel performance.
- OpenVDB’s previous object-oriented IO design had limitations, leading to the introduction of a new VDB IO codec framework for improved efficiency.
- Zstandard compression is recommended as a best-in-class fast lossless codec, replacing older methods like LZ4 for better performance.
- Future plans include direct reading of new VDB files into NanoVDB, vectorization, mutable grids, and ongoing community collaboration.
- The talk emphasizes the importance of breaking changes only once per decade to future-proof the format without frequent disruptions.
- Performance benchmarks illustrate the impact of storage tiers and compression on loading times for large volumetric datasets.
Chapters
- 00:00Introduction to OpenVDB and file format changes
- 01:08History and need for file format update
- 02:08Design considerations for file formats
- 03:07Storage tiers and their impact on IO
- 04:20OS paging, prefetching, and memory mapping
- 08:25Comparison with other open source formats
- 11:37OpenVDB IO design and new codec framework
- 16:26Future work and summary
Full Transcript — Download SRT & Markdown
Speaker A
going to go into too much detail here. It's a sparse hierarchical volumetric data structure. It has a rich set of tools and it handles IO.
Speaker A
Okay, so VDB's already been talked about a little bit, so I'm not going to go into too much detail here. It's a sparse hierarchical volumetric data structure. It has a rich set of tools and it handles IO.
Speaker A
So, the file format has essentially been unchanged for more than a decade now. We haven't really made any substantial changes to it since 2014 and what's become apparent is that it's fallen behind compared to some of the more
Speaker A
It was first open sourced by DreamWorks back in 2012 and as has been mentioned, it was the first project to be accepted into the ASWF in 2018.
Speaker A
something that we do lightly. It's not something that we plan to do regularly, but once every sort of 10 years or so, we feel like it's justified to really make a change if it's going to set up the format better for the future.
Speaker A
So, the file format has essentially been unchanged for more than a decade now. We haven't really made any substantial changes to it since 2014 and what's become apparent is that it's fallen behind compared to some of the more
Speaker A
different hardware techniques and how things kind of work. So, this is very simplified, but essentially I've divided this into three different tiers: cold, warm, and hot storage tiers. So, if you're using a shared server like NFS, Isilon, NetApp, something like that, you
Speaker A
cutting-edge techniques and technologies that are available. We've tried to work around some of the limitations to an extent, but we took the difficult decision recently to make a breaking change to the file format. So, this is not
Speaker A
Usually, these servers have a sort of faster cache in front of that. Maybe you get 500 megabits per second in that kind of warm tier, which is roughly similar to having an SSD on your local workstation. And then finally, you have
Speaker A
something that we do lightly. It's not something that we plan to do regularly, but once every sort of 10 years or so, we feel like it's justified to really make a change if it's going to set up the format better for the future.
Speaker A
at the bottom here we've got some example times of if if you were to load a gigabyte file off disk, sort of 20 seconds, 2 seconds, and then sort of a fraction of a second depending on which of those tiers of storage you kind of
Speaker A
So, before we get started, I'm going to go through some of the sort of parameters and the things we have to think about when designing different file formats. So, this is kind of a little bit of a summary of
Speaker A
dominates over everything else. But, if we're also doing heap allocation where we're sort of decompressing data, uh that's also a consideration. Uh and if you look at the hot storage, the hot tier there, uh you can see that read is
Speaker A
different hardware techniques and how things kind of work. So, this is very simplified, but essentially I've divided this into three different tiers: cold, warm, and hot storage tiers. So, if you're using a shared server like NFS, Isilon, NetApp, something like that, you
Speaker A
Okay, there's also OS paging and prefetching. So, the typical size of an OS page is 4 kilobytes. If you retrieve a single byte, that will load the whole page into the page cache.
Speaker A
might get around 50 megabits per second to kind of get the data off disk.
Speaker A
If you were to load 2 megabytes from that 1 gigabyte file, let's put 1 megabyte at the beginning and 1 at the end, and then you seek over the rest of the file, it it loads in a fraction of a
Speaker A
Usually, these servers have a sort of faster cache in front of that. Maybe you get 500 megabits per second in that kind of warm tier, which is roughly similar to having an SSD on your local workstation. And then finally, you have
Speaker A
bytes as it would do if you read the entire file. Another technique to be aware of is memory mapping. So, uh essentially you still pay the same cost in bringing data from the shared server over to your local workstation on a page cache. Um
Speaker A
the hot tier, which is the page cache, which is essentially a region of your system memory that can be used for storing data. If you are lucky enough to have an NVMe drive as well, that's also super fast as well. And then
Speaker A
the page cache rather than allocating new memory for it. And there's different trade-offs involved there. One of the key things to note is that you can't really compress the data in your page cache because you would need to
Speaker A
at the bottom here we've got some example times of if you were to load a gigabyte file off disk, sort of 20 seconds, 2 seconds, and then sort of a fraction of a second depending on which of those tiers of storage you kind of
Speaker A
other open source projects have made. We're obviously building on some of the decisions that people have made and looking at other projects for inspiration here. So it's interesting to learn about what tradeoffs they made and why.
Speaker A
happen to land on. Uh and the other thing to think about is IO bound versus compute bound. So, on the sort of the cold storage, you can see that by far the biggest component of that is the read times that sort of
Speaker A
of our studio storage is EXR data, but in particular I wanted to highlight their use of codecs to manage disk space cost. So a whole bunch of different codecs have been released over the over the last 25 years. Some lossless, some lossy, and
Speaker A
dominates over everything else. But, if we're also doing heap allocation where we're sort of decompressing data, uh that's also a consideration. Uh and if you look at the hot storage, the hot tier there, uh you can see that read is
Speaker A
There's also Alembic. Alembic, the initial implementation was HDF5, but performance was too slow particularly in a parallel context. And so Agar was was introduced to resolve the the read locking performance problems to massively improve the the performance of
Speaker A
actually the minority of the time. It's actually a small portion. So, we want to be mindful of different types of storage and to be fast in all different uh all different cases.
Speaker A
It uses zero copy memory mapping that we talked about and as a result there's no data compression.
Speaker A
Okay, there's also OS paging and prefetching. So, the typical size of an OS page is 4 kilobytes. If you retrieve a single byte, that will load the whole page into the page cache.
Speaker A
this. So bulk float data is typically stored using zero copy memory mapping which is the same technique that Alembic uses.
Speaker A
Uh and there's a little example here. If you were to load, say, a 1 gigabyte file, maybe that takes you 20 seconds.
Speaker A
And I wanted to sort of use this recompression test as a way of seeing how compressed are these file formats in reality. So, this is a very simple example. It's 25 million polygons, a little sort of deformed patch of ocean
Speaker A
If you were to load 2 megabytes from that 1 gigabyte file, let's put 1 megabyte at the beginning and 1 at the end, and then you seek over the rest of the file, it loads in a fraction of a
Speaker A
source file is 70% larger than than using this kind of compression. So, there's quite a big gap between those two. Uh and that's primarily because the float positions are stored uncompressed in the file.
Speaker A
second. However, if you take that same 2 megabytes and you put 1 byte and spread it through the file 2 kilobyte intervals, each of those bytes lands in an OS page. So, essentially it takes the same amount of time to read those two
Speaker A
So, each node handles its own IO. It also uses uh memory mapping as well.
Speaker A
bytes as it would do if you read the entire file. Another technique to be aware of is memory mapping. So, uh essentially you still pay the same cost in bringing data from the shared server over to your local workstation on a page cache. Um
Speaker A
Uh this results in some slower performance because of the use of atomics to try and reduce uh race conditions.
Speaker A
but, you might want to maybe say heap allocate some memory uh to copy and optionally decompress the data as you read it. Um but, you can also do uh a technique called zero copy read, which is where you read the data directly from
Speaker A
Uh back in 2014 with VDB 3, we introduced Blosc and LZ4 to improve the compression ratio and get faster read and write performance. And that's become sort of the ubiquitous standard that everyone's used so since then.
Speaker A
the page cache rather than allocating new memory for it. And there's different trade-offs involved there. One of the key things to note is that you can't really compress the data in your page cache because you would need to
Speaker A
We're so essentially removing the whole of the tree IO. We're removing delay loading and memory mapping. And And introducing this concept where VDBs can be written and read by multiple different codecs. And so, VDB will provide a number of these codecs, but
Speaker A
decompress it every time you access that data if you wanted to do that. So then this being kind of an open source forum, I thought it would be good to look at some of the decisions that some of the
Speaker A
There's also a number of other ways that we've improved the file format, for example, adding the ability to be able to interleave topology and buffers for for the same VDB grid.
Speaker A
other open source projects have made. We're obviously building on some of the decisions that people have made and looking at other projects for inspiration here. So it's interesting to learn about what tradeoffs they made and why.
Speaker A
you want it represented as a half grid in memory, you can At the moment, you have to read it as a float grid, and then convert it. Now, you can do that conversion on the fly as it's being
Speaker A
Uh so first off is OpenEXR obviously started in the late 1990s at ILM and was the first project to be kind of open source within the VFX industry. Sort of set the model as to how to kind of open source. Um lots
Speaker A
involves a dictionary slider win sliding window matching phase and an entropy coding stage. And let's look at some of these algorithms that are involved.
Speaker A
of our studio storage is EXR data, but in particular I wanted to highlight their use of codecs to manage disk space cost. So a whole bunch of different codecs have been released over the over the last 25 years. Some lossless, some lossy, and
Speaker A
Uh and as you can see here, this sort of section of these uh the first section of the orange uh Z characters being kind of compressed. Basically, what happens is that there's a small sliding window, you look back and try and find cases where
Speaker A
this was a big inspiration for some of the changes that we wanted to make to to BDB.
Speaker A
first section and kind of unpacking something that's already been used. And almost all of the lossless algorithms, as I mentioned, is uh is derived from from this kind of foundation.
Speaker A
There's also Alembic. Alembic, the initial implementation was HDF5, but performance was too slow particularly in a parallel context. And so Agar was introduced to resolve the read locking performance problems to massively improve the performance of
Speaker A
you simply just kind of compress them down so that you can, uh, you can store them with fewer characters.
Speaker A
the read times. Um Alembic is very efficient for instancing and sequence archives in particular where you can deduplicate data.
Speaker A
tree so that you have longer prefixes, longer codes for the for the less frequent data, and then shorter ones for the higher higher frequency data.
Speaker A
It uses zero copy memory mapping that we talked about and as a result there's no data compression.
Speaker A
matching LZ77 variant and range coding. Uh, LZ4 is interesting because it has no entropy encoding. It's purely about performance, so they sort of strip that bit out, and it's just pure LZ77. And then more recently, Zstandard was introduced, uh, which adds a new type of
Speaker A
And then of course there's USD. So USD has a number of different formats. The USDC is a binary format. The C here stands for crate, not compressed. Um USDZ is an uncompressed zip archive and how USD thinks about data is like
Speaker A
choice because it offers really good compression speed, really good decompression speeds, and very competitive compression ratios as well.
Speaker A
this. So bulk float data is typically stored using zero copy memory mapping which is the same technique that Alembic uses.
Speaker A
compressible as it is. However, if you were to do a delta transformation, the difference between each number and the previous one, uh, you can you can compress that data much more effectively than the than the original.
Speaker A
Uh but integer data and float as integer data, which is float data that can be precisely represented as integers, is stored using LZ4 compression.
Speaker A
quite simple, so I'm going to run through this algorithm. Uh so this is what a floating point value would look like. One bit of sign, eight bits exponent, 23 bits of mantissa, and then this is how it breaks up into these
Speaker A
And I wanted to sort of use this recompression test as a way of seeing how compressed are these file formats in reality. So, this is a very simple example. It's 25 million polygons, a little sort of deformed patch of ocean
Speaker A
Now if we compare three similar floating point values, what you can see is that those first two bytes, they barely change in terms of the actual values.
Speaker A
here. And what I'm doing is I'm compressing it with byte shuffle and LZMA and seeing what the resulting file format disk footprint is. So, we can see how compressed they are. And you can see that even in the USD case that the
Speaker A
exposes more opportunity to make use of the lossless compression algorithms. So what we've done, we have introduced a new lossless codec for VDB. Uh one of the big changes is that we're actually compressing topology, which was previously uncompressed. We've stripped
Speaker A
source file is 70% larger than using this kind of compression. So, there's quite a big gap between those two. Uh and that's primarily because the float positions are stored uncompressed in the file.
Speaker A
the buffer section as well by using a larger block size and switching from LZ4 to Zstandard compression.
Speaker A
So, onto VDB. So, up until now, VDB has used an object-oriented IO design.
Speaker A
doing IO in the background whilst we're also doing compute. Uh in this in this graph here, I've tried to represent kind of what what this might look like. This is on the on the Y axis here, you have
Speaker A
So, each node handles its own IO. It also uses memory mapping as well.
Speaker A
then caps out at 250 million voxels per second. This new algorithm we've been able to significantly increase that cap. So, now we can process as much as 900 million voxels per second, and this effectively shifts the ratio between IO bound and
Speaker A
But, it's used slightly differently. It's used for delay loading, not so much for zero copy, which means that we still load the data off disk as it's needed.
Speaker A
Another thing that we've done is rather using Blosc to perform our byte shuffling, we're actually doing that natively as part of the format itself.
Speaker A
Uh this results in some slower performance because of the use of atomics to try and reduce race conditions.
Speaker A
the byte zero and byte one, that is effectively the BFloat16 representation. So, it's a smaller kind of representation. It doesn't have as much precision as the as the full floating point. We can read that data really quickly to give us a preview of what the
Speaker A
Um not that atomics themselves, but the fact that it limits the opportunity to use inlining and auto vectorization as well.
Speaker A
entire VDB. So, that's that's another interesting benefit that we've been able to introduce. And then let's look at the recompression test. The same thing again. We are just we're taking this Disney data set uh 1.5 billion voxels, and we're recompressing
Speaker A
Uh back in 2014 with VDB 3, we introduced Blosc and LZ4 to improve the compression ratio and get faster read and write performance. And that's become sort of the ubiquitous standard that everyone's used so since then.
Speaker A
VDB, and when we've recompressed it, it's less than 2% larger than this heavily compressed version of it. So, that's our target. That's what we want to a- aim towards.
Speaker A
So, what are we changing? What are we doing with the new VDB? We are introducing a VDB IO codec framework.
Speaker A
We're intending to do a 13.1 release soon where we're adding this new codec IO framework. We've been using it at ILM for a number of months now.
Speaker A
We're so essentially removing the whole of th
Speaker A
looking at uh VDB 14 uh and beyond to add lossy codecs as well. And a quick note on backwards compatibility. We've added backwards compatible lossless codecs. So even once we've transitioned to this new this new codec API, uh we'll be able to read VDBs
Speaker A
that were written right back to VDB 1. Uh we'll be able to write VDBs that are still readable with older readers as well. Um but of course with the new lossless codecs, these will only produce VDBs that are readable from VDB 14
Speaker A
onwards. So at that point, um we'll need people to upgrade their readers. So this will remain opt-in for the foreseeable future. And when enough readers have been upgraded, uh we'll switch this to being opt-out.
Speaker A
Uh so then just to start kind of wrapping up here, future work. Uh one of the things we want to do is want we want to read new VDBs directly into NanoVDB.
Speaker A
Slightly changing how the topology and the buffer data is is organized means that we no longer need to read VDBs into OpenVDB and then convert them to NanoVDB. We can we can kind of read them directly.
Speaker A
Uh we also want to improve the tooling around selective loading and some of the Houdini tools as well. Um and we want to look at improving our lossy compression uh codecs. We've been experimenting with discrete cosine transform. It's looking
Speaker A
very promising. Uh we'd like to explore adding some of these things as well. But this where I said that the the lossless compression field is relatively small.
Speaker A
There's only so much work that you can kind of do in the space of lossless compression. Lossy is wide open. So there's a whole bunch of different applications, real-time, targeted level set compression, neural compression, a whole bunch of stuff that can be done
Speaker A
there and that's where we sort of invite the community to help explore some of these areas that maybe other people care more about than than we do.
Speaker A
So, just kind of in summary, what I wanted to highlight from this, Zstandard is the best in class for fast lossless compression. It is worth upgrading from LZ4 to Zstandard to get the benefits of this this new format. The other thing
Speaker A
I'll mention, floating point exponents are heavily compressible. Let's please stop writing them to file formats uncompressed so that we can reduce the size of our files. And I don't know about other studios, but certainly ILM, we are almost always at
Speaker A
the limit of our file servers and how much we can fit in on on on our desks and how much our disks are taking. So, anything we can do in this space is super important.
Speaker A
And then finally, the most exciting part of this whole project for me is being able to expose this codec framework so that other people can write codecs and contribute them to our library instead of developing competing file formats. So,
Speaker A
this is really an invitation to other people to contribute ideas to try and improve on some of the existing formats that that we're putting out, existing codecs that we're putting out and see if we can we can make this better together.
Speaker A
And then finally, just to wrap up, we have an OpenVDB Birds of a Feather tomorrow. This is just one small part of what's been happening in the world of VDB over the last year or so. We're also going to be talking about SIMD
Speaker A
vectorization, NanoVDB mutable grids, FVDB updates, improvements to the VDB tool, and of course a lot of discussion and Q&A from the community as well.
Speaker A
Thank you very much.
Topics:OpenVDBvolumetric datafile formatcompressionZstandardmemory mappingIO performanceNanoVDBvisual effectsAcademy Software Foundation
