Max compositing performance, movie files or numbered stills?

[Mistyk]

Hi,

For compositing work, is it preferable to have the clips stored as continuous movie files (like avi) or as numbered stills? The reason for asking is that a continous large movie file would likely benefit more from the increased STR of a stripe-set, than lots of small files would. Is this correct? The clips are non-compressed PAL with an alpha channel as well as deep rasters, so they're quite heavy.

Can many small files benifit from high STR as long as they are sequentially stored on disk? If so, would it be necessary to defrag extremely often to keep them grouped? What am thinking, I guess that no single defragger is capable of organising numbered files? Does this mean that continuous movie files are the only way to go for maximum performance?

Then again perhaps compositing in general does not benefit much if at all from high STR, since even if continuous movie files are used, many of them will be retrieved in parallel. Therefore the drive heads would have to go all over the place to find all of the files and the streams would repeatedly be interrupted. Thus you would need one stripe-set per clip to keep the reads sequential, which would be ridiculous. Am I totally off here?

With this reasoning, maybe it doesn't matter if you store the clips as stills or movie files. The only thing that would matter then is low latency/seek/whatever. Please tell me what you think.

Thank you in advance!

 

[Simon F]

If you want the stills "uncompressed" to maintain quality, take a look at huffyuv, which is a lossless, compressed AVI format/codec (possibly PC only). It typically gets around 50% compression.

 

[Mistyk]

Thank you Simon F.

I'm leaning towards using numbered stills for compositing and, when that's done, save the final composited output as a single uncompressed movie file for the sake of playback speed.

I'm also investigating the possibility to use "Windows Compressed Folders" to keep the numbered stills tightly grouped. This way the seek times should be lowered, since the drive heads don't have to look for the files across the entire disk array. Credit goes to a StorageReview forum member, who shared the clever idea of using compressed folders.

Please comment on the approach whether you think it's reasonable or crap.

 

[Simon F]

Thank you Simon F.

I'm leaning towards using numbered stills for compositing and, when that's done, save the final composited output as a single uncompressed movie file for the sake of playback speed.

I'd strongly recommend you look at Huffyuv if you're going to do the latter. The decompression (and compression for that matter) is realtime for video resolution on anything >celeron 400, whereas you are more likely to hit bottlenecks due to disk I/O.

 

[Mistyk]

Sounds awfully tempting, but what worries me a tad is that Huffyuv is lossy in the sense that the RGB video has to be converted to YUV (or some variant thereof). The video is initially RGB since it is not captured, but created within the realms of the computer. Not sure what part of the RGB->YUV conversion that's actually lossy, but it would be nice to have the orignal video completely intact. That way one could it as a template for any further compression/conversion/compositing or transfer to tape/CD/DVD.

On the one hand the author of Huffyuv claims that RGB video would have to be converted to YUV anyway if it's destined to end up as mpeg2 or other lossy codecs. On the other hand, perhaps there are other formats available now or in the future, which offer greater quality than today's lossy formats, and that demands RGB video. Then one would have to convert the YUV video (which was originally RGB) to RGB once again. Since one such compression is lossy, it would seem a little bad to do two of them. Especially as you'll end up with what you had from the beginning (RGB), though slightly degraded quality-wise.

Maybe I'm being silly. Am I?

 

[Simon F]

You don't have to convert to YUV - it has a completely lossless RGB mode as well.

 

[Mistyk]

You don't have to convert to YUV - it has a completely lossless RGB mode as well.

How on earth did I miss that? Thank you for pointing it out! It said on the site that it was possible and advisable to convert the RGB video prior to compressing it, so I mistakenly thought that it was necessary.

Have been investigating Huffyuv and its RGB compression and it looks good thus far. Haven't been able to try out the NTFS folder compression and its performance advantages as of yet, due to the lack of an NTFS formatted harddrive.

Is Virtualdub the best/only player/converter in the field or are there others, which are better or similar? When any file is loaded in Vdub, nothing can be seen except for the grey background and two grey frames around the areas where the loaded image/video files ought to show up. One has to press the rewind button or play button for the image/video to appear. Is this normal?