About Image compression, streaming, the meaning of life...

[archie4oz]

Marc, since Archie was too lazy to post it

Yeah go and post the dandy english version that I just found for you instead of making him work from the hackish JIS version..

Oh and forget everything I said about RGB... My head was in subtractive color models not additive..

 

[Fafalada]

Here you go Squeak:
http://homepage.mac.com/archie4oz/.Pictures/Fafalada/akik_lum4_16.png
http://homepage.mac.com/archie4oz/.Pictures/Fafalada/akik_lum4_64.png
The images are 4.5bit/texel and 4.125bit texel respectively.

Thanks to Jpeg artifacts being recognized as luminance changes this image is a really bad scenario for this compression, and with source being a pretty pic I didn't have the heart to post the 2bit version
Temple pic fares better(I should post it a bit later), but it's not ideal either - this compression is at its best with images with lots of long color gradients and high contrast colors(like skymaps) and don't like noisy detail much.

I meant, who first came up with the idea of interleaving two CLUT images?

I suppose the closest you can get to that is to credit whoever created the first graphic chip to support bitplanes - so that would date what, 20-25 years back? Someone better familiar with history can perhaps dig up a name.

 

[Fafalada]

Okay, and for the temple pics.
http://homepage.mac.com/archie4oz/.Pictures/Fafalada/temple_4_16.png
http://homepage.mac.com/archie4oz/.Pictures/Fafalada/temple_2_16.png

Considering it's image with lots of noise detail, the 2bit image does pretty good in terms of size/quality.

 

[Guden Oden]

Thanks to Jpeg artifacts being recognized as luminance changes this image is a really bad scenario for this compression, and with source being a pretty pic I didn't have the heart to post the 2bit version

The change between the yellow fuzzy flowers in the background and her right arm (lower left corner of image) shows stair-stepping, esp. in the higher compressed version. The girl's skin - esp. lips - look a bit anemic in the higher compressed version too. It actually makes her smile look a little different.

Anyway, why not break up colorful images into smaller pieces? I mean, GS don't like large polys anyway, right? So, what happens if you butcher that girl up in like three horizontal chunks?

 

[Simon F]

Here you go Squeak:
http://homepage.mac.com/archie4oz/.Pictures/Fafalada/akik_lum4_16.png
http://homepage.mac.com/archie4oz/.Pictures/Fafalada/akik_lum4_64.png
The images are 4.5bit/texel and 4.125bit texel respectively.

.. and score RMSE values of 9.54 and 12.88 respectively....

Okay, and for the temple pics.
http://homepage.mac.com/archie4oz/.Pictures/Fafalada/temple_4_16.png
http://homepage.mac.com/archie4oz/.Pictures/Fafalada/temple_2_16.png

and get 12.72 and 18.98 (!) respectively.


Of course, RMSE is not really an ideal measure of quality ... one day I intend to research some ideas for a better, but still simple, perceived quality metric.

Well separate Luma and Chroma (with lower chroma bandwidth) has been used in colour TV transmission since day one and that is definitely realtime

If we are talking public transmissions then yes, but if we are talking colour TV in general, I think Bairds mechanical television, from the nineteen thirties had a spinning translucent disc with the primary colours, thus no separate luma-channel.

Interesting. I didn't know that. Ironic, I suppose, since I used to work for the company I believe he set up, i.e, "Cintel".

 

[Guden Oden]

Of course, RMSE is not really an ideal measure of quality

Hm, for a second I thought you wrote, "RMSE is not really a measure of quality", and I'd actually agreed with you on that. The temple pics may get huge error values in your program, but to the human eye they still look surprisingly good, especially considering the noisyness like Fafalada noted. So the 2bpp image is a bit grainy, but I only really thought about it when comparing with the image above.

one day I intend to research some ideas for a better, but still simple, perceived quality metric.

Don't really understand how you could make a computer program that measures percieved quality, as that is a subjective thing which differs from person to person. In the end, it'd just be a number anyway...

 

[Dio]

Not everything is subjective. For example, we've all agreed that the noisy background has crappy MSE after DXTC but still looks fine.

 

[Squeak]

Here you go Squeak

Thank you very much.
I think it looks very decent, considering the compression ratio, the 2bit picture is especially impressive.
Of course, when mapped to actual geometry, and depending on the tessellation of that, the bit-cost is going to increase a bit, but not by much.

It was predictable that RMS wouldn’t “like†this kind of compression, but to the human eye the artefacts and errors look more acceptable than Block Truncation errors for example.
Actually the artefacts look more “analogueâ€, a bit like a bad photograph, with graininess and “off key†colours.

Maybe the “anaemic†look could be cured by tweaking the brightness of either texture a bit, and the blockyness could probably be improved, by doctoring the colour texture manually?

I meant, who first came up with the idea of interleaving two CLUT images?

I suppose the closest you can get to that is to credit whoever created the first graphic chip to support bitplanes - so that would date what, 20-25 years back? Someone better familiar with history can perhaps dig up a name.

Well the Amiga is nineteen years old, and it had bitplanes, so the invention is probably a quite a bit older than that, because usually inventions tend to take some years from conception, untill they get used in consumer electronics.
I once read that Jay Miner got inspiration from flight simulators, for the some of the graphics modes, so maybe there?

Interesting. I didn't know that. Ironic, I suppose, since I used to work for the company I believe he set up, i.e, "Cintel".

A bit off topic, but for those it might interest: Specwise one of his experimental colour models was actually, in some ways superior to current television standards, it had 600 lines and did progressive scan.
The superiority was only on paper of course. Reaction time of the light used, was very slow so the horizontal resolution was pretty low, and it used simple back projection, on a glass plate. Among other things, that brought the quality down considerably.

Anyway, why not break up colorful images into smaller pieces? I mean, GS don't like large polys anyway, right? So, what happens if you butcher that girl up in like three horizontal chunks?

What about the seams where the bilinear lacks? You could fix that partially, by overlapping the textures by one alphablended texel on the colour map, at the expense of one CLUT entry, but the luminance map already only has 4 colours as it is, so you couldn’t do the same there.
Actually I once suggested something similar: http://www.beyond3d.com/forum/viewtopic.php?p=62480&highlight=clut#62480
(thirteen posts from the top).

 

[Simon F]

Here you go Squeak

Thank you very much.
I think it looks very decent, considering the compression ratio, the 2bit picture is especially impressive.

I thought the other 2bpp methods, eg VQ (and PVR-TC, but I can't display that) looked better (and scored better with RMS)

Of course, when mapped to actual geometry, and depending on the tessellation of that, the bit-cost is going to increase a bit, but not by much.

It was predictable that RMS wouldn’t “likeâ€￾ this kind of compression, but to the human eye the artefacts and errors look more acceptable than Block Truncation errors for example.
Actually the artefacts look more “analogueâ€￾, a bit like a bad photograph, with graininess and “off keyâ€￾ colours.

Well that's effectively the trick used by stochastic super-sampling, isn't it? Errors are still present in the image but since they are high-frequency noise (instead of low frequency) it's less annoying.

 

[Fafalada]

The change between the yellow fuzzy flowers in the background and her right arm (lower left corner of image) shows stair-stepping, esp. in the higher compressed version. The girl's skin - esp. lips - look a bit anemic in the higher compressed version too. It actually makes her smile look a little different.

The stairstepping is partially a result of using bilinear filtering during compression. I could choose a different filter strategy which would remove those but may have other sideeffects on different images.
The "anemic" is a direct result of using a lower frequency color map, which could again be remedied to an extent by adjustments to generation of the map and actual resulting map.
In other words, the current algorythm I use to compress has a lot of room for hand tuning which isn't really desirable for a general purpose image compression.
It works for me though because I use it on a very specific subset of images - skymaps.

I thought the other 2bpp methods, eg VQ (and PVR-TC, but I can't display that) looked better (and scored better with RMS)

I think the Temple image looks a bit worse in VQ, at least displayed on NTSC. Of course it depends on how you look at it, but to me the higher chroma loss in VQ is more noticeable then the more noisy luma.
Either way, images I usually target in 2bit tend to look close to or better then 8bit(avoiding banding artifacts is a big deal with skies), and running a few RMS tests with compressonator the numbers are close to 8bit/S3, so that's about all I could ask - save memory without perceptible quality loss.
I do believe with some work on compression tool, results could be improved for more general image types too, but that's outside my scope of interest (and time constraints).

Of course, when mapped to actual geometry, and depending on the tessellation of that, the bit-cost is going to increase a bit, but not by much.

The two maps share the same UV, so no, actually there's no extra overhead for that.

Maybe the ?anaemic? look could be cured by tweaking the brightness of either texture a bit, and the blockyness could probably be improved, by doctoring the colour texture manually?

As I replied to Guden, there's a bunch of tweaks that could work to improve results, but integrating that into automated compression scheme is a whole different matter.

What about the seams where the bilinear lacks? You could fix that partially, by overlapping the textures by one alphablended texel on the colour map, at the expense of one CLUT entry, but the luminance map already only has 4 colours as it is, so you couldn?t do the same there.

Splitting the texture up will only be beneficial to LUT methods if the newly created maps encapsulate smaller regions of values. If luminance is randomly distributed across the whole image, I gain nothing by cutting it up (in terms of quality) - same for regular CLUT.

 

[akira888]

Anyway, why not break up colorful images into smaller pieces? I mean, GS don't like large polys anyway, right? So, what happens if you butcher that girl up in like three horizontal chunks?

When you're creating 3D models you usually map one color texture onto a quite large number of triangles by assigning each vertex a u/v (or s/t depending on the notation) coordinate on that texture to be sent to the interpolators, and then the per-pixel coords from the interpolators are sent to the rasterization and shading unit where the texture is actually sampled. You can also generate texcoords procedurally. It's a common misconception to believe that one texture maps onto one triangle.

 

[Guden Oden]

Akira,

I know a texture can be stretched across several polys, but in this case, a portrait, you wouldn't use that texture as a character skin for example, that'd looked very weird. So if it were to appear in a game for example, it'd be in the form of a poster or a photograph or such, and then what would be the point in splitting it up in multiple polygons unless the texture itself is also split up in chunks? If you know what I mean...

On most hardware it would probably be better with large polys rather than small.

 

[akira888]

Guden:

Sorry about that, I must have misread.

edit: I now see Faf said it much better than I did.

 

[Squeak]

Here you go Squeak

Thank you very much.
I think it looks very decent, considering the compression ratio, the 2bit picture is especially impressive.

I thought the other 2bpp methods, eg VQ (and PVR-TC, but I can't display that) looked better (and scored better with RMS)

Well of course, PVR-VQ is an official TC method and technology, which has had hundreds of man-hours of labour and fine-tuning put in to it. The luminance compression method is more akin to a hack, with potential for futures optimisations. If PVR-VQ didn’t perform noticeably better, that would have been course for concern.

Of course, when mapped to actual geometry, and depending on the tessellation of that, the bit-cost is going to increase a bit, but not by much.

The two maps share the same UV, so no, actually there's no extra overhead for that.

But don’t you need to resend geometry for the second pass (with other TC schemes, you only need one pass)? Otherwise you would have invented a method of effectively applying two textures in one pass. Now that would be something!

What about the seams where the bilinear lacks? You could fix that partially, by overlapping the textures by one alphablended texel on the colour map, at the expense of one CLUT entry, but the luminance map already only has 4 colours as it is, so you couldn?t do the same there.

Splitting the texture up will only be beneficial to LUT methods if the newly created maps encapsulate smaller regions of values. If luminance is randomly distributed across the whole image, I gain nothing by cutting it up (in terms of quality) - same for regular CLUT.

I think Guden Oden was talking about dicing the textures to fit the texture buffer, and the 32 pixel polys GS likes the best? But you would probably need to do trilinear at the splits, and in that case you would thrash the buffer anyway.

 

[MfA]

Of course, RMSE is not really an ideal measure of quality ... one day I intend to research some ideas for a better, but still simple, perceived quality metric.

In the mean time take a look at SSIM, it does quite well and is computationally simple (as long as you dont go for the naive implementation). The old version of the metric has some nice demo pictures picked to point out weaknesses in MSE. His recent papers with methodology to cross compare metrics are also quite interesting.

Personally I think per pixel weighted MSE with weights determined by local luminance and variance could work quite well (for the level of complexity).

Marco

PS. if anyone is interested in an implementation of SSIM to compare images give me a shout.

 

[Fafalada]

But don?t you need to resend geometry for the second pass (with other TC schemes, you only need one pass)? Otherwise you would have invented a method of effectively applying two textures in one pass. Now that would be something!

I don't need to store extra UV anywhere, hence the memory cost is not increased in any way. If I used a texture across a lot of geometry I would uncompress it into intermediate buffer on GS instead and use a single pass to render.
Lastly, the method of applying two textures in one pass is already built into hardware - it's called trilinear filtering

I think Guden Oden was talking about dicing the textures to fit the texture buffer, and the 32 pixel polys GS likes the best? But you would probably need to do trilinear at the splits, and in that case you would thrash the buffer anyway.

If GS performance is really so critical in your application, it's easier(and usually much more efficient too) to tesselate your geometry enough so that UVs on each polygon only hit a single page.
Personally I was always under impression this issue is overblown though, as mipmapping takes care of making your texture fetching more cache friendly for wast majority of rendered stuff.

Marco,
do you really need to ask after all the stuff talked in this thread? I'm sure I'm not the only one that would like to give it a whirl, and I'd be happy to implement my own version but I won't have time for "free work" on my schedule until end of this month...

 

[Simon F]

I thought the other 2bpp methods, eg VQ (and PVR-TC, but I can't display that) looked better (and scored better with RMS)

Well of course, PVR-VQ is an official TC method and technology, which has had hundreds of man-hours of labour and fine-tuning put in to it.

VQ? Oh yes, there was a lot of time spent developing that compressor but it's pretty much dead now. The indirection required in VQ-based schemes (including simple colour palette systems) makes it unattractive for HW implementation.

As for PVR-TC, I actually spent a lot of my initial development time just making sure the HW decode was quite inexpensive. As for its compressor, probably less time to date has been spent on it than on the VQ tool, so I'm still hoping to raise the quality further, especially for the 2bpp mode.

The luminance compression method is more akin to a hack, with potential for futures optimisations.

I wasn't saying it was optimal, just commenting on its current performance. Of course separating luma and chroma in some form or another is a well known technique - lots of systems use, eg JPEG.

 

[Simon F]

Of course, RMSE is not really an ideal measure of quality ... one day I intend to research some ideas for a better, but still simple, perceived quality metric.

In the mean time take a look at SSIM, it does quite well and is computationally simple (as long as you dont go for the naive implementation). The old version of the metric has some nice demo pictures picked to point out weaknesses in MSE. His recent papers with methodology to cross compare metrics are also quite interesting.

Gosh, I don't know. The "salt and pepper" examples seem to score well but I found them rather disturbing. The blocky JPEG score seemed more reasonable, though.

Personally I think per pixel weighted MSE with weights determined by local luminance and variance could work quite well (for the level of complexity).

That sounds a bit closer to what I was considering, but I won't comment just yet in case I make a fool of myself

 

[Squeak]

Lastly, the method of applying two textures in one pass is already built into hardware - it's called trilinear filtering

Is trilinear even realistically useable on PS2, let alone on whole textures?
If it only halves fillrate (like one would guess), it should still be very useful in many situations.

 

[MfA]

Personally I think per pixel weighted MSE with weights determined by local luminance and variance could work quite well (for the level of complexity).

That sounds a bit closer to what I was considering, but I won't comment just yet in case I make a fool of myself

A very old idea I never tried out was to combine this with weighted MSE.

I have a hard time coming up with something which could capture meanshift and contrast stretching as well as SSIM which isnt an awfully lot like SSIM though. On the other hand, standard SSIM does not capture colour distortion well (for his video metric he simply weighted the chrominance planes by 1/10th each, but in MPEG coding at most bitrates colour distortion simply isnt a big deal ... Ill add some parameters to use either RGB or YUV and let you choose your own weights).

Ill put a link in here to a SSIM util tomorrow