Need Help with Divx & Xvid Encoding

Khronus

Newcomer
I've got an AIW 9600XT that I've started using recently to record tv shows and movies with. I record in the max res I can which is MPEG2 (720x480 48khz/16bit, 8mbit) and a 30 min show averages about 1.5gb. I've tried out Dr. Divx as a compression tool and it can shrink that 1.5gb file to around 380mb, then I use VirtualDub to edit out the commercials. Once its compressed and edited its about 22min long and 300mb. But I keep comparing it to my Bittorrent downloads for say Naruto which is also 22min but 170mb, or a Stargate SG1 episode (recorded from SkyOne in Xvid) which is 42min and 350mb.

Can anyone point me in the right direction, do I need different tools or what? My AIW records everything as a .mpg, so I have to use Dr Divx to convert to an .avi so that VirtualDub can even open it to edit commercials. Plus every avi file I open in VirtualDub I get an error that there's a problem with the audio, and when it fixs it the audio is now out of sync. So then I have to export the audio to a .wav file and then add it back in to fix audio sync, cut commercials and save, then recombine.....argh. Oh and if anyone knows where to find a good audio codec package download please let me know, I think I'm missing some important ones. Sorry for the long drawn out post, I'm kinda frustrated atm.
 
Short primer on the innards of encoding audio and video:

The first digital media player, CDI (Philips, the Netherlands :D ) used their recently invented CD to store movies, which gave them a bitrate of 150KB/sec to work with. (640 MB over 60 minutes.) They invented MPEG2 to make it possible to watch a movie. (And MPEG3 as well of course, both are "layers" of the original MPEG protocol.)

That bitrate is constant, and the amount of bits per frame as well. So, each frame consists of 150 / 60 fps kilobyte of information. Which is actually VERY little.

RealMedia came up with a nice idea: why use the same amount of data for each frame? Some frames require more data to look decent, while others require less. If you spread the amount of data around, you can have each frame just use what it needs and save the surplus for the first future frame that does need it.

In other words: they buffer the constant bitstream, so they can use the right amount of data for each frame. This decreased their bitrate quite a bit, while increasing the overall quality of the media significantly.

Newer compression methods, like DivX, XdiV, Ogg and Windows Media from version 6 onward use this variable bitrate. And the maximum sustained bitrate has gone up very much since the CDI, of course.

So, if you want it to look better and use less memory, don't use MPEG2.
 
Now you made it, he's prolly more confused than before :LOL:

Just record in MPEG1 format, 512x384. The quality isn't much lower than MPEG2 but a lot smaller. That would be about SVCD quality.
 
I haven't used Dr Divx for quite a while (not since version 1.0 IIRC), but I did find it quite a straightforward encoding app, albeit somewhat bloated and a bit buggy then, however as you outlined, using it as an intermediatary process is not going to give the best results. There are myriad tools and applications out there that can do what you want and a number of different ways of going about doing it. Since you've used VirtualDub, I'll outline my recommendations with it in mind.

Khronus said:
...Can anyone point me in the right direction, do I need different tools or what? My AIW records everything as a .mpg, so I have to use Dr Divx to convert to an .avi so that VirtualDub can even open it to edit commercials.
If you are comfortable using VirtualDub, then I would recommend trying VirtualDubMod which, as the name suggests, is a modification of the VirtualDub source to allow for the import of MPEG2/VOB streams, muxing more than one audio stream and other formats like ogg vorbis, as well as support for the Matroska container, amongst other things. You can also download it, as well as a host of other useful (mostly open source) tools at Doom9.net.

VirtualDubMod will allow you to open the 1.5GB MPEG2 capture and cut out the commercials (set each "in" and "out" mark to the nearest keyframe -- i.e hold "shift" & the right or left arrow key to adjust to the nearest point at which you wish to cut. Alternatively, press the "next keyframe" or "previous keyframe" close to the point you wish to mark. Once you have a selection (indicated in the timeline as a coloured bar corresponding to your in and out marks) press the delete key to cut that section. Proceed through the whole stream and delete sections/commericials as required. Once done, the stream can be filtered (cropped, resized, de-noised, etc) if necessary and encoded.

I generally use XviD with custom matrixes (depending on content) for the video encoding, although DivX with the right settings should produce decent results. You could also try VP6.2, which is great at low bitrates (<800kbps) and for Anime material. There are a number of bitrate calculators available (check Doom9) to work out the necessary video bitrate for a given encode size or you can just use a normal calculator :) With the right settings, you should be able to get fairly good encodes with 350MB for 40-42 minute programmes and 175MB for 20-22 minutes programmes.

Khronus said:
...Plus every avi file I open in VirtualDub I get an error that there's a problem with the audio, and when it fixs it the audio is now out of sync.
Yeah, that's probably because Dr Divx encoded the audio in Variable Bit Rate (VBR) mp3, which while playable in the AVI container, is essentially a hack and causes syncing issues. When prompted by VirtualDub, do not let it fix the audio sync, just adjust the framerate in VDub to "change so that video and audio durations match" and then set the audio and video to "Direct Stream Copy" after editing out the commericals.

Khronus said:
...Oh and if anyone knows where to find a good audio codec package download please let me know, I think I'm missing some important ones.
Well, there are a number of options here and while I find ogg vorbis great at low bitrates (<80kbps), for wider compatibility, the LAME mp3 codec is recommended. While you could use VBR settings (-alt.standard giving a fairly transparent encode) for simplicity I would recommend using 128kbit CBR Joint Stereo or 192kbit CBR Stereo or similar.

Another possibility if you have Nero 6, would be to download the latest update packages and give NeroVision a try. I'm not certain, but if you are able to import an MPEG2 stream and possibly cut out the commercials, then you could encode the stream with Nero's AVC/H.264 codec, which seems to give very good results and should improve over current MPEG4 ASP codecs (DivX/XviD, etc) as it matures/transitions into the market.

Cheers,


BrynS
 
Khronus said:
I've got an AIW 9600XT that I've started using recently to record tv shows and movies with. I record in the max res I can which is MPEG2 (720x480 48khz/16bit, 8mbit) and a 30 min show averages about 1.5gb. I've tried out Dr. Divx as a compression tool and it can shrink that 1.5gb file to around 380mb, then I use VirtualDub to edit out the commercials. Once its compressed and edited its about 22min long and 300mb. But I keep comparing it to my Bittorrent downloads for say Naruto which is also 22min but 170mb, or a Stargate SG1 episode (recorded from SkyOne in Xvid) which is 42min and 350mb.

Can anyone point me in the right direction, do I need different tools or what? My AIW records everything as a .mpg, so I have to use Dr Divx to convert to an .avi so that VirtualDub can even open it to edit commercials. Plus every avi file I open in VirtualDub I get an error that there's a problem with the audio, and when it fixs it the audio is now out of sync. So then I have to export the audio to a .wav file and then add it back in to fix audio sync, cut commercials and save, then recombine.....argh. Oh and if anyone knows where to find a good audio codec package download please let me know, I think I'm missing some important ones. Sorry for the long drawn out post, I'm kinda frustrated atm.


Do you have a cable box with an IEEE port? Supposedly your cable provider has to offer that option if you're paying for digital.*

Secondly: Cable box aside do you have the option to record in the TS (transport stream) format? I ask because the bitrate you are recording at/filesize suggests there are a lot of null packets in the stream.

Null packets are needed for some people who record to tape as their decks don't support a variable bitrate. They need a steady bit rate stream whether the packets carry useful information or not.

Also, some digital feeds include more than one channel. For example there could be a weather channel embedded in the stream.

There are freeware tools available for stripping the null packets and excess channels out of transport stream files. These same tools can be used to edit out commercials.


DivX: Most DivX encodes are at less than 720x480. Generally speaking, the higher the rez the more bitrate you need.

Most DivX encodes use "b-frames" which help reduce file size in two ways.

They reduce on mostly reduntant data and the quantizer level used for them is set to a maximum quality of "4" IIRC.

As to setting filesizes: What frontend/gui are you using for the encoder? You should be offered the option to set the file size plus other options.

*With IEEE you can record to your hardrive.


Stargate SG1


Ah, I did some research here. It seems that these encodes generally follow what is known as "scene rules". The scene rules dictate that a DivX encode of a typical hour long (counting commercials which are of course edited out) broadcast must be 350 megs or less in size so that two episodes can be burnt to a CDR.

Some more info on such a typical scene compliant encode.


624x352 at 23.976 fps (progressive)

An average of 970 kbps and with 2 channel lame mp3 at 156 kb/s.


Perhaps you can use these settings as a rough guideline.

Ah, here's an essential link for any enthusiast.

http://www.avsforum.com/avs-vb/index.php?s=acede53a2713d7839cb074ed14b5c64c
 
DiGuru said:
Short primer on the innards of encoding audio and video:

The first digital media player, CDI (Philips, the Netherlands :D ) used their recently invented CD to store movies, which gave them a bitrate of 150KB/sec to work with. (640 MB over 60 minutes.) They invented MPEG2 to make it possible to watch a movie. (And MPEG3 as well of course, both are "layers" of the original MPEG protocol.)

That bitrate is constant, and the amount of bits per frame as well. So, each frame consists of 150 / 60 fps kilobyte of information. Which is actually VERY little.

RealMedia came up with a nice idea: why use the same amount of data for each frame? Some frames require more data to look decent, while others require less. If you spread the amount of data around, you can have each frame just use what it needs and save the surplus for the first future frame that does need it.

In other words: they buffer the constant bitstream, so they can use the right amount of data for each frame. This decreased their bitrate quite a bit, while increasing the overall quality of the media significantly.

Newer compression methods, like DivX, XdiV, Ogg and Windows Media from version 6 onward use this variable bitrate. And the maximum sustained bitrate has gone up very much since the CDI, of course.

So, if you want it to look better and use less memory, don't use MPEG2.


I think perhaps you are referring to the VCD (MPEG-1) standard. That standard matches up with both the timeline you are referring to as well as the characteristics you mentioned.

MPEG-2 supports a variable bitrate, though it doesn't demand it.

Apologies if I'm mistaken.
 
Actually variable bitrate is not new. It exists way before VCD. Actually, an encoder needs to take care to produce a constant bitrate result. If you don't care about constant bitrate, you can just use the same quality, and the resulting bitrate will vary.

The reason why VCD uses constant bitrate is due to hardware constraints: the CD readers at that time are not good at seeking (and you can't just "pause" the motor), and you don't want to spend a lot on a buffer, so variable bitrate is out. The bitrate of VCD is designed to fit a 1x CD-ROM (75 sectors per second, 2324 bytes per sector).

A better "variable bitrate" method is to analyse the video first, and estimate the quality factor for each frames to make the average bitrate approaches a pre-assigned value. This method requires two pass, but obviously you can't do a two pass encoding in realtime. Some realtime encoders try to buffer a certain amount of video (say, 2 seconds or 60 frames) and do a two-pass variable bitrate encoding on them. Therefore, you get a constant average bitrate for every 2 seconds, but within the 2 seconds, the bitrate may vary.

Why DivX/Xvid/... are better than VCD/MPEG-2 (at least at comparable bitrate) is mostly because the new features (mostly from MPEG-4). Variable bitrate plays very little role because DVD already supports variable bitrate.
 
Babel-17 said:
I think perhaps you are referring to the VCD (MPEG-1) standard. That standard matches up with both the timeline you are referring to as well as the characteristics you mentioned.

MPEG-2 supports a variable bitrate, though it doesn't demand it.

Apologies if I'm mistaken.

You're right. CDI and VCD are MPEG-1, which is basically low bitrate / low quality. MPEG-2 is meant for broadcast, and is meant to scale from a 1.5 Mbit/sec bitrate upward, not downward. And MPEG-3 is the sound compression layer from MPEG-3, that was meant for HDTV. And MPEG-4 is the successor of MPEG-2, which should be able to scale back to low bitrates as well.

I was trying to make it into a really short story, and I messed it up a bit. Sorry about that.
 
DiGuru said:
And MPEG-3 is the sound compression layer from MPEG-3, that was meant for HDTV.

MP3 (MPEG Audio Layer 3) is from MPEG-1, not MPEG-3. MPEG-3 is canned very early (be replaced by MPEG-2). MPEG-2 added some extra features to audio (multi-channel, low bitrate and lower sample rate, etc.).
 
pcchen said:
DiGuru said:
And MPEG-3 is the sound compression layer from MPEG-3, that was meant for HDTV.

MP3 (MPEG Audio Layer 3) is from MPEG-1, not MPEG-3. MPEG-3 is canned very early (be replaced by MPEG-2). MPEG-2 added some extra features to audio (multi-channel, low bitrate and lower sample rate, etc.).

Doh! :oops:
 
Thanks for all the great info guys, I'm gonna try it tonight. The VirtualDubMod alone sounds like a godsend! Unfortunately I'm not on digital cable Babel-17, but are you telling me that if I was I could run a firewire cable from the cable box to my Shuttle and record the broadcast digitally direct from the cable box?
 
Khronus said:
Thanks for all the great info guys, I'm gonna try it tonight. The VirtualDubMod alone sounds like a godsend! Unfortunately I'm not on digital cable Babel-17, but are you telling me that if I was I could run a firewire cable from the cable box to my Shuttle and record the broadcast digitally direct from the cable box?

Is the Shuttle your PC? If so then yes, exactly.

Even HDTV digital signal signals of 1280x720 and 1920x1080. You even get the sound too. You don't need the HD box, just a digital one. You wouldn't be stealing the HD channels as they are included with your digital package. They are offered for free to those who are paying for the digital premium channels.

But no analog channels as IEEE is strictly digital.

Afaik this is 100% legal as you won't get any channels you aren't paying for and is no different than using a regular recording device.

Not every cable system offers a box that supports this function though.

Some people have a dickens of a time getting it to work, for others it takes less than an hour.

Read the following thread carefully.

http://www.avsforum.com/avs-vb/show...readid=403695&perpage=20&pagenumber=1

My 2 cents. Get a good quality IEEE cable that isn't longer than it has to be. It doesn't have to be a "Monster" cable. There are far less expensive alternatives. Use a good quality IEEE card or make sure your onboard one is good. Record to a dedicated defragmented drive. Keep other cpu/memory/data transfer intensive tasks to a minimum while recording.

Depending on the quality of the cable signal in your area and the quality of your setup you may or may not get a recording mostly free of annoying dropouts and other glitches.

A couple of things that maybe aren't stressed that well in the above thread. You can't use CapDVHS and the VLC streaming player at the same time. You'll lock your system and get a blue screen if you do.

You will want a nightly build of the VLC player and not the last stable version. The performance difference is large.

The latest nightly is usually good to go.

http://vthr.via.ecp.fr/~videolan/build/win32/

It's important to read through the thread.

You'll possibly want at least two other freeware tools.

HDTVtoMPEG2 1.11 and mpeg2repair.
 
Had a chance to play with VirtualDubMod and I have to say it rocks!! I still have audio sync issues but I found a way to fix them very easily. I set the video to Full Processing Mode, then go into Framerate and set "Change so video and audio durations match" and that fixes my audio problems.
I'm pretty sure now that its because my poor old 2ghz P4 can't compress it fast enough so I'm dropping frames. The longer the video. the more out of sync it gets.... ah well next upgrade that should go away ;)
 
About the audio sync problem, it's pretty nasty when you have analog sources.

The biggest problem is from clock mismatch. The clock on the TV card (which is synced to the broadcast signal) is different from the clock on the sound card. When you tell the sound card to record a 48kHz audio, the resulting sampling rate is usually slightly different, e.g. 48.01 kHz or 47.99 kHz. This 0.02% difference may not sound very large, but when you records a lengthy file, it will appear. Normally a 0.3 second mismatch is very obvious to a viewer. With a 0.02% difference, you only need to record for 1,500 seconds (25 minutes) to have a 0.3 second mismatch. There is a program here which can help you determine the clock skew of your sound card.

The easiest way to solve this mismatch is, just like you said, to change the frame rate so the video and audio durations match. However, if you want to maintain the frame rate to 29.97 (or 25 fps for PAL), such as for DVD authoring, you'll need to make the audio longer to match the duration of the video. I wrote a simple program for it. Some MPEG-2 encoders can adjust for it, too (changes frame rate like 29.967 fps to 29.97 fps by dropping frames or changing audio length).

Since I mostly recorded the TV show and F1 racing to DVD, my workflow is like:

1. Record the analog signals to a motion JPEG encoded file (lossless compression is too large and makes compression slow)
2. Use VirtualDub to edit the video file.
3. Change the frame rate to match video and audio in VirtualDub, and save the audio to a WAV file.
4. Use my simple program to stretch the audio to correct length (basically the number of frames divides by 29.97).
5. Change the frame rate back to 29.97 in VirtualDub, and make a frame server.
6. Use an MPEG-2 encoder to compress the video (without audio).
7. (Optional) Transcode the "stretched" audio to AC-3.
8. Author the DVD with the video and audio files.
9. Use a DVD player software to check the result.
10. Burn the result into a DVD-R.
 
THANK GOD someone mentioned it!!
I've been watching TV programs dloaded from the net, all on Divx/Xvid format, and many have synch problems. Is there ANY way to solve this, apart from the very annoying "rewind for a second" option to bring everything back to normal?
 
If your cpu is pushed near to its limit you can experience synch issues. Found that out messing around with some very taxing ffdshow/Avisynth filters. Once I got much above 90% cpu useage synch was shot.

I'm going to guess that players try to compensate for sloppy encodes and that compensation is the first thing to get dropped when the framerate gets compromised by high cpu useage.


The VLC player nightly builds use less cpu cycles for mp4 (DivX, Xvid, others) files than anything else I've ever used. Try the latest nightly with one of your problem files.

Even if the issue isn't cpu related it can be a good idea trying this as the VLC player nightlies run from the folder you extract them to and contain all the libraries it needs. So you eliminate system configuration issues, pretty much, when you test a file with it.
 
Sorry to bump an old thread but I have an update on my situation. I ended up creating a custom recording profile in the ATI MMC tv software that allows me to record it in MPEG4 and saves them in AVI format. I'm encoding at 640x480 and now a 1hr tv show is 1.9GB recorded. I open it up in VirtualDubMod and since it's a native mpeg4 avi I no longer get the audio sync problems. So I just edit my commercials and then do a fast recompress and the now commercial-free 42 min tv show is 1.33GB

The only problem I have left is my compressed files are larger than I've set in the Xvid Target Size. I set it to 358400 (350MB) but the smallest I've ever gotten was 480MB, turning on Quarter Pixel only increased the size by 19MB. The following url is what I used to configure my Xvids options so they match up with the suggested settings at the bottom of the page titled "17 I'm a newbie. What settings do you recommend?"
http://www.gromkov.com/faq/conversion/xvid_options.html
Should I try out GordianKnot, keep using VirtualDubMod, any suggestions?
 
Back
Top