TMPEG 2-pass engine. A key to faster CQ prediction.
 

Hi all,

First of all, **** THIS IS EXPERIMENTAL (and not for newbies!) ***
What we are about to do, is to use TMPEG's 2-pass engine as a "helper" for CQ encoding.
Today, I said to myself: If TMPEG can properly calculate the final file size of a movie by doing 2 passes, then why can't I do a small 2-pass (20 seconds or so) encode, which will have the correct average bitrate, and use the encoded file size as a reference footprint to encode the same clip in CQ mode, and tune the parameters until I achieve the same footprint :!:
Then, in theory, the full encode (in CQ mode) should be identical in final file size.
So let the game begin :)

Load your source into CalcuMatic, and get your average bitrate as usual for the movie to be processed.
Now follow this steps:

(1) Encode about 20 seconds or so (selected source range) of a mixed high action/normal action clip, anywhere in your movie where you have some of the highest visible action, as 2-pass VBR with MIN bitrate 300 and MAX=Your Max (2,500 for me), and the average bitrate that CalcuMatic gave you.
(2) With Bitrate Viewer, open the encoded sample and note the PEAK bitrate and the file size.
(3) Change to CQ mode, and set the CQ value to 100.
(4) Encode the same source range, and start lowering the MAX bitrate until the PEAK bitrate is as close as possible to the one recorded on the 2-pass.
(5) Now encode the same clip, and start lowering CQ value until you get as close to the same file size as the one created with the 2-pass encode.
(6) Once you get a close file size, you've found your CQ, and now you set back your MAX bitrate to your usual 2,000Kbps or 2,500Kbps, and encode your movie with the CQ found.

Your final file size should never exceed the maximum calculated by the average bitrate.
When we found the CQ, we return the MAX bitrate to the high value we normally use, thus increasing the distance of MAX bitrate from CQ value. This will now expand the dynamic range of the encoder, and give you lower file size on average scenes and higher bitrate peaks on high action scenes.
But the final file size should never exceed the calculated file size, because we calculated a "worse case" scenario on a high action scenes.

Note: If your movie's highest action doesn't go too much above the calculated MAX during the 2-pass run, you might want to leave the MAX bitrate to the value found on step 4. This will guarantee that the final file size will be right on target, just like if a 2-pass VBR encode had been done. But you will still have better quality with the CQ encode, because of the CQ bitrate distribution.

If this method proves to be correct, after some more tweakings, then the next version of CQMatic will do this automatically. So bye bye to long CQMatic loops trying to find the correct CQ value. Let's use the 2-pass encoding engine that TMPEG provides ( in a way it was never designed :!: ), and we should have a correct CQ value in less than 10 minutes :D
Please let me know your results :cool:

-kwag

Speed is always good, like they say, Time is money.

Sounds good, Kwag.
I'll do a test this evening (PAL :roll: ).

Fine tunning
 

Finished my first encode ( Red Planet at 704x480 ), with a file size of ~50MB lower than wanted ( I did expect that, but quality is still excelent ).
Now I'm compensating by using a MIN bitrate of 25% lower than average (not 300Kbps). This will push the CQ curve up, and increase the average bitrate on low lighted/low action scenes.
Currently encoding K19 at 352x240 ( so I can see the results faster )
Then I'll encode Red Planet again, with the corrections, to see if the results are consistent.

-kwag

Funny :wink:
Yesterday I had some of the same thoughs....
Videoencoding improvments????

But your thoughts were 12 hours after I posted. Look at the time stamp ;)
But I read your post, and it's not the same thing :)

-kwag

Finished encode of K19 at 352x240.
Wanted size: 703,106KB
Final Encoded: 687,563KB
Things are looking very good :)
Currently encoding "The Bourne Identity" at 528x480.
4 hours to go.....

-kwag

I'd like to know if there will be a viewable quality advantage by doing this in comparison to our CQmatic way.
8O :D

Quality is the same, because we're still encoding in CQ mode :D

-kwag

More final tweaks.
 

"The Bourne Identity" at 528x480 done.
Wanted size: 719,786KB
Encoded: 745,296KB

3.4% over. Now to do some minor adjustments with MIN bitrate.
I used 25% under average for MIN. I'll now test with 30% under average for MIN ( for safety margin ).
Now on to encode "Red Planet" at 704x480......

-kwag

Hi,

I tried that on "the beauty and the beast", with those values:
2 pass VBR :
- res 704x576
- min: 300
- max: 1800
- Aver.: 1089
And it gave me a 1174 peak (and this is the highest motion scene, as I got macroblock in my first encode just right there).
Filesize: 3909Kb.

Swith to CQ, and I had to lower the max bitrate until 1200 to get this max. bitrate value (with 1800, it give me a peak of 1600).

The CQ I encode this movie before was 56,6, but with a 1200 max. bitrate value, the 60 CQ value give me a 3605 Kb file, which is a way smaller than the reference sample.

I'll try again to see where is the error...

Fabrice

Hi Fabrice,

After you get the same MAX bitrate in CQ as you got in the 2-pass run, then you start to lower the cq value until you get the same file size that you got in the 2-pass. After you get that, you set your MAX again up to your 1,800Kbps and encode your movie.

-kwag

Re: TMPEG 2-pass engine. A key to faster CQ prediction.
 

Hi Kwag,

i'm having some adjust problems here.
Some infos:
Movie k-pax (Source is KDVD), 20 seconds Source Range, average bitrate from CalcuMatic 794.
2-pass (old style) is giving me a Peak of 1180, Average 782.
Now there's my problem in finding the correct Max-value to achieve a peak of 1180. From 400-850 @max bitrate there's no change in peak, it's always 1192 :roll:
Min bitrate is 350.
2-pass filesize is 2.128KB.
When i'm doing an CQ-encode @maxbitrate=850 with the same source range the CQ doesn't matter, file size remains the same. Only if i change max to 400 there's a siginificant change in filesize.
Any change in CQ is ignored, file size remains the same.
Can you suggest anything, please :?:

EDIT: Maybe the problem is the already compressed KDVD :roll:

Hi Krassi,

Did you pick a high action scene for source range :?:
On K-Pax, maybe the part where everyone is running to see the Bluebird :idea:

-kwag

Its not easy with that movie :lol: I'll try again with another and longer source range.

I'll scan the whole source movie with bitrate viewer to find the time where the highest bitrates are used :idea:

I'm thinking something else right now, and I'm about to try it :!:
I'm going to try a 2-pass of the slicing that CQMatic creates. That is, the 2 minute slice of the movie. Then, that will leave a sample of a correct size ( properly calculated file size, based on average bitrate ), and then I'll run prediction with CQ, to target that file size.
So it will be one 2 minute 2-pass cycle, and the next cycles as usual, with CQ.
I think the 2 pass needs more footage than 20 seconds to properly map the material.
So with the 2 minute chunks that CQMatic creates, this should give a better accurate mapping of the source.
And this way, there's no fumbling with peak bitrates, etc., and can easily be automated with minor modifications to CQMatic.

Edit: You can try this yourself by running CQMatic as usual. Load your project file, etc., and after clicking on "Execute", start the process and cancel it. Kill TMPEG. Then manually load the file "CQMatic.tpr" in the directory where you have CQMatic installed.
That file will already have your project in time slices. So just use 2-pass (old) to create your sample, and then change to CQ and try to match the file size.

-kwag

Gentlemen ...
Im right now in the office so I can't actively attend this workout.
But I read that sometimes in here its difficult for some people to "catch" useful scenes in TmpgEnc to get the "peak" parts.

Open your import d2v-.avs in Vdubmod .... cut out everything exept about 20 sec of some "peak" szenes and go to the editor to add the "trim"'s to your existing .avs ... paste the trims there and open this again in tmpg to make your "see the peak" encodings.

I did this before too, but with another intention, ... to find out the maxbitrate for CQ mode and to do a prediction by using the known CQ matic way.

So maybe this gives precise scenes to figure it out? Hmmmm :roll:

I've predicted a CQ of ~77 this way (targetting for 1600 MB). I'll post the results of the final encode later.
I'm testing without filters to decrease encoding time.

Hi incredible,

With the "Short 2-pass/Multiple CQ" cycle I just described, there is no need to find peaks :)
I'll implement this new technique into CQMatic soon. I'm just double checking that the manual process gives accurate results.
I theory, a dual pass (if properly implemented by the encoder) will give an accurate final file size.
So that's why I came up with the idea of implementing a dual pass on the first CQMatic cycle, and then resume the following cycles as CQ, matching the file size found by the dual pass.
This way, CQMatic will have a result much more accurate than a single random piece of a movie.
This should work :!:
At least in theory :roll:
If it does, I'll roll out a new CQMatic version in a matter of hours :mrgreen:

-kwag