Quantcast TMPGEnc 2-Pass Engine. a Key to Faster CQ Prediction. - digitalFAQ.com Forums [Archives]
  #1  
10-19-2003, 11:51 PM
kwag kwag is offline
Free Member
 
Join Date: Apr 2002
Location: Puerto Rico, USA
Posts: 13,537
Thanks: 0
Thanked 0 Times in 0 Posts
Hi all,

First of all, **** THIS IS EXPERIMENTAL (and not for newbies!) ***
What we are about to do, is to use TMPEG's 2-pass engine as a "helper" for CQ encoding.
Today, I said to myself: If TMPEG can properly calculate the final file size of a movie by doing 2 passes, then why can't I do a small 2-pass (20 seconds or so) encode, which will have the correct average bitrate, and use the encoded file size as a reference footprint to encode the same clip in CQ mode, and tune the parameters until I achieve the same footprint
Then, in theory, the full encode (in CQ mode) should be identical in final file size.
So let the game begin

Load your source into CalcuMatic, and get your average bitrate as usual for the movie to be processed.
Now follow this steps:

(1) Encode about 20 seconds or so (selected source range) of a mixed high action/normal action clip, anywhere in your movie where you have some of the highest visible action, as 2-pass VBR with MIN bitrate 300 and MAX=Your Max (2,500 for me), and the average bitrate that CalcuMatic gave you.
(2) With Bitrate Viewer, open the encoded sample and note the PEAK bitrate and the file size.
(3) Change to CQ mode, and set the CQ value to 100.
(4) Encode the same source range, and start lowering the MAX bitrate until the PEAK bitrate is as close as possible to the one recorded on the 2-pass.
(5) Now encode the same clip, and start lowering CQ value until you get as close to the same file size as the one created with the 2-pass encode.
(6) Once you get a close file size, you've found your CQ, and now you set back your MAX bitrate to your usual 2,000Kbps or 2,500Kbps, and encode your movie with the CQ found.

Your final file size should never exceed the maximum calculated by the average bitrate.
When we found the CQ, we return the MAX bitrate to the high value we normally use, thus increasing the distance of MAX bitrate from CQ value. This will now expand the dynamic range of the encoder, and give you lower file size on average scenes and higher bitrate peaks on high action scenes.
But the final file size should never exceed the calculated file size, because we calculated a "worse case" scenario on a high action scenes.

Note: If your movie's highest action doesn't go too much above the calculated MAX during the 2-pass run, you might want to leave the MAX bitrate to the value found on step 4. This will guarantee that the final file size will be right on target, just like if a 2-pass VBR encode had been done. But you will still have better quality with the CQ encode, because of the CQ bitrate distribution.

If this method proves to be correct, after some more tweakings, then the next version of CQMatic will do this automatically. So bye bye to long CQMatic loops trying to find the correct CQ value. Let's use the 2-pass encoding engine that TMPEG provides ( in a way it was never designed ), and we should have a correct CQ value in less than 10 minutes
Please let me know your results

-kwag
Reply With Quote
Someday, 12:01 PM
admin's Avatar
Site Staff / Ad Manager
 
Join Date: Dec 2002
Posts: 42
Thanks: ∞
Thanked 42 Times in 42 Posts
  #2  
10-19-2003, 11:58 PM
Gamecraze Gamecraze is offline
Free Member
 
Join Date: Oct 2003
Posts: 42
Thanks: 0
Thanked 0 Times in 0 Posts
Speed is always good, like they say, Time is money.
Reply With Quote
  #3  
10-20-2003, 01:17 AM
Krassi Krassi is offline
Free Member
 
Join Date: Mar 2003
Location: Germany
Posts: 390
Thanks: 0
Thanked 0 Times in 0 Posts
Sounds good, Kwag.
I'll do a test this evening (PAL ).
Reply With Quote
  #4  
10-20-2003, 01:23 PM
kwag kwag is offline
Free Member
 
Join Date: Apr 2002
Location: Puerto Rico, USA
Posts: 13,537
Thanks: 0
Thanked 0 Times in 0 Posts
Finished my first encode ( Red Planet at 704x480 ), with a file size of ~50MB lower than wanted ( I did expect that, but quality is still excelent ).
Now I'm compensating by using a MIN bitrate of 25% lower than average (not 300Kbps). This will push the CQ curve up, and increase the average bitrate on low lighted/low action scenes.
Currently encoding K19 at 352x240 ( so I can see the results faster )
Then I'll encode Red Planet again, with the corrections, to see if the results are consistent.

-kwag
Reply With Quote
  #5  
10-20-2003, 02:10 PM
azel azel is offline
Free Member
 
Join Date: Jul 2003
Posts: 120
Thanks: 0
Thanked 0 Times in 0 Posts
Funny
Yesterday I had some of the same thoughs....
Videoencoding improvments????
Reply With Quote
  #6  
10-20-2003, 02:14 PM
kwag kwag is offline
Free Member
 
Join Date: Apr 2002
Location: Puerto Rico, USA
Posts: 13,537
Thanks: 0
Thanked 0 Times in 0 Posts
Quote:
Originally Posted by azel
Funny
Yesterday I had some of the same thoughs....
Videoencoding improvments????
But your thoughts were 12 hours after I posted. Look at the time stamp
But I read your post, and it's not the same thing

-kwag
Reply With Quote
  #7  
10-20-2003, 04:07 PM
kwag kwag is offline
Free Member
 
Join Date: Apr 2002
Location: Puerto Rico, USA
Posts: 13,537
Thanks: 0
Thanked 0 Times in 0 Posts
Finished encode of K19 at 352x240.
Wanted size: 703,106KB
Final Encoded: 687,563KB
Things are looking very good
Currently encoding "The Bourne Identity" at 528x480.
4 hours to go.....

-kwag
Reply With Quote
  #8  
10-20-2003, 04:14 PM
incredible incredible is offline
Free Member
 
Join Date: May 2003
Location: Germany
Posts: 3,189
Thanks: 0
Thanked 0 Times in 0 Posts
Send a message via ICQ to incredible
I'd like to know if there will be a viewable quality advantage by doing this in comparison to our CQmatic way.
Reply With Quote
  #9  
10-20-2003, 04:19 PM
kwag kwag is offline
Free Member
 
Join Date: Apr 2002
Location: Puerto Rico, USA
Posts: 13,537
Thanks: 0
Thanked 0 Times in 0 Posts
Quote:
Originally Posted by incredible
I'd like to know if there will be a viewable quality advantage by doing this in comparison to our CQmatic way.
Quality is the same, because we're still encoding in CQ mode

-kwag
Reply With Quote
  #10  
10-20-2003, 08:37 PM
kwag kwag is offline
Free Member
 
Join Date: Apr 2002
Location: Puerto Rico, USA
Posts: 13,537
Thanks: 0
Thanked 0 Times in 0 Posts
"The Bourne Identity" at 528x480 done.
Wanted size: 719,786KB
Encoded: 745,296KB

3.4% over. Now to do some minor adjustments with MIN bitrate.
I used 25% under average for MIN. I'll now test with 30% under average for MIN ( for safety margin ).
Now on to encode "Red Planet" at 704x480......

-kwag
Reply With Quote
  #11  
10-21-2003, 12:03 AM
fabrice fabrice is offline
Free Member
 
Join Date: Mar 2003
Location: Madrid-Spain
Posts: 515
Thanks: 0
Thanked 0 Times in 0 Posts
Hi,

I tried that on "the beauty and the beast", with those values:
2 pass VBR :
- res 704x576
- min: 300
- max: 1800
- Aver.: 1089
And it gave me a 1174 peak (and this is the highest motion scene, as I got macroblock in my first encode just right there).
Filesize: 3909Kb.

Swith to CQ, and I had to lower the max bitrate until 1200 to get this max. bitrate value (with 1800, it give me a peak of 1600).

The CQ I encode this movie before was 56,6, but with a 1200 max. bitrate value, the 60 CQ value give me a 3605 Kb file, which is a way smaller than the reference sample.

I'll try again to see where is the error...

Fabrice
Reply With Quote
  #12  
10-21-2003, 12:19 AM
kwag kwag is offline
Free Member
 
Join Date: Apr 2002
Location: Puerto Rico, USA
Posts: 13,537
Thanks: 0
Thanked 0 Times in 0 Posts
Hi Fabrice,

After you get the same MAX bitrate in CQ as you got in the 2-pass run, then you start to lower the cq value until you get the same file size that you got in the 2-pass. After you get that, you set your MAX again up to your 1,800Kbps and encode your movie.

-kwag
Reply With Quote
  #13  
10-21-2003, 04:23 AM
Krassi Krassi is offline
Free Member
 
Join Date: Mar 2003
Location: Germany
Posts: 390
Thanks: 0
Thanked 0 Times in 0 Posts
Quote:
Originally Posted by kwag
(4) Encode the same source range, and start lowering the MAX bitrate until the PEAK bitrate is as close as possible to the one recorded on the 2-pass.
(5) Now encode the same clip, and start lowering CQ value until you get as close to the same file size as the one created with the 2-pass encode.
Hi Kwag,

i'm having some adjust problems here.
Some infos:
Movie k-pax (Source is KDVD), 20 seconds Source Range, average bitrate from CalcuMatic 794.
2-pass (old style) is giving me a Peak of 1180, Average 782.
Now there's my problem in finding the correct Max-value to achieve a peak of 1180. From 400-850 @max bitrate there's no change in peak, it's always 1192
Min bitrate is 350.
2-pass filesize is 2.128KB.
When i'm doing an CQ-encode @maxbitrate=850 with the same source range the CQ doesn't matter, file size remains the same. Only if i change max to 400 there's a siginificant change in filesize.
Any change in CQ is ignored, file size remains the same.
Can you suggest anything, please

EDIT: Maybe the problem is the already compressed KDVD
Reply With Quote
  #14  
10-21-2003, 06:18 AM
kwag kwag is offline
Free Member
 
Join Date: Apr 2002
Location: Puerto Rico, USA
Posts: 13,537
Thanks: 0
Thanked 0 Times in 0 Posts
Hi Krassi,

Did you pick a high action scene for source range
On K-Pax, maybe the part where everyone is running to see the Bluebird

-kwag
Reply With Quote
  #15  
10-21-2003, 06:21 AM
Krassi Krassi is offline
Free Member
 
Join Date: Mar 2003
Location: Germany
Posts: 390
Thanks: 0
Thanked 0 Times in 0 Posts
Quote:
Originally Posted by kwag
Hi Krassi,

Did you pick a high action scene for source range
On K-Pax, maybe the part where everyone is running to see the Bluebird

-kwag
Its not easy with that movie I'll try again with another and longer source range.
Reply With Quote
  #16  
10-21-2003, 06:36 AM
Krassi Krassi is offline
Free Member
 
Join Date: Mar 2003
Location: Germany
Posts: 390
Thanks: 0
Thanked 0 Times in 0 Posts
I'll scan the whole source movie with bitrate viewer to find the time where the highest bitrates are used
Reply With Quote
  #17  
10-21-2003, 06:37 AM
kwag kwag is offline
Free Member
 
Join Date: Apr 2002
Location: Puerto Rico, USA
Posts: 13,537
Thanks: 0
Thanked 0 Times in 0 Posts
I'm thinking something else right now, and I'm about to try it
I'm going to try a 2-pass of the slicing that CQMatic creates. That is, the 2 minute slice of the movie. Then, that will leave a sample of a correct size ( properly calculated file size, based on average bitrate ), and then I'll run prediction with CQ, to target that file size.
So it will be one 2 minute 2-pass cycle, and the next cycles as usual, with CQ.
I think the 2 pass needs more footage than 20 seconds to properly map the material.
So with the 2 minute chunks that CQMatic creates, this should give a better accurate mapping of the source.
And this way, there's no fumbling with peak bitrates, etc., and can easily be automated with minor modifications to CQMatic.

Edit: You can try this yourself by running CQMatic as usual. Load your project file, etc., and after clicking on "Execute", start the process and cancel it. Kill TMPEG. Then manually load the file "CQMatic.tpr" in the directory where you have CQMatic installed.
That file will already have your project in time slices. So just use 2-pass (old) to create your sample, and then change to CQ and try to match the file size.


-kwag
Reply With Quote
  #18  
10-21-2003, 07:02 AM
incredible incredible is offline
Free Member
 
Join Date: May 2003
Location: Germany
Posts: 3,189
Thanks: 0
Thanked 0 Times in 0 Posts
Send a message via ICQ to incredible
Gentlemen ...
Im right now in the office so I can't actively attend this workout.
But I read that sometimes in here its difficult for some people to "catch" useful scenes in TmpgEnc to get the "peak" parts.

Open your import d2v-.avs in Vdubmod .... cut out everything exept about 20 sec of some "peak" szenes and go to the editor to add the "trim"'s to your existing .avs ... paste the trims there and open this again in tmpg to make your "see the peak" encodings.

I did this before too, but with another intention, ... to find out the maxbitrate for CQ mode and to do a prediction by using the known CQ matic way.

So maybe this gives precise scenes to figure it out? Hmmmm
Reply With Quote
  #19  
10-21-2003, 08:07 AM
Krassi Krassi is offline
Free Member
 
Join Date: Mar 2003
Location: Germany
Posts: 390
Thanks: 0
Thanked 0 Times in 0 Posts
Quote:
Originally Posted by kwag
Edit: You can try this yourself by running CQMatic as usual. Load your project file, etc., and after clicking on "Execute", start the process and cancel it. Kill TMPEG. Then manually load the file "CQMatic.tpr" in the directory where you have CQMatic installed.
That file will already have your project in time slices. So just use 2-pass (old) to create your sample, and then change to CQ and try to match the file size.


-kwag
I've predicted a CQ of ~77 this way (targetting for 1600 MB). I'll post the results of the final encode later.
I'm testing without filters to decrease encoding time.
Reply With Quote
  #20  
10-21-2003, 08:08 AM
kwag kwag is offline
Free Member
 
Join Date: Apr 2002
Location: Puerto Rico, USA
Posts: 13,537
Thanks: 0
Thanked 0 Times in 0 Posts
Hi incredible,

With the "Short 2-pass/Multiple CQ" cycle I just described, there is no need to find peaks
I'll implement this new technique into CQMatic soon. I'm just double checking that the manual process gives accurate results.
I theory, a dual pass (if properly implemented by the encoder) will give an accurate final file size.
So that's why I came up with the idea of implementing a dual pass on the first CQMatic cycle, and then resume the following cycles as CQ, matching the file size found by the dual pass.
This way, CQMatic will have a result much more accurate than a single random piece of a movie.
This should work
At least in theory
If it does, I'll roll out a new CQMatic version in a matter of hours

-kwag
Reply With Quote
Reply




Similar Threads
Thread Thread Starter Forum Replies Last Post
TMPGEnc: Using SSRC as an external audio engine syk2c11 Video Encoding and Conversion 3 06-01-2004 05:28 AM
My new car engine kwag Off-topic Lounge 6 10-03-2003 10:05 PM
Multi pass size prediction girv Avisynth Scripting 14 07-14-2003 04:51 PM
Faster prediction method ARnet_tenRA Avisynth Scripting 19 04-12-2003 09:11 AM
Encoding single-pass KVCD vs. two-pass logan555 Video Encoding and Conversion 1 12-04-2002 09:18 AM

Thread Tools



 
All times are GMT -5. The time now is 06:47 PM  —  vBulletin © Jelsoft Enterprises Ltd