TMPGEnc 2-Pass Engine. a Key to Faster CQ Prediction. - Page 5

Jellygoose · #81 11-12-2003, 10:10 AM

Quote:

Originally Posted by incredible

---------------------------------- Next step ------------------------------------

Other successful prediction yesterday evening using this method:

"Gladiator" · 480x576 · mpeg1 · 149min · GOP = 84 (1-41-1) · 112kbit Audio

muxed mpg. size = 783MB ... 795 wanted (CDr80)

What kind of GOP is that

audioslave · #82 11-13-2003, 06:54 PM

WOW!!! This is amazing guys

I just encoded "Half Past Dead" with your exellent prediction method. Here's the results:

Quote:

Half Past Dead (2003)
=================================================
Video: 480x576, 25fps
MIN bitrate 300, MAX bitrate 2500, MPEG-1
Motion search precision: High quality (slow)
Audio: 128kbps (dual channel)

-------------------------------------------------
Prediction (gop=25, gl=3, sa=5)
-------------------------------------------------
Wanted sample size* 15 938 kB
1
sampler_GOPbased_CQ70,00_Offset00.m1v 14 828 kB
sampler_GOPbased_CQ70,00_Offset30.m1v 16 368 kB
2
sampler_GOPbased_CQ70,10_Offset00.m1v 15 021 kB
sampler_GOPbased_CQ70,10_Offset30.m1v 16 568 kB
3
sampler_GOPbased_CQ70,20_Offset00.m1v 15 189 kB
sampler_GOPbased_CQ70,20_Offset30.m1v 16 743 kB
4
sampler_GOPbased_CQ70,15_Offset00.m1v 15 106 kB
sampler_GOPbased_CQ70,15_Offset30.m1v 16 656 kB
5
sampler_GOPbased_CQ70,17_Offset00.m1v 15 128 kB
sampler_GOPbased_CQ70,17_Offset30.m1v 16 684 kB
6
sampler_GOPbased_CQ70,18_Offset00.m1v 15 147 kB
sampler_GOPbased_CQ70,18_Offset30.m1v 16 699 kB
-------------------------------------------------
Average
-------------------------------------------------
1
sampler_GOPbased_CQ70,10_Offset00.m1v 15 021 kB
sampler_GOPbased_CQ70,10_Offset30.m1v 16 568 kB
Average:** ~ 15 795 kB = Below
2
sampler_GOPbased_CQ70,20_Offset00.m1v 15 189 kB
sampler_GOPbased_CQ70,20_Offset30.m1v 16 743 kB
Average:** 15 966 kB = Above
3
sampler_GOPbased_CQ70,15_Offset00.m1v 15 106 kB
sampler_GOPbased_CQ70,15_Offset30.m1v 16 656 kB
Average:** 15 881 kB = Below
4
sampler_GOPbased_CQ70,17_Offset00.m1v 15 128 kB
sampler_GOPbased_CQ70,17_Offset30.m1v 16 684 kB
Average:** 15 906 kB = Below
5
sampler_GOPbased_CQ70,18_Offset00.m1v 15 147 kB
sampler_GOPbased_CQ70,18_Offset30.m1v 16 699 kB
Average:** 15 923 kB ~ MATCH!
-------------------------------------------------
Wanted video size: 830 864 kB
Encoded video size: 825 224 kB

Difference in size: - 5 640 kB
Difference in percent: - 0,68 %
-------------------------------------------------

* Wanted sample size:
Frames per sample/Frames per second/8*Average bitrate*Safety margin
Example for "Half Past Dead" - PAL
(Calculated with CQMatic for a 90 min CD-R):
2775/25/8*1172,1*0,98=15937,62975
So in this case the sample size should be 15 938 kB.

**Average:
Offset00+Offset30/2

As you can see I added a "Saftey margin" to the sample size formula. Just to make up for the few extra megs in the muxing process. Total (muxed) file size got 925 440 kB ~ 903 MB

Awesome!
The script I used look like this (Thank you incredible

):

Quote:

################################################## ##
######### GOP length based slicing script #########
################################################## ##
#
################### The source ####################
#
MPEG2Source("D:/Rippat/Half/Half.d2v")
#
####### important variables to set! ################
#
off=0 # offset in seconds when the sampling on the stream starts defaults = "0" or "30"
gop= 25 # Gop seq length set in mpeg encoder/ Example: DVD =15(PAL), 18(NTSC)
gl=3 # GOP seq length multiplicator, default is gl=3 means more frames per slice
sa=5 # Total size of sample stream! 2= 5% of the whole movie and 1=10%, default is 2! (do not use 0!! This would cause a division by Zero!!)
#
################## Setting the offset #################
#
Trim(round(framerate()*off), framecount())
#
### In case of our workout we resize to 352x288 just for encoding speed##
#
BlindPP(cpu=4)
Deen("a2d", 1, 5, 8 )
MergeChroma(blur(1.5))
MergeLuma(blur(0.1))
ASharp(1, 4)
BicubicResize(448, 416, 0, 0.6, 11, 0, 698, 576)
TemporalSoften(1, 5, 8, 3, 2)
UnDot()
BlockBuster(method="noise", detail_min=1, detail_max=10, variance=0.3, seed=1)
DCTFilter(1, 1, 1, 1, 1, 1, 0.5, 0)
AddBorders(16, 80, 16, 80)
#
############### The smart sampler routine ###############
#sampler(samples=(round(((framecount())/10)/(gop*gl))/sa), Length=(gop*gl))

You (all involved) deserves kudos for an excellent work. Thank you very much!

P.S.
Now all we need is for someone to make an automated program for this prediction method...

azel · #83 11-14-2003, 10:57 AM

I read some replies, and there are a lot of bright thoght. My english is terrible, but I hope you will understand.

I have also some ideas to predict CQ, I have posted a post about I before but I got no response...

First we can take a look at how TMPGEnc PLUS do 2 pass (new method) encoding:
-First it encode the first pass with a current CQ value. Then TMPGEnc can calculate a motion factor out off avg bitrate and the bitrate off each GOP.
-Then it does the second pass. It use the results from the first pass and calculate a of the GOPs. The CQ value will variate a litle bit from GOP to GOP, becouse the last GOP can be bigger or less than the last calculated.

To get all the information of bitrate for each GOP etc. try to encode a litle sample and turn the logg on!!!

In the logg we get all the information we need, and I think the logg can be very usefull when doing prediction.

So one method can be:
-Encode the a sample (5 % of the movie) in 2-pass encoding. Then goto the encoderlogg, and then take avg CQ value of each CQ value for each GOP from the second pass.
-Then do a couple of samples with the standard method.
This method will incearse time of prediction.

I hope someone have somthing to say..... Just try to encode a litle sample and look at the encoderlog. If TMPGEnc can predict CQ value in two passes, why can't we??

incredible · #84 11-14-2003, 11:52 AM

Quote:

I have also some ideas to predict CQ, I have posted a post about I before but I got no response...

Azel, I hope you don't mind and I understood you very well ...

but I think this Thread here is getting total confused IF we start another prediction method within this thread. Cause IF someone Replies "I got it too" .... we never can say to which method his success belongs to, ... see what I mean?

So if you have an Idea to give a new way of prediction, open an new thread again and I'm shure someone will reply!
I will!

incredible · #85 11-14-2003, 06:02 PM

@ Audioslave

You don't need to do the Ping and pongs when trying to get close to the needed size you have to match! This takes too much time

You only do pings as long as you get very close ... then you perform a "safety" pong finally. NOW you can see how much the difference takes and if needed you can then perform another pong and if also needed a ping. So ... finetuning will be only needed if you're very close to the reference size.

Is this your only test prediction using this ping-pong offest method you did until now?
I would apriciate it, if you still participate a bit more by doing some other predictions ... just to see YOUR succes/failure average

audioslave · #86 11-14-2003, 06:59 PM

@ incredible

Yes, you're absolutely right about my excessive ping-ponging. But since it was my first try with this method I had to check to be sure everything was OK!

I'm on my way to try to predict another movie with the "Ping-Pong" prediction. Will post my results as soon as I'm done!

incredible · #87 11-15-2003, 07:48 AM

Another Movie, encoded using the Offset PingPong Routine Above
BUT again another Test using CCE!

Muxed 786MB! -- 795MBs wanted (CCE only supports rounded Q values)

nicksteel · #88 11-15-2003, 07:53 AM

Quote:

Originally Posted by incredible

Another Movie, encoded using the Offset PingPong Routine Above
BUT again another Test using CCE!

Muxed 786MB! -- 795MBs wanted (CCE only supports rounded Q values)

Could you post a simple "how to" for doing this. So many posts I've become a little confused.

Thanks,

NickSteel

incredible · #89 11-15-2003, 08:10 AM

Some postings above ... Krassi & I gave an explanation how to do this .. just have a look

incredible · #90 11-17-2003, 08:16 AM

Well my friends ... I did a lot more prediction/full encode tests using my script above. I didn't want to continue posting all precise values cause ... I think it will start boring

Testing on Movie Sources where I even didn't achieve almost. 90% acuracy when using TOK or CQmatic before ... this here gave me more acurate results.
This Weekend I did another 8 encodings.
So seen the average as a whole (including the test-encodings in the past) 3 movies failed ... one resulted as a 767MB muxed .mpg, one as 786mb and one as 798MB.

So if someone's there (maybe the author of TOK!) who knows programming so this could be done automatically (Like TOK, which also uses the adding of AVS-Sampler lines) ... it would be very helpful to start the next step: Testing the computer automated process.

BTW: What I figured out is... when using a first 2pass on the sample ... this gives me a better reference size to match! I did also the method to do sample reference size determining using the manual calculation like Audioslave and Krassi do incl. a safety margin of factor 0,98 but my first 2pass method results more acurate for me

Abond · #91 11-17-2003, 12:13 PM

How much time in average you need for prediction?

incredible · #92 11-17-2003, 01:01 PM

Watch the thread some postings above!

Quote:

Originally Posted by Thread some postings above

Quote:

Originally Posted by bman

Thank's Krassy & Incredible !
I think that now is clear for me what to do so as soon as I'll get home I'll try this new method .
One more quastion : How do u now what CQ u should start encoding ( or it do not metter ) and how long it takes to find RIGHT CQ value this way .
I assume that as u are running all stuff manually than it should take an about half or 3/4 of hour .
Is my assumption right ?
If so than it's really precise but not so fast way (yet )!

Maybe some day whan it'll be implemented in automathic prog than .....
Anyway , great job .
I LOVE this site

bman

Well the CQ to start with is based on my assumption as I also use in CQmatic my assumed CQ to start with.
How long it takes? Well if you start using a very near CQ as I did in the reports above ... you'll match it faster

3/4 of hour??? No!
First ... also when using automatic program codes like CQ the prediction time depends on your encoding size, CPU and so on.
Try first using an encoding size of 352x288! As default in the script --- this gives you a faster prediction to do your test workout.
So in my case and my CPU as I remember every turn took about 1-2 minutes (352x28

and as you see the reports above (wich show the count of turns) you can assume how much it took, including the manual setting of off=x and re-safing the .avs every time...

EDIT: And again, meanwhile Krassi found the english words much faster

Abond · #93 11-18-2003, 03:40 AM

Sorry, probably I was not clear. Firstly, I meant how much time it take for you, incredible, and second I thought of results as a % from the full encode. For example, 2h movie full encode 10h, prediction 2h, another movie 100 min, full encoding 8h, prediction 90 min - well in average it makes for the prediction about 20% of full encoding time . Something like that...

incredible · #94 11-18-2003, 04:38 AM

Quote:

Originally Posted by Abond

Sorry, probably I was not clear. Firstly, I meant how much time it take for you, incredible, and second I thought of results as a % from the full encode. For example, 2h movie full encode 10h, prediction 2h, another movie 100 min, full encoding 8h, prediction 90 min - well in average it makes for the prediction about 20% of full encoding time . Something like that...

As I said in the quote above ...

In case of MY CPU (XP1800 DLT3C JIUHB @ 2530 Mhz real) every turn when using 352x288 takes 1-2min, so 480x576 would take abou 2-3min.
As you see in my reports in Average I need about 5-10 turns, .. and so ... yes, in case of a 100min movie it would take about 10-20min (480x576), when doing this routine manually --- refering to my CPU.

So 10-20% prediction time theoretically it will take.
But ... just try it, cause I don't know your CPU ... it also depends on if you directly go very near when choosing the first CQ step. This depends on your feelings for CQs.

Abond · #95 11-18-2003, 04:54 AM

A-ah, well, I just asked to compare with the prediction I am currently using. It is a bit different approach. Thank you.

incredible · #96 11-18-2003, 05:55 PM

I updated the prediction script:

When running in Workout-Mode you will see this part within the picture above:

Explanation:

- "Frames of Movie total"
= Speaks for itself, but anyway y added the length in seconds and minutes ...

- "x Samples*x Frames (GOP=gop*gl) = ca x% of Movie total
= The auto calculated Prediction size depending on the values you set in the beginning of the script and therefore the resulted amount of % of the whole movie which will be sampled

- "Frames of Sampler = x ...."
= The auto calculated size of the sampler in frames, seconds and minutes

- "Offset set to x sec."
= Here you see the offset in seconds to perform the "ping" and the "pong" which you set as a value at the beginning of the script REFERING to:
- "Offset intervall should be...."
= Here you see how the "ping" and the "pong" values (variable "off")should be set individually to each source and settings in the script! ..... Nice!

The script below is in workout mode status ... so the subs will be seen.
Its important so you can see the recommended Offset-values at the end of the subs!
Keep them in mind and afterwards deactivate the lines beginning with "Subtitle...." and "\.......", after this you can do your first step.
Because if you do an encoding after predictionning WITHOUT Subs (shure) you also have to do the Prediction WITHOUT the subs! So the Pixelinformation at Prediction and encoding will be constant!

These Days I'll write a "function" Version of this script so you'll be able to safe the script as .avsi ... and by just adding it to the AVS Plugins Folder you can call this Prediction by using a simple command in your AVS Script incl. values to set like the gop, ml,and so on, like
slicer(0,24,3,5,sub="true")
.... so stay tuned.

And here comes the new script:

Code:

###############################################################
##############  GOP length based slicing script ###############
############# Including calculating Offset Sizes ##############
###############################################################
#
########################  The source ##########################
# 
mpeg2source("D:\GLADIATOR\gladiator.d2v")
# 
################## important variables to set! ################
# 
off=0 # offset in seconds when the sampling on the stream starts (watch the recommended Offsets in the subs!)
gop= 24  # Gop seq length set in mpeg encoder/ Example: DVD =15(PAL), 18(NTSC)
gl=2 # GOP seq length multiplicator, default is gl=2 means more frames per slice 
sa=5 # Total size of sample stream! 5= 2% of the whole movie and 2=5%, default is 2! (do not use 0!! This would cause a division by Zero!!) 
#
######################## Setting the offset #################### 
#
Trim(round(framerate()*off), framecount()) 
#
### Resizing for workout ## 
#
GripCrop(480,576, source_anamorphic=true)
GripSize(resizer="BicubicResize") 
Gripborders()
#
pr=sampler(samples=(round(((framecount())/10)/(gop*gl))/sa), Length=(gop*gl))
############# subtitels just for the testing workout ############ 
Subtitle(text_color=$999999 ,size=16,
\         "Frames of movie total : "+String(framecount())+" "+
\         "= ca. "+String(Round(framecount()/Framerate()))+" Sec. "+
\         "= ca. "+String(ceil((framecount()/Framerate())/60))+" min.",10,13) 
Subtitle(text_color=$999999 ,size=16,
\         String(round(((framecount())/10)/(gop*gl))/sa)+" Samples * "+
\         String(gop*gl)+" Frames (GOP="+String(gop)+"*"+String(gl)+")"+
\         " = ca. "+String(100/10/sa)+"% of Movie total",10,28) 
Subtitle(text_color=$999999 ,size=16,
\         "Frames of Sampler = "+String(Framecount(pr))+
\         " = ca. "+String((Framecount(pr)/25))+" sec."+
\         " = ca. "+String(round((Framecount(pr)/60)/Framerate()))+" min.",10,43)
Subtitle(text_color=$999999 ,size=16,
\         "Offset set to : "+String(off)+" sec.  "+
\         "(Offset intervall should be: 0 sec. / "+String(Round(((framecount()/Framerate())/60)/(round(((framecount())/10)/(gop*gl))/sa)*60)/2)+" Sec.)",10,58 ) 
#################### The smart sampler routine ################## 
sampler(samples=(round(((framecount())/10)/(gop*gl))/sa), Length=(gop*gl))

nicksteel · #97 11-20-2003, 09:14 AM

Hi, Incredible.

I've been having difficulties predicting a 82 min 720x480 PVR250 MPEG2 tv capture. CQMatic constantly predicts a CQ resulting in a small mpg.

CalcuMatic set to 128, 80Min CD results in 740MB muxed (Headac3he mp2 12

.

I wish to try the method above. I will use trim statements from VirtualDub-MPEG, the MA optimal script with Telecide and Decimate and Kwag's x3 template reset to 704x480. I wish to use the maximum sampling size.

What changes should I make in the above script?

incredible · #98 11-20-2003, 10:09 AM

Post your script!

Ill add the needed lines ....

Bravucon · #99 12-17-2003, 10:39 AM

Novice question

How can I set "Gop seq length" = 15 for KDVD-Pal template in CEE ?? where is this option? Is in Video setting ?

anybody can say to me the best settings for "Video setting"?

thxs in advance

mistermickster · #**100** 12-18-2003, 04:33 AM

@Kwag,

I'd just like to mention my findings on this and maybe an idea

When this method was first brought up, I thought I'd experiment a little.

CalcuMatic gave me an Avg of 1672 and a (0.57 * Avg) of 953. Doing the calculation, a 2 minute clip gave me a sample size of 24.49MB. I actually ran TMPGEnc in VBR mode (using the CQMatic selection), and that gave me a sample size of only 22MB

A prediction only run of CQMatic gave me a sample size nearer the 24MB, that's a difference of 8%. Over the size of a full encode, that's a big difference.

Would it be possible, and would it be beneficial, to have CQMatic do the first pass VBR

It may help with hitting final size more accurately, and stop CQMatic from going over wanted size

What do you think