Quantcast TMPGEnc 2-Pass Engine. a Key to Faster CQ Prediction. - Page 4 - digitalFAQ.com Forums [Archives]
  #61  
11-07-2003, 04:25 PM
incredible incredible is offline
Free Member
 
Join Date: May 2003
Location: Germany
Posts: 3,189
Thanks: 0
Thanked 0 Times in 0 Posts
Send a message via ICQ to incredible
@VMesquita

I just changed the script also into a GOP based version:
Code:
####################################################
#########  GOP length based slicing script #########
####################################################
#
###################  The source ####################
# 
mpeg2source("H:\AMERBEAUTY\ambeauty.d2v")
# 
####### important variables to set! ################
# 
off=0 # offset in seconds when the sampling on the stream starts defaults = "0" or "30"
gop= 15  # Gop seq length set in mpeg encoder/ Example: DVD =15(PAL), 18(NTSC)
gl=3 # GOP seq length multiplicator, default is gl=3 means more frames per slice 
sa=2 # Total size of sample stream! 2= 5% of the whole movie and 1=10%, default is 2! (do not use 0!! This would cause a division by Zero!!) 
#
################## Setting the offset ################# 
#
Trim(round(framerate()*off), framecount()) 
#
### In case of our workout we resize to 352x288 just for encoding speed## 
#
GripCrop(352,288) # your encoding size
GripSize(resizer="BilinearResize") 
GripBorders() 
Letterbox(0,0,16,16) 
#
############# subtitels just for the testing workout ############ 
Subtitle("Frames total movie/unsliced : "+String(framecount()),10,13) 
Subtitle("samples : "+String(round(((framecount())/10)/(gop*gl))/sa),10,28) 
Subtitle("sample length  : "+String(gop*gl),10,43) 
Subtitle("= ca. "+String(100/10/sa)+"% of Movie total",10,58) 
Subtitle("offset  : "+String(off)+" sec.",10,73) 
############### The smart sampler routine ############### 
sampler(samples=(round(((framecount())/10)/(gop*gl))/sa), Length=(gop*gl))
Seems more acurate!!

Full movie got 721 MB (just testing)

the GOP method gives me an Q=12 and therefore an average of the ping-pong-outputs which is 36.18 MB!
So the total sampler size takes 5% of the whole movie... 36.18*2=72,36*10=723,6MB!!!!!!!!!! Matches almost the full encode, now I've to see if the overflow depends on the "round" AVS comand in the script, maybe "int" would be better ... first I gonna try on a different movie...
All testings where made first using 300min, 3000max ... next movie I choose 500min, 5000max
Reply With Quote
Someday, 12:01 PM
admin's Avatar
Site Staff / Ad Manager
 
Join Date: Dec 2002
Posts: 42
Thanks: ∞
Thanked 42 Times in 42 Posts
  #62  
11-07-2003, 07:41 PM
incredible incredible is offline
Free Member
 
Join Date: May 2003
Location: Germany
Posts: 3,189
Thanks: 0
Thanked 0 Times in 0 Posts
Send a message via ICQ to incredible
-------------------------------- nest step ---------------------------------

Another Movie testing ... "The Matrix"

Refering Offset ping-pong settings:
I did 1-2-3-4 "Pings" and hit the nedded sample filesize, then I performed a "pong" and another one .. average then was right! Ok not that exact cause CCE only allows rounded Q values!

And thats what came out finally: 649MB! Which gets in relation almost exact close to the predicted sample!
649+117(Audio)=766MB
Well for a prediction which only allows rounded Q Values to me it seems good. Cause that's what was almost predicted

Tomorrow testing on other Movies also using Mainconcept and for shure again TmpgEnc
Reply With Quote
  #63  
11-08-2003, 02:18 AM
kwag kwag is offline
Free Member
 
Join Date: Apr 2002
Location: Puerto Rico, USA
Posts: 13,537
Thanks: 0
Thanked 0 Times in 0 Posts
Quote:
Originally Posted by Krassi
@Kwag:
Do you think it is possible to make CQMatic take a 2-pass VBR sample at the beginning, which has a doubled sample size or similar. After that you could let it do the job like it is know doing
Yes. I already had stated that idea on a previous post.
But only when we get very accurate manual predictions with this method.
Once it's proven to work reliably (manually), then I'll integrate the method into CQMatic.

-kwag
Reply With Quote
  #64  
11-08-2003, 08:37 AM
incredible incredible is offline
Free Member
 
Join Date: May 2003
Location: Germany
Posts: 3,189
Thanks: 0
Thanked 0 Times in 0 Posts
Send a message via ICQ to incredible
----------------------------------- Next Step ---------------------------------

Encoding using TmpgEnc:

Here's the report:

__________________________________________________ _________________________

Now I did set the variable sa=5 ... (100/10/sa) .... therefore the sample stream will be 2%!!! of the whole video. To see how acurate this method works by using only a 2% moviesample ... therefore less prediction time.

I first I calculated the Avrg.Bitrate using Calcumatic and did a 2pass on the sample just to see how much the 5% stream hast to take finally!
(But I think you don't need 2pass cause you can obtain this value based on an mathematic formula .. but here I wanted to test again including 2pass at the beginning)

Then aproaching needed filesize using "pings" (=offset set to 0) and therefore changing CQs until it matches.
Finally I perfome a "pong" (offset in script set to 30 sec!) to be safe!
Changing CQs again until the average of "ping" & "pong" ca. matches finally the needed result.
... encoding "full" size (offset set to 0 and deactivating the sampler line in the AVS script). Muxed result matches very well the needed size to fit on one CD-R80.

Now testing on 480x576 ...... using diff. max and min Bitrates like 64/2000

... to be continued
Reply With Quote
  #65  
11-08-2003, 12:31 PM
incredible incredible is offline
Free Member
 
Join Date: May 2003
Location: Germany
Posts: 3,189
Thanks: 0
Thanked 0 Times in 0 Posts
Send a message via ICQ to incredible
----------------------------------- Next Step ---------------------------------

Other Movie encoding using my Script above.
Encoding again using TmpgEnc: 480x576 mpeg1! 64/2000kbit and 2% Moviesample size!
(variable sa=5 ... (100/10/sa) .... =sample stream=2%)

Here's the report:
__________________________________________________ _________________________


__________________________________________________ _________________________

It seems thats the way!!
Using The Offset Ping Pong Script icluding the auto generation of a 2% Moviesample works as wonderful as working with an auto generated 5% Moviesample.
Reply With Quote
  #66  
11-09-2003, 01:06 PM
Krassi Krassi is offline
Free Member
 
Join Date: Mar 2003
Location: Germany
Posts: 390
Thanks: 0
Thanked 0 Times in 0 Posts
Hi incredible,
sorry for getting back so late.
You've achieved great results.
Have you made a second VBR encoding on your offset prediction
I'm doing some tests right know.

@Kwag
Have you seen my post above on calculating the right size
What do you think
Reply With Quote
  #67  
11-09-2003, 03:38 PM
Krassi Krassi is offline
Free Member
 
Join Date: Mar 2003
Location: Germany
Posts: 390
Thanks: 0
Thanked 0 Times in 0 Posts
Here are the test results with incredible's script and a sa=5 (2% sample of movie length)

VBR mode
VBR sample: 13.174 KB
CQ1: 74.3 offset 0
CQ2: 73.1 offset 30

CQ average: 73.7

Full encoding:
Wanted VS: 684080 KB
Resulted VS: 656206 KB


Calculated Sample (Frames of Sample/framerate)/8*(average bitrate)

13559.4 KB sample

13.488 with 74.7 offset 0
13.527 with 74.2 offset 30

CQ average: 74.45

Full encoding:
Wanted VS: 684080 KB
Resulted VS:680409 KB
Reply With Quote
  #68  
11-09-2003, 03:46 PM
incredible incredible is offline
Free Member
 
Join Date: May 2003
Location: Germany
Posts: 3,189
Thanks: 0
Thanked 0 Times in 0 Posts
Send a message via ICQ to incredible


Now we should test what will serve better:

more frames per slice or more slices containing less frames within the 2%
I think if we will integrate this pingpong-offset way in CQ matic there should be a window whre the user can determine this ... just in case of for example "YDifferenceToNext()" based avs script routines like MA?!

Today I testet it on "Gladiator" @ 480x576 ... 8MBs less than wanted
(1-41-1 GOP based Template so GOPlength= 84! as value in the script therefore less slices)

Quote:
Originally Posted by Krassi
Have you made a second VBR encoding on your offset prediction?
No, definitely not needed cause both would base identical upon the avrg.Bitrate and the 2%-sample

And thats why its not really needed to do a 2pass before ... as we talked in the chat ... and also VMesquita said ... its a simple math. calculation based upon Mediasize.
In my workouts above I was just too lazy to calculate (got my brain in already other kvcd-Template optimization projects *g*) so I just did a quick 2pass.
Reply With Quote
  #69  
11-09-2003, 09:49 PM
kwag kwag is offline
Free Member
 
Join Date: Apr 2002
Location: Puerto Rico, USA
Posts: 13,537
Thanks: 0
Thanked 0 Times in 0 Posts
Quote:
Originally Posted by incredible
I said the "default" calculation of my sampler AVS routine gives me a 5% of the whole movie output which should match 5% of 785(-Audiosize)MB IF CQ is set right!
No. The 5% you encoded will not be the same size as another 5% of some other part of the movie. If you got a final size close to your wanted, you were lucky

-kwag
Reply With Quote
  #70  
11-09-2003, 09:57 PM
kwag kwag is offline
Free Member
 
Join Date: Apr 2002
Location: Puerto Rico, USA
Posts: 13,537
Thanks: 0
Thanked 0 Times in 0 Posts
Quote:
Originally Posted by incredible
... its a simple math. calculation based upon Mediasize.
Please see this thread, where a similar approach was tried, and it didn't work.
http://www.kvcd.net/forum/viewtopic.php?t=4964

-kwag
Reply With Quote
  #71  
11-10-2003, 05:41 AM
incredible incredible is offline
Free Member
 
Join Date: May 2003
Location: Germany
Posts: 3,189
Thanks: 0
Thanked 0 Times in 0 Posts
Send a message via ICQ to incredible
Quote:
Originally Posted by kwag
Quote:
Originally Posted by incredible
I said the "default" calculation of my sampler AVS routine gives me a 5% of the whole movie output which should match 5% of 785(-Audiosize)MB IF CQ is set right!
No. The 5% you encoded will not be the same size as another 5% of some other part of the movie. If you got a final size close to your wanted, you were lucky

-kwag
Ok ... lets see it in the case if we do a 2pass before .. as I also did.
The 2% "reference"-output as it's based on the sampling routine in my script gives me an output of 2% of the whole amount of frames of the movie and .... the 2pass result will give me a size which would fit the needed final size (in this case 2% of 785) cause its based on avrg.Bitrate calculation.
So i.e. I calculate using Calcumatic ... to fit 2%movie on 2%media I need an average of i.e. 700kbit. and thats the point .... the 2% reference encoding is not made using CQ which really would give me on each slice diff. sizes as you mentioned.

And thats also what Krassi and I talked about yesterday in the Chat.
2% will be a sliced-amount of minutes which as to fit to a determined size (in this case as it makes the sense: 2% of 785) and thats what 2pass should calculate correct if Tmpegs internal calculation does this correct!?.
No matter if the reference-samplestream includes high motion or less moving dark scenes --- the result will base upon average bitrate and thats why it ca. matches. Maybe within the sample encoding the allocation of the bitrates will differ a lot in comparosion to another sliced sample but the output will be ca. the same 2pass

And as Krassi also said ... to do this calculation the 2pass of tmpgenc is not needed cause it also refers to a formula where the average bitrate is the main component of the calculation to fit the result to the wanted size.
And that's also a point ... can we believe tmpgEnc's internal calculation to fit an amount of frames refering to average bitrate to a needed final size?
And to be shure Krassi used a mathematical formula.

I think IF there would be a size-difference in encoded 2%-reference streams this will not belong to different slice-contents but to tmpgEnc internal 2pass avrg. bitrate calculation. And so I did yesterday again a new test:

- No 2pass before done to get the refered reference-size but used Krassis formula also based on an avrg.bitrate value determined using Calcumatic.

- Movie "The english Patient", ca.150mins 480x576mpeg1 using a gop length of 84! (1-41-1) to test. Therefore the variable Gop in the script was determinet to 84! but I choosed a GOPlength multiplicator of 1 cause 84 frames each slice are enough.

-Again sliced-sampler size calculation set to 2% in the script.

- ca. 9 ping-pongs where needed. In this case pong-offset was set to 40 sec! Cause of less slices (larger gop lenght!) where auto-calculated by the script to get still 2% of the movie.

- Final encoding and therfore muxed output was a 787 MB! mpeg stream, 795 where wanted.

I can't proof that this is really Luck BUT Im Going to to more encodings to see the average acuracy of this way .... just to be safe.



But Kwag .... I think the thread finally NOW does not base only on "2pass or not" , it changed ALSO to verify a "offset ping-pong" calculation method (which by the way is not new) ... so if there still will be a precise calculation needed refering to generate a acuarate first-reference sample ... then this will be another workout. There i agree with you definitely, thats right.
I don't say THIS IS the right method to determine the first-reference sample stream! (although my tests do match)
BUT even if we got a precise method to determine the right referenze stream size ... we still have to do prediction turns to match the right reference size ... and IMHO this will be done best by using offset-intervalls in CQ matic which is faster than larger samples and even more precise cause the offset-method offers more DIFFERENT scene contents in the slices within the whole prediction.
And exactly that's it what I wanted to say in the other thread where you replied.

But as I said ... Im not at the end ... Ill do more and more tests and maybe if this will end by seeing that there will be needed a right first reference sample calcu-method we continue by figure that out.
Belongig to acurate predictions I will not give up
Reply With Quote
  #72  
11-10-2003, 07:10 AM
bman bman is offline
Free Member
 
Join Date: Apr 2002
Posts: 356
Thanks: 0
Thanked 0 Times in 0 Posts
First of all I want to thank to Incredible and Krassy for this interesting thread .
I hope it'll give us perfect file size prediction method.

@ Incredible
I'm trying to follow this thread and to understand how exectly are u predicting file size and I have question (with your permission ) :
1. This method coould be good for predicting sizes for vcd,svcd,xvcd and DVD's too ??? ( as I see it answer should be - YES , or I'm wrong???)
2. are u runung off=0 and off=30 scripts in manual way or u use some prgram code ?
3. Which encoder do u prefer for best results ???(TMPG ,CCE ,MCE???)
4. Did u tryed to compare CCE vs Tmpg for prediction accuracy with your method???

I know that u have made many tests and I'm just currious about results of prediction with diff encoders .
To make it more clear , can u make mini HOW-To and put it here ???
It'll be so helpful ( for me for sure ).

Results are wery promissing , Keep going !!!!
bman
Reply With Quote
  #73  
11-10-2003, 07:22 AM
Krassi Krassi is offline
Free Member
 
Join Date: Mar 2003
Location: Germany
Posts: 390
Thanks: 0
Thanked 0 Times in 0 Posts
Hi bman,

thanks for your input
We are changing the offset manually, but here's a short how-to:

1) Take incredible's script, make your adaptions (e.g. Resizing Values and sa)
2) Change sa to 5 to reflect a 2% sample of the movie
2a) Change the Variable GOP to the amount of frames within your Goplength you're encoding with .... like 24 in case of standard KVCD
3) Make a 2-pass VBR encoding on that script or calculate it with the following formula: (Frames of Sample/framerate)/8*(average bitrate).
Tip: You can see the Frames of the sample when you start an encoding of the sample. You can calculate the average bitrate with CalcuMatic.
4) Try to find a CQ matching this sample size. If found note it and
5) Change the offset value to 30 and redo a prediction until sample size is matiching again
6) Take the average of this value and do a full encode. You only need to comment the sampler line.
7) Post the results

CCE gives about the same accuracy, the only problem is that the CQ values aren't precise enough.I'm using TMPGEnc for now.
My range is min: 300 to max 5000. So that's targeting for KDVD.

EDIT: Edited Mini-Howto with point 2a. Thanks incredible
Reply With Quote
  #74  
11-10-2003, 08:01 AM
incredible incredible is offline
Free Member
 
Join Date: May 2003
Location: Germany
Posts: 3,189
Thanks: 0
Thanked 0 Times in 0 Posts
Send a message via ICQ to incredible
Quote:
Originally Posted by bman
I'm trying to follow this thread and to understand how exectly are u predicting file size and I have question (with your permission ) :
You're very welcome to participate

First, I can only give you answers based on my experiences, .... maybe Kwag is right ... maybe I'm right .... maybe another Member will figure out the right way using a total diffenrent method ... we don't know by now.
And I hope I use the correct english words so noone will get confused.
Quote:
1. This method coould be good for predicting sizes for vcd,svcd,xvcd and DVD's too ??? ( as I see it answer should be - YES , or I'm wrong???)
As the logic of this way is based on for example a 2%sample of the whole movie (outputted by the script) and that you (as I see) could determine a reference encoding-size-sample based on also 2% of the whole video which gives you the needed size and therefore which we have to match .... it should be independed from what you're going to encode, ... KVCD KsVCD, KDVD. So "if" this way works it logically will be independed. So the encoded reference sample will take more size in case of for example a DVD encoding which usually can take more Average Bitrate within the stream. And the average Bitrate you can calculate by using Calcumatic as it supports also DVD and independend media/final stream sizes. Therefore you have to match the larger sized reference-sample on your hd.
Quote:
2. are u runung off=0 and off=30 scripts in manual way or u use some prgram code ?
Yes, Im doing that manually by changing the "off"-value cause until now I don't know how to develope the right program code which performs this automatically .... and in here we're talking about new ways to implementate in a already existing program code .... CQ matic
Quote:
3. Which encoder do u prefer for best results ???(TMPG ,CCE ,MCE???)
Do you mean Prediction results?? Well as TmpgEnc offers that you can determine more preciser CQ values then CCE (As I saw CCE only accepts rounded Q values) this gave me a more acurate final encoding result ... but it's also working using CCE! The only disadvantage by using CCE is that you could get less acuracy in your result ---- but I love CCE as it performs fast and as its my first choice when encoding DVD video streams.
Mainconcept is even worth when predictionning cause of less amount of possible Q values!
Quote:
4. Did u tryed to compare CCE vs Tmpg for prediction accuracy with your method???
See lines above ....
Quote:
I know that u have made many tests and I'm just currious about results of prediction with diff encoders .
The lines above also do explain why ...



Oh ... I see now Krassi already answered ...


AND Bman....... :

As Krassi already posted a how to.
- Use the GOP refering script!
- Use the same logic of offset "pin-pong" as I did in my reports above!

You also can figure out the amount of frames of your script generated sample by opening the script in Vdub .... switch to the last frame and see the framecount within the generated sample below in Vdub.
It should take about 2% of the whole amount of frames in case of the "unscliced" source if varaible sa=5 (5= 2%, 2=5%, 1=10% sample size to match!) Based on (100/10)/sa = sample size

To see if the offset is rigth I inculded subs in the script in case of trying and playing ... just to get a feeling for this!

So if you manual predict including subs ... therefore you also have to full-encode including the subs! Later you can perform a manual real-world prediction by deactivating the sub- lines in the avs script.
Reply With Quote
  #75  
11-10-2003, 09:34 AM
bman bman is offline
Free Member
 
Join Date: Apr 2002
Posts: 356
Thanks: 0
Thanked 0 Times in 0 Posts
Thank's Krassy & Incredible !
I think that now is clear for me what to do so as soon as I'll get home I'll try this new method .
One more quastion : How do u now what CQ u should start encoding ( or it do not metter ) and how long it takes to find RIGHT CQ value this way .
I assume that as u are running all stuff manually than it should take an about half or 3/4 of hour .
Is my assumption right ?
If so than it's really precise but not so fast way (yet )!
Maybe some day whan it'll be implemented in automathic prog than .....
Anyway , great job .
I LOVE this site
bman
Reply With Quote
  #76  
11-10-2003, 09:47 AM
Krassi Krassi is offline
Free Member
 
Join Date: Mar 2003
Location: Germany
Posts: 390
Thanks: 0
Thanked 0 Times in 0 Posts
Quote:
Originally Posted by bman
Thank's Krassy & Incredible !
Your welcome
Quote:
One more quastion : How do u now what CQ u should start encoding ( or it do not metter ) and how long it takes to find RIGHT CQ value this way .
I assume that as u are running all stuff manually than it should take an about half or 3/4 of hour .
Actually, it's not that long. But it depends on your power machine
So a good start in CQ is 60, but you can guess what you want at start. If it's not a long movie you could start with 70...
Quote:
Maybe some day whan it'll be implemented in automathic prog than .....
We have to alot more work on this before to make sure everything is going right.
Quote:
I LOVE this site
bman
Right
Reply With Quote
  #77  
11-10-2003, 09:59 AM
incredible incredible is offline
Free Member
 
Join Date: May 2003
Location: Germany
Posts: 3,189
Thanks: 0
Thanked 0 Times in 0 Posts
Send a message via ICQ to incredible
Quote:
Originally Posted by bman
Thank's Krassy & Incredible !
I think that now is clear for me what to do so as soon as I'll get home I'll try this new method .
One more quastion : How do u now what CQ u should start encoding ( or it do not metter ) and how long it takes to find RIGHT CQ value this way .
I assume that as u are running all stuff manually than it should take an about half or 3/4 of hour .
Is my assumption right ?
If so than it's really precise but not so fast way (yet )!
Maybe some day whan it'll be implemented in automathic prog than .....
Anyway , great job .
I LOVE this site
bman
Well the CQ to start with is based on my assumption as I also use in CQmatic my assumed CQ to start with.
How long it takes? Well if you start using a very near CQ as I did in the reports above ... you'll match it faster
3/4 of hour??? No!
First ... also when using automatic program codes like CQ the prediction time depends on your encoding size, CPU and so on.
Try first using an encoding size of 352x288! As default in the script --- this gives you a faster prediction to do your test workout.
So in my case and my CPU as I remember every turn took about 1-2 minutes (352x28 and as you see the reports above (wich show the count of turns) you can assume how much it took, including the manual setting of off=x and re-safing the .avs every time...

EDIT: And again, meanwhile Krassi found the english words much faster
Reply With Quote
  #78  
11-12-2003, 09:04 AM
incredible incredible is offline
Free Member
 
Join Date: May 2003
Location: Germany
Posts: 3,189
Thanks: 0
Thanked 0 Times in 0 Posts
Send a message via ICQ to incredible
---------------------------------- Next step ------------------------------------

Other successful prediction yesterday evening using this method:

"Gladiator" 480x576 mpeg1 149min GOP = 84 (1-41-1) 112kbit Audio

muxed mpg. size = 783MB ... 795 wanted (CDr80)
Reply With Quote
  #79  
11-12-2003, 09:34 AM
nicksteel nicksteel is offline
Free Member
 
Join Date: Nov 2002
Posts: 863
Thanks: 0
Thanked 0 Times in 0 Posts
Quote:
Originally Posted by incredible
---------------------------------- Next step ------------------------------------

Other successful prediction yesterday evening using this method:

"Gladiator" 480x576 mpeg1 149min GOP = 84 (1-41-1) 112kbit Audio

muxed mpg. size = 783MB ... 795 wanted (CDr80)
Hi, Incredible.

Could you also test with MPEG2? I've found that you can't simply treat MPEG2 the same as MPEG1 and just subtract 25MB for SVCD overhead.
Reply With Quote
  #80  
11-12-2003, 09:41 AM
incredible incredible is offline
Free Member
 
Join Date: May 2003
Location: Germany
Posts: 3,189
Thanks: 0
Thanked 0 Times in 0 Posts
Send a message via ICQ to incredible
That will be the next testing ...!
Reply With Quote
Reply




Similar Threads
Thread Thread Starter Forum Replies Last Post
TMPGEnc: Using SSRC as an external audio engine syk2c11 Video Encoding and Conversion 3 06-01-2004 05:28 AM
My new car engine kwag Off-topic Lounge 6 10-03-2003 10:05 PM
Multi pass size prediction girv Avisynth Scripting 14 07-14-2003 04:51 PM
Faster prediction method ARnet_tenRA Avisynth Scripting 19 04-12-2003 09:11 AM
Encoding single-pass KVCD vs. two-pass logan555 Video Encoding and Conversion 1 12-04-2002 09:18 AM

Thread Tools



 
All times are GMT -5. The time now is 02:31 PM  —  vBulletin Jelsoft Enterprises Ltd