Quantcast A New Way to Calculate CQ... Maybe? - digitalFAQ.com Forums [Archives]
  #1  
01-25-2003, 03:48 PM
titans_ca titans_ca is offline
Free Member
 
Join Date: Sep 2002
Location: Toronto
Posts: 22
Thanks: 0
Thanked 0 Times in 0 Posts
I was tinkering with file prediction yesterday, and wished that there were an easier method of calculating the correct CQ. then an idea hit me, and I tried some tests it seem to have merit so far, so I'm just gonna throw this out on the table, maybe someone will come up with something cool

What I did was I used KVCDx3-MPEG-2-NTSCFilm template to encode short clips of 1000 frames increasing CQ slowly from 60 up to 61, going like 60.1, 60.2, 60.3 etc.
I did this for two different sources, and then put the file sizes into excel and ploted some graphs, the results I found was prettey interesting.

Note1: The two sources I used are divx animes, not dvd rips of movies so whether or not the result I found applies there needs to be tested.
Note2: Also I used 1000 frames because I wanted as little variable between the two comparisons as possible.

Here's the AVS script I used to encode the test files

Quote:
LoadPlugin("C:\Movies Conversion\Filters\MPEG2DEC2.dll")
LoadPlugin("C:\Movies Conversion\Filters\fluxsmooth.dll")
LoadPlugin("C:\Movies Conversion\Filters\blockbuster.dll")
LoadPlugin("C:\Movies Conversion\Filters\legalclip.dll")
LoadPlugin("C:\Movies Conversion\Filters\GripFit_Preview.dll")
LoadPlugin("C:\Movies Conversion\Filters\Convolution3D.dll")
LoadPlugin("C:\Movies Conversion\Filters\DustV5.dll")
LoadPlugin("C:\Movies Conversion\Filters\sampler.dll")

AviSource("E:\Complete\Noir\Noir - 02.avi")
converttoyuy2()
LegalClip()
GripCrop(544, 480, overscan=2)
GripSize()
SpaceDust()
Convolution3d(preset="animeLQ")
Blockbuster(method="noise", variance=.5, seed=1)
GripBorders()
LegalClip()

Sampler(samples=40, length=25) <--- Takes a sample of 1000 frames
## MPEG size = ((Total frames/MovieTimeInMinutes)/24) * MPEG sample file size ##

So the result I found was that when you increases CQ the file size also increases in a near linear fashion. And this occured in both clips.
And the increases in both files seem to be parallel.

Here's the data that I got:

Quote:
CQ
60.00
60.10
60.20
60.30
60.40
60.50
60.60
60.70
60.80
60.90
61.00

First File
4,595,709
4,595,986
4,596,291
4,596,537
4,596,886
4,597,171
4,597,510
4,597,876
4,598,175
4,598,485
4,598,734

Second File
3,881,973
3,882,332
3,882,642
3,882,931
3,883,255
3,883,513
3,883,800
3,884,128
3,884,493
3,884,799
3,885,157

Adjusted First file: (this is values from First file minusing the average difference between first and second file which is 713,677)
3,882,042
3,882,319
3,882,624
3,882,870
3,883,219
3,883,504
3,883,843
3,884,209
3,884,508
3,884,818
3,885,067
( I posted the data this way so it's easier to cut and paste into excel)

if you paste these values into excel and graph them you'll see what I mean, sorry but I don't know how to post a image, otherwise I'd post the graphs.

So what I was wondering is:
1. does this pattern holds true for other templates?
2. Does it hold true for DVD movies?
3. What about higher CQ, is the pattern the same?
4. would the result be consistent if we used the sampler?
5. Will changes in filters affect the file size?

Because if the pattern is the same in all cases, we can pool our data and find out what's the equation for the line, and just encode one sample and we can use the existing information to extrapolate the CQ required.

But then again this might be too much of a hassle, since there are so many variable to test.

of course this is just an idea, maybe someone with better math background can tell me if there's anything wrong with my logic. or someone can write a software to make the process automatic. But since math and programing are not my strong points I'll leave those to the experts . if you think this idea has some merit to it, I'd be glad to do more tests and post the results.

Thanks for reading my rambling , and I just want to say to kwag, sansgrip and all the others great work everyone, hopefully this could make it even better, or maybe a little bit easier.
Reply With Quote
Someday, 12:01 PM
admin's Avatar
Site Staff / Ad Manager
 
Join Date: Dec 2002
Posts: 42
Thanks: ∞
Thanked 42 Times in 42 Posts
  #2  
01-26-2003, 11:03 AM
titans_ca titans_ca is offline
Free Member
 
Join Date: Sep 2002
Location: Toronto
Posts: 22
Thanks: 0
Thanked 0 Times in 0 Posts
Yeah figure out how to post the images.

So, any comments?


Reply With Quote
  #3  
01-26-2003, 11:14 AM
kwag kwag is offline
Free Member
 
Join Date: Apr 2002
Location: Puerto Rico, USA
Posts: 13,537
Thanks: 0
Thanked 0 Times in 0 Posts
Hi titans_ca,

Great graph and test . The only problem I see is that this is on a one step increment of CQ. The CQ vs. file size is not linear if you try CQ values from 1 to 100. It's (sort of) linear from 1 to about 50. After 50, the curve starts to flatten. I just said 50, but that was with the original KVCD matrix and GOP. I believe this curve has changed with the new KVCD-BETA-1a notch matrix and 1-5823-2-1-24 GOP. Also the different resolutions affect the curve. It's very unpredictable

-kwag
Reply With Quote
  #4  
01-27-2003, 04:11 AM
gonzopdx gonzopdx is offline
Free Member
 
Join Date: Dec 2002
Location: Corvallis, OR
Posts: 86
Thanks: 0
Thanked 0 Times in 0 Posts
I've also noticed a sharp spike in file size when going from 79.9 to 80.. sometimes between 70-70.9 will be linear, growing in very small amounts (500-ish kb or less), then will suddenly leap to over 2 megs larger when you use 80.

This isn't consistent with every source, but just goes to show that it really is unpredictable.
Reply With Quote
  #5  
01-27-2003, 12:51 PM
titans_ca titans_ca is offline
Free Member
 
Join Date: Sep 2002
Location: Toronto
Posts: 22
Thanks: 0
Thanked 0 Times in 0 Posts
I did some further tests , going from CQ 60-80 (60, 61, 62, 63 etc.)
and you are right the relation is no longer linear, but it's not totally random either there's definitly a function that governs the changes in the file size.



now if we can find that function, then finding the correct CQ would be easy.

So I'm wondering when the CQ is incresed, what does it improve to get the larger file size? or in other words what factors does CQ effect? anyone knows?
Reply With Quote
  #6  
02-27-2003, 02:19 PM
PyRoMaNiA PyRoMaNiA is offline
Free Member
 
Join Date: Jan 2003
Location: a PAL land (UK)
Posts: 408
Thanks: 0
Thanked 0 Times in 0 Posts
Looks like everyone's forgotten about this thread....which is a shame as it looks like a good idea.....

Seems to me that the CQ, although it is not totally linear, MUST follow some kind of pattern, assuming that it is the same clip with the same script every time...so if we had samples from CQ 0.1 to 100, then wouldn't it be easy to make a spreadsheet in Excel where all we had to do was enter the current CQ and current size? But then I'm sort of new to this stuff and I could be completely wrong....

I'll go make some samples with some short clip.............
Reply With Quote
  #7  
02-27-2003, 03:39 PM
ARnet_tenRA ARnet_tenRA is offline
Free Member
 
Join Date: Jan 2003
Location: Illinois, USA
Posts: 73
Thanks: 0
Thanked 0 Times in 0 Posts
I have already done this with samplepoints at every 5 CQs. The graph is not linear over the entire CQ range. Here is a graph:

I do prediction the following way. I encode a sample at CQ 50 and then I look at the chart I have created for the next best guess based on how far I am from the desired size. After that I use the predict2 function and enter the values for these two samples and I get very close to the final CQ with this.

Here is my chart of CQ values by 5s:
Code:
CQ	Scale Factor
0	  0.910172095
5	  0.915119357
10	 0.922162579
15	 0.927555593
20	 0.933867258
25	 0.941058657
30	 0.953374653
35	 0.96019634
40	 0.970685007
45	 0.983586023
50	 1
55	 1.025307964
60	 1.063866729
65	 1.121782825
70	 1.161976976
75	 1.322610111
80	 1.484164068
85	 1.787561288
90	 1.919594168
95	 1.956110637
100	1.992761229
Regards, Tenra
Reply With Quote
  #8  
02-27-2003, 10:04 PM
rendalunit rendalunit is offline
Free Member
 
Join Date: Apr 2002
Location: san jose, Ca
Posts: 1,148
Thanks: 0
Thanked 0 Times in 0 Posts
Quote:
Originally Posted by ARnet_tenRA
The graph is not linear over the entire CQ range. Here is a graph:
I have also encoded samples throughout the whole cq range at every 5 increments and I'm shocked at how similar my results are to your graph It is almost linear to cq 50 then increases slightly more until cq 70 to ~85 are the highest jumps, then starts to level off. I'm currently encoding samples at every whole cq and when I'm done I'll graph the 100 samples and post it.

I'm just kind of throwing out an idea here but if the graphs of the whole cq range of encodings usually follow that pattern then maybe two formulas can be derived from the graph: a formula based on a linear equation from cq0 - 50 and another formula for the non- linear equation of 50-100. Even if they're not exact, maybe they can be used to come close Or how about dividing it into 3 equations: one for cq0-cq70, another for cq70-cq85, and another for cq85-100 and then calculate a new cq with the three equations.
Just an idea.

ren
Reply With Quote
  #9  
02-27-2003, 11:06 PM
kwag kwag is offline
Free Member
 
Join Date: Apr 2002
Location: Puerto Rico, USA
Posts: 13,537
Thanks: 0
Thanked 0 Times in 0 Posts
I just had an idea. I hope it's not a crappy one

Why don't we take an X-pass calculator. MovieStacker for example , and find out the average bit rate just like if we were going to do an x-pass vbr encoding.
Here's what I just tried. Say MIN=300, MAX=1,800, and average is found by MovieStacker and say it's 889Kbps. We do a test encode at a CQ=100. Say 15% or less of the movie. We look at the file with bit rate viewer, and look at the average bit rate. If our wanted average is 889Kbps, but we got 1,110Kbps with CQ=100, we do 889 / 1,110=0.800 and then 0.800 * 100=80.090. So 80.9 would be our new CQ value.

Which just so happens to be close to a value I just tested recently. I had encoded "Death to Smoochie" with KVCD 352x240 (for fun and test!) and the file size came out to 772,973MB with audio at a CQ of 80. So the formula above, should be extremely close We would be using average bit rate instead of file size for prediction.
Any thoughts on this

Now we just need a little program to read our sample and tell us the average bit rate, so we don't need to open bit rate viewer

Anyone want to try this

Edit: Just confirmed with the same movie, now at 528x480, and I hit the sampler file size target on just one run of 25%
That is, 25% of the sampler and read average bit rate to get new CQ. Then did the complete sampler, and my sampler file size hit the wanted file size (~11MB) dead on

-kwag
Reply With Quote
  #10  
02-27-2003, 11:20 PM
ARnet_tenRA ARnet_tenRA is offline
Free Member
 
Join Date: Jan 2003
Location: Illinois, USA
Posts: 73
Thanks: 0
Thanked 0 Times in 0 Posts
I could be wrong, but I don't think that there is any difference using the filesize and the average bitrate.

Say a 60sec file of 8,682KB and its average bitrate is 8682/60=144.7KBps

If we encode again and get 8,840KB then its average is 8840/60=147.33KBps.

8840/8682=1.0181986
147.33/144.7=1.0181986

The differences are the same whether you use KB/s (bitrate) or KB/file (filesize)

Let me know if my thinking is wrong on this.

-Tenra
Reply With Quote
  #11  
02-27-2003, 11:23 PM
kwag kwag is offline
Free Member
 
Join Date: Apr 2002
Location: Puerto Rico, USA
Posts: 13,537
Thanks: 0
Thanked 0 Times in 0 Posts
Quote:
Originally Posted by ARnet_tenRA

Let me know if my thinking is wrong on this.

-Tenra
No, your thinking is correct! I agree 100% with you, but the result I'm getting is extremely sharp, and with less math juggling

-kwag
Reply With Quote
  #12  
02-27-2003, 11:26 PM
ARnet_tenRA ARnet_tenRA is offline
Free Member
 
Join Date: Jan 2003
Location: Illinois, USA
Posts: 73
Thanks: 0
Thanked 0 Times in 0 Posts
Kwag,

Actually I think that your formula may be good. I was just saying that it should be able to work for both average bitrate and filesize.

So when you did your prediction did you start with CQ100?

Regards, Tenra
Reply With Quote
  #13  
02-27-2003, 11:32 PM
ARnet_tenRA ARnet_tenRA is offline
Free Member
 
Join Date: Jan 2003
Location: Illinois, USA
Posts: 73
Thanks: 0
Thanked 0 Times in 0 Posts
Kwag,
Also check out the SampleClip function I created. It allows you to shorten your sample by any factor you desire. It should make it easy to produce a smaller sample for your tests.

For instance:
SampleClip() produces the regular 1sec/min of video.
SampleClip(10) produces 1/10th of the above.

Let me know what you think, Tenra
Reply With Quote
  #14  
02-27-2003, 11:37 PM
kwag kwag is offline
Free Member
 
Join Date: Apr 2002
Location: Puerto Rico, USA
Posts: 13,537
Thanks: 0
Thanked 0 Times in 0 Posts
Quote:
Originally Posted by ARnet_tenRA
Kwag,

Actually I think that your formula may be good. I was just saying that it should be able to work for both average bitrate and filesize.

So when you did your prediction did you start with CQ100?
Yes. Start with CQ=100. And I just ran another 20% prediction against S1M0NE, and got a CQ of 69.12, which is VERY close to what I recall I had used on my previous encode
Got that value on one run
Just run MovieStacker and open your .d2v. Use the suggested average bit rate under "Video stream".
Running the complete sampler now to see what the file size will be. It should also be slightly above ~11MB.

Edit: Wanted file size 11MB. Got 10.797MB (Good enough!) and that was with 20% of sampler, so maybe 25% of a run is a good number
Going to Run a test on a third movie now


-kwag
Reply With Quote
  #15  
02-27-2003, 11:59 PM
ARnet_tenRA ARnet_tenRA is offline
Free Member
 
Join Date: Jan 2003
Location: Illinois, USA
Posts: 73
Thanks: 0
Thanked 0 Times in 0 Posts
All right, I think I am a convert.

You can use file size like I said, but who cares? A sample of ANY size can be used with the kbps method. I just did Shrek and found out how good this actually works. A larger sample size is only for increased security, unlike before when it was pretty important.

Thanks Kwag!

That said, we need to run more tests to see if the formula needs any tweaking.

To run 25% of a sample use SampleClip(4).
This will get you a quarter of a sample.

Regards, Tenra
Reply With Quote
  #16  
02-28-2003, 12:01 AM
kwag kwag is offline
Free Member
 
Join Date: Apr 2002
Location: Puerto Rico, USA
Posts: 13,537
Thanks: 0
Thanked 0 Times in 0 Posts
Quote:
Originally Posted by ARnet_tenRA

Let me know what you think, Tenra
Hi Tenra,

Your formula works very good. It still has to do several runs to reach a CQ value (that's not your fault ), because of the nature of CQ vs. file size non-linearity. The thing I see with the method I'm testing, is that MovieStacker is already giving us a very precise average bit rate, and we only have to do one ( very short ( ~25-30% ) run and we have a final CQ
With file size based formulas, it usually takes (with good luck ) at least two full sampler encodes. If this method works, we can forget about even doing a single full size sampler, because on one short sampler test run, we get the final CQ value.
Or the worst case would be just running one full sampler. That should be precise enough

-kwag
Reply With Quote
  #17  
02-28-2003, 12:08 AM
kwag kwag is offline
Free Member
 
Join Date: Apr 2002
Location: Puerto Rico, USA
Posts: 13,537
Thanks: 0
Thanked 0 Times in 0 Posts
Quote:
Originally Posted by ARnet_tenRA
To run 25% of a sample use SampleClip(4).
This will get you a quarter of a sample.

Regards, Tenra


Got it . And I think 25% is good enough, maybe 30% for insurance.
Doing a test on another movie now ("The Last Castle"), which gave me a CQ of 60.746. Let's see if I get the ~11MB wanted
Running full sampler now. Will post result here. Hold on for result in ~15 minutes, because I forgot to change FaeryDust for SpaceDust ..........



Final file size 11,000kB EXACTLY

Edit: BTW, VirtualDub can be used to find out the average bit rate too. Just open the .m1v file with Vdub, and look at "File information". There is a small difference between the information and the one bitrate viewer shows , so we have to find out which is the correct one. I have a feeling that Vdub is the correct one

-kwag
Reply With Quote
  #18  
02-28-2003, 03:38 AM
kwag kwag is offline
Free Member
 
Join Date: Apr 2002
Location: Puerto Rico, USA
Posts: 13,537
Thanks: 0
Thanked 0 Times in 0 Posts
Anyone else getting accurate results, or is it just me

-kwag
Reply With Quote
  #19  
02-28-2003, 07:47 AM
GFR GFR is offline
Free Member
 
Join Date: May 2002
Posts: 438
Thanks: 0
Thanked 0 Times in 0 Posts
Loads of fun!

This curve shows some interesting points.

First, below CQ=50 you don't save too much file size if you drop CQ, so there's little point in going below CQ=50.

Then from CQ=50 to CQ=70, the slope of the curve is not too steep, and you can up CQ without increasing file size too much, but to decrease file size you have to drop CQ quite a bit.

From CQ=70 to CQ=90, the slope of the curve is steep so if you up CQ the file size will increase a lot, on the other way you can drop file size without sacrificing too much CQ.

From CQ=90 to CQ=100 the file size doesn't change a lot (problably visual quality does not change much too).

The shape of the curve is what's called a "sigmoidal curve" in signal processing (it's shaped like an "S"). One such curve is the hyperbolic tangent (tanh). BTW DivX 1-bas quality based behaves with a similar curve.

Size(CQ)=a+b*tanh((c*CQ)-d)

Where the parameters a,b,c,d make the shape generic.

Parameter d sets the "center" of the "S" (~CQ=80 in your curve), parameter c sets the width of the "S", parameters a and b scale the curve to fit the max and min file size.

If we chose good values for a,b,c,d we can fit the formula to your experimental data quite well.

The problem is that I think every movie will have its own a,b,c,d set!

There are several optimization techniques to find the right a,b,c,d for a movie. Since that's four parameters you should need four test runs (that can be 25% "short samples"). You enter this data in excel and use the "Solve" tool to find out a,b,c,d, then you use the formula to estimate the file size for any CQ.

Another option is to collect a large database of size X CQ curves (we can use a sticky in the forum to collect the data) and then train a neural net that will perform the estimation (I can train the net and create a GUI). The potential advantage of the neural net is that it MAY be able to learn any correlation that MAY exist between a,b,c,d, so MAYBE we need less test runs instead of four to create a good model of the curve.

Quote:
Originally Posted by ARnet_tenRA
I have already done this with samplepoints at every 5 CQs. The graph is not linear over the entire CQ range. Here is a graph:

I do prediction the following way. I encode a sample at CQ 50 and then I look at the chart I have created for the next best guess based on how far I am from the desired size. After that I use the predict2 function and enter the values for these two samples and I get very close to the final CQ with this.

Here is my chart of CQ values by 5s:
Code:
CQ	Scale Factor
0	  0.910172095
5	  0.915119357
10	 0.922162579
15	 0.927555593
20	 0.933867258
25	 0.941058657
30	 0.953374653
35	 0.96019634
40	 0.970685007
45	 0.983586023
50	 1
55	 1.025307964
60	 1.063866729
65	 1.121782825
70	 1.161976976
75	 1.322610111
80	 1.484164068
85	 1.787561288
90	 1.919594168
95	 1.956110637
100	1.992761229
Regards, Tenra
Reply With Quote
  #20  
02-28-2003, 09:15 AM
ARnet_tenRA ARnet_tenRA is offline
Free Member
 
Join Date: Jan 2003
Location: Illinois, USA
Posts: 73
Thanks: 0
Thanked 0 Times in 0 Posts
Quote:
Originally Posted by GFR
The problem is that I think every movie will have its own a,b,c,d set!
You are correct. I gathered the values in my table by averaging the numbers from 4 movies. None of the movies was exactly these numbers.

Regards, Tenra
Reply With Quote
Reply




Similar Threads
Thread Thread Starter Forum Replies Last Post
KVCD: How to calculate required bitrate? pappukanghi Video Encoding and Conversion 8 10-13-2004 11:47 AM
Bitrates: Does Calcumatic (and others) calculate correctly average bitrate Prodater64 Video Encoding and Conversion 3 08-25-2004 03:55 AM
BBMpeg: Calculate exact output size? azel Video Encoding and Conversion 3 07-22-2003 06:15 PM
DVD2SVCD: Can IfoEdit calculate frame count and movie length ? black prince Video Encoding and Conversion 5 12-08-2002 12:19 AM
Program to Calculate CQ file size? marky Video Encoding and Conversion 2 06-16-2002 05:10 PM

Thread Tools



 
All times are GMT -5. The time now is 06:04 AM  —  vBulletin Jelsoft Enterprises Ltd