A new way to calculate CQ... maybe?
 

I was tinkering with file prediction yesterday, and wished that there were an easier method of calculating the correct CQ. then an idea hit me, and I tried some tests it seem to have merit so far, so I'm just gonna throw this out on the table, maybe someone will come up with something cool :D

What I did was I used KVCDx3-MPEG-2-NTSCFilm template to encode short clips of 1000 frames increasing CQ slowly from 60 up to 61, going like 60.1, 60.2, 60.3 etc.
I did this for two different sources, and then put the file sizes into excel and ploted some graphs, the results I found was prettey interesting.

Note1: The two sources I used are divx animes, not dvd rips of movies so whether or not the result I found applies there needs to be tested.
Note2: Also I used 1000 frames because I wanted as little variable between the two comparisons as possible.

Here's the AVS script I used to encode the test files


So the result I found was that when you increases CQ the file size also increases in a near linear fashion. And this occured in both clips.
And the increases in both files seem to be parallel.

Here's the data that I got:

( I posted the data this way so it's easier to cut and paste into excel)

if you paste these values into excel and graph them you'll see what I mean, sorry but I don't know how to post a image, otherwise I'd post the graphs.

So what I was wondering is:
1. does this pattern holds true for other templates?
2. Does it hold true for DVD movies?
3. What about higher CQ, is the pattern the same?
4. would the result be consistent if we used the sampler?
5. Will changes in filters affect the file size?

Because if the pattern is the same in all cases, we can pool our data and find out what's the equation for the line, and just encode one sample and we can use the existing information to extrapolate the CQ required.

But then again this might be too much of a hassle, since there are so many variable to test.

of course this is just an idea, maybe someone with better math background can tell me if there's anything wrong with my logic. :roll: or someone can write a software to make the process automatic. But since math and programing are not my strong points I'll leave those to the experts :) . if you think this idea has some merit to it, I'd be glad to do more tests and post the results.

Thanks for reading my rambling :) , and I just want to say to kwag, sansgrip and all the others great work everyone, hopefully this could make it even better, or maybe a little bit easier.

Yeah figure out how to post the images.

So, any comments? :?:


http://www.digitalfaq.com/archives/error.gif

Hi titans_ca,

Great graph and test :). The only problem I see is that this is on a one step increment of CQ. The CQ vs. file size is not linear if you try CQ values from 1 to 100. It's (sort of) linear from 1 to about 50. After 50, the curve starts to flatten. I just said 50, but that was with the original KVCD matrix and GOP. I believe this curve has changed with the new KVCD-BETA-1a notch matrix and 1-5823-2-1-24 GOP. Also the different resolutions affect the curve. It's very unpredictable :roll:

-kwag

I've also noticed a sharp spike in file size when going from 79.9 to 80.. sometimes between 70-70.9 will be linear, growing in very small amounts (500-ish kb or less), then will suddenly leap to over 2 megs larger when you use 80.

This isn't consistent with every source, but just goes to show that it really is unpredictable.

I did some further tests :) , going from CQ 60-80 (60, 61, 62, 63 etc.)
and you are right the relation is no longer linear, but it's not totally random either there's definitly a function that governs the changes in the file size.

http://www.digitalfaq.com/archives/error.gif

now if we can find that function, then finding the correct CQ would be easy.

So I'm wondering when the CQ is incresed, what does it improve to get the larger file size? or in other words what factors does CQ effect? anyone knows?

Looks like everyone's forgotten about this thread....which is a shame as it looks like a good idea.....

Seems to me that the CQ, although it is not totally linear, MUST follow some kind of pattern, assuming that it is the same clip with the same script every time...so if we had samples from CQ 0.1 to 100, then wouldn't it be easy to make a spreadsheet in Excel where all we had to do was enter the current CQ and current size? But then I'm sort of new to this stuff and I could be completely wrong.... :roll:

I'll go make some samples with some short clip............. :D

I have already done this with samplepoints at every 5 CQs. The graph is not linear over the entire CQ range. Here is a graph:
http://www.digitalfaq.com/archives/error.gif
I do prediction the following way. I encode a sample at CQ 50 and then I look at the chart I have created for the next best guess based on how far I am from the desired size. After that I use the predict2 function and enter the values for these two samples and I get very close to the final CQ with this.

Here is my chart of CQ values by 5s:
Regards, Tenra

I have also encoded samples throughout the whole cq range at every 5 increments and I'm shocked at how similar my results are to your graph 8O 8O It is almost linear to cq 50 then increases slightly more until cq 70 to ~85 are the highest jumps, then starts to level off. I'm currently encoding samples at every whole cq and when I'm done I'll graph the 100 samples and post it.

I'm just kind of throwing out an idea here but if the graphs of the whole cq range of encodings usually follow that pattern then maybe two formulas can be derived from the graph: a formula based on a linear equation from cq0 - 50 and another formula for the non- linear equation of 50-100. Even if they're not exact, maybe they can be used to come close :?: Or how about dividing it into 3 equations: one for cq0-cq70, another for cq70-cq85, and another for cq85-100 and then calculate a new cq with the three equations.
Just an idea.

ren

I just had an idea. I hope it's not a crappy one 8O

Why don't we take an X-pass calculator. MovieStacker for example 8) , and find out the average bit rate just like if we were going to do an x-pass vbr encoding.
Here's what I just tried. Say MIN=300, MAX=1,800, and average is found by MovieStacker and say it's 889Kbps. We do a test encode at a CQ=100. Say 15% or less of the movie. We look at the file with bit rate viewer, and look at the average bit rate. If our wanted average is 889Kbps, but we got 1,110Kbps with CQ=100, we do 889 / 1,110=0.800 and then 0.800 * 100=80.090. So 80.9 would be our new CQ value.

Which just so happens to be close to a value I just tested recently. I had encoded "Death to Smoochie" with KVCD 352x240 (for fun and test!) and the file size came out to 772,973MB with audio at a CQ of 80. So the formula above, should be extremely close :!: We would be using average bit rate instead of file size for prediction.
Any thoughts on this :?: :?:

Now we just need a little program to read our sample and tell us the average bit rate, so we don't need to open bit rate viewer :wink:

Anyone want to try this :!:

Edit: Just confirmed with the same movie, now at 528x480, and I hit the sampler file size target on just one run of 25% :mrgreen:
That is, 25% of the sampler and read average bit rate to get new CQ. Then did the complete sampler, and my sampler file size hit the wanted file size (~11MB) dead on :mrgreen:

-kwag

I could be wrong, but I don't think that there is any difference using the filesize and the average bitrate.

Say a 60sec file of 8,682KB and its average bitrate is 8682/60=144.7KBps

If we encode again and get 8,840KB then its average is 8840/60=147.33KBps.

8840/8682=1.0181986
147.33/144.7=1.0181986

The differences are the same whether you use KB/s (bitrate) or KB/file (filesize)

Let me know if my thinking is wrong on this. 8O

-Tenra

No, your thinking is correct! I agree 100% with you, but the result I'm getting is extremely sharp, and with less math juggling :!:

-kwag

Kwag,

Actually I think that your formula may be good. I was just saying that it should be able to work for both average bitrate and filesize.

So when you did your prediction did you start with CQ100?

Regards, Tenra

Kwag,
Also check out the SampleClip function I created. It allows you to shorten your sample by any factor you desire. It should make it easy to produce a smaller sample for your tests.

For instance:
SampleClip() produces the regular 1sec/min of video.
SampleClip(10) produces 1/10th of the above.

Let me know what you think, Tenra

Yes. Start with CQ=100. And I just ran another 20% prediction against S1M0NE, and got a CQ of 69.12, which is VERY close to what I recall I had used on my previous encode :mrgreen:
Got that value on one run :mrgreen:
Just run MovieStacker and open your .d2v. Use the suggested average bit rate under "Video stream".
Running the complete sampler now to see what the file size will be. It should also be slightly above ~11MB.

Edit: Wanted file size 11MB. Got 10.797MB (Good enough!) :D and that was with 20% of sampler, so maybe 25% of a run is a good number :idea:
Going to Run a test on a third movie now :)

-kwag

All right, I think I am a convert.

You can use file size like I said, but who cares? A sample of ANY size can be used with the kbps method. I just did Shrek and found out how good this actually works. A larger sample size is only for increased security, unlike before when it was pretty important.

Thanks Kwag! :D

That said, we need to run more tests to see if the formula needs any tweaking.

To run 25% of a sample use SampleClip(4).
This will get you a quarter of a sample.

Regards, Tenra

Hi Tenra,

Your formula works very good. It still has to do several runs to reach a CQ value (that's not your fault :) ), because of the nature of CQ vs. file size non-linearity. The thing I see with the method I'm testing, is that MovieStacker is already giving us a very precise average bit rate, and we only have to do one ( very short ( ~25-30% ) run and we have a final CQ :D
With file size based formulas, it usually takes (with good luck ) at least two full sampler encodes. If this method works, we can forget about even doing a single full size sampler, because on one short sampler test run, we get the final CQ value.
Or the worst case would be just running one full sampler. That should be precise enough 8)

-kwag

:mrgreen:

Got it :!: . And I think 25% is good enough, maybe 30% for insurance.
Doing a test on another movie now ("The Last Castle"), which gave me a CQ of 60.746. Let's see if I get the ~11MB wanted :D
Running full sampler now. Will post result here. Hold on for result in ~15 minutes, because I forgot to change FaeryDust for SpaceDust :twisted: ..........

:ole:

Final file size 11,000kB EXACTLY :mrgreen:

Edit: BTW, VirtualDub can be used to find out the average bit rate too. Just open the .m1v file with Vdub, and look at "File information". There is a small difference between the information and the one bitrate viewer shows :!:, so we have to find out which is the correct one. I have a feeling that Vdub is the correct one :wink:

-kwag

Anyone else getting accurate results, or is it just me :? 8O

-kwag

Loads of fun!

This curve shows some interesting points.

First, below CQ=50 you don't save too much file size if you drop CQ, so there's little point in going below CQ=50.

Then from CQ=50 to CQ=70, the slope of the curve is not too steep, and you can up CQ without increasing file size too much, but to decrease file size you have to drop CQ quite a bit.

From CQ=70 to CQ=90, the slope of the curve is steep so if you up CQ the file size will increase a lot, on the other way you can drop file size without sacrificing too much CQ.

From CQ=90 to CQ=100 the file size doesn't change a lot (problably visual quality does not change much too).

The shape of the curve is what's called a "sigmoidal curve" in signal processing (it's shaped like an "S"). One such curve is the hyperbolic tangent (tanh). BTW DivX 1-bas quality based behaves with a similar curve.

Size(CQ)=a+b*tanh((c*CQ)-d)

Where the parameters a,b,c,d make the shape generic.

Parameter d sets the "center" of the "S" (~CQ=80 in your curve), parameter c sets the width of the "S", parameters a and b scale the curve to fit the max and min file size.

If we chose good values for a,b,c,d we can fit the formula to your experimental data quite well.

The problem is that I think every movie will have its own a,b,c,d set!

There are several optimization techniques to find the right a,b,c,d for a movie. Since that's four parameters you should need four test runs (that can be 25% "short samples"). You enter this data in excel and use the "Solve" tool to find out a,b,c,d, then you use the formula to estimate the file size for any CQ.

Another option is to collect a large database of size X CQ curves (we can use a sticky in the forum to collect the data) and then train a neural net that will perform the estimation (I can train the net and create a GUI). The potential advantage of the neural net is that it MAY be able to learn any correlation that MAY exist between a,b,c,d, so MAYBE we need less test runs instead of four to create a good model of the curve.

You are correct. I gathered the values in my table by averaging the numbers from 4 movies. None of the movies was exactly these numbers.

Regards, Tenra

Kwag,

Can I make a sticky post to collect some data like Tenra's table? I think with the high activity here I'd be able to have a working file size curve estimator in a month.

Sure you can :D Fine, but did you read my posts on the previous page of this thread, regarding average bit rate vs. file size :!: 8)

-kwag

Kwag,

Here is the new prediction function to give you video bitrate:

I've read it, but I still don't get why average bitrate would be any different from file size, after all average bit rate = file size (in kb) / sample length (in s). That is, file size is in direct proportion with bit rate. My guess is that you were just lucky :). I'm gonna test it anyway.

BTW I always use FitCD/MovieStacker to get my target file size, because it takes in account things like muxing, multiple subtitles, multiple audios, extras, etc.

Anyway if it's discovered that bitrate is more precise for some reason, we can collect bit rate X CQ tables instead.

@ARnet_tenRA
Would you post how this function would used in a script?

-Yoda

Just place this function at the bottom of your script or in another file and import it.

Then at the bottom of your script , just before SampleClip or Sampler place Prediction()

The first frame of the AVS file is now the output of this function.

Regards, Tenra

This was my point before and it is still true. For prediction, average bitrate and filesize will work equally well. The reason that average bitrate is nice is because you don't have to first determine a target filesize. You just look at the bitrate of your sample (no matter the filesize) and use that for future predictions.

-Tenra

What can I say, BEAUTIFUL :mrgreen:

-kwag

What is the desired avg. bitrate :?:

Here's three samples of Heist
The problem i see with determining the cq from the bitrate is that it's not proportional to it. However the bitrate is directly proportional to the filesize but that doesn't help to find the cq value 8O .

ren

Hi ren,

Open your .d2v with MovieStacker, and look at the suggested average bit rate. Say it is 890Kbps. With your numbers above, you would take the average produced with CQ=100. So 890/1767=50.36 So 50.36 would be the suggested CQ value ( kinda low for one CD :? )
But you don't need to run a full sampler. Just ~30% should do.

-kwag

-kwag

@Tenra,
Just ran an average bit rate sample on the movie "xXx" using your script. Bingo :!: target again on the first run :lol:

-kwag

@Kwag

On MovieStacker the GOP options is set to 15, Need I change to 24, or that's not important?
Also, the avg bitrate for The Last Castle for me was 1015, is that correct? Could you give me the settings (resolution, cq, 1 or 2 cds) of The Last Castle?
Thanks!
LaBomba

hey kwag,

MovieStacker gave a recommended avg bitrate of 841. So 841/1767=.47 or cq=47 <- This is way off! I've already encoded a full sample of Heist with every whole cq value and the one that came closest to the desired filesize of 11.54mb with 128kbps audio was cq=62 8O

ren

8O Are you using the same script you used for CQ=62 :?:, or have you made any changes :?:
Can you run your sampler again with a CQ of 47 to see what is the actual file size :?:

-kwag

The sample with cq-47 was 8.906 I used the same script with all the samples.

ren

@Tenra,

I think that what we need is your "Scale" factor table, but instead of the center being 50, it has to be CQ=100 equals the factor of 1.0
So then we encode the sampler at CQ=100 and take the average bit rate, then we adjust with the scale factor. This would still only require one sampler run.

@Ren,
The above should then work for the non-linearity conditions.

The scale would be something like this:

Now, i'm on to make a scale 8)

Edit: See you all in about a couple of hours, because I'm doing a CQ spreadsheet from 100 down to 1 in steps of one, and each sample is ~1 minute 8O :lol:
I'll post the result when done.

-kwag

The problem I see with these values is that your CQ100 seem way out of proportion with the CQ0 and CQ50 values.

Using the scale I had posted before you should only have about 17.778MB and 1194 as the avg. bitrate for CQ100. ie: double CQ50

Your CQ100 is triple CQ50 and that is why the formula is not working.

-Tenra

@Tenra.
Everytime i use your SampleClip() (or SampleClip(n)) the TMPGEnc shows me the number of frames of the complete movie. Although it codes not the complete movie it needs is to much time. Also the file is huge.
Whats wrong?

I just wish CQ would be a linear scale :cry:
Then we wouldn't have all this trouble :x

-kwag

First, make sure that on the audio input line you remove the avs script. (Only have something listed on the video line and the output line.

Second, make sure that you have the newest version from my sig below, I tend to make updates often. If you have the newest version you can disregard the first suggestion since I have fixed it.

-Tenra