Size/CQ curve: a one pass CQ estimation method
 

Hi,

Playing with Tmpgenc, and encoding the same movie sample 20 times, I 'discover' (if that's not new, just say it to me) that the size/CQ curve is quite 'repetitive':
http://www.digitalfaq.com/archives/error.gif
Don't know if it's possible to 'simulate' this curve to be able to make better and faster size estimation, but it look like a "sin(x) + x" curve, with jump in size each 6 CQ.

I'll go until CQ 80 and try to do the same with another movie, to see if it occurs the same.

Mi sample is one generated by CQMatic 1.1.11, and has got 2925 frames.

CU

Tamano=?

Hmmm... really a nice sin(x)+x curve... or a cos(x)+x... (ahhhhh! you brought back school... man... I'm on vacation :lol: )

Re: CQ/Size curve
 

Hi fabrice,

Between 80 and 90, you'll see a very steep curve, where a very little change in CQ, causes a large change in file size. It's really painfull for prediction :x :lol: [quote]

@Razorblade:

Tamaño in Spanish means "size" in English :)

-kwag

this forum is great!

now i'm learning Spanish !
:lol:

Tamaño = size = tamanho(portuguese)

:wink:

hey kwag,

I did tons of testing with this and I saw the steep curve between cq70 and cq 85 then from 85 to 100 it was more gradual 8O I'll look for my xls files but they might be gone because of a recent reformat :roll:

Thnx,
ren

Hi,

Sorry for this spanish word :oops: I was so excited about that, that I forgot what I was writing.
Here is the full chart
http://www.digitalfaq.com/archives/error.gif

Kwag, as always, you're right, but I think this is possible to estimate this curve with a equation.

However, I did this curve with the Spiderman movie, using the sample CQMatic generate, and I validate it with a script with a sampler line, with the same movie but other resolution, and I got the same CQ (with two simples CQ=b/c*d rules!), and with just ONE encoding.

I'm validating this one with another movie, to see if an encoding and 2 mathematical operation, I can get the right CQ!
The precision is lower if the CQ is greater than 78 but with more datas between 78 and 90 (with a measure each 0,5 CQ), it's possible to get the rigth CQ.

How do I do? I create a table with the 'theorical size' of the sample for each CQ, encoding a sample at a 63 CQ, and calculating this size this way:
encoded size * real CQ63 size / encoded CQ63 size
Then, with the desired sample size, I know what is the CQ Interval, and I do a linear regresion between those 2 CQ values (That's where it could be better with more values).

For example:
I want a sample size of 72658 Mb
The m1v size, encoded at 63 is 62495 Mb.
In my generated table, I see that this sample size is between 70 and 71:
and the final CQ is 70,33.
(this is a 91 min movie, at 528x576)

I just ran CQMatic, which gives me a 70,21 value! :)

IT WORKS! :lol:

CU

Yep, it works all right, until the next version of TMPEG :!:
If you want to try it, just go back and encode with one of the 1.x versions of TMPEG, and you'll see what I mean. :roll:
Not even that :!:
Just go back to say, version 2.53 or so, and take a look at the curve :?
If TMPEG was constant throughout versions, we could correct with a table. Sadly, this is not the case, not to mention that the linearity is also different for different resolutions.

-kwag

Hi,

About the tmpgenc history, I just have seen that they officially change CQ between the 2.01 and the 2.02.

I test that with a 352x288, 352x576 and 528x576 encoding, and it work!
I have to check with a previous version of tmpgenc (I got the 2.57), to see if it's ok or not...

To validate that, just give me the sample size at a CQ of 63, and the wanted sample size, and I'll give you the CQ I obtain (Please, choose one with a CQ lower than 78! :) ).

Fabrice

Thank you very much for posting that chart Fabrice :!: I'm looking at your chart and I see a gradual increase in size between cq=85 and cq=100. That doesn't look like a steep curve to me :!:

maybe is not important ... or maybe is...

but i can confirm:
from CQ63 to CQ64 and to CQ65, the size really encrease,
i did samples using CQs: 60, 62, 63, 63.20, 64 and 65.
i got proportional resuts like the chart!

:wink:

hi Frabrice..

w/ reference to your chart...
When you performed your encoding clips, what was your source ??

* DVD rip, and..
* AVS script, w/ sampler ??
* VFAPI's puseudo .avi to TMPG ??

Just curious :roll:

Thanks,
-vhelp

From 80 to ~86, it's almost vertical 8O
That's what I meant for steep, which means that a very little change in CQ, makes a huge difference in file size. That hell for predicting CQ :!:

Edit:
Then, from ~85 to ~90, it's almost flat, meaning that a change in CQ causes almost no change in file size.

-kwag

Yes that's true but I was referring to 85-100. As you can see there's not a steep curve between cq 85-100 so I see no reason why the limit for ToK and CQMatic shouldn't be 100.

http://www.digitalfaq.com/archives/i.../2003/08/5.jpg

ren

@ ren..

I'm w/ you on this.. I've don'e zillions of small samples, though maybe not
in the more exact or priciness of methods that are used today :wink: but
I've zillions of them. I, like you, have all but deleted them for obvious
reasons. Fabrices chart brings back some memories of these LONG encoding
nights and charting in EXCEL !! A great tool, if I may say so. That's the
tool I use mostly these days for quick formulas and things. Anyways, I'll
have to do some reading on TMPG's newer features, as it's ben enhanced,
and I've ignored them since v2.53 cause I was pretty satisfied with it.
I hesitated using sample() via AVS script due to the color space issues that
I have ben plagued with for some time now, but I think it's time to play
around w/ newer ideas :!: ..and using EXCEL is great, specially w/ the
Filter (those of you filter guru's like myself) and using various tricks like
separating data elements vs. using VB/Delphi/C++ for all that, and much
much more. Well, I'm boring you all.. hehe.. just getting wormed up here.

Anyone playing around w/ "Starship Troopers" w/ these tools and things ??
It's got great quality output - really. Gonna play around w/ some dark
scenes. Anyways..

@ Kwag..
Partdon my stupidity.. but, graphi'wise, is this whats called "Linear" or some
thing else ??

-vhelp

The problem is that after 90, because there's almost zero change in file size ( for a given CQ change ), prediction takes very long. The program will try to calculate a higher CQ, and expect a larger file size. Then after the run, there's no file size change :!:
So the program continues increasing CQ in small steps, on and on and on, until it reaches 100, and still the same file size. So that's why we clamp CQ at 90, because there's just no more file size change after that :D

Edit: Now I remembered :!:
After 90, I is almost vertical, and not flat.
It's still a nightmare for prediction, if it's flat or almost vertical. But after 90, it really is a steep vertical line, and although there's an increase in file size, there's no more visible quality appreciated after 90. That was the reason for the limit of 90.

-kwag

Hi everybody,

@vhelp: I did it using the samples generated by CQMatic 1.1.11, with a script reading a d2v file (DVDRip). This is a 116 min movie, en 352x576 (PAL). Max bitrate 2000, min 300.

As I changed something in my script (I don't remember what), I have to reencode all the samples to get a more complete curve (that don't mean that it doesn't work. It just mean that like every experiment, it have to be done in the same conditions to be valid): I just wanted to be more 'complete' between 70-73 y 79-84, to get a more accurate CQ.

I wrote a small VC++ application (it's very very raw), and if you give me a wanted sample size and the size of the sample encoded at CQ 63, I'll give you the CQ calculaded by this way.

Viewing the chart, I agree with Kwag: It seem that after 91, there is other vertical line.
If someone know what kind of equation is that (Something like ), it would be the best, as it would be easier to calculate some values, intead of using the old linear method between to known CQ.

Fabrice

Now this is REALLY getting hairy 8O :lol:
Thanks fabrice :cool:
-kwag

Hi Frabrice..

I think that the above, could be done in Excel too !!
But, as for your second formula/equation (hated algebra) for:
That too, can be done in Excel. In fact, you could chart results, either
based on a single value, or graph estimates of various results, based on
the two items needs to drive the formulas !!

If you post the formula, I'm sure that others here can conjure up a snipplet :)

Interesting indeed :!:
-vhelp

Hi,

@kwag: what's the translation of this is getting hairy. Google translate it as "esto realmente está consiguiendo melenudo" :lol: 8O

@vhelp: I'm not using excel, but Openoffice. Basically the same.
Does Excel have a way to find automaticaly an equation from datas?

This equation is one that give similar results to the size/CQ curve. Try it! :) Today I've been obliged to resolve a problem with 2 equations and 2 variables. Don't know if I could derivate this equation, or extract the x variable... :lol:

I'm still encoding with a 2.510 version, to verify if it work with a old version, but the Cq 63 encoding gives the same file size :mrgreen: ...

Fabrice

"Peludo" :mrgreen:

-kwag