Size/CQ Curve: a one Pass CQ Estimation Method - digitalFAQ.com Forums [Archives]
 digitalFAQ.com Forums [Archives] Size/CQ Curve: a one Pass CQ Estimation Method

#1
08-15-2003, 05:12 AM
 fabrice Free Member Join Date: Mar 2003 Location: Madrid-Spain Posts: 515 Thanks: 0 Thanked 0 Times in 0 Posts
Hi,

Playing with Tmpgenc, and encoding the same movie sample 20 times, I 'discover' (if that's not new, just say it to me) that the size/CQ curve is quite 'repetitive':

Don't know if it's possible to 'simulate' this curve to be able to make better and faster size estimation, but it look like a "sin(x) + x" curve, with jump in size each 6 CQ.

I'll go until CQ 80 and try to do the same with another movie, to see if it occurs the same.

Mi sample is one generated by CQMatic 1.1.11, and has got 2925 frames.

CU
Someday, 12:01 PM
 Site Staff / Ad Manager Join Date: Dec 2002 Posts: 42 Thanks: ∞ Thanked 42 Times in 42 Posts
#2
08-15-2003, 05:53 AM
 Razorblade2000 Free Member Join Date: Aug 2002 Posts: 323 Thanks: 0 Thanked 0 Times in 0 Posts
Tamano=?

Hmmm... really a nice sin(x)+x curve... or a cos(x)+x... (ahhhhh! you brought back school... man... I'm on vacation )
#3
08-15-2003, 11:17 AM
 kwag Free Member Join Date: Apr 2002 Location: Puerto Rico, USA Posts: 13,537 Thanks: 0 Thanked 0 Times in 0 Posts
Quote:
 Originally Posted by fabrice I'll go until CQ 80 and try to do the same with another movie, to see if it occurs the same.
Hi fabrice,

Between 80 and 90, you'll see a very steep curve, where a very little change in CQ, causes a large change in file size. It's really painfull for prediction [quote]

Tamaño in Spanish means "size" in English

-kwag
#4
08-15-2003, 11:27 AM
 jorel Invalid Email / Banned / Spammer Join Date: Aug 2002 Location: Brasil - MG - third stone from the sun Posts: 5,570 Thanks: 0 Thanked 0 Times in 0 Posts
this forum is great!

now i'm learning Spanish !

Tamaño = size = tamanho(portuguese)

#5
08-15-2003, 12:09 PM
 rendalunit Free Member Join Date: Apr 2002 Location: san jose, Ca Posts: 1,148 Thanks: 0 Thanked 0 Times in 0 Posts
Quote:
 Originally Posted by kwag Between 80 and 90, you'll see a very steep curve,
hey kwag,

I did tons of testing with this and I saw the steep curve between cq70 and cq 85 then from 85 to 100 it was more gradual I'll look for my xls files but they might be gone because of a recent reformat

Thnx,
ren
#6
08-15-2003, 12:17 PM
 fabrice Free Member Join Date: Mar 2003 Location: Madrid-Spain Posts: 515 Thanks: 0 Thanked 0 Times in 0 Posts
Hi,

Sorry for this spanish word I was so excited about that, that I forgot what I was writing.
Here is the full chart

Kwag, as always, you're right, but I think this is possible to estimate this curve with a equation.

However, I did this curve with the Spiderman movie, using the sample CQMatic generate, and I validate it with a script with a sampler line, with the same movie but other resolution, and I got the same CQ (with two simples CQ=b/c*d rules!), and with just ONE encoding.

I'm validating this one with another movie, to see if an encoding and 2 mathematical operation, I can get the right CQ!
The precision is lower if the CQ is greater than 78 but with more datas between 78 and 90 (with a measure each 0,5 CQ), it's possible to get the rigth CQ.

How do I do? I create a table with the 'theorical size' of the sample for each CQ, encoding a sample at a 63 CQ, and calculating this size this way:
encoded size * real CQ63 size / encoded CQ63 size
Then, with the desired sample size, I know what is the CQ Interval, and I do a linear regresion between those 2 CQ values (That's where it could be better with more values).

For example:
I want a sample size of 72658 Mb
The m1v size, encoded at 63 is 62495 Mb.
In my generated table, I see that this sample size is between 70 and 71:
Code:
```CQ	Ref. size	Gener. size
69	12661	69292,34
70	12882	70501,85
71	14065	76976,28
72	14983	82000,4```
and the final CQ is 70,33.
(this is a 91 min movie, at 528x576)

I just ran CQMatic, which gives me a 70,21 value!

IT WORKS!

CU
#7
08-15-2003, 12:32 PM
 kwag Free Member Join Date: Apr 2002 Location: Puerto Rico, USA Posts: 13,537 Thanks: 0 Thanked 0 Times in 0 Posts
Yep, it works all right, until the next version of TMPEG
If you want to try it, just go back and encode with one of the 1.x versions of TMPEG, and you'll see what I mean.
Not even that
Just go back to say, version 2.53 or so, and take a look at the curve
If TMPEG was constant throughout versions, we could correct with a table. Sadly, this is not the case, not to mention that the linearity is also different for different resolutions.

-kwag
#8
08-15-2003, 12:47 PM
 fabrice Free Member Join Date: Mar 2003 Location: Madrid-Spain Posts: 515 Thanks: 0 Thanked 0 Times in 0 Posts
Hi,

About the tmpgenc history, I just have seen that they officially change CQ between the 2.01 and the 2.02.

I test that with a 352x288, 352x576 and 528x576 encoding, and it work!
I have to check with a previous version of tmpgenc (I got the 2.57), to see if it's ok or not...

To validate that, just give me the sample size at a CQ of 63, and the wanted sample size, and I'll give you the CQ I obtain (Please, choose one with a CQ lower than 78! ).

Fabrice
#9
08-15-2003, 01:08 PM
 rendalunit Free Member Join Date: Apr 2002 Location: san jose, Ca Posts: 1,148 Thanks: 0 Thanked 0 Times in 0 Posts
Thank you very much for posting that chart Fabrice I'm looking at your chart and I see a gradual increase in size between cq=85 and cq=100. That doesn't look like a steep curve to me
#10
08-15-2003, 01:28 PM
 jorel Invalid Email / Banned / Spammer Join Date: Aug 2002 Location: Brasil - MG - third stone from the sun Posts: 5,570 Thanks: 0 Thanked 0 Times in 0 Posts
maybe is not important ... or maybe is...

but i can confirm:
from CQ63 to CQ64 and to CQ65, the size really encrease,
i did samples using CQs: 60, 62, 63, 63.20, 64 and 65.
i got proportional resuts like the chart!

#11
08-15-2003, 02:52 PM
 vhelp Free Member Join Date: Jan 2003 Posts: 1,009 Thanks: 0 Thanked 0 Times in 0 Posts
hi Frabrice..

* DVD rip, and..
* AVS script, w/ sampler ??
* VFAPI's puseudo .avi to TMPG ??

Just curious

Thanks,
-vhelp
#12
08-15-2003, 03:09 PM
 kwag Free Member Join Date: Apr 2002 Location: Puerto Rico, USA Posts: 13,537 Thanks: 0 Thanked 0 Times in 0 Posts
Quote:
 Originally Posted by rendalunit That doesn't look like a steep curve to me
From 80 to ~86, it's almost vertical
That's what I meant for steep, which means that a very little change in CQ, makes a huge difference in file size. That hell for predicting CQ

Edit:
Then, from ~85 to ~90, it's almost flat, meaning that a change in CQ causes almost no change in file size.

-kwag
#13
08-15-2003, 03:21 PM
 rendalunit Free Member Join Date: Apr 2002 Location: san jose, Ca Posts: 1,148 Thanks: 0 Thanked 0 Times in 0 Posts
Quote:
 Originally Posted by kwag From 80 to ~86, it's almost vertical That's what I meant for steep, which means that a very little change in CQ, makes a huge difference in file size. That hell for predicting CQ
Yes that's true but I was referring to 85-100. As you can see there's not a steep curve between cq 85-100 so I see no reason why the limit for ToK and CQMatic shouldn't be 100.

ren
#14
08-15-2003, 03:23 PM
 vhelp Free Member Join Date: Jan 2003 Posts: 1,009 Thanks: 0 Thanked 0 Times in 0 Posts
@ ren..

Quote:
Originally Posted by rendalunit
Quote:
 Originally Posted by kwag Between 80 and 90, you'll see a very steep curve,
hey kwag,

I did tons of testing with this and I saw the steep curve between cq70 and cq 85 then from 85 to 100 it was more gradual I'll look for my xls files but they might be gone because of a recent reformat
I'm w/ you on this.. I've don'e zillions of small samples, though maybe not
in the more exact or priciness of methods that are used today but
I've zillions of them. I, like you, have all but deleted them for obvious
reasons. Fabrices chart brings back some memories of these LONG encoding
nights and charting in EXCEL !! A great tool, if I may say so. That's the
tool I use mostly these days for quick formulas and things. Anyways, I'll
have to do some reading on TMPG's newer features, as it's ben enhanced,
and I've ignored them since v2.53 cause I was pretty satisfied with it.
I hesitated using sample() via AVS script due to the color space issues that
I have ben plagued with for some time now, but I think it's time to play
around w/ newer ideas ..and using EXCEL is great, specially w/ the
Filter (those of you filter guru's like myself) and using various tricks like
separating data elements vs. using VB/Delphi/C++ for all that, and much
much more. Well, I'm boring you all.. hehe.. just getting wormed up here.

Anyone playing around w/ "Starship Troopers" w/ these tools and things ??
It's got great quality output - really. Gonna play around w/ some dark
scenes. Anyways..

@ Kwag..
Partdon my stupidity.. but, graphi'wise, is this whats called "Linear" or some
thing else ??

-vhelp
#15
08-15-2003, 03:28 PM
 kwag Free Member Join Date: Apr 2002 Location: Puerto Rico, USA Posts: 13,537 Thanks: 0 Thanked 0 Times in 0 Posts
Quote:
 Originally Posted by rendalunit Yes that's true but I was referring to 85-100. As you can see there's not a steep curve between cq 85-100 so I see no reason why the limit for ToK and CQMatic shouldn't be 100.
The problem is that after 90, because there's almost zero change in file size ( for a given CQ change ), prediction takes very long. The program will try to calculate a higher CQ, and expect a larger file size. Then after the run, there's no file size change
So the program continues increasing CQ in small steps, on and on and on, until it reaches 100, and still the same file size. So that's why we clamp CQ at 90, because there's just no more file size change after that

Edit: Now I remembered
After 90, I is almost vertical, and not flat.
It's still a nightmare for prediction, if it's flat or almost vertical. But after 90, it really is a steep vertical line, and although there's an increase in file size, there's no more visible quality appreciated after 90. That was the reason for the limit of 90.

-kwag
#16
08-15-2003, 04:45 PM
 fabrice Free Member Join Date: Mar 2003 Location: Madrid-Spain Posts: 515 Thanks: 0 Thanked 0 Times in 0 Posts
Hi everybody,

@vhelp: I did it using the samples generated by CQMatic 1.1.11, with a script reading a d2v file (DVDRip). This is a 116 min movie, en 352x576 (PAL). Max bitrate 2000, min 300.

As I changed something in my script (I don't remember what), I have to reencode all the samples to get a more complete curve (that don't mean that it doesn't work. It just mean that like every experiment, it have to be done in the same conditions to be valid): I just wanted to be more 'complete' between 70-73 y 79-84, to get a more accurate CQ.

I wrote a small VC++ application (it's very very raw), and if you give me a wanted sample size and the size of the sample encoded at CQ 63, I'll give you the CQ calculaded by this way.

Viewing the chart, I agree with Kwag: It seem that after 91, there is other vertical line.
If someone know what kind of equation is that (Something like
Code:
`abs(sin(1000/x)) * 1,3 ^(x/10)  + 1,5 ^(x/8)`
), it would be the best, as it would be easier to calculate some values, intead of using the old linear method between to known CQ.

Fabrice
#17
08-15-2003, 04:49 PM
 kwag Free Member Join Date: Apr 2002 Location: Puerto Rico, USA Posts: 13,537 Thanks: 0 Thanked 0 Times in 0 Posts
Quote:
 Originally Posted by fabrice If someone know what kind of equation is that (Something like Code: `abs(sin(1000/x)) * 1,3 ^(x/10) + 1,5 ^(x/8)` ), it would be the best, as it would be easier to calculate some values, intead of using the old linear method between to known CQ. Fabrice
Now this is REALLY getting hairy
Thanks fabrice
-kwag
#18
08-15-2003, 04:54 PM
 vhelp Free Member Join Date: Jan 2003 Posts: 1,009 Thanks: 0 Thanked 0 Times in 0 Posts
Hi Frabrice..

Quote:
 Originally Posted by fabrice Hi everybody, @vhelp: I did it using the samples generated by CQMatic 1.1.11, with a script reading a d2v file (DVDRip). This is a 116 min movie, en 352x576 (PAL). Max bitrate 2000, min 300. As I changed something in my script (I don't remember what), I have to reencode all the samples to get a more complete curve (that don't mean that it doesn't work. It just mean that like every experiment, it have to be done in the same conditions to be valid): I just wanted to be more 'complete' between 70-73 y 79-84, to get a more accurate CQ. I wrote a small VC++ application (it's very very raw), and if you give me a wanted sample size and the size of the sample encoded at CQ 63, I'll give you the CQ calculaded by this way. Viewing the chart, I agree with Kwag: It seem that after 91, there is other vertical line. If someone know what kind of equation is that (Something like Code: `abs](sin(1000/x)) * 1,3 ^(x/10) + 1,5 ^(x/8)` ), it would be the best, as it would be easier to calculate some values, intead of using the old linear method between to known CQ. Fabrice
I think that the above, could be done in Excel too !!
But, as for your second formula/equation (hated algebra) for:
Code:
`abs](sin(1000/x)) * 1,3 ^(x/10)  + 1,5 ^(x/8)`
That too, can be done in Excel. In fact, you could chart results, either
based on a single value, or graph estimates of various results, based on
the two items needs to drive the formulas !!

If you post the formula, I'm sure that others here can conjure up a snipplet

Interesting indeed
-vhelp
#19
08-15-2003, 05:34 PM
 fabrice Free Member Join Date: Mar 2003 Location: Madrid-Spain Posts: 515 Thanks: 0 Thanked 0 Times in 0 Posts
Hi,

@kwag: what's the translation of this is getting hairy. Google translate it as "esto realmente está consiguiendo melenudo"

@vhelp: I'm not using excel, but Openoffice. Basically the same.
Does Excel have a way to find automaticaly an equation from datas?

This equation is one that give similar results to the size/CQ curve. Try it! Today I've been obliged to resolve a problem with 2 equations and 2 variables. Don't know if I could derivate this equation, or extract the x variable...

I'm still encoding with a 2.510 version, to verify if it work with a old version, but the Cq 63 encoding gives the same file size ...

Fabrice
#20
08-15-2003, 05:39 PM
 kwag Free Member Join Date: Apr 2002 Location: Puerto Rico, USA Posts: 13,537 Thanks: 0 Thanked 0 Times in 0 Posts
Quote:
 Originally Posted by fabrice Hi, @kwag: what's the translation of this is getting hairy. Google translate it as "esto realmente está consiguiendo melenudo"
"Peludo"

-kwag

 Similar Threads Thread Thread Starter Forum Replies Last Post girv Avisynth Scripting 14 07-14-2003 04:51 PM Garedryl Convertir y Codificar Video (Español) 1 07-08-2003 05:11 PM kwag Video Encoding and Conversion 31 03-04-2003 12:42 PM logan555 Video Encoding and Conversion 1 12-04-2002 09:18 AM Pioneer Video Encoding and Conversion 0 11-21-2002 01:50 PM