Size/CQ curve: a one pass CQ estimation method
 

Hi,

Playing with Tmpgenc, and encoding the same movie sample 20 times, I 'discover' (if that's not new, just say it to me) that the size/CQ curve is quite 'repetitive':
http://www.digitalfaq.com/archives/error.gif
Don't know if it's possible to 'simulate' this curve to be able to make better and faster size estimation, but it look like a "sin(x) + x" curve, with jump in size each 6 CQ.

I'll go until CQ 80 and try to do the same with another movie, to see if it occurs the same.

Mi sample is one generated by CQMatic 1.1.11, and has got 2925 frames.

CU

Tamano=?

Hmmm... really a nice sin(x)+x curve... or a cos(x)+x... (ahhhhh! you brought back school... man... I'm on vacation :lol: )

Re: CQ/Size curve
 

Hi fabrice,

Between 80 and 90, you'll see a very steep curve, where a very little change in CQ, causes a large change in file size. It's really painfull for prediction :x :lol: [quote]

@Razorblade:

Tamaño in Spanish means "size" in English :)

-kwag

this forum is great!

now i'm learning Spanish !
:lol:

Tamaño = size = tamanho(portuguese)

:wink:

hey kwag,

I did tons of testing with this and I saw the steep curve between cq70 and cq 85 then from 85 to 100 it was more gradual 8O I'll look for my xls files but they might be gone because of a recent reformat :roll:

Thnx,
ren

Hi,

Sorry for this spanish word :oops: I was so excited about that, that I forgot what I was writing.
Here is the full chart
http://www.digitalfaq.com/archives/error.gif

Kwag, as always, you're right, but I think this is possible to estimate this curve with a equation.

However, I did this curve with the Spiderman movie, using the sample CQMatic generate, and I validate it with a script with a sampler line, with the same movie but other resolution, and I got the same CQ (with two simples CQ=b/c*d rules!), and with just ONE encoding.

I'm validating this one with another movie, to see if an encoding and 2 mathematical operation, I can get the right CQ!
The precision is lower if the CQ is greater than 78 but with more datas between 78 and 90 (with a measure each 0,5 CQ), it's possible to get the rigth CQ.

How do I do? I create a table with the 'theorical size' of the sample for each CQ, encoding a sample at a 63 CQ, and calculating this size this way:
encoded size * real CQ63 size / encoded CQ63 size
Then, with the desired sample size, I know what is the CQ Interval, and I do a linear regresion between those 2 CQ values (That's where it could be better with more values).

For example:
I want a sample size of 72658 Mb
The m1v size, encoded at 63 is 62495 Mb.
In my generated table, I see that this sample size is between 70 and 71:
and the final CQ is 70,33.
(this is a 91 min movie, at 528x576)

I just ran CQMatic, which gives me a 70,21 value! :)

IT WORKS! :lol:

CU

Yep, it works all right, until the next version of TMPEG :!:
If you want to try it, just go back and encode with one of the 1.x versions of TMPEG, and you'll see what I mean. :roll:
Not even that :!:
Just go back to say, version 2.53 or so, and take a look at the curve :?
If TMPEG was constant throughout versions, we could correct with a table. Sadly, this is not the case, not to mention that the linearity is also different for different resolutions.

-kwag

Hi,

About the tmpgenc history, I just have seen that they officially change CQ between the 2.01 and the 2.02.

I test that with a 352x288, 352x576 and 528x576 encoding, and it work!
I have to check with a previous version of tmpgenc (I got the 2.57), to see if it's ok or not...

To validate that, just give me the sample size at a CQ of 63, and the wanted sample size, and I'll give you the CQ I obtain (Please, choose one with a CQ lower than 78! :) ).

Fabrice

Thank you very much for posting that chart Fabrice :!: I'm looking at your chart and I see a gradual increase in size between cq=85 and cq=100. That doesn't look like a steep curve to me :!:

maybe is not important ... or maybe is...

but i can confirm:
from CQ63 to CQ64 and to CQ65, the size really encrease,
i did samples using CQs: 60, 62, 63, 63.20, 64 and 65.
i got proportional resuts like the chart!

:wink:

hi Frabrice..

w/ reference to your chart...
When you performed your encoding clips, what was your source ??

* DVD rip, and..
* AVS script, w/ sampler ??
* VFAPI's puseudo .avi to TMPG ??

Just curious :roll:

Thanks,
-vhelp

From 80 to ~86, it's almost vertical 8O
That's what I meant for steep, which means that a very little change in CQ, makes a huge difference in file size. That hell for predicting CQ :!:

Edit:
Then, from ~85 to ~90, it's almost flat, meaning that a change in CQ causes almost no change in file size.

-kwag

Yes that's true but I was referring to 85-100. As you can see there's not a steep curve between cq 85-100 so I see no reason why the limit for ToK and CQMatic shouldn't be 100.

http://www.digitalfaq.com/archives/i.../2003/08/5.jpg

ren

@ ren..

I'm w/ you on this.. I've don'e zillions of small samples, though maybe not
in the more exact or priciness of methods that are used today :wink: but
I've zillions of them. I, like you, have all but deleted them for obvious
reasons. Fabrices chart brings back some memories of these LONG encoding
nights and charting in EXCEL !! A great tool, if I may say so. That's the
tool I use mostly these days for quick formulas and things. Anyways, I'll
have to do some reading on TMPG's newer features, as it's ben enhanced,
and I've ignored them since v2.53 cause I was pretty satisfied with it.
I hesitated using sample() via AVS script due to the color space issues that
I have ben plagued with for some time now, but I think it's time to play
around w/ newer ideas :!: ..and using EXCEL is great, specially w/ the
Filter (those of you filter guru's like myself) and using various tricks like
separating data elements vs. using VB/Delphi/C++ for all that, and much
much more. Well, I'm boring you all.. hehe.. just getting wormed up here.

Anyone playing around w/ "Starship Troopers" w/ these tools and things ??
It's got great quality output - really. Gonna play around w/ some dark
scenes. Anyways..

@ Kwag..
Partdon my stupidity.. but, graphi'wise, is this whats called "Linear" or some
thing else ??

-vhelp

The problem is that after 90, because there's almost zero change in file size ( for a given CQ change ), prediction takes very long. The program will try to calculate a higher CQ, and expect a larger file size. Then after the run, there's no file size change :!:
So the program continues increasing CQ in small steps, on and on and on, until it reaches 100, and still the same file size. So that's why we clamp CQ at 90, because there's just no more file size change after that :D

Edit: Now I remembered :!:
After 90, I is almost vertical, and not flat.
It's still a nightmare for prediction, if it's flat or almost vertical. But after 90, it really is a steep vertical line, and although there's an increase in file size, there's no more visible quality appreciated after 90. That was the reason for the limit of 90.

-kwag

Hi everybody,

@vhelp: I did it using the samples generated by CQMatic 1.1.11, with a script reading a d2v file (DVDRip). This is a 116 min movie, en 352x576 (PAL). Max bitrate 2000, min 300.

As I changed something in my script (I don't remember what), I have to reencode all the samples to get a more complete curve (that don't mean that it doesn't work. It just mean that like every experiment, it have to be done in the same conditions to be valid): I just wanted to be more 'complete' between 70-73 y 79-84, to get a more accurate CQ.

I wrote a small VC++ application (it's very very raw), and if you give me a wanted sample size and the size of the sample encoded at CQ 63, I'll give you the CQ calculaded by this way.

Viewing the chart, I agree with Kwag: It seem that after 91, there is other vertical line.
If someone know what kind of equation is that (Something like ), it would be the best, as it would be easier to calculate some values, intead of using the old linear method between to known CQ.

Fabrice

Now this is REALLY getting hairy 8O :lol:
Thanks fabrice :cool:
-kwag

Hi Frabrice..

I think that the above, could be done in Excel too !!
But, as for your second formula/equation (hated algebra) for:
That too, can be done in Excel. In fact, you could chart results, either
based on a single value, or graph estimates of various results, based on
the two items needs to drive the formulas !!

If you post the formula, I'm sure that others here can conjure up a snipplet :)

Interesting indeed :!:
-vhelp

Hi,

@kwag: what's the translation of this is getting hairy. Google translate it as "esto realmente está consiguiendo melenudo" :lol: 8O

@vhelp: I'm not using excel, but Openoffice. Basically the same.
Does Excel have a way to find automaticaly an equation from datas?

This equation is one that give similar results to the size/CQ curve. Try it! :) Today I've been obliged to resolve a problem with 2 equations and 2 variables. Don't know if I could derivate this equation, or extract the x variable... :lol:

I'm still encoding with a 2.510 version, to verify if it work with a old version, but the Cq 63 encoding gives the same file size :mrgreen: ...

Fabrice

"Peludo" :mrgreen:

-kwag

@ frabrice:

I'm not sure what CQ 63 stands for 8O

Is it your "pivot" at pointing to a final value, based on a scale ??
ie, your chart above (page 1) maybe ??

I experimented w/ the algo in Excel, just to see if I could at least get a num
out of it. I had trouble w/ the "^" char, as I understood it, "^" means
"power of". In Excel it means (assuming I'm understanding the help file)
that "^" is the Exponent.. same as "Power of" ??

here's the Excel formulat format, placed under my Fabrice TAB :)
* =ABS( SIN ( 1000/E5 ) * 1.3 ^( E5/10 )+1.5 ^( E5/8 ))

--> where X=Cell(E5) = 300
--> E5[300] --> result: 4012066.95
--> E5[150] --> result: 2022.290361
--> E5[050] --> result: 15.99550055
.
.
etc. etc.

If by chance, my Excel of "^" is used incorrectly in the formula, please feel
free to correct me.

Thanks,
-vhelp

Hi,

What an honor: I got a tab in your excel Spreadsheet ! :ole:
I use the 63 value, because it's in the middle of a vertical line, so it gives, I think, better results. It could be 71 too, for the same reason.

Your formula is correct, because a CQ 50 gives me a 16 value, with this equation. It should be 8917, in this case...

Just finished encoding my sample with the CQ calculated, and with Tmpgenc 2.510, it gives me a variation of 2%, just because the CQ is in the 'jump' zone, and I have to get more datas in this zones, to get more accurate results.

I encode samples from CQ 90 to CQ 100, and it's almost linear: only a 2% file size increase...

Fabrice

Hi Fabrice..

For some strange reason, I didn't get e-mail noticifcation of your last response..
sorry I didn't get to it sooner !! Anyways..

I'm glad things are finally working out for you :)

EDIT: - - I moved the ..off topic.. snip :)

keep plugging away,
-vhelp

Weel,

After a very short night 8) , I have the first 'proof of concept' version of my estimation program. It's quite small, but not as small as CQMatic! :)

http://coutadeurf.en.telepolis.com/CQEstim.exe

It's quite easy to use:
- encode a sample at a 63 CQ
- give the program the estimated sample size (or total frames, samples frames and video size) and the encoded sample size
- and estimate!

With the CQMatic sampling way, you get your CQ in 3 minutes! 8O

Tell me your success and failed story.

And just remember, this is a very alpha version!

Fabrice

@vhelp: sorry you moved the off-topic: I had the solution (in FitCD source! :) )

Hey Fabrice,

I tested and tried you program. I noticed you used MB vs KB for you numbers. Also in the program you allow only whole numbers. I have a sample size @63 of 8.91 MB do I use 8 or 9 for the sample size?

Racer99

Oooops, you're right :oops:
Should have been Kb ... 8O (or I'm doing the validation with a 62 Gb sample size! :lol: )

I change the text in the application, and I'll change the exe file (you can use the exe you get putting Kb instead of Mb).

Thanks,

Fabrice

Edit: link ok.

@ Fabrice..

Sorry, but I didn't want to seem like I was starting a new topic in the middle
of yours 8)

If you want, please PM me your solutions. However, I too, did some laborous
hard working as well (off topic) I found that I could just use the following in
my calcs ie, (480 * 2.35) or (480 * 1.85) etc etc., but I'm not sure it's 100%
accurate. In any case, ad not to go too off-topic, PM me what you have and
maybe we can compare :lol:

I finally went ta bed after 3:30am here..
Thanks for all your help and assist,
-vhelp

That's weird 8O All the tests I did with dvd sources when graphed, had a leveling off after cq=85 that was almost horizontal.

I'll do more testing on that.

Let me know ren,
After so many hours of coding/fixing/debugging CQMatic, I really can't tell the difference from 85 CQ curve to 85 proof Bacardi Rum :lol: :mrgreen:

-kwag

The 85 CQ curve doesn't catch fire :lol:

Could you modify CQMatic to make 50 samples at every cq between 50 and 100 for me? (j/k) :D

ren

Sure :!: :D
But then, why not from 1 to 100 :idea:
Then we can graph different resolutions :idea:

-kwag

Yeah! If you could modify cqmatic to do that, that would be great :!:

I did that manually with "Heist" and it was tedious to make 100 tprs :wink:

This graph shows what I meant about the linear pattern from cq 85-100 with this particular movie

http://www.digitalfaq.com/archives/i.../2003/08/2.png

Hi ren,

And that's exactly why we had settled for a ceiling of 90 :)
After a value of around 88, there's no more noticeable file size change for a given CQ change :!:
So it's a loss of time trying to find CQ above 90, because there's no quality increase at all :!:
The curve pattern will apply to any movie, with only different file sizes for a given CQ.

-kwag

Hi,

You're right Kwag: this pattern is for every movies I tried.

I'm encoding a 116 min movie, with the CQ given by cqestim (what a ugly name! :) ), and using the sampler line:
which gives me the best results with 2 PAL movies (better than more shorter samples). The estimated diference was less than 1%. Let's see tomorrow how accurate it is with this one.

Fabrice

Hay ren,

would you happen to have those number laying around somewhere's ??

* Filesize (mb)
* CQ curve

Really appreciate, K?

Thanks,
-vhelp

hi vhelp,

here's the numbers from the "Heist" samples .xls file;

Some are missing because of mistakes I made in naming the .tpr :oops:

ren

Thanks ren..

Ok, one more thing.. who's pointing my browser to this location, after I
pressed REFRESH key to update this page, I got this popped up:

* http://login.passport.net/uilogin.srf?id=2

What gives ??
Happended to me on once before, but I ignored it. Now, it's a 2nd time 8O

-vhelp

Billy Gates :mrgreen:
Kill your MSN client :!: It's running, right :?:

Hi Kwag..

I'm not sure. But, I have IE 5.0, if that's what you mean :?
-vhelp

This little icon in your tray: :arrow: http://www.digitalfaq.com/archives/i.../2003/08/8.jpg Kill it :!:

-kwag