Bitrates: CQmatic Version 1.2.03 Reports

kwag · #1 11-10-2003, 10:08 PM

My first encode, to test the latest version of CQMatic.

Movie: General's Daughter.
Resolution: 352x240 ( No filters. Direct processing from .d2v file )

Data from CalcuMatic:
Movie Time: 117 minutes.
Average Bitrate: 801.1
Wanted final video size: 706,336KB

I used 801Kbps as value into CQMatic:

First run, with X3 prediction selected. CQ found = 81.68
Second run, with X1 prediction selected. CQ found = 82.19

First encode (X3), with CQ 81.68: 700,782KB for 0.786% accuracy.

Second encode (X1), with CQ 82.19: 713,458 for 0.998% accuracy.

Here's the X3 (higher new precision) option log:

Code:

Prediction cycle #8
Encoder started...
Process time: 8.22 minutes.
Encoder end.
File size difference  = 1.005513
Low fence: 81.675888
High fence: 81.956734
Last CQ = 81.68
Current CQ = 81.82
CQ difference = 0.140427
Final CQ value is 81.68
Encoding set to Full encode.
Full encode start...
CQMatic complete!
Total minutes of process: 65.95
Process ended at 19:10:15
On 11/10/2003

And here's the X1 (default precision) log:

Code:

Prediction cycle #7
Encoder started...
Process time: 2.60 minutes.
Encoder end.
File size difference  = 0.994517
Low fence: 81.818085
High fence: 82.186234
Last CQ = 82.19
Current CQ = 82.00
CQ difference = 0.184074
Final CQ value is 82.19
Encoding set to Full encode.
Full encode start...
CQMatic complete!
Total minutes of process: 19.57
Process ended at 19:31:26
On 11/10/2003

Edit: As we can see, the X3 prediction was closer to target, and just under the wanted size, which is a goog thing. The X1 prediction went over target. Now, this has to be tested on other movies
In my case, there was a difference (variance) of 12,676KB between both encodes, at the different CQs. Clearly, the X3 being ~12MB lower than the X1 prediction is a big difference. Now let's test this new X3 setting on some flicks

Note: I used 801Kbps for average, where the required was actually 801.1Kbps
This means that the final result, if I had used 801.1Kbps, would have been even more accurate

-kwag

nicksteel · #2 11-11-2003, 05:51 AM

I ran x3 on a 720x480 4:3 TV MPEG2 capture for 352x480 SKVCD.

CalcuMatic 89:41 minutes
128 audio
Custom 775
ABR 998.6
0.57 569.2

Final results:

mp2 (Headac3he) 84,082
m2v 751,001
muxed 845,571

It continues to be very difficult for me to predict MPEG2 encodes.

Also, when I enable GripBorders(), I cannot get TEMPGEnc to set aspect ratio (tried all combinations) and get bars on top and bottom of film.

kwag · #3 11-11-2003, 11:44 AM

Hi nicksteel,

I think the biggest problem is AviSynth and the scripts.
I just finished another encode, "Count of Monte Cristo", which was a PAIN to predict with ToK, acp and older versions (~1.0x) of CQMatic.
I always got about 80MB lower final size with this particular movie.
I predicted with X3, and I used a MIN bitrate of 300 and MAX of 2,500, and the final size was 690,764KB with a wanted size of 712,072KB.
The CQ calculated in X3 prediction was 78.85.
After running prediction again in X1 mode, I got a CQ of 78.6, which will obviously create a lower final size.
This was processing the .d2v directly, without any filters.
I'm currently 76% (about 40 minutes left) into the encode with CQ=78.6, just to see how much lower the the final file size will be compared to the X3 prediction.
I don't think there's much more I can do to control the behaviour of prediction, specially when AviSynth is in the way.

Edit: Result of X1 prediction: 677,128KB.
This clearly shows the new longer prediction (X3) is more accurate. That's a difference of 13,636KB for the small change in CQ from 78.85 to 78.6.
A 0.25 change in CQ

-kwag

nicksteel · #4 11-12-2003, 09:24 AM

I'm now trying to use the audio bitrate value in CalcuMatic (as it is based upon audio length) to arrive at some method for predicting SKVCD. The results are erratic and final size for a film can be 25-50 MB too large with x3 when just rerunning several times with smaller custom MB sizes. Not a very scientific effort, as I am just selecting larger than wanted (12

bitrates to try and establish some pattern. It appears that CalcuMatic values used with x3 are not directly applicable to mpeg2, although things may be working well with mpeg1 files.

With x1, I always used custom 775 with Audio Bitrate 192 to predict 800/128 encodes. This is not very accurate, but I've found no other alternative so far.

I'm hoping that all the ongoing efforts to enhance prediction will address MPEG2 as well as MPEG1.

Any suggestions?

andybno1 · #5 11-12-2003, 11:02 AM

so is it best to predict using d2v files when doin dvd predicts?

Dialhot · #6 11-12-2003, 11:12 AM

Quote:

Originally Posted by nicksteel

With x1, I always used custom 775 with Audio Bitrate 192 to predict 800/128 encodes. This is not very accurate, but I've found no other alternative so far.

I really do not understand. I always use cutom 807 MB and 128 for audio and I always have something right in the target. I have to do my final encode in 112 insteed of 128 for only 10% of all my encoding.

The 2 or 3 real failure were whith very long movie (last one was 2h12). That is due to CQMAtic taking 120 samples whatever the lenght of the movie. For movie longer than 2horus, that make less than one sample per minute. For me that is the real problem of CQMatic : adjusting the number of samples according to the length of the movie (at least for extrem long ones).

Note: I always use min bitrate of 64 and max from 1800-2300 according to the length of the movie.

nicksteel · #7 11-12-2003, 12:06 PM

Quote:

I ran x3 on a 720x480 4:3 TV MPEG2 capture for 352x480 SKVCD.

CalcuMatic 89:41 minutes
128 audio
Custom 775
ABR 998.6
0.57 569.2

Final results:

mp2 (Headac3he) 84,082
m2v 751,001
muxed 845,571

From my post above. You can see that even with 775 custom and 128 audio that my final mpeg2 was way over. I must be doing something wrong, although I use new MPEG2DEC3.dll, TMPGEnc 2.520 and avisynth 2.5. I have this problem with 4:3 or 16:9 MPEG2 tv captures. Most of my encodes are of 80-90 minutes.

kwag · #8 11-12-2003, 12:29 PM

Hi nicksteel,

Have you tried a prediction from your capture without any scripts

I just want to see what variables are in the way

I just finished "Bourne Identity" predicted in X3 mode, full screen, processed directly from .d2v using MIN=500, MAX=2,500:
Wanted: 719,786KB
Final: 724,396KB
0.636% diff.

-kwag

nicksteel · #9 11-12-2003, 12:35 PM

Quote:

Originally Posted by kwag

Hi nicksteel,

Have you tried a prediction from your capture without any scripts

I just want to see what variables are in the way

I just finished "Bourne Identity" predicted in X3 mode, full screen, processed directly from .d2v using MIN=500, MAX=2,500:
Wanted: 719,786KB
Final: 724,396KB
0.636% diff.

-kwag

Since I'm working with:

mpeg2source("h:\tigger\tigger.d2v")
Trim(11106,43702)+Trim(54556,85605)+Trim(96176,113 980)+Trim(123469,153060)+Trim(163146,197039)+Trim( 206525,222861)

Can I make a script file with only the above for test? I am really interested in MPEG2 prediction, as the PVR250 captures look good as KSVCD and 704x480 MPEG2.

kwag · #10 11-12-2003, 04:56 PM

Quote:

Originally Posted by nicksteel

Since I'm working with:

mpeg2source("h:\tigger\tigger.d2v")
Trim(11106,43702)+Trim(54556,85605)+Trim(96176,113 980)+Trim(123469,153060)+Trim(163146,197039)+Trim( 206525,222861)

Can I make a script file with only the above for test?

Yes, I believe so

I guess your "trims" are for commercials, right

Quote:

I am really interested in MPEG2 prediction, as the PVR250 captures look good as KSVCD and 704x480 MPEG2.

I have an MPEG-2 captured movie, which I did with my DMR-E80, and I haven't encoded it to KVCD yet.
I'll give it a try and see how CQMatic does. I'll encode to MPEG-2, and see what size I get.

-kwag

nicksteel · #11 11-12-2003, 05:01 PM

Try it with the optimal ma for clean dvd template at 704x480 with kvcdx3Mpeg2. My captures are clean enough for this on analog cable. Let me know!

kwag · #12 11-13-2003, 08:36 PM

Hi nicksteel,

I finished a full encode with the MA script, just as it is and without any modifications, using CQMatic and X3 mode.
I encoded a capture from the SCI-FI channel with my DMR-E80, and created a .d2v from it.
So I encoded 704x480 MPEG-2 29.97fps Interlaced.
I set CalcuMatic to "Custom", and entered a value of 1,400MB.
With audio at 128Kbps, I got a calculation of 1,319,072KB wanted size. It's a 2 hour footage.
After hours ( Many hours

) of prediction and encode ( I have a P4 @1.6Ghz

) the final file size is 1,303,325KB

I also encoded the same thing yesterday at 352x480, directly from the .d2v, and the file size was 727,316KB ( wanted 718,988 for one CD and audio at 112Kbps )

Here's my full log:

Code:

http://www.kvcd.net
CQMatic Version 1.2.03
Copyright Softronex Corporation, 2003.
All rights reserved.
Time: 07:15:18 Date: 11/13/2003
Ready!
Project: K:\rosswell.tpr

Creating: CQMatic.tpr

f:\temp\rosswell.m2v
Project resolution: 704x480
Execute.
Movie Time: 119
Average Bitrate: 1474.2
Full Encode mode
Executing Prediction Phase...
Process started at 07:16:17
On 11/13/2003
CQ set for prediction
Setting up initial sampling.
Using CQ of 60.00
Prediction cycle #1
Encoder started...
Process time: 37.02 minutes.
Encoder end.
File size difference  = 0.866200
Low fence: 2.000000
High fence: 60.000000
Last CQ = 60.00
Current CQ = 51.97
CQ difference = 8.028019
Using CQ of 51.97
Prediction cycle #2
Encoder started...
Process time: 36.23 minutes.
Encoder end.
File size difference  = 0.942350
Low fence: 2.000000
High fence: 51.971981
Last CQ = 51.97
Current CQ = 48.98
CQ difference = 2.996189
Using CQ of 48.98
Prediction cycle #3
Encoder started...
Process time: 30.55 minutes.
Encoder end.
File size difference  = 1.025078
Low fence: 48.975792
High fence: 51.971981
Last CQ = 48.98
Current CQ = 50.47
CQ difference = 1.498093
Using CQ of 50.47
Prediction cycle #4
Encoder started...
Process time: 29.62 minutes.
Encoder end.
File size difference  = 0.992234
Low fence: 48.975792
High fence: 50.473885
Last CQ = 50.47
Current CQ = 49.72
CQ difference = 0.749046
Using CQ of 49.72
Prediction cycle #5
Encoder started...
Process time: 35.38 minutes.
Encoder end.
File size difference  = 1.024513
Low fence: 49.724838
High fence: 50.473885
Last CQ = 49.72
Current CQ = 50.10
CQ difference = 0.374523
Using CQ of 50.10
Prediction cycle #6
Encoder started...
Process time: 25.78 minutes.
Encoder end.
File size difference  = 1.020819
Low fence: 50.099361
High fence: 50.473885
Last CQ = 50.10
Current CQ = 50.29
CQ difference = 0.187260
Final CQ value is 50.10
Encoding set to Full encode.
Full encode start...
CQMatic complete!
Total minutes of process: 194.60
Process ended at 10:30:53
On 11/13/2003

The full encode took 10 hours 29 minutes, so 3 hours and 14 minutes for prediction is not bad, considering the full encode time. And with these conditions ( Slow CPU, 704x480 and MA script ), it would have taken ~21+ hours for a 2-pass encode

Anyway, I need a faster CPU

-kwag

Dialhot · #13 11-14-2003, 05:30 AM

@Nicksteel:
Can you make a try without all your trimming instructions ? Perhaps that this combined to the source range editing done by CQMatic to do its samples completely messed up the process.

incredible · #14 11-14-2003, 06:00 AM

@ Nick

I never had problems using Trim()'s....

Nearly 70% of all my OneCD Encodings base on captures, where I shurely do apply Trim() in my scripts too ....

Maybe you have a problem in general

Did you set "Detect Scene Change" in TmpgEnc????
If yes TURN this OFF!! Cause it could reset uncontrolled the GOPseq structure too often (and we don't know which Threshold is used by TmpgEnc to "see" a scene change as real!).
Because a untouched Full length GOP structure based sample gives a "+" in acuracy during prediction.

And IMHO that's also a Problem of CQ matic if I understood this right: .... it everytime takes the same Sample-length!! (now depending on the X factor) and the same amount of samples (as already DialHot mentioned) ... in case of diff. movie lengths in total

We should fix this ... as I figured out there in the "new prediction" Thread using my GOP based script ... I tried a lot of individual settings and GOPseq-length-based sample slicing gave me an extra acuracy. And of course as already mentioned to set how many samples will be taken ... and not only the sample length by changing the X facor

Hmmmm ...

?

kwag · #15 11-14-2003, 11:14 AM

Quote:

Originally Posted by incredible

Did you set "Detect Scene Change" in TmpgEnc????
If yes TURN this OFF!! Cause it could reset uncontrolled the GOPseq structure too often (and we don't know which Threshold is used by TmpgEnc to "see" a scene change as real!).

Not really, because all of my encodes are done with "Scene Change Detect" Enabled. So that shouldn't be an issue, specially with X3 mode, because of the size of each snapshot taken (15 seconds).

-kwag

nicksteel · #16 11-14-2003, 02:28 PM

I'm having better luck with predictions for MPEG2 now. Most of the problem was the anamorphic statement (thanks Phil). The encodes for SVCD are still a bit over, but seem to be ok using audio of 192 for real 128. This allows SVCD overhead. The trim statements give no problem. I'll post some results later.

THANKS, guys.

kwag · #17 11-15-2003, 12:35 AM

Hi Guys,

Forget it

CQMatic just can't deal with TMPEG's unlinearity ( un-predictability )

I just encoded "K-19", using this script:

Code:

LoadPlugin("C:\Filters25\unfilter.dll") 
Mpeg2Source("F:\K19\VIDEO_TS\k19.d2v" )
Limiter() 
undot() 
BicubicResize(352, 144, 0, 0.6, 24, 57, 672, 366)
STMedianFilter(3, 3, 1, 1 ) 
AddBorders(0, 48, 0, 48)
LetterBox(16, 16, 16, 16)

And the final file size SUCKS

Wanted: 703,196KB
Encoded: 612,125KB

So there we have it

I can't deal with TMPEG's idiocrasies any more

I'm tired of this

Why

Because TMPEG does whatever it feels like with CQ

If you want close to 100% prediction, go 2-pass. Period

Just make sure you set your MAX bitrate to about 2,000Kbs ( unless you're doing DVD's), because it you don't, 2-pass will look like crap. If you set your MIN to 0.57 * average and MAX to ~2,000Kbps, 2-pass should look ok.
I just hope, for the benefit of all that the developer of TMPEG will make CQ linear (some day

)
QMatic development is now "Freezed" until some other, more accurate form of prediction can be achieved

-kwag

nicksteel · #18 11-15-2003, 06:53 AM

Quote:

I just hope, for the benefit of all that the developer of TMPEG will make CQ linear (some day )

This would solve a lot of difficulties. Maybe some day!

incredible · #19 11-15-2003, 07:59 AM

Quote:

Originally Posted by kwag

I'm tired of this

Kwag .... we can do whatever we want

There will NEVER be a near 100% (seen in average of all results!) prediction ... but you know that.

To rise up the sample lenght by using X factors is in fact an advantage, so you didn't waste your time man!

Even using the pingpong method, .... even it gives me great results on near 10 movies already encoded ... in a few encodings there also will be a failure

non linear CQ as you already said above.
But no matter which of the Apps. we use - its the right way to give in average a good method to get very near to a needed final size. No matter if using TOK, CQmatic, Pingpong, bimbam, tictac .. whatever

Levantate! So keep on developing CQmatic

Shure CQ isn't linear, but .... is it really reponsable for such a very big difference as seen in your last result above.

jorel · #20 11-15-2003, 11:50 AM

Kwag and friends:

i encode harry potter 5 times(testing scripts quality)!

the last was using CalcuMatic106 and CQMatic1203.
320x240 mpeg1,CQ_vbr...
X1 prediction=CQ20,44
X3 prediction=CQ21

movie with 153minutes, 40 seconds.
min 300,max2000,mpeg1,audio 128-48k,CQ21(X3 prediction),CQ_vbr
....forget something?

800mb

( 819,....kb! )
PERFECT.......100%.......is 800mb(not 799 or 801)

Kwag...why you are "tired"?
how can i "forget it" ?