Bitrates: Weird Behaviour on a Movie!

[kwag]

Ok, I've encoded about 8 different movies now with versions 1.1.07, and I've double checked the results with 1.1.08b, and found basically the same CQ as 1.1.07. So I assume 1.1.08b is the same (or better) than 1.1.07.
On all the movies I've encoded, the final file size has been within 1% to 2% Which is just excelent Until yesterday, when I processed the movie "The Siege"
The Wanted file size is 718,395KB.
My first encode, final size came out to 635,883KB
So I thought I did something wrong, and I encoded the movie again with different settings. Here they are, with the file names detailing the specs:

siege-cqm-1.1.07-cq-53.0155-min-300-max-2500.m1v 635,883KB
siege-cqm-1.1.07-cq-49.991-min-473.1-max-2000.m1v 660,671KB
siege-cqm-1.1.07-cq-44.96-min-500-max-2500.m1v 660,765KB
siege-cqm-1.1.09-cq-45.69-min-500-max-2500.m1v 677,123 KB

The last test, a modified version of CQMatic (1.1.09) not-released, made to double the samples per minute, and again, the file size was still below. So there must be something wrong with the source. I've double checked the .d2v, and it seems correct. The .d2v was created with "Force FILM", and it was 99% progressive. The only thing I will try tomorrow, is re-creating the .d2v WITHOUT "Force FILM", and processing with Telecide() and Decimate() in a script. BTW, all the encodes above were created "directly processing the .d2v". There was no AviSynth script at all. This is the ONLY movie that has failed. I also tested it with ToK, and the CQ found by ToK was even lower ( ~40), so I know that the problem is not CQMatic. Maybe it's a color space issue
I don't know, but I just wanted to post this issue, in case it happens to anyone else.
Tomorrow I'll post the results doing IVTC in AviSynth, and see if it makes a difference.

-kwag

[audi2honda]

Hi kwag I've had similiar low CQ and low filesize results on my last few encodes but I didn't mention it because it is hybrid material. Mine are about 75% film which I process with no forc film then use telecide and decimate. These keep coming out smaller then expected. Regular movies...so far so good

[kwag]

Hi audi2honda,

How much smaller did they come out
I'm currently predicting the same movie with Telecide() and Decimate().
Then I'll let it encode, and see the final file size.
As long as the file size is "smaller" than target, I don't mind. But not too small
I can tolerate minus ~30MB, but no more
More that that, would be a waste of space, and some visual quality loss.

-kwag

[audi2honda]

They've come out roughly 40MB to small. I'm doing another one right now so I'll let you know exactly when it's done.

These are TV episodes on DVD so without telecide and decimate they have very bad interlacing.

[vhelp]

hi Kwag..

This just came to now (earlier actually) but I'm on a hunch that its something
to do w/ the framerate. In any event. I'm also hunching that you should
perform the IVTC externally, THEN feed the PURE 23.976 fps source:
* no vearing left/right in FPS, and
* no IVTC issues, cause all that was worked out when source was processed.

Once the above is complete, and you have (what I would call, the equivalent
to a dvd2avi .D2V file, process it as you did all those actual .D2V files

But, I think the color space may be playing only sligtely in this one.
I think it's the framerate and IVTC that dvd2avi w/ this particular source.

Why I recommend an external IVTC ???
Becuase you can't guarentee that the IVTC decision process will behave
EXACTLY as it did in one test. A different frame could be chosen at any
given moment, and for whatever the reasons. But, by performing the
external IVTC to a separate file (no filtering, for fastest process time) you
will in effect, have a progressive source, and no other decisions have to be
made upon it during "prediction"

Sorry, didn't see this post till now.
-vhelp

[Dialhot]

I have to say that I has the same problem yesterday with 1.1.07 :

Wanted movie size 750 Mb.

Predicted CQ with TOK : 73.18 - final size : 714 Mb
Predicted CQ with CQmatic 67.20* - final size : 702 Mb

(*) the difference is normal as min-max were diff between the two test (64-2000 for Tok, 350-2000 for CQQMatic; average = 610).

A little question abot CQMAtic : how does it compute the number of samples to take ?

With a video of 42 minutes (25 fps, PAL) it takes 4200 and with a 168 min one, it takes only 3100 ?

[kwag]

Quote:

Originally Posted by Dialhot

A little question abot CQMAtic : how does it compute the number of samples to take ?

Hi Phil,

CQMatic takes 120 seconds of sampling, no matter how long the movie is. It divides the movie in 120 "window" snapshots, each of the relative proportion to the total movie time. For example. If the movie is 90 minutes, it will take 120 snapshots of 1.333 seconds each.
If it's a 30 minute movie, it takes 120 snapshots of 4 seconds each. The idea is clear. Maintain a consistent number of samples, and the shorter the movie, the "wider" the sampling to compensate for the play time. I settled for a maximum of 120 samples, because it's a good balance between accuracy and time to predict. Please note that I already tried 240 samples on a test version I quickly cooked up, and in this case of "The Siege" movie, the result was almost the same
So it's not a matter of sampling. Something is WAY wrong in the movie. Either color space, or something else. This is really good, because now I have a movie with "consistent" results, that I can start to look into, as to why this strange behaviour. So I believe it's not a matter of the sampling algorithm I'm using. It's something that is screwing up TMPEG on the full encode (or giving TMPEG "glitches" on the sampling phase, which makes it lower the CQ during prediction). I'll be playing with this movie, as a model, to see where the problem is. This is the ONLY movie that has given me this problem. And because every other movie is less than 2% accurate, I strongly believe the problem is in the movie. If this wasn't the case, then the percent of accuracy would fluctuate on all other movies. But it doesn't

Quote:

With a video of 42 minutes (25 fps, PAL) it takes 4200 and with a 168 min one, it takes only 3100 ?

Check again
For a 42 minute movie, it takes 120 / movie time = 2.85
So that's 120 samples of 2.85 seconds per minute (in your case). That's a VERY wide window snapshot for prediction, per minute. So accuracy should be higher, the shorted the film is. That was part of the design

-kwag

[Dialhot]

Quote:

Originally Posted by kwag

And because every other movie is less than 2% accurate, I strongly believe the problem is in the movie. If this wasn't the case, then the percent of accuracy would fluctuate on all other movies. But it doesn't

I don't want to complain, nor blame you (), but in my case the accuracy is less than 7% (702Mb for 750 waited).
And I don't even use the MA script but an "old fashion" static filtering script.

Quote:

Check again

I will

Quote:

So accuracy should be higher, the shorted the film is. That was part of the design

Great. That is a good news. I do a lot of TV shows that range between 25 and 42 minutes.

[kwag]

Quote:

Originally Posted by Dialhot

Quote:

I don't want to complain, nor blame you (), but in my case the accuracy is less than 7% (702Mb for 750 waited).
And I don't even use the MA script but an "old fashion" static filtering script.

Well, that's what I mean
In most cases, it's either right on target, or ~50 MB off in some wierd movies, like that one you have, and the one I have
But if the differences were unstable, like 2%, 3%, 6%, 8%, 15%, then that would really mean a problem.
Right now, almost every movie ends right on target, and some about 7% off.
My "The Siege" movie, also ended up ~8% under. So you see, there is a pattern of <2% almost every time, or ~8 off in some cases

-kwag

[kwag]

I think I got it, and it's not a source problem It's another kwag problem
Currently running some tests. Expect an update in the next hour ( if my current prediction tests hit a wanted target )

-kwag

[kwag]

Quote:

Originally Posted by kwag

Expect an update in the next hour ( if my current prediction tests hit a wanted target )

-kwag

Long hour wasn't it
Everything was on schedule, until my wife de-scheduled me for other more important things ( for her, not for me )
Let's give it a whirl
http://www.kvcd.net/CQMatic-1.1.08d.exe

-kwag

[jorel]

hey Kwag,
the 1108d give me small CQ.
i need CQ63,25 and got CQ62,52!

http://www.kvcd.net
CQMatic Version 1.1.08d
Copyright Softronex Corporation, 2003.
All rights reserved.
Time: 23:18:59 Date: 08/12/2003
Ready!
Project: C:\CQMatic\1106.tpr

Creating: CQMatic.tpr

D:\La Luna\ToK\2\VIDEOKwagPhilpred2.m2v
Project resolution: 480x480
Execute.
Movie Time: 31
Average Bitrate: 1074
Prediction Only mode
Executing Prediction Phase...
Process started at 23:19:11
On 08/12/2003
CQ set for prediction
Setting up initial sampling.
Using CQ of 60.00
Prediction cycle #1
Encoder started...
Process time: 2.10 minutes.
Encoder end.
File size difference = 1.015823
Low fence: 60.000000
High fence: 90.000000
Last CQ = 60.00
Current CQ = 60.95
CQ difference = 0.949368
Using CQ of 60.95
Prediction cycle #2
Encoder started...
Process time: 1.85 minutes.
Encoder end.
File size difference = 1.014951
Low fence: 60.949368
High fence: 90.000000
Last CQ = 60.95
Current CQ = 61.86
CQ difference = 0.911274
Using CQ of 61.86
Prediction cycle #3
Encoder started...
Process time: 1.83 minutes.
Encoder end.
File size difference = 1.014144
Low fence: 61.860641
High fence: 90.000000
Last CQ = 61.86
Current CQ = 62.74
CQ difference = 0.874947
Using CQ of 62.74
Prediction cycle #4
Encoder started...
Process time: 1.82 minutes.
Encoder end.
File size difference = 0.988164
Low fence: 61.860641
High fence: 62.735588
Last CQ = 62.74
Current CQ = 62.30
CQ difference = 0.437473
Using CQ of 62.30
Prediction cycle #5
Encoder started...
Process time: 1.85 minutes.
Encoder end.
File size difference = 1.013726
Low fence: 62.298115
High fence: 62.735588
Last CQ = 62.30
Current CQ = 62.52
CQ difference = 0.218739
Using CQ of 62.52
Prediction cycle #6
Encoder started...
Process time: 1.87 minutes.
Encoder end.
CQMatic complete!
Total minutes of process: 11.32
Process ended at 23:30:30
On 08/12/2003

less samples per minute?

was faster but don't give me the perfect CQ like others 1108 versions!

[kwag]

@ALL,

Here's something I just found out. On all the movies I've done with CQMatic, sample GOP has been mostly sticking to 24 frames. Here's a screenshot of K19's prediction sample, which after the full encode, was almost exactly on the wanted size:



And here's the sample of "The Siege". Look at the difference



But now look closely at the GOP of the FULL encode of "The Siege" ( snapshot from about an hour into the movie) :



So there you have it
TMPGEnc is screwed up on some sources
I'll make some changes now on CQMatic, to disable the scene change detect while predicting. In the mean time, you can "Uncheck" the option, before you save your project. Let's see how this affects our final file size

-kwag

[kwag]

@All,

Thread closed. Problem solved in CQMatic 1.1.11 with new sampling

-kwag