CQMatic 1.4.00 Release Candidate #2
 

http://www.digitalfaq.com/archives/error.gif

What's new in this version :?:

1) Completely re-written prediction engine. Sampling now depends on movie length, so it's now dynamic.
You should be able to throw just about anything at CQMatic, and it will know "how" to encode "whatever" material (PAL/NTSC, etc) :D

2) Prediction is now done in "phases". Then, using a formula and some tricks (don't ask ;) ), it determines the correct target CQ.

3) Prediction is MUCH faster :!:
On my Pentium 4 @2.8Ghz, I am averaging a full prediction cycle in about 20 to 25 minutes minutes, on a average movie of 1 hour, 45 minutes (on linear range, CQ=~50 to CQ=~85)
Below CQ=~50, you all know it takes longer, and I can't do a thing about it :roll:
The first phase, in theory, should be the longest.
All subsequent phases uses the previous found CQ, which is closer to final value, instead of starting every phase at CQ=60, as in previous versions.
Still, you should count on about 1/5 of your total movie time, for prediction time.
Still, it beats the hell out of any multipass encode :mrgreen:

4) Removed "Override CQ" option. It's now done automagically ;)

5) Added progress bar, to give you a visual idea where the process is.

Here you have it: http://up1.fastuploads.info/CQMatic-1.4.00-RC-2.exe.zip

Please report any problems on this same thread.
If everything goes well, and no bugs are found, I will cut the software as 1.4.00 Release :)
If there are inconsistencies, then there might be more release candidates before final release.

Thanks to everyone, and all who BETA tested :cool:
Enjoy :!:
-kwag

Great Tool Karl!

I know you got your hands into many things and this would be just one of many in a long line of requests, but what about taking some of this functionality to predict with NuEnc and it's CQ function.

Right now I am doing it manually. It's super simple and easy to determine as it seems mostly linear with Peter Cheat's code modificatins. Right now I am running commandlines via batch files.

Greg a.k.a. Racer99

I will, once libavcodec based encodes prove to be more stable. Their current state is still not useable for production environments. I will support it, once I define a very simple API file, where you will put your basic parameters, similar to a .tpr file in TMPEG, and then your batch file. Just to make it transparent and similar to the current process.Thanks :)
-kwag

RC-3 is up.
http://up1.fastuploads.info/CQMatic-1.4.00-RC-3.exe.zip

You might want to give this a try.
RC-2 does 5 test phases.
RC-3 does 7 phases.

I've had extremely good results with RC-2, but on ONE case, I got a file size over the wanted.
This was due to the damn CQ (un)linear range in TMPEG, when target is in the range of ~69 to ~70, where just one click of CQ makes a huge diffference in final file size.
With two aditional phases, this should zero in more on the final precision for these particular cases.
I'm taking a constant prediction time of around 37 minutes with RC-3, for a full movie, which takes me about 2.5 hours to encode.
That's about a 25% (worst case) overhead for prediction, which I consider ok, because what I want is final size to be within a calculated target of no more that -2% to +2%
As I've said before, if your final CQ will fall in a range of ~50 or less, you can count on a VERY long time to predict due to TMPEG's behaviour, and I can't do a thing about that :roll:

-kwag

YES !

Movie length : 146 min - Target half DVD
Wanted Size : 2109062
Encoded Size: 2078620 :!:

Note: RC-2 found CQ= 81.50 and hit the target.
Beta-4 found CQ=89.5 and was a little over (above 2200000)
Beta-2 found CQ=80.6 and was a lot under (19000000)
Beta-1 found CQ=79.8 and was a lot under (17800000)

:ole:

Now predict with RC-3, please :!:
It should be very close, on every run.

-kwag

Mi results, with RC3.

Test bed movie: Red Planet.
Time 102 minutes, 39 seconds.

Previously encoded with a CQ of 65.81 (from older version of CQMatic) , which rendered a file size of 709,325KB.
My wanted file size, based on CalcuMatic, is 722,734KB.

Test runs :arrow:
Edit: The above were seven individual runs :!: Not a single 7 phase prediction.

I assume that If I encode with all of those CQ values, the final target will be well within the +-2%, as in the calculations in CQMatic :)
Remember that the maximum deviation of the final target should be no more than 2% below, or 2% above target. It's really a 4% "window", trying to keep the final size within that bracket.

Seems CQMatic 1.4.00 (release) will finally hit the road later today ;)
I just compiled it, and fixed the progress bar to reflect the new 7 phases, and everything else is identical to RC-3.
Just waiting for more feedback, to see if we post it, or if there's a need for a RC-4 (hopefully not :!: )

-kwag

The link is not working :( .
Are thousands of us trying to download it?.
I'll try again in several minutes... if there's no problem with the link :?:

I guess that site is down :!:
Well, here you have the release ;)
http://www.kvcd.net/downloads/CQMatic-1.4.00.exe

-kwag

nice! :wink:

Thanks ginoboy ;)

And that's exactly 1.4645 % difference :cool:

I just re-did "Rambo III", with another script, and look:
Target is 2 CDs, full screen movie (KVCD)

Wanted: 1,520,640KB
Encoded: 1,541,067KB

For a 1.3433 % accuracy :)

-kwag

@Kwag

It´s very cool, Mr. Karl Wagner! :D More one "goal" for you! I´ll download it and I´ll let it a shot! 8)

Keep up on work, mate! :wink:

My Best Wishes,

Thanks danpos :)
Please test it, and let me know if it fails somewhere :!:

-kwag

Another test :!:
Just for the hell of it, I encoded (my daughter's DVD) Harry Potter Prisoner Of Azkaban, at 352x340. The DVD is the full screen version.
I just wanted to see how accurate the encode was going to be, because I really don't do any 352x240 encodes anymore :lol:
But her TV is 27", so 352x240 look ok, if done properly.

Anyway, here are the results:

Movie time: 141 minutes, 39 seconds.
Target: One CD 8O :lol:
Wanted file size: 689,974KB
Encoded file size: 672,345KB

And I'll say NOT BAD, for a 2 hour, 21 minute Full Screen movie, on one CD :!:
I only used this script:
Yes, it could be WAY better with proper filtering, but I just wanted to test CQMatic :cool:
Little sample here: www.kvcd.net/downloads/hp.Sample.m1v

-kwag

I'm not very confident with value #2. Can you encode with this one to be sure ?

I sure will :!:
Starting now.

-kwag

BTW what is important is to tell in the stricky post (I guess you updated it with new release) is that new engine works on a random sample and 2 laucnches won't give the same value. So if an encoded is over to far from the target, just restart a CQ guess and the new value will have great chances to be the good one.

Yes, you are correct. Every run will give different, but close values.
And it's very possible that CQMatic could fail, if it gets in that non linear range of TMPEG, and will produce a file size much more that the 2% wanted :!:
But the chances are now much lower than in previous versions.
It's really a nightmare to predict in that area :roll:

Edit: Sticky updated with note.

-kwag

@Dialhot

Result of encode at CQ=69.54
File size: 714,913KB ;)

Which clearly explains that the area of 65.x to 69.x is almost flat :!:

Look what hapened on Harry Potter, on two separate runs:

CQ = 69.384. File size: 629,105KB

CQ = 70.440. File size: 672,345KB

The CQ difference is only 1.056, but the file size difference is 43,240KB :!:

And that my friends, is what I call a VERY sloppy CQ curve in TMPEG :x
The only way I think that I could correct this in CQMatic, is charting a complete CQ curve, in steps of .1, and then depending on the final calculated CQ value, compare the value to the chart, and adjust it so it doesn't overshoot the file size.
But the problem is that every version of TMPEG has a different CQ curve, and this would be a nightmare to maintain.
So my advice is: If your file size is way off, run CQMatic one more time. Chances are you'll get the correct file size on the second run.

-kwag

You won't believe me but... the 146 min long movie I use for testing is... Harry Potter :-D

:lol:
But I used a one CD target, at 352x240, so that could also be a problem.
I've tested 704x480 NTSC and 704x576 PAL, without any problems.

-kwag

TMPGEnc has always been disappointing in this area.

I would like to see more emphasis on NuEnc, as soon as Peter Cheat can release a stable and bug free version of it.

A combination of CQ Matic + NuEnc would be great.

I don't like to use TMPGEnc at all, because it is simply inaccurate. Predicting the CQ Value with TMPGEnc is always a gamble.

We all should leave the chapter TMPGEnc behind us...

Tmpgenc (MPEG1) gives the best quality whaetever the encoder you try to challenge with it. I have perhaps 1 movie on 20 that gives problem with CQMatic. People having more problems than that never understood that this the the MA script that is the guilty :!:
With a static (linear) script, the accuracy of CQ prediction is very effective.

And the visual quality one ?

TMPEG is still the king, as far as MPEG-1 quality and mpeg (stream) compliance.
NuEnc and all other avcodec based front ends are still very buggy.
Sure I'll add support for NuEnc. But only when it reaches stable useability :)

-kwag

Hey Phil,

I just re-did Harry Potter, but this time the target was KDVD 704x480.

Wanted size: 1,818,424 KB
Encoded size: 1,808,832 KB

Diff: 0.53% :!:


-kwag

For my part I did not have a closer result with official release than with the RC2. But there were close.

Found a bug
 

Okay, this has only happened to me once and only once. I set up to do a full encode of a movie -- The Bourne Supremacy (Fullscreen NTSC) if it helps -- and at phase 2 of cycle 3 I was presented with an interesting error. I can't remember the exact text, but it was something along the lines of "Illegal floating point operation." Something like that. It occured halfway through the phase, not at the beginning. Needless to say the next phase was using a significantly higher CQ value, and I had to restart the process.

The bug hasn't happened in this run thus far, so I don't know if it's a major problem. But it's there.


Oh yes, I'd like to request an option to add cycles to the prediction process as a means to increase accuracy(sp?). I usually leave my computer encoding while I sleep or while I'm at school, so if the process takes 6 hours I'm not going to mind.

Hi yukichigai,

The only time I've seen that floating point error was once, and I believe the probem was in my source. I haven't experienced that problem for months. It's not related to CQMatic. It's either the ".avs script/.d2v/source" combination.
As for the increased accuracy, yes I could add an option where the final algorithm that CQMatic uses will try to focus down on a narrower final file size target percentage.
This of course could take 2 to 3 times the current prediction time, but as you say, it's an option for leaving ovenight and forgetting about it :)
I'll add something for the next version, later this weekend or early Monday.
A "deep-mode" prediction option. ;)

-kwag

This is a well know tmpgenc"s bug that you can solve, when you encode manually, by changing a little the CQ (add or remove 0.1).
If this happens when you use CQMatic, then you are unlucky...

Well, here you have it: http://www.kvcd.net/downloads/CQMatic-1.4.07.exe


Added: Deep prediction option.
If this option is checked, the final size for prediction is 0.5% to wanted file size. That means that samples will target a sample size within 0.5% of wanted file size, per cycle.
If this option is not checked (default), then 2% file size precision is used for prediction.

-kwag

Muchos Gracias
 

Sweet, thank you very much. This will come in handy.

Also, I found another bug, maybe. After I got a good render of the movie I tried to go back and do an SKVCD (352x480 MPEG-2) render. CQMatic flipped out. The prediction cycle, the ENTIRE cycle, took under a second, and the final CQ value was always 89.something. This produced a file that was too large, so it wasn't an accurate prediction.

So, yeah, you might wanna take a look at that.

Thanks again for the Deep Prediction option. This will make further projects much easier.

What do you mean by "Go back" ?

If you opened tmpgenc, changed some settings, and saved again the project then you forgot to RESET the range editing. This must be done anytime !

I made sure to make a completely new project. I started from scratch. It still flipped out.

There was no range editing involved. If I ever need to edit the range I render I do it with DVD2AVI, so as not to make CQMatic freak out.

I arrives that I neverf close CQMatic for a complete week. Doing encoding after encoding. I can tell you that if you do everything correctly, it works !

So explain better what you call "going back" ???

By "going back" I just meant "went to encode the Bourne Supremacy again".

As far as what I did, I opened the d2v file again, loaded the SKVCD template, made sure it was set to 4:3 display, full screen video arrangement, no source range edits, then saved it as a text project. I did exactly what I did for the KVCD encode, except I loaded up the SKVCD template instead.

And then you load the new project into CQMatic and start the process ?
That's what I'am doing anytime !
Look at your log file, if all size are negative, then you have a range editing problem.

Note: perhaps the problem is when the new project has the same name than the previous. I never did that.

I dunno what the problem was, but after a number of times hitting "new project" it cleared up. Huzzah.

Anyway, I used the new Deep Prediction option on The Village (Fullscreen):

Target filesize: 679520kb (663.59mb)
Actual filesize: 704676kb (688.16mb)

Oddly enough, it didn't seem to take any extra time. Not sure if this is indicative of the overshoot or what.

More problems
 

Once again, I've gotten the odd "Illegal floating point operation" error, and not just once. It only seems to occur with the Deep Prediction option checked.

This is not a CQMatic problem :!:
This is a TMPGEnc problem, with a particular source/script combination :!:

Quick test: Note down the CQ value that was used when the error appeared. Run TMPGEnc at that same CQ value, with the same script, and the error should re-appear.

-kwag

Not necessary because this problem often occurs at a scene change so if you encode the whole film you can have no problem has the scene changes are differetn during prediction (encoding of a sliced sample).

But as you tell, this is not related to CQMatic.