KVCD Predictor
 

I've been working on it and have most of it done, just need to implement the various "Get this from there..." functions. Here's a screengrab to whet your appetites:

http://www.digitalfaq.com/archives/error.gif

You can expect the first release today :).

Wow...That looks GREAT!
Kudo`s to you, and all that put in the work!...http://www.digitalfaq.com/archives/error.gif





***************************************
They can put a man on the moon,
but they still can`t put a tin pan in a microwave.......UNKNOWN

Beautiful :lol:
You could calculate the audio file size automatically, by just dividing the number of frames by frame rate. That will give you total number of seconds. Then the user just selects audio bit rate ( 128, 160, 192, etc. ), and multiply bit rate by seconds, etc to get audio file size automatically :idea:
Anyway, that looks AWESOME :) :)

-kwag

I was going to do that, and then I thought "what if they're using VBR?" but I guess none of the templates call for that. I'll implement it now :).

When DVD players support VBR MP2 audio, then we'll do a audio file prediction formula, just like we did for video :lol:
Same principle. Different samples at different time intervals ;)

-kwag

I don't think that's necessary... there are not a whole lot dvd players to support that, and you don't save a lot of MBytes with VBR sound anyway...

First release
 

Yes, it's the moment you've all been waiting for ;). Here's KVCD Predictor 0.1 (and source code), ready for you all to pound on.

Please read the accompanying documentation. It's pretty easy to install, but you will need the .NET runtime if you don't have it already. There's a link in the readme.

I've established it's fairly well-behaved and doesn't seem to crash unexpectedly or get drunk and insult the in-laws, but I've not done extensive testing on the actual algorithm and the results it produces. This is where you guys come in :). Feed it as many values as you can think of, work out the result by hand and see if it matches what KVCD Predictor comes up with. Do some encodes with its recommended sample size and see if the final encode will really fit on that number of CDs.

Bear in mind it uses the exact length of the CD, which can be calculated with the following formula:

bytes = seconds * 75 * 2324

where seconds is, obviously, 60 * minutes, or 4800 for an 80 minute disc. Thus an 80 minute disc holds precisely

4800 * 75 * 2324 = 836640000 bytes

Also be aware that any reference to "megabyte" or "kilobyte" means 1048576 bytes and 1024 bytes respectively, so that if Windows Explorer says your audio file is 95.6mb, that's the value you enter. The 1024/1000 difference does not effect the final calculation, which is all done in bytes anyway (if you don't know what I'm talking about you can ignore this whole paragraph).

In this version the "Get movie length from source file..." option doesn't work, because it's going to be a lot more complicated than I imagined (for programmers amongst you, I have to learn how to write a managed C++ class to access the Video for Windows framework. Any hints appreciated :)). I'm going to work at it though, and hopefully it'll be ready for 0.2.

So, pound away. Use it and abuse it and let me know what happens. It's likely some tweaking will have to be done, but that's what an alpha release is for.

Have fun :).

P.S. Kwag -- could you please mirror the binary package for those who can't access my site at the moment?

Edit: Removed links to old versions.

Oh, and by the way, feel free to suggest better names ;).

Re: First release
 

Mirroring in the next five minutes ;)
I'll post a note in the "What's Happening" section and on the main page.

Thanks :D
-kwag

Re: First release
 

Excellent. I'm very eager to hear some feedback :).

Main page at www.kvcd.net updated.
Mirrors updated.
"Whats Happening" updated.
;)

-kwag

Great job SansGrip! :D

One thing I notice is the est. sample size is ~24mb less than 800mb and multiples thereof with scale factor @ 1.0

i'm using this formula:
MPEG size = ((Total frames/Framerate)/100) * (MPEG sample file size * scale factor)

again, good work and sorry if my math is wrong! :wink:
-ren

Hi SansGrip,

I just tested the program, and the audio calculation is perfect, but not the video. My #frames is 173097 / 23.976 / 100 = 72.195. My audio at 128Kbps, selecting one CD of 80 minutes, gives 112MB. That is correct.
Now for the video. 800 -112 = ~688MB for video. That's the estimated size. So if we reverse the formula: 688 / .95 / 72.195 = 10.03MB for the sample size, but the program calculates 8.87MB.

-kwag

Hi SansGrip,

KVCD file predictor needs .NET to run and the above quote is from Micro-
soft. :( So myself and others will not be able to use your fine program.
I use Win ME. :)

-black prince

New version
 

@rendalunit & kwag

Yep, you're right. Here's 0.1a ;).

(In case you're curious, I was adjusting the media size to compensate for the system stream, which is what you do for bitrate calculation but evidently not file size prediction :). I just took it out and did a couple of tests and reversing the result manually seems to be correct now.)

@black_prince

The SDK (Software Development Kit) requires Windows 2000/XP, but the actual framework (the runtime libraries) work on the other versions too. You should be able to find a link to it on that page I provided -- if not, you can get it through Windows Update (or here for the English version) and I'll fix the link in the doc.

Edit: Removed links to old version.

Hi SansGrip.

Almost!, it's still a little low. The result for the example I posted above, should give a sample size of 10.03MB. But the result is now 9.03MB :!:

-kwag

I'm really living up to my nickname tonight ;). I'll look at the source and try to figure out what I did wrong...

Edit: I'm multiplying by the scaling factor instead of dividing. 0.1b coming up ;).

Another new version??
 

Here's 0.1b, which should now produce the correct results...

At least, on kwag's test data it produces 10 -- not 10.03 -- which, after stepping through the code, is actually the more accurate value given that an 80-minute CD isn't quite 800mb (it's more like 797mb).

Hopefully this is the last update tonight ;).

Edit: Removed links to old version.

BINGO :lol: *** Jack Pot ***, you hit it right on the nose :mrgreen:
Updating mirror now....

Edit: ... Done!

-kwag

...which leads me to the question, should there be a facility for specifying a custom CD size, for those who like to overburn?

Also, wouldn't it be more logical and intuitive for a "scale" factor to be multiplicative? That is, instead of dividing by 0.95, multiplying by 1.05?

Also, @kwag, what do you think of the idea in the readme of some kind of formula to give a guess as to a good CQ/bitrate for producing a sample of the desired size? Too difficult?

Sheesh... Three versions later ;). Anyway, now it's producing the right answers, let's see how it holds up to some real vigorous testing...

Edit: You forgot to update the source code link in the announcement :). It should be 0.1b_src.

That's a good idea!

The reason for the .95 came to be, because the sampling taken on a one second basis, always produces ~2% to ~3% lower file size than predicted. The reason is the KVCD GOP, always gives smaller file size than a standard GOP, where there is less compression. So on tighter GOP's, like the standard IBBP ( Not IBBBP as in KVCD ), the prediction is closest to 1.0, and maybe above 1.0, for very active movies with lots of I frames inserted at the point of scene changes. So on KVCD's it will always be a factor below 1.0, and 1.0 or above for standard mpeg GOP structures.
I don't think it would be too difficult ;)
Maybe taking instead of 100 initial one second snapshots, take 10 one second shots. Then increase to 20. That should be close enough, but on each step, adjust the CQ_VBR value. Then jump to the 100 samples and zero in, just like on a binary search. Ex: if CQ_VBR at 50 is too high, divide by two, so now it's 25. Else, multiply by two, so it's 75. That way it should zero in on a CQ_VBR target pretty fast ;) Maybe one pass of 10 one pass of 20 and one or two passes of 100 snapshots would hit the target pretty close.
Hope I explained myself correctly :roll:

-kwag

:oops: Sorry.
Done. :o

-kwag

Note to self:
 

@SansGrip,

From your readme file:

"0.1b Note to self: Learn difference between multiplying and dividing. "

I'm still laughing at this one :mrgreen: :mrgreen: :mrgreen: :lol:

Ok, I hope to get that in for 0.2 along with "Get movie length from source file...".

This is what I've been doing, but I didn't really mean what you thought I meant. I think 8O.

What I meant was, it would be nice if KVCD Predictor could display, along with the maximum sample size, a guess as to what CQ and bitrate you'd want to try in order to hit that target. I'm guessing to do this it would need to know the template you're using and maybe whether the movie is "low action", "high action" etc.

That's why I asked if it would be too difficult ;).

Edit: Though, thinking about it, it shouldn't be too hard to implement that kind of binary search via some kind of wizard interface, with it saying "Encode with CQ 25, min bitrate 500" then you tell it the size of the file, then it says "In that case, try CQ 22, min bitrate 400" and so on... Stupid idea?

I think that would involve something like doing a quick scan on the actual material to be encoded. Then determining, based on the bit rate activity (high/low), what kind of movie it is. Sort of a "Quick Pass" similar to the prediction samples, but actually analyzing the material. Then it would have to be compared to some pre established constant references. Like which material is low action, and which is high action. Which of course could be done by running the program against many movies :idea: and creating a "fingerprint" of a DVD or movie :idea: for the encoding bit rate to be used.

-kwag

No, not a stupid idea at all :D . But it might be better to keep the bit rate boundaries fixed (MIN/MAX) and only adjust/slide the CQ_VBR value. This way it's only one factor involved. Not three ( CQ_VBR, MIN, MAX ) wich then also change the linearity curve of the CQ_VBR. Like MIN=300, MAX=1,150, CQ_VBR=30 gives a larger file size than MIN=300, MAX=2000, CQ_VBR=30 on a low action scene 8O . Because we changed the MAX from 1,150 to 2,000, and kept the CQ_VBR value constant, the scale in the encoder changed drastically, and produces a worse result. This example only on low action. On high action, then of course the file size is larger with the MAX=2000. Don't know about other encoders, but this is the behaviour of TMPEG.

-kwag

@SansGrip,

When you enter the time length of the movie does it default to 23.976 fps? If so then if it's 25 or 29.97 then you have to enter # of frames and fps so maybe the fps should be a "must enter" value- or you could have a choice of FILM, PAL, or NTSC on the main screen.

Here's a crazy idea- what if FitCD could be modified so that it will write a line into the .avs with the #of frames and framerate (or you could do it manually), something like this:

# 150000, 23.976

then KVCD Predictor could read this info from the .avs so you won't have to enter anything!

here's crazier part:

then when you've adjusted the CQ to get the target sample size then KVCD Predictor would go into the avisynth script and remove or comment out the file prediction lines so that you're ready to go! :?:

just and idea :)
-ren

The "Enter movie length in frames and frames per second..." dialog box defaults to 23.976, but the length is actually stored internally as hours, minutes, seconds. Before calculating the target sample size this is converted into seconds, since that's what matters in the formula. The (frames / fps) part of the formula just gives you the length of the movie in seconds anyway.

So there's no need to enter an fps unless you want to specify the movie length in frames, and thus no need for a film/PAL/NTSC choice.

Once "Get movie length from source file..." is implemented, you'll be able simply to browse to either an AVI, AVS or D2V file (perhaps even MPEG at some point, if there's any demand for it) and KVCD Predictor will read the frames/fps information directly from the file. You'll also be able just to drag 'n' drop the video file onto the window to do the same thing.

I was actually trying to think of a way to get rid of the "add some lines to script" step of the whole thing, but the only "solution" I could come up with was automatic generation of a new, temporary script to use just for encoding the sample. But it saves such little effort for the user that I'm not sure it's worth my effort to implement and debug it ;).

Sounds like motion estimation to me. Ouch!

I like to keep variables to a minimum. Fewer opportunities for me to mix up my mathematical symbols :D.

As for the wizard-assisted binary search for the best CQ, that sounds like an interesting idea to me. I imagine something like this:

1. Specify path of sample file. Let x=initial CQ value, say 25
2. Tell user to encode with CQ of x
3. Wait for "done" button to be pressed
4. Read file size
5. If within threshold of target, stop
6. If low, let x=new higher value
7. If high, let x=new lower value
8. Goto 2

The fun part will be figuring out the new higher/lower value. It might not be quicker for an expert (read: kwag) who can get it right in a couple of tries, but it might help us normal folk (read: me) who take five or six attempts to get close :P.

It could even suggest a couple of fractional values towards the end, to really squeeze the most out of it.

How does that sound? Feasible? Useful? Worth the effort?

@SansGrip and Kwag,

Getting all kinds of errors trying to execute KVCD file prediction after
insatlling Microsoft.NET:

Then I tried to reinstall microsoft.net and could not remove it
or re-install it. :?:

I don't suppose a standalone version could be created :?:
I can't seem to get this to work with .NET.

-black prince

You bet :D Sounds Feasible AND Useful AND Worth the effort ;)
Here's a simple formula I've been using to zero in faster on the prediction. Say I need 10MB for target and I'm encoding at a starting CQ_VBR of 25, but on the first try I get 6MB. So just do 10/6 =1.6666 and then 25 * 1.66666 = ~41. Run again with CQ_VBR=41. It usually takes me two or three runs (sometimes four) to hit the target. Never fails.
And that's WAY better than waiting for two or three X-pass VBR ;)

-kwag

Very weird. Theoretically if you have the runtime installed you should just be able to double-click it and go.

I asked Dr Google and he came up with a bunch of other people with this kind of problem. One suggested solution I read was uninstalling the .NET framework (through Add/Remove Programs), then installing MDAC 2.7 from here, then reinstalling .NET.

Sorry you're having problems with this :(. Unfortunately I don't have a copy of Windows ME to test it on, but I can try it on my Windows 98 laptop tomorrow. Hopefully I'll have the same problem, and will figure out how to fix it, and can then tell you :).

Well I could rewrite it in plain C++ and MFC (just like in the bad old days before running water), but that would probably take three-to-four times as long as it did to write it in C# (and the same goes for any future enhancements, of which I hope there will be many). I'd like to try to get .NET working on your machine before I consider the horrors of Windows programming in C++ :D.

Then let's do it! :D

Sounds perfect for automation. Not even I could mis-code that algorithm :!:. It's late, but... I think I might start coding it right now :mrgreen:.

The problem with file size predition is that file size and Q don't have a linear relationship, this relationship is not well-known and it varies from movie to movie... This looks like a typical problem where you could give a neural network a try.

I'd imagine a network with 2 inputs: the size of the prediction at a default Q, and a given Q at which you want the estimate. The output of the network would be the approximate file size for the new Q.

To train the network one would need a large database with the filesize prediction for several Q's, for lots of movies (this could be a collaborative effort). The network may take some time to train, but once trained (and it would be trained only once at design time), it runs imediately. So you'd run one file size prediction encode at a default Q, give this number to the network and tweak a slider until the chosen Q gives you the desired file size.

If this estimative turns to be accurate enough, that's the fastest prediction you can get; but, we can collect the data, train the network and find out that it's no good and just forget about it.

This may be worth looking at:

http://www.informatik.uni-mannheim.d...ephan1995a.pdf
http://www.informatik.uni-mannheim.d...ephan1995b.pdf

Predictor is a great tool. Thanks Sansgrip. I get my movie length from Fitcd which nicely shows it in total seconds. Is it possible to have the program accept this single value rather than have to make several entries. This probably sounds trivial :)

Hi SansGrip,

SansGrip wrote:
Installing .NET says delete product code: {3BD60997-A7E4-4A55.....} before it
will install. Add/Remove reports it can't find file "netfx.mis" and terminates
removing .NET. Tried to use regedit to remove product code from above.

Found it in HKEY_Local_machine\Software\NET Framework Setup\
Full\v 1.0.....\. Deleted this and rebooted. install .NET still says product
code must deleted. So, I can't get rid of old .NET and new .NET won't
install. :( This is frustrating :? I'll just wait to a standalone is available.

-black prince

Hi SansGrip and Kwag,

Don't let my problems with .NET, Flux, and NoMo worry you for now. Keep
working towards your goals and I will solve this eventually. :D
I am not trying to discourage you in any way. I want to see more
improvements just as bad as you. :) I have other pressing problems
unrelated to KVCD, but I still will keep watching your progress and
when I can will add layman's comments and suggestions. 8)

-black prince