File prediction experiment #1,000,000 :)
 

*** Post edited, see bottom for note ***

Here we go again :)
Another thread that might be a hit, or a flush down the toilet :lol: Well see :cool:
Since last night, something has been bouncing in my head. I recall my previous attempt about a month ago, to create a program that would read all the VOBs, and find out the bitrate per frame. After further analysis, I came to the conclusion that it wouldn't work, because the GOP in the VOBs is not the same as the GOP in our encoded MPEG files. We use a longer GOP in KVCDs. So I discarded the idea. But last night, suddenly another idea came into my mind :!:
If we are encoding VOBs, we can know what is the compressibility of a VOB, and we have a constant :!:. Here's my experiment:
Encoding by using the source compressibility ratio, and scaling to the encoder compressibility ratio.

Take a program like MPEG2Cut http://www.kvcd.net/MPEG2Cut-1.15.rar
Cut off EXACTLY 1 minute of your first VOB from the beginning. Just make it as close to one minute as you can. Save this piece to your hard disk. Now, open TMPGEnc and use the tools to "Demux" the video stream of the piece you just saved. We need this step, because the VOB has other streams, and we need precise video only size so we need to strip it.
Now open your newly demuxed .m2v with Vdub, and note the total number of frames. This is very important :!:
Now open your regular .avs file with TMPEG (the one you're going to use to encode the movie), and select start frame 0 and the end frame from the step above. We are going to encode exactly the same clip segment that we cut off the VOB.

What is the file size (sampler) that we need :?: Simple :)
Go into MovieStacker and open your .d2v. We need to get the compressibility ratio of our movie to be encoded. Look at MovieStacker's recommended "Space left for elementary video stream". Divide that size by the number of minutes of the movie, and again by the average bitrate that MovieStacker gives you.
You end up with this: KVCDFactor = ElementaryVideoSize/MovieMinutes/AverageBitRate

Now just divide the size of your demuxed .m2v size by the KVCDFactor, and you have the sample size that you must create with TMPEG ;)

Encode the selected frame range with TMPEG and find your CQ value to make the clip size as close as the sample file size found above.

What I'm trying to do here, is to use the compression factor from the source material, as a scaling factor for our movie to be encoded.
Is this accurate :?: Hell, I don't know yet, but theoretically it should :!:
I've done a couple of tests with various movies, and so far, the CQ I end up, seems very accurate based on my previous encodes with regular file size prediction. If this method is indeed accurate, then we just need to automate the cutting of the VOBs somehow, or find another way to extract the file size from a one minute clip in the source. Note that in theory, it shouldn't matter what piece you extract from the source, as long as it's the same piece to be encoded in TMPGEnc :!:
I've chosen the first minute starting at frame 0, because it's easier. If you start somewhere in the middle of the movie, then you would have to find out the starting frame, which wouldn't be 0. So it's just easier to start from the beginning.
I hope this works, if not, it was worth a try :roll:
And if I screwed up somewhere in the process, please let me know 8O

Note: I guess I did screw up :!:
It's easier than I described above. The sampler size is determined like this:
Open the .m2v you demuxed with Vdub. Go to File/File Information, and look at the average bitrate. Divide the file size of the .m2v by the average bitrate reported in File Information, and multiply the result by the average bitrate that MovieStacker wants :!:
That's your wanted file size for the sampler.

And remember: Specifications subject to change WITH notice :mrgreen:

Edit:
The above should be correct, but must be multiplied by the factor difference of the .m2v and the "to be encoded" movie factor. That is KVCDFactor / (.m2v size/.m2v Avg.Bitrate )
I'm currently in a long encode of K19 with the current script, using CQ of 66 ( given by this process)

-kwag

hi Kwag..

I was just checing this thread out.

In what way would this simplify the process ??

Actually, I checked out MPEG2Cut-1.15 and its based on dvd2avi.
But, I think I understand what you mean. Right now, it's combersome to
have to:
* smartRip your vobs, then
* open up a MPEG2Cut-1.15 and CUT approx 1 minute***

*** hay, have you figured out a way to "calculate" the one minute CUT in
.......MPEG2Cut-1.15 yet ??
Well, how about you try this.. find your spot you'd like to CUT, and
from that point, (using ">") start count 5 to 6 Crsr-Rt's and
save. This has given me 1 to 1m 30sec, give or take. Cause, the pictures
are based on "I" frames only, and those vary, from what I understand.
But, heck, it's a good gauge at least.

On another note, I did a test CUT of clip from K-Pax and in vdubMOD(4,971k)
and in BV(5,872k) was reported, and on another test, vdubMOD(5,125k) and
in BV(6,054k) So, I don't know exactly what gives, but maybe you can
better explain the difference.

Wait a minute, I think I have an idea for "simplifying" the CUTting..
There is a command-line version of DVD2AVI :wink: Maybe, we could
first see if MPEG2Cut-1.15 has this ability, if not, lets play w/ dvd2avi !!

Hope I've help some :) If anything, you've now got more options ta play with :)
Good luck so far though..
-vhelp

That you have to do anyway to encode your movie, so the cut is after you process your .d2v as usual :)You use the slider in MPEG2Cut to the one minute point, and mark that as the end. Then save the segment. It's a piece of cake :!:That's exactly what you do.I don't think MPEG2Cut has command line options. I may be wrong. But the cut doesn't have to be done with MPEG2Cut only :!: You can use MPEG2-VCR or any other MPEG editor.

-kwag

Here's the test (dressed up) formula:

And then:

Anyone care to revise this :roll:


-kwag

.
.
http://www.digitalfaq.com/archives/error.gif

Is there errors in it ??
-vhelp

Hi Kwag,

Firstly very interesting concept and something worth pursuing :)

However I think there is a flaw in the formulas. I'm no mathematical genius but it seems that there is a lot of negation going on. Let me *try* to explain (and maybe I'm completely of the wall here) :!:

KVCDFactor is as follows:
You are then multiplying this by:
(Demuxed .m2v file size / Demuxed .m2v Average Bitrate) * MovieStacker's Average Bitrate to get the desired sample size.

Lets consider (Demuxed .m2v file size / Demuxed .m2v Average Bitrate) from the sample forumula.

As you are multiplying this against the KVCDFactor, it negates the same division in the KVCDFactor formula leaving KVCDFactor as:

KVCDFactor = (MovieStacker's Elementary Video Stream / Total Minutes of Movie / MovieStacker's Average Bitrate)

So the sample formula changes to:
Sample size = MovieStacker's Average Bitrate * KVCDFactor

which gives the same result as your original formulas. :!:

Taking this a step further again you are also multiplying the MovieStacker's Average Bitrate * KVCDFactor in the sample forumula which again negates the same value in the KVCDFactor formula.

Leaving KVCDFactor = (MovieStacker's Elementary Video Stream / Total Minutes of Movie )

If that is the case the formula for the sample is
Sample = KVCDFactor :!:

This means that all the Demuxed .m2v information is all in vain and we are not utilising it in any way.

Maybe I got it all wrong but I tried this and the result is the same. Hope it made sense as I think I've confused myself as well :wink:

I got lost in the middle of kwag's first post :)

If you want to know how well will a movie compress, I think there's a tool that can make it dummy-friendly :)

When you first open a disc with DVDShrink, it spents a few minutes "analyzing" the disc so that the % of shrinkage for each compression level can be estimated. Perhaps this % shrinkage / compression level numbers can be used some way to make KVCDs.

Just finished reading this thread for the umteenth time, and....HUH???
(he sez, as he scratches his head and goes to re-read the KISS principle again... :lol: )


*******************************
The Devil`s always.....in the Details!

Let's put some real numbers in :)
MovieStacker Video size = 699,875.78KB
MovieStacker Movie time is 138 minutes.
MovieStacker Average bitrate is 683Kbps.
The .m2v sample size is 31,555KB.
Average bitrate of .m2v is 4,206Kbps.
So then KVCDFactor = (699875.78/138/683) / (31555/4206)
So KVCDFactor is 0.9897
This is our multiplying compressibility ratio of source to our encoder. So if you use Sample size = MovieStacker's Average Bitrate * KVCDFactor, your result would be 675.9651 which is not correct :)

-kwag

Ok, just looked at my first encode result on K19. Final file size is 722,875KB, and wanted size by MovieStacker was 699,875.78KB. 3.18% difference
Not bad for first try. My wanted sample size from the formula was 5,071KB, and my actual size with the CQ I encoded was 5,106KB
So my sample was about 0.7% over the size of wanted. So if we apply a negative 0.7% to the final size of 722,875KB, the result should have been 717,814.875KB :!:
So it really means that the sampler must be as close to the wanted sampler size, and on the "under" side for insurance. Remember that we are not estimating CQ anymore by sampling an encode, we are actually using the source size/bitrate ratio to our advantage and applying the ratio to our encoder. This should be by far better that any sampling encode to find out CQ :!:
I'm on my second encode now (The Boondock Saints). ~4 hours to go :cool:
I'm just checking to see if the results are consistent.

-kwag

It would, but the percent is for the total size of the movie, and we need only a small portion. That's the reason for demuxing the small clip, to extract only the video portion, so that the average bitrate/file size ratio is correct :)

-kwag

Hi kwag,

What I'm saying is that if you just did Moviestacker Video Size/ MovieStacker Movie Time you would get exactly the same sample size as your formula (barring rounding differences).

Your original KVCDFactor formula was
KVCDFactor = (MovieStacker's Elementary Video Stream / Total Minutes of Movie / MovieStacker's Average Bitrate) / ( Demuxed .m2v file size / Average Bitrate of demuxed.m2v)

But you can take away the bit for ( Demuxed .m2v file size / Average Bitrate of demuxed.m2v) because it is also in the sample formula as a multiplication. They cancel each other out.

Therefore both formulas are amended as follows using the process of negation.

In your example with real numbers you are still doing ( Demuxed .m2v file size / Average Bitrate of demuxed.m2v) in your KVCDFactor formula but you can see the results are the same!

e.g.

KVCDFactor = (699875.78/138/683) = 7.4254225
Sample = 683 * 7.4254225 = 5071.56Mb

You will see that this is the same as your original formula:
KVCDFactor = (699875.78/138/683) / (31555/4206) = 0.9897
Sample = (31555/4206) * 683 * 0.9897 = 5071.34 (difference due to rounding)

I hope you can see you are multiplying (31555/4206) with the KVCDFactor, which you can remove from both formulas as they cancel each other out.

My last point was that if you use the process of negation further you can also remove the MovieStacker Average bitrate from the formulas as well because they are cancelling each other out in the same way.

I hope this makes sense and if I'm completely mistake then accept my apologies :oops:

Hi Fiz,

Yes, you're right, and I'm too tired 8O :)
That's why I previously said "Anyone care to revise this" :lol:
Now I need some rest :!:
The idea is to apply the source to target ratio correctly.
Maybe we can get this right :cool:

Thanks,
-kwag

Hi kwag,

You deserve a rest, you've been working flat out and it's much appreciated. :) Sometimes we can get so enthusiastic about an idea that we overlook things, especially when tiredness kicks in 8O

You're right I think this concept has potential and I will look to see if I can do some testing as well when I get home from work.

Cheers,

Fiz.

@ Fiz and Kwag:

please,post how to do it using numbers as samples
like Kwag posted..."some real numbers"...

can you please?
:D

So maybe after refinement, the formula simplifies to:
CompressionFactor = (MovieStacker's Elementary Video Stream / Total Minutes of Movie / MovieStacker's Average Bitrate) / ( Demuxed .m2v file size / Average Bitrate of demuxed.m2v)

And finally:

Sample size = (MovieStacker's Elementary Video Stream / Total Minutes of Movie ) * CompressionFactor :idea:

Edit: I believe this is now correct :!:

-kwag

This is good. This is really good :mrgreen:
Finished "The Boondock Saints".
MovieStacker wanted file size: 722,779.89KB
Encoded file size: 725,451KB

I encoded with a sampler of 6,534KB, and the required size with the corrected formula (previous post) should have been 6,494KB. The difference would have been about positive 0.7%. So if I apply a minus 0.7% to 725,451KB, the result is 720,372.843KB for an accuracy of 0.33% against the wanted size of 722,779.89KB :mrgreen:
Now on to encode "The Bourne Identity" :cool:

-kwag

Encoding "The Bourne Identity" with CQ=55.75.
Wanted sample size = 5975.24KB
Sample size with CQ of 55.75 is 5,938KB
Results in ~6 hours :? ( Got to get a faster processor :!: )

-kwag

Hi Kwag,

Good work :D I'm trying the new formula on the movie signs.

Here are my formula values


MovieStacker Video Size = 717784.65
Total Time of Movie = 102
MovieStacker Average Bitrate = 947

Demuxed .m2v Size = 33945
Demuxed .m2v Average Bit rate = 4596.


I get

Compression Factor = (717784.65 / 102 / 947) / (33945 / 4596) = 1.00611
Therefore my sample should be (717784.65 / 102) * 1.0061 = 7080

Question: Is a compression factor over 1 going to affect anything? Does it mean it's not so compressible.

I'm doing the sample encodes to get the closest CQ for 7,080Kb and will let you know.

Fiz.

Edit:
OK heres the spooky thing. TOK tells me the CQ should be 70.235 for this movie. But with this new prediction I get a CQ of 80.5 for my 7,080kb 1 minute target. :!:

I'm just wondering if it's because we take this sample from the start of the movie and for Signs, the start credits are dark.

I'm going to let the full encode run at cq 80.5 overnight (4.5 hours to encode) and will post my result tomorrow.

Hi Fiz,
Your calculations are correct :)
Try to get the closest value , but "under" sample size. Just to leave some insurance margin. It seems that in your case, the compressibility of your movie is not the same as the ones I'm trying, and that's why the factor is positive. So it will be a good test to see what your final size will be :!:
Let me know what CQ you got and how close to sample size. Then when you do your full encode, let's see how far off it's the file from the wanted size in MovieStacker. If you get <1% accuracy, I think we have a winner, and this will supercede all other file prediction methods :mrgreen:
I'm already thinking on how to automate the cutting process 8)

-kwag