Crazy idea just poped into my mind!
 

I just had a crazy :idea: :D
We all know that file prediction is not very accurate, because every movie has different content at different parts of the movie.
Well, while I was just testing DIKumciser, this occured to me:
Take all VOBs of a movie, and join them.
After that run DIKumsizer on the VOB, and cut it in at least 400 pieces.
That's an easy task, because it's automatic. Just divide the size of the total VOBs by 400, and uses that size to cut.
Now here's the good part, which will obviously need a program to do it automatically, but it can be done. ( :idea: DIKumpaster :mrgreen: )
Join all 400 parts again, but randomly :!: :D
Now you'll have a VOB that you can do prediction on, because it will contain VERY random clips from all the movie, making prediction extremely accurate.
What we are doing is changing the order of ALL the movie, so even if it's a movie that half of it is drama, and the second half is action, when we join all parts in a random fashion, we are actually making a VOB that is well distributed.
After doing prediction and finding the correct CQ value on the joined VOB, we discard the VOB and proceed to encode the actual VOBs with the found CQ.
This way, we've "normalized" the file, and our predictions should be way up in the 99% precision ;)

-kwag

I dont understand anything about these predicting methods, but is it possible to some sort of avisynth script jump around vob randomly to archieve same effect ? That would save a lot of space :)

Yes, but I'm not sure if a random variable can be assigned in a avs script, to pick out random segments.
Actually, a better way, and also simple like what you just described, would be to "create" a file of 1% of the total of the movie, with a modified "Dikumciser" program, that would pick out random points of the original VOBs, and this 1% file will be already a random mpeg from the complete movie.
Then it's a matter of encoding the file, until the final target size is wanted (1/100 of the total size), with the found CQ.
As you said, this will save a lot of space and time :)

-kwag

What you intend to do could be done also by avisynth, ...

cutting the whole stream into slices, .... reorder these slices randomly and again slicing that random puzzled stream for prediction :idea: :?:

Hmm,
Is it just me or is Incredible already warming up for a nice evening of scripting :?: :lol:
Now what would he call that new function :?:
He already has a function named Slicer...
If I know Incredible I think that the hard part of it will be to name the new "baby" 8)
C ya

Yes, and I see there's random function in AviSynth, but I'm pretty sure this is going to be painly slow, doing it in a script.
It will be extremely fast to do it with an external program, because of the fast binary seeking functions being done.

Just figure it like this:

(1) Run X program to creata the random 1% VOB file from all VOBs.
(2) Create a .d2v for that file.
(3) Encode that 1% until you get 1/100 of wanted final size.
(4) Encode your movie with the CQ found on step 3.

If I can do step 1 with a modified DIKumciser, there's no AviSynth script that will ever be as fast as that. So the 1% file will be produced in a matter of seconds.
Same for step 2, because it's a very small file.
Step 3 is a linear encode, because we don't need to do any jumping, or selecting areas anymore, because the clip already contains all random parts.
Most of this could be automated, as a matter of fact, with CQMatic :)

-kwag

For me that's idea can't work.
What will surely happen is that you will probably predict a CQ to low. Why ? Because your randomizing in the sequences will create movement and scene changing where there actually aren't.

Let's imagine a sequence 10 minutes long with slow camera movement and a a scene change every minutes. You cut it in 20 part of 30 seconds each and you randomize them. You will create a 10 minute sequence of slow movement camera but with a scene change every 30s !

I really don't see how this method could lead to find a better CQ.

I see your point Phil, unless scene change detection is disabled for both prediction and final encode.
I'm going to do a little test by making a new program from DIKumciser, that will take just one VOB, and create a small 1-2% file out of it.
Then I'll run prediction on it, and encode, and see if the final size matched a target.
What I'm trying to do, as I explained, is to put the maximum amount of footage and cram it on a small percent, so that this small piece matches real average of the activity of the complete movie.
One more thing. No AviSYnth script can cut (sample) at correct sequence headers. Only at specified frames. But I can cut at exact sequence headers, and that should make a hell of a difference for prediction ;)

-kwag

@ kvcd,

:idea: can you create an .AVS script that carries out an "index" of these so
called "sequence headers" somehow ??
.
.
Maybe I'm thinking :roll: a "reference table" or something like that - I don't
know, but if it helps in the pile of ideas and stuff, think about it a little then.

But, in a way, I'm also w/ Phil on this too. But, I'm open to your new ideas..
as always.. Keep them up too 8)

-vhelp

@Kwag,

Kwag wrote:
Ocassionally, A file size prediction is perfect using the manual process
and I wanted to compare the sample test file in terms of I,P, and
B frame types to the original DVD. This may be another crazy idea,
but the I, P, and B frame types seemed to be proportational to the
DVD. For example, if the DVD has a total number of frames of say
200,000 and "I" frame type is 70,000 or 35%, "P" type is 100,000 or
50% and "B" type is 30,000 or 15%, as a makeup. My test file of 1%
(2,000 frames) of the DVD would have "I" as 700 or 35%, "P" as
1,000 or 50% and "B" as 300 or 15%. Apply and average bitrate
to the test file for each frame and test for CQ using CQMatic it should
be more accurate. Even a test of 0.01% of the DVD using the same
percentages as the DVD should be accurate. IfoEdit can create a file
of all frame numbers and frame type. Using a batch command to find
total count for each "I", "P" and "B". :) The goal is to create a more
closer sample test file as the original. What do you think about this
idea and if it's crazy then I'll join the club :)

-BP

It does sound crazy! I'll be honest with you though and tell you I've been using cqtester since November '03 and the filesize prediction has been very accurate for my dvd encodes :D

@ren,

Tried CQTester and it's prediction was closer than CQMatic. I have a
question. What are your Tmpgenc settings for Min, Max bitrate or
do you use the default from the KVCD template of max=1800 and
min=300. CQTester does some curious things, such as, the sample
frames changed for each re-test even though the number of sample
frames was consistant. CQTester calls for Audio size in MB. Do you
use Headac3he to project the audio size? If you have any special
tweaks for CQTester, please share :D

Thanks

-BP

hey bp,

I just use the stock min/max settings in the templates. CQtester calculates the mb for 128 kbps audio so I leave that or I change it to the value gotten by Moviestacker. It also adds padding (mb multiplex) and i change that to 0. I don't really understand the settings in cqtester (offset, etc..) so I just leave all the default settings and it works for me. However, I think Incredible did explain its' usage in the other thread.

ren

@ren,

Thanks for info. I'll check incredible's post. :)

-BP

hey bp,

Incredibles post is in the "cqtester" thread and also check out the instructions by cqtester's author here:

http://www.boraxsoft.de/CQTester_eng.shtml

ren

Well, here it is.
You guys tease me, so here you go :twisted: :lol:

http://www.kvcd.net/downloads/CQMatic-1.3.00.exe

Changes:

Prediction slices are now random :!:, sort of what I proposed at the start of this thread ;)
That is, instead of linear samples, the samples are chosen randomly from the complete movie.
When you start prediction, you'll see that the encoding sequence doesn't follow the cronologic order of the movie.

Note: THIS IS EXPERIMENTAL :!:

Hint: If you want to increase accuracy, run CQMatic once, in prediction only mode.
Then reload the .tpr project (this will re-initialize a new random set), and run prediction once again. Note the final CQ value. You may average both resulting values, to achieve a higher accuracy.
Or just simply run in X3 mode.

Let me know your results, PLEEEEESE :roll:

-kwag

That's exactly what I did, but without the need to use AviSynth ;) :D

-kwag

@Kwag,

Kwag wrote:
Testing now and will report tommorrow (Sunday) with results.
I still believe my theory of keeping I, P, and B frames
the same percentage in sample test file the same for entire
movie. :) I will test CQTester vs CQMatic and report :wink:

-BP

Test with this better:
http://www.kvcd.net/downloads/CQMatic-1.3.01.exe
Note: Use X1 mode only, and run at least TWO full predictions to get average CQ.
After the first run, reload your project file so that the random frames are reinitialized.

-kwag

@Kwag,

CQMatic vs CQTester

General Information:

Movie: The Transporter (Action)
Time : 92 minutes
Type: DVD Full Screen (4:3 NTSCFilm)

Common to both methods

Avisynth script: Motion Adapative
Tmpgenc Template: KVCD 352X240 Plus NTSCFilm
No CD-R’s: 1 80 minute CD-R (700mb)
Desired Audio File size: 86MB @ 128 kb
Desired Video File size: 709MB
Total A/V file size: 795MB

Results of video file size prediction

CQTester v0.92

Max Bitrate: 1800
Min Bitrate: 300
CQ prediction: 80.60

CQMatic v1.301

Average Bitrate: 1058 (CalcuMatic)
Max Bitrate: 2000
Min Bitrate: 603 (CalcuMatic)
CQ Prediction: 79.86

Comments:
Both results were close. CQMatic v1.301 is experimental and showing more
promise. I will have actual file sizes after full encode (about 8 to 10 hours).
They both may be off from the desirable video file size of 709MB. This
challenge certainly is creating improvements. Testing 3+ hour
movies will tell even more about accuracy.

-BP