Crazy idea just poped into my mind!
 

I just had a crazy :idea: :D
We all know that file prediction is not very accurate, because every movie has different content at different parts of the movie.
Well, while I was just testing DIKumciser, this occured to me:
Take all VOBs of a movie, and join them.
After that run DIKumsizer on the VOB, and cut it in at least 400 pieces.
That's an easy task, because it's automatic. Just divide the size of the total VOBs by 400, and uses that size to cut.
Now here's the good part, which will obviously need a program to do it automatically, but it can be done. ( :idea: DIKumpaster :mrgreen: )
Join all 400 parts again, but randomly :!: :D
Now you'll have a VOB that you can do prediction on, because it will contain VERY random clips from all the movie, making prediction extremely accurate.
What we are doing is changing the order of ALL the movie, so even if it's a movie that half of it is drama, and the second half is action, when we join all parts in a random fashion, we are actually making a VOB that is well distributed.
After doing prediction and finding the correct CQ value on the joined VOB, we discard the VOB and proceed to encode the actual VOBs with the found CQ.
This way, we've "normalized" the file, and our predictions should be way up in the 99% precision ;)

-kwag

I dont understand anything about these predicting methods, but is it possible to some sort of avisynth script jump around vob randomly to archieve same effect ? That would save a lot of space :)

Yes, but I'm not sure if a random variable can be assigned in a avs script, to pick out random segments.
Actually, a better way, and also simple like what you just described, would be to "create" a file of 1% of the total of the movie, with a modified "Dikumciser" program, that would pick out random points of the original VOBs, and this 1% file will be already a random mpeg from the complete movie.
Then it's a matter of encoding the file, until the final target size is wanted (1/100 of the total size), with the found CQ.
As you said, this will save a lot of space and time :)

-kwag

What you intend to do could be done also by avisynth, ...

cutting the whole stream into slices, .... reorder these slices randomly and again slicing that random puzzled stream for prediction :idea: :?:

Hmm,
Is it just me or is Incredible already warming up for a nice evening of scripting :?: :lol:
Now what would he call that new function :?:
He already has a function named Slicer...
If I know Incredible I think that the hard part of it will be to name the new "baby" 8)
C ya

Yes, and I see there's random function in AviSynth, but I'm pretty sure this is going to be painly slow, doing it in a script.
It will be extremely fast to do it with an external program, because of the fast binary seeking functions being done.

Just figure it like this:

(1) Run X program to creata the random 1% VOB file from all VOBs.
(2) Create a .d2v for that file.
(3) Encode that 1% until you get 1/100 of wanted final size.
(4) Encode your movie with the CQ found on step 3.

If I can do step 1 with a modified DIKumciser, there's no AviSynth script that will ever be as fast as that. So the 1% file will be produced in a matter of seconds.
Same for step 2, because it's a very small file.
Step 3 is a linear encode, because we don't need to do any jumping, or selecting areas anymore, because the clip already contains all random parts.
Most of this could be automated, as a matter of fact, with CQMatic :)

-kwag

For me that's idea can't work.
What will surely happen is that you will probably predict a CQ to low. Why ? Because your randomizing in the sequences will create movement and scene changing where there actually aren't.

Let's imagine a sequence 10 minutes long with slow camera movement and a a scene change every minutes. You cut it in 20 part of 30 seconds each and you randomize them. You will create a 10 minute sequence of slow movement camera but with a scene change every 30s !

I really don't see how this method could lead to find a better CQ.

I see your point Phil, unless scene change detection is disabled for both prediction and final encode.
I'm going to do a little test by making a new program from DIKumciser, that will take just one VOB, and create a small 1-2% file out of it.
Then I'll run prediction on it, and encode, and see if the final size matched a target.
What I'm trying to do, as I explained, is to put the maximum amount of footage and cram it on a small percent, so that this small piece matches real average of the activity of the complete movie.
One more thing. No AviSYnth script can cut (sample) at correct sequence headers. Only at specified frames. But I can cut at exact sequence headers, and that should make a hell of a difference for prediction ;)

-kwag

@ kvcd,

:idea: can you create an .AVS script that carries out an "index" of these so
called "sequence headers" somehow ??
.
.
Maybe I'm thinking :roll: a "reference table" or something like that - I don't
know, but if it helps in the pile of ideas and stuff, think about it a little then.

But, in a way, I'm also w/ Phil on this too. But, I'm open to your new ideas..
as always.. Keep them up too 8)

-vhelp

@Kwag,

Kwag wrote:
Ocassionally, A file size prediction is perfect using the manual process
and I wanted to compare the sample test file in terms of I,P, and
B frame types to the original DVD. This may be another crazy idea,
but the I, P, and B frame types seemed to be proportational to the
DVD. For example, if the DVD has a total number of frames of say
200,000 and "I" frame type is 70,000 or 35%, "P" type is 100,000 or
50% and "B" type is 30,000 or 15%, as a makeup. My test file of 1%
(2,000 frames) of the DVD would have "I" as 700 or 35%, "P" as
1,000 or 50% and "B" as 300 or 15%. Apply and average bitrate
to the test file for each frame and test for CQ using CQMatic it should
be more accurate. Even a test of 0.01% of the DVD using the same
percentages as the DVD should be accurate. IfoEdit can create a file
of all frame numbers and frame type. Using a batch command to find
total count for each "I", "P" and "B". :) The goal is to create a more
closer sample test file as the original. What do you think about this
idea and if it's crazy then I'll join the club :)

-BP

It does sound crazy! I'll be honest with you though and tell you I've been using cqtester since November '03 and the filesize prediction has been very accurate for my dvd encodes :D

@ren,

Tried CQTester and it's prediction was closer than CQMatic. I have a
question. What are your Tmpgenc settings for Min, Max bitrate or
do you use the default from the KVCD template of max=1800 and
min=300. CQTester does some curious things, such as, the sample
frames changed for each re-test even though the number of sample
frames was consistant. CQTester calls for Audio size in MB. Do you
use Headac3he to project the audio size? If you have any special
tweaks for CQTester, please share :D

Thanks

-BP

hey bp,

I just use the stock min/max settings in the templates. CQtester calculates the mb for 128 kbps audio so I leave that or I change it to the value gotten by Moviestacker. It also adds padding (mb multiplex) and i change that to 0. I don't really understand the settings in cqtester (offset, etc..) so I just leave all the default settings and it works for me. However, I think Incredible did explain its' usage in the other thread.

ren

@ren,

Thanks for info. I'll check incredible's post. :)

-BP

hey bp,

Incredibles post is in the "cqtester" thread and also check out the instructions by cqtester's author here:

http://www.boraxsoft.de/CQTester_eng.shtml

ren

Well, here it is.
You guys tease me, so here you go :twisted: :lol:

http://www.kvcd.net/downloads/CQMatic-1.3.00.exe

Changes:

Prediction slices are now random :!:, sort of what I proposed at the start of this thread ;)
That is, instead of linear samples, the samples are chosen randomly from the complete movie.
When you start prediction, you'll see that the encoding sequence doesn't follow the cronologic order of the movie.

Note: THIS IS EXPERIMENTAL :!:

Hint: If you want to increase accuracy, run CQMatic once, in prediction only mode.
Then reload the .tpr project (this will re-initialize a new random set), and run prediction once again. Note the final CQ value. You may average both resulting values, to achieve a higher accuracy.
Or just simply run in X3 mode.

Let me know your results, PLEEEEESE :roll:

-kwag

That's exactly what I did, but without the need to use AviSynth ;) :D

-kwag

@Kwag,

Kwag wrote:
Testing now and will report tommorrow (Sunday) with results.
I still believe my theory of keeping I, P, and B frames
the same percentage in sample test file the same for entire
movie. :) I will test CQTester vs CQMatic and report :wink:

-BP

Test with this better:
http://www.kvcd.net/downloads/CQMatic-1.3.01.exe
Note: Use X1 mode only, and run at least TWO full predictions to get average CQ.
After the first run, reload your project file so that the random frames are reinitialized.

-kwag

@Kwag,

CQMatic vs CQTester

General Information:

Movie: The Transporter (Action)
Time : 92 minutes
Type: DVD Full Screen (4:3 NTSCFilm)

Common to both methods

Avisynth script: Motion Adapative
Tmpgenc Template: KVCD 352X240 Plus NTSCFilm
No CD-R’s: 1 80 minute CD-R (700mb)
Desired Audio File size: 86MB @ 128 kb
Desired Video File size: 709MB
Total A/V file size: 795MB

Results of video file size prediction

CQTester v0.92

Max Bitrate: 1800
Min Bitrate: 300
CQ prediction: 80.60

CQMatic v1.301

Average Bitrate: 1058 (CalcuMatic)
Max Bitrate: 2000
Min Bitrate: 603 (CalcuMatic)
CQ Prediction: 79.86

Comments:
Both results were close. CQMatic v1.301 is experimental and showing more
promise. I will have actual file sizes after full encode (about 8 to 10 hours).
They both may be off from the desirable video file size of 709MB. This
challenge certainly is creating improvements. Testing 3+ hour
movies will tell even more about accuracy.

-BP

Update for CQMatic vs CQTester

Desired video file size: 709MB

CQTester final video file size: 677MB

CQMatic is still encoding and expected to be done in 6 hours. :)

-BP

Hi bp,

Thanks for the tests :)
Please keep those figures handy, so you can compare when I release the latest version, which I'm currently testing ( 1.3.02 )
I have one more movie to encode, to confirm accuracy.
So far, I hit 0.661% accuracy 8O on my first try ( Movie: Point of no return )
Wanted: 725,722KB
Final: 720,919

Coincidence :?:
Maybe, maybe not :!:
If I get the same result with the movie I'm currently encoding (Payback), then I'll post this new version, which will be 1.3.02.
If not, then ignore this post, as I'll be back to the drawing board :x

-kwag

Kwag wrote:
This is the Kwag we know and love. Never giving up and always
searching for answers. You brought us KVCD templates, Notch-
Matrix, Motion Adapative avs script and more. You and others at
this forum have raised the level of VCD backups. I hope you never
lose the desire to find answers. :)

-BP

I never have, I never will :D
But the answer is clearly not TMPEG (at least anymore!)
I think you'll see a big storm and a lot of brainstorming in the forum, the following days, related to MEncoder ;)
I'm sure many good things will show up :)

Thanks,
-kwag

@All,

Played with MEncoderME GUI for a week now and it needs more time
to mature. The 2 pass hangs up on second pass, the script needs
changing to create m1v KVCD (352X240). It's fast, but when I
changed the script to encode mpeg-1 the picture quality is now
really bad (i.e. very blocky, gibbs effect or chrona around objects, etc).
I went back to CQTester and Tmpgenc for now. I finally understand
how to tweak CQTester and my video file sizes are almost perfect
after encoding five movies. It's slower then MEncode, but I can use
more of the CD for video after calculating audio with Headac3he. :)

-BP

@ BP

Can you post or write a little guide on the optimal settings for cq tester,i would like to try it.


Thanx

@bigggt,

Here's a quick step by step for 352x240 for 1 CDR using cqtester:

1) DVD Decrypter (movie only)

2) DVD2AVI v 1.77.3 (check for force film) create .d2v and .ac3 files

3) Headac3he - create .mp2 with you desired audio bitarte (keep
track of file size)

4) Moviestacker - select resolution and overscan for avs script

5) CalcuMatic - select .d2v and keep track of min bitrate and avg bitrate
make note of predicted video and audio file sizes to compare with
final files.

6) Tmpgenc - select avs script and kvcd template
a) set max bitrate 2000, min bitrate (from calcumatic)
b) open source in adv tab and using cut/edit select the whole movie
c) go to file save to "tpr text file" for .tpr
d) close Tmpgenc

7) CQTester -
a) select .tpr file from Tmpgenc
b) set no. cd's to 1
c) set MB pro CD to 800
d) set Audio MB to .mp2 file size from headac3he
e) set mulitplex offset to 0
f) click on "Opt + Enc" button

8 ) wait until CQTester has finished and check file size created for .m1v

9) Use "Mplex" to mutliplex audio + video

10) Burn mpg file from mutliplex

Note:
My results for actual video file size were 2% to 3% less than predicted
for five movies I tested with various lengths of 90min up to 180min.

-BP

I agree bp as much as i love the MEncoderME GUI I have a lot of PAL DVD Source and interlaced and the oput of Mencoder in KDVD mode is very blocky and not as sharp as say CCE. Having said that it still early days with MEncoder and im sure it will improve with time.

@Kwag

Hi Karl one thing I would like to know about CQMatic is that it has a radio button for CCE but it never was implemented and cant seem to be selected.

Will CQMatic ever be able to use CCE what with the trouble TMPGEncoder is giving these days, I know CCE is only good for SKVCD and KDVD but I would love to able use this tool for calculating purposes as I intend backing up a lot of my DVD to KDVD and use CCE and it would be nice to have a calculator to work out the necessary figures. :)

my first doubt BP:

forced film in all situations?
:?
thanks!
:wink:

Well, the thing is that there are so many other tools that already work with CCE, that I really don't want to re-invent the wheel :!:
You can use CalcuMatic for that matter, and just feed in the values to CCE manually.
Anyway, I can clearly see that we are all slowly moving towards MEncoder/ffmpeg, so I think that in a short time, we'll be using tools like vmensquitas "MencodeME", QuENC, and who knows what other tools that will show up, supporting MEncoder.
But to answer your question, no, that button will stay grayed out. Maybe it will be turned on again in the future, but it won't be for CCE ;)

-kwag

@jorel,

"Check for force film" means determine if you need to use force
film, not use in all cases. I created this quick and dirty step by
step process mainly for experienced members who were curious
about how to fit in and use CQTester. These settings work good for me
and I wanted to share :)

-BP

Thanks for the info Karl its much appreciated. :)

I like the sound of QueEnc ive heard so many positive things about that and and it looks like its coming along nicely.

I do love Vemesquita's MEncoderME but I have issues with that at the moment.

I have PAL DVD Sources and they are interlaced and when I try to make a KDVD or KVCD the results are aweful.

I dont have any webspace to upload a sample to but if I did I could show you a sample. Maybe I can post some screenshots to show you what I mean.

In the meantime I usually use CCE for my KDVD as for me at present it produces the best quality.

Sorry I know this is off-topic. :oops:

i see what you mean BP! (my poor english misunderstand it)
thanks!
:wink:

ps:
of course, this way seems cool!
:D

Thanx BP for the guide,simple to understand(exactly what i like),i will try and see if i can get it to work :D

Although i say its simple doesn't mean i won't have problems :roll:

Thanx again

Tom

edit-here is my first problem when saving tpr project do you save as ES(video only)because when i do this it says an error about the audio and won,t allow me to select any of the options on cq tester like Test CQ or Opt CQ.

If i save it as audio and video it works but i can't choose the audio size

I just can't figure out how to get the audio part,i have my mp2 made up but can't seem to get this part

Thanx

@bigggt,

Did you remember to do this:

Tmpgenc --> Settings --> Advanced Tab --> Source Range -->
Cut Editing --> "cut except currently selected area".

This sets up a range for CQTester (the whole movie as above) to
begin selecting from. CQTester is creating video only
So set Tmpgenc for ES (video only). I use Headac3he to create
my audio file before this step and when the video is done (.m1v)
I use "MPlex" an audio-video multiplexer to finish the mpeg-1
file (.mpg) 8) For "MPlex" do a search on all forums in KVCD and
download this gem of a program :wink:

-BP

http://www.kvcd.net/downloads/mplex.exe :cool:

-kwag

Thanx BP and thanx Karl for the link

Yes BP i did thise settings many times but kept getting the error under project details NO audio and could not select anything other then run tmpge,settings and Quit

edit-just tried again and now its working :evil: ,must have done something wrong

Thanx again

As far as mplex is it better then bbmpeg.

@bigggt,

bigggt wrote:
If you look in the bottom right of the File tab where it says
Save Settings, click on this button to save your settings for
the next multiplex :wink:

In Settings Tab here's what I use for 1 CDR 80min 700mb

VCD ---> check
VBR ---> check
Write Program End Code ---> check
Pad VCD audio ---> check
Multiplex Audio ---> check
Force Mux Rate = 0
Max Size = 795

-BP

Its the same engine, based on the mjpegTools commandline compiling "mplex.exe" ... just a USerInterface added.

But youre right its more easy to get into its settings than if open BBmpeg

Thanx Guys,i did realize it was pretty much the same as bbmpeg and Iused the same settings just didn't know if there was a difference because i heard mplex mentioned in a few threads lately

Tom