Quantcast Crazy Idea Just Poped into my Mind! - digitalFAQ.com Forums [Archives]
  #1  
03-16-2004, 03:32 PM
kwag kwag is offline
Free Member
 
Join Date: Apr 2002
Location: Puerto Rico, USA
Posts: 13,537
Thanks: 0
Thanked 0 Times in 0 Posts
I just had a crazy
We all know that file prediction is not very accurate, because every movie has different content at different parts of the movie.
Well, while I was just testing DIKumciser, this occured to me:
Take all VOBs of a movie, and join them.
After that run DIKumsizer on the VOB, and cut it in at least 400 pieces.
That's an easy task, because it's automatic. Just divide the size of the total VOBs by 400, and uses that size to cut.
Now here's the good part, which will obviously need a program to do it automatically, but it can be done. ( DIKumpaster )
Join all 400 parts again, but randomly
Now you'll have a VOB that you can do prediction on, because it will contain VERY random clips from all the movie, making prediction extremely accurate.
What we are doing is changing the order of ALL the movie, so even if it's a movie that half of it is drama, and the second half is action, when we join all parts in a random fashion, we are actually making a VOB that is well distributed.
After doing prediction and finding the correct CQ value on the joined VOB, we discard the VOB and proceed to encode the actual VOBs with the found CQ.
This way, we've "normalized" the file, and our predictions should be way up in the 99% precision

-kwag
Reply With Quote
Someday, 12:01 PM
admin's Avatar
Site Staff / Ad Manager
 
Join Date: Dec 2002
Posts: 42
Thanks: ∞
Thanked 42 Times in 42 Posts
  #2  
03-16-2004, 03:47 PM
Critter Critter is offline
Free Member
 
Join Date: Jun 2003
Posts: 36
Thanks: 0
Thanked 0 Times in 0 Posts
I dont understand anything about these predicting methods, but is it possible to some sort of avisynth script jump around vob randomly to archieve same effect ? That would save a lot of space
Reply With Quote
  #3  
03-16-2004, 03:52 PM
kwag kwag is offline
Free Member
 
Join Date: Apr 2002
Location: Puerto Rico, USA
Posts: 13,537
Thanks: 0
Thanked 0 Times in 0 Posts
Yes, but I'm not sure if a random variable can be assigned in a avs script, to pick out random segments.
Actually, a better way, and also simple like what you just described, would be to "create" a file of 1% of the total of the movie, with a modified "Dikumciser" program, that would pick out random points of the original VOBs, and this 1% file will be already a random mpeg from the complete movie.
Then it's a matter of encoding the file, until the final target size is wanted (1/100 of the total size), with the found CQ.
As you said, this will save a lot of space and time

-kwag
Reply With Quote
  #4  
03-16-2004, 04:09 PM
incredible incredible is offline
Free Member
 
Join Date: May 2003
Location: Germany
Posts: 3,189
Thanks: 0
Thanked 0 Times in 0 Posts
Send a message via ICQ to incredible
What you intend to do could be done also by avisynth, ...

cutting the whole stream into slices, .... reorder these slices randomly and again slicing that random puzzled stream for prediction
Reply With Quote
  #5  
03-16-2004, 04:24 PM
rds_correia rds_correia is offline
Free Member
 
Join Date: Apr 2003
Location: Chinese Democracy starts now!
Posts: 2,563
Thanks: 1
Thanked 0 Times in 0 Posts
Hmm,
Is it just me or is Incredible already warming up for a nice evening of scripting
Now what would he call that new function
He already has a function named Slicer...
If I know Incredible I think that the hard part of it will be to name the new "baby"
C ya
__________________
Rui
Reply With Quote
  #6  
03-16-2004, 05:20 PM
kwag kwag is offline
Free Member
 
Join Date: Apr 2002
Location: Puerto Rico, USA
Posts: 13,537
Thanks: 0
Thanked 0 Times in 0 Posts
Quote:
Originally Posted by incredible
What you intend to do could be done also by avisynth, ...

cutting the whole stream into slices, .... reorder these slices randomly and again slicing that random puzzled stream for prediction
Yes, and I see there's random function in AviSynth, but I'm pretty sure this is going to be painly slow, doing it in a script.
It will be extremely fast to do it with an external program, because of the fast binary seeking functions being done.

Just figure it like this:

(1) Run X program to creata the random 1% VOB file from all VOBs.
(2) Create a .d2v for that file.
(3) Encode that 1% until you get 1/100 of wanted final size.
(4) Encode your movie with the CQ found on step 3.

If I can do step 1 with a modified DIKumciser, there's no AviSynth script that will ever be as fast as that. So the 1% file will be produced in a matter of seconds.
Same for step 2, because it's a very small file.
Step 3 is a linear encode, because we don't need to do any jumping, or selecting areas anymore, because the clip already contains all random parts.
Most of this could be automated, as a matter of fact, with CQMatic

-kwag
Reply With Quote
  #7  
03-16-2004, 05:35 PM
Dialhot Dialhot is offline
Free Member
 
Join Date: May 2003
Posts: 10,463
Thanks: 0
Thanked 0 Times in 0 Posts
For me that's idea can't work.
What will surely happen is that you will probably predict a CQ to low. Why ? Because your randomizing in the sequences will create movement and scene changing where there actually aren't.

Let's imagine a sequence 10 minutes long with slow camera movement and a a scene change every minutes. You cut it in 20 part of 30 seconds each and you randomize them. You will create a 10 minute sequence of slow movement camera but with a scene change every 30s !

I really don't see how this method could lead to find a better CQ.
Reply With Quote
  #8  
03-16-2004, 05:50 PM
kwag kwag is offline
Free Member
 
Join Date: Apr 2002
Location: Puerto Rico, USA
Posts: 13,537
Thanks: 0
Thanked 0 Times in 0 Posts
I see your point Phil, unless scene change detection is disabled for both prediction and final encode.
I'm going to do a little test by making a new program from DIKumciser, that will take just one VOB, and create a small 1-2% file out of it.
Then I'll run prediction on it, and encode, and see if the final size matched a target.
What I'm trying to do, as I explained, is to put the maximum amount of footage and cram it on a small percent, so that this small piece matches real average of the activity of the complete movie.
One more thing. No AviSYnth script can cut (sample) at correct sequence headers. Only at specified frames. But I can cut at exact sequence headers, and that should make a hell of a difference for prediction

-kwag
Reply With Quote
  #9  
03-17-2004, 12:29 PM
vhelp vhelp is offline
Free Member
 
Join Date: Jan 2003
Posts: 1,009
Thanks: 0
Thanked 0 Times in 0 Posts
@ kvcd,

can you create an .AVS script that carries out an "index" of these so
called "sequence headers" somehow ??
.
.
Maybe I'm thinking a "reference table" or something like that - I don't
know, but if it helps in the pile of ideas and stuff, think about it a little then.

But, in a way, I'm also w/ Phil on this too. But, I'm open to your new ideas..
as always.. Keep them up too

-vhelp
Reply With Quote
  #10  
04-07-2004, 01:36 PM
black prince black prince is offline
Free Member
 
Join Date: Jul 2002
Posts: 1,224
Thanks: 0
Thanked 0 Times in 0 Posts
@Kwag,

Kwag wrote:
Quote:
Yes, but I'm not sure if a random variable can be assigned in a avs script, to pick out random segments.
Actually, a better way, and also simple like what you just described, would be to "create" a file of 1% of the total of the movie, with a modified "Dikumciser" program, that would pick out random points of the original VOBs, and this 1% file will be already a random mpeg from the complete movie.
Then it's a matter of encoding the file, until the final target size is wanted (1/100 of the total size), with the found CQ.
As you said, this will save a lot of space and time
Ocassionally, A file size prediction is perfect using the manual process
and I wanted to compare the sample test file in terms of I,P, and
B frame types to the original DVD. This may be another crazy idea,
but the I, P, and B frame types seemed to be proportational to the
DVD. For example, if the DVD has a total number of frames of say
200,000 and "I" frame type is 70,000 or 35%, "P" type is 100,000 or
50% and "B" type is 30,000 or 15%, as a makeup. My test file of 1%
(2,000 frames) of the DVD would have "I" as 700 or 35%, "P" as
1,000 or 50% and "B" as 300 or 15%. Apply and average bitrate
to the test file for each frame and test for CQ using CQMatic it should
be more accurate. Even a test of 0.01% of the DVD using the same
percentages as the DVD should be accurate. IfoEdit can create a file
of all frame numbers and frame type. Using a batch command to find
total count for each "I", "P" and "B". The goal is to create a more
closer sample test file as the original. What do you think about this
idea and if it's crazy then I'll join the club

-BP
Reply With Quote
  #11  
04-07-2004, 02:47 PM
rendalunit rendalunit is offline
Free Member
 
Join Date: Apr 2002
Location: san jose, Ca
Posts: 1,148
Thanks: 0
Thanked 0 Times in 0 Posts
It does sound crazy! I'll be honest with you though and tell you I've been using cqtester since November '03 and the filesize prediction has been very accurate for my dvd encodes
Reply With Quote
  #12  
04-08-2004, 04:03 PM
black prince black prince is offline
Free Member
 
Join Date: Jul 2002
Posts: 1,224
Thanks: 0
Thanked 0 Times in 0 Posts
@ren,

Tried CQTester and it's prediction was closer than CQMatic. I have a
question. What are your Tmpgenc settings for Min, Max bitrate or
do you use the default from the KVCD template of max=1800 and
min=300. CQTester does some curious things, such as, the sample
frames changed for each re-test even though the number of sample
frames was consistant. CQTester calls for Audio size in MB. Do you
use Headac3he to project the audio size? If you have any special
tweaks for CQTester, please share

Thanks

-BP
Reply With Quote
  #13  
04-09-2004, 11:38 AM
rendalunit rendalunit is offline
Free Member
 
Join Date: Apr 2002
Location: san jose, Ca
Posts: 1,148
Thanks: 0
Thanked 0 Times in 0 Posts
hey bp,

I just use the stock min/max settings in the templates. CQtester calculates the mb for 128 kbps audio so I leave that or I change it to the value gotten by Moviestacker. It also adds padding (mb multiplex) and i change that to 0. I don't really understand the settings in cqtester (offset, etc..) so I just leave all the default settings and it works for me. However, I think Incredible did explain its' usage in the other thread.

ren
Reply With Quote
  #14  
04-09-2004, 02:18 PM
black prince black prince is offline
Free Member
 
Join Date: Jul 2002
Posts: 1,224
Thanks: 0
Thanked 0 Times in 0 Posts
@ren,

Thanks for info. I'll check incredible's post.

-BP
Reply With Quote
  #15  
04-09-2004, 04:27 PM
rendalunit rendalunit is offline
Free Member
 
Join Date: Apr 2002
Location: san jose, Ca
Posts: 1,148
Thanks: 0
Thanked 0 Times in 0 Posts
hey bp,

Incredibles post is in the "cqtester" thread and also check out the instructions by cqtester's author here:

http://www.boraxsoft.de/CQTester_eng.shtml

ren
Reply With Quote
  #16  
04-09-2004, 06:39 PM
kwag kwag is offline
Free Member
 
Join Date: Apr 2002
Location: Puerto Rico, USA
Posts: 13,537
Thanks: 0
Thanked 0 Times in 0 Posts
Well, here it is.
You guys tease me, so here you go

http://www.kvcd.net/downloads/CQMatic-1.3.00.exe

Changes:

Prediction slices are now random , sort of what I proposed at the start of this thread
That is, instead of linear samples, the samples are chosen randomly from the complete movie.
When you start prediction, you'll see that the encoding sequence doesn't follow the cronologic order of the movie.

Note: THIS IS EXPERIMENTAL

Hint: If you want to increase accuracy, run CQMatic once, in prediction only mode.
Then reload the .tpr project (this will re-initialize a new random set), and run prediction once again. Note the final CQ value. You may average both resulting values, to achieve a higher accuracy.
Or just simply run in X3 mode.

Let me know your results, PLEEEEESE

-kwag
Reply With Quote
  #17  
04-09-2004, 07:44 PM
kwag kwag is offline
Free Member
 
Join Date: Apr 2002
Location: Puerto Rico, USA
Posts: 13,537
Thanks: 0
Thanked 0 Times in 0 Posts
Quote:
Originally Posted by Critter
but is it possible to some sort of avisynth script jump around vob randomly to archieve same effect ? That would save a lot of space
That's exactly what I did, but without the need to use AviSynth

-kwag
Reply With Quote
  #18  
04-10-2004, 02:29 PM
black prince black prince is offline
Free Member
 
Join Date: Jul 2002
Posts: 1,224
Thanks: 0
Thanked 0 Times in 0 Posts
@Kwag,

Kwag wrote:
Quote:
Well, here it is.
You guys tease me, so here you go
Testing now and will report tommorrow (Sunday) with results.
I still believe my theory of keeping I, P, and B frames
the same percentage in sample test file the same for entire
movie. I will test CQTester vs CQMatic and report

-BP
Reply With Quote
  #19  
04-10-2004, 02:58 PM
kwag kwag is offline
Free Member
 
Join Date: Apr 2002
Location: Puerto Rico, USA
Posts: 13,537
Thanks: 0
Thanked 0 Times in 0 Posts
Quote:
Originally Posted by black prince
@Kwag,

Kwag wrote:
Quote:
Well, here it is.
You guys tease me, so here you go
Testing now and will report tommorrow (Sunday) with results.
I still believe my theory of keeping I, P, and B frames
the same percentage in sample test file the same for entire
movie. I will test CQTester vs CQMatic and report

-BP
Test with this better:
http://www.kvcd.net/downloads/CQMatic-1.3.01.exe
Note: Use X1 mode only, and run at least TWO full predictions to get average CQ.
After the first run, reload your project file so that the random frames are reinitialized.

-kwag
Reply With Quote
  #20  
04-10-2004, 11:34 PM
black prince black prince is offline
Free Member
 
Join Date: Jul 2002
Posts: 1,224
Thanks: 0
Thanked 0 Times in 0 Posts
@Kwag,

CQMatic vs CQTester

General Information:

Movie: The Transporter (Action)
Time : 92 minutes
Type: DVD Full Screen (4:3 NTSCFilm)

Common to both methods

Avisynth script: Motion Adapative
Tmpgenc Template: KVCD 352X240 Plus NTSCFilm
No CD-R’s: 1 80 minute CD-R (700mb)
Desired Audio File size: 86MB @ 128 kb
Desired Video File size: 709MB
Total A/V file size: 795MB

Results of video file size prediction

CQTester v0.92

Max Bitrate: 1800
Min Bitrate: 300
CQ prediction: 80.60

CQMatic v1.301

Average Bitrate: 1058 (CalcuMatic)
Max Bitrate: 2000
Min Bitrate: 603 (CalcuMatic)
CQ Prediction: 79.86

Comments:
Both results were close. CQMatic v1.301 is experimental and showing more
promise. I will have actual file sizes after full encode (about 8 to 10 hours).
They both may be off from the desirable video file size of 709MB. This
challenge certainly is creating improvements. Testing 3+ hour
movies will tell even more about accuracy.

-BP
Reply With Quote
Reply




Similar Threads
Thread Thread Starter Forum Replies Last Post
Crazy frog Prodater64 Off-topic Lounge 6 08-13-2005 12:50 PM
Some crazy ideas! Prodater64 Off-topic Lounge 4 04-05-2004 05:48 AM
Set of 4 Mencoder tests: mind loaning us some CPU cycles? bilu Video Encoding and Conversion 38 04-04-2004 02:27 PM
video goes crazy slow and is doubled in size to 640x480? Gamecraze Video Encoding and Conversion 7 11-04-2003 03:00 PM
Crazy Idea for better quality #2... Jellygoose Avisynth Scripting 14 01-08-2003 11:06 PM

Thread Tools



 
All times are GMT -5. The time now is 03:07 PM  —  vBulletin Jelsoft Enterprises Ltd