Experimental Revised prediction!
 

Here are the results and analisys on my K-Pax encode:

Wanted video file size: 714,303KB
Encoded video file size: 741,735KB

A difference of ~+3.69%

And this is GOOD :!:
Here's the pattern I see. On action movies, file size will be ~3% smaller MAX.
On low action movies (like K-Pax), file size will be ~3% larger MAX.
So normal "average" films should target ~100% accuracy :!:
There's something I did notice, and that is that just before the end credits, the file size was just at about 714MB, which was the wanted size. So the end credits is REALLY what's throwing off the calculations :)
I was watching TMPEG as it was encoding the end credits, and the file size was soaring high on the vertical scroll of the credits. "THAT'S OUR MAIN PROBLEM" The function "YDifferenceToNext()" reports a VERY LOW value on the end credits (verified with Vdub adding "nf" variable from script as subtitle), even though the scroll is a very high difference to next frame :!: But because it's mainly a black screen with text scrolling vertically, the returned value is very small, and we are loosing A LOT of compression there :!:
So my suggestions are as follow:

(1) Cut off end credits with AviSynth's function "Trim", and encode with a factor of 1.0. OR take into consideration the type of movie you're about to encode. If it's a low action film, subtract 3% (multiply by 0.97) the wanted sampler size.
If it's an action film, add 3% (multiply by 1.03) the wanted sampler size.
If it's an average film, don't add or subtract anything :)
(2) If using ToK, the same applies, BUT:
(a) Use a factor of 0.97 for low action movies.
(b) Use a factor of 1.03 for action movies.
(c) Use a factor of 1.0 for regular (average movies)

*** PAL people, could you please test this please ***
Please use the "Video.en1" file I uploaded earlier ( www.kvcd.net/video.en1 ) and copy it to your ToK OPT directory.
*** PLEASE, let us know your results :!: ***

Please keep replies related to file prediction :!:

-kwag

Maybe I`m talkin` outta` my hat here...but here goes.
Is there a script line we could use, to call extra compression at a given
start/end point of frames, (beginning/end of credits) within the "Adaptive" script?

Maybe this would compensate for some of the error?


*******************************
The Devil`s always.....in the Details!

Well, you can just use the "trim" function, and just cut off the end credits :idea:
You can use Vdub to find your last movie frame (where credits start ), and trim from there to the end.
And that's exactly what I plan to do from now on :!:
I've seen some credits go beyond 10 minutes, and that's ridiculous :D
It's just that much compression we're loosing from the main movie :!:

-kwag

just a suggestion to make it even easier. Instead of using trim() and Vdub why not just hack off the credits in DVD2AVI when you initially create your project file. That's what I do.

Yep, you're right :)

-kwag

@kwag

Are you using the shorter GOP's for this prediction or are you using the "old" ones - NTSCFilm 24, PAL 25 & NTSC 30? If you're using the shorter ones, could you please post the right values for the different sources? Come to think of it, please post the shorter GOP's even if you're not using them :wink: ! If I use the shorter GOP's the file size will be bigger, right? And how does it affect the quality of the movie?

Kwag,
the new video.en1 that you post is only for PAL?
:?

This was done with the regular GOP, or actually "Auto GOP" as used in ToK. I uploaded the "video.en1" file that I used, so you can use it in ToK.
Also I'm correcting the post above, because it was too late and I posted the factors in reverse and offset of 6% instead of 3% :oops:
The factor should be 0.97 (-3%) for low action and 1.03 (+3%) for high action films.

-kwag

ToK selects PAL or NTSC automatically depending on the source :)
The changes I did are basically to enable scene change detection, and I also enabled 3:2 pulldown when encoding MPEG-2. So now ToK can make correct MPEG-2 files for DVDs ;)

-kwag

great news... thanks!
:wink:

Because of this , we can use this configuration file for PAL videos. Indeed I've did this modification myself some weeks ago because I need to do a NTSC video. But when I switch back to a PAL video, TMPGEnc complained with "3:2 pulldown is a feature allowed only for NTSC video encoding" (or something like this).

Now I have 2 video.en1 that I have to switch manually following what is the type of the video :-(

Yes, I know what you mean :!:
This is something that would be great if hedix integrates it into ToK, so that it will automatically switch his configuration files, depending on what the target is :)

-kwag

i got this advice in TMPGEN with the new video.en1 in ToK:

"for encode mode,3:2 pulldown when playback can not be used if
the flame rate is not 23.976 or24fps"
:?

my source is 29,97!

I pretty much lnow the differece between an action film n a non action film, but how can you classified a high action from a low action film from toks point of view. :?: i mean we didn't have to worried about prediction
been off before, so its better to be sure now :!:

Yes I agree, and I'm looking for a solution so we can determine the factor to apply to a movie, depending on the action/activity etc. :)
This is where the fun part begins :mrgreen:

-kwag

The post above still says... :?

and my error remains in tmpgenc with the new video.en1!

help please.

i see this differences between the "old and the new video.en1 :

old:
Job.MPEG.Video.DC_prec=0
Job.MPEG.Video.VideoEncodeMode=MPEGVideoEncoder_Vi deoEncodeMode_Interlace
Video.AutoForceIPict=False

new:
Job.MPEG.Video.DC_prec=1
Job.MPEG.Video.VideoEncodeMode=MPEGVideoEncoder_Vi deoEncodeMode_Progressive_32Pulldown
Video.AutoForceIPict=True

:? Fixed :!:

Should read:
(a) Use a factor of 0.97 for low action movies.
(b) Use a factor of 1.03 for action movies.

-kwag

I'm confused are these new video.en1 settings for NTSC and PAL or just PAL?

On what type of material do I need to add 3:2 pulldown?

Hi audi2honda,

The changes I did on the file apply to NTSC and PAL only on MPEG-1. On MPEG-2, the 3:2 pulldown applies to NTSC. So for PAL people, they must change that. The best and easiest way is to load TMPEG and set all the parameters you want, and then save as an ascii project. Then copy the respective parameters from the saved file, and edit the file "video.en1". I would keep separate copies, depending on what you want to encode. One file for MPEG-1, and another one for MPEG-2, another for DVD, etc.

-kwag

Just as a test I just did a super file prediction. :!: I used the old longer style and used a prediction factor of 1.0. Length of sample 72 and 2 samples per minute. The prediction took 4.5 hours on my P4 1.8ghz machine :!: :!:

And after the encode I still came in 100MB under target for a 2CD encode of a 150minute movie. :cry: :(

I thought even if the MA script was throwing off perdiction this level of sampling could get an accurate prediction, but I guess now.

Hi audi2honda,

I know how you feel :!:
Don't worry, we will get this bastard called "prediction" eventually :lol:

And here's something I just found out problem :!:
I just ran prediction using SansGrip's "Sampler()", and the sample prediction file size of my "K-Pax" movie came out to 12,048KB
The same sample using the manual formula, which uses "SelectRangeEvery()", came out to: 11,652 AND THERE'S OUR DIFF :!:
So there is something going on with the way SelectRangeEvery() works, that doesn't affect Sampler the same way :!:
The value of 12,048, for me is too high, so I would need to drop CQ down to get to the ~11,652, and then my target should be more on the nose.
So I'm now finding the correct CQ with sampler() instead of the manual functions. Then I'll encode again :roll:

-kwag

Update :!:
The new CQ found by manual prediction and using SansGrip's "Sampler()" is now 65.32 :D
The old CQ found by ToK or manual prediction with the following lines was was 67.33 :!:
An THAT'S why the target was ~3% over.
Quite a difference 8)
Now encoding at 65.32 :!:

-kwag

i'll be waiting :D When I get in the office in the morning fire up a few more tests using the sampler method. I don't know exactly how to use SansGrips sampler(), but i'll do some reading.

So r u saying the problem might be tok :?:
8O

Yes and no :!:
The previous version of ToK used "Sampler()", but if was dropped in favor of the "SelectRangeEvery()" because sampler was very inconsistent with PAL material. So now, when I tried the sampling functions to use "Sampler()" again, it seems that we're back on track :)
Right now, the most accurate prediction will be using SansGrip's "Sampler" dll :!:, and using the old and tried manual prediction method.

Edit: This is for NTSC material. I don't know what to do about PAL :?

-kwag

HEDIX where are you? Help!!!! :D

So whats the deal use :?:

Sampler(samples=100, length=24) and find the CQ__ point. :?:

Yep :!:, but I simply call Sampler()
That will use the default frame rate, and take one sample for every minute of the movie. If I'm encoding a DVD, which usually is 23.976fps, it will take 24 frames per sample automatically.

-kwag

Manual File prediction refresher
 

I don't know about you all, but I've been so dependent on ToK for calculations, If forgot how to determing CQ prediction. Could some point me to the right post so that I can refresh my memory. Thx!

Racer99

Use moviestacker to find out the sampler size you need. Simply open your .d2v, set the number of CDs you want, and set your audio bitrate. Now on the "Audio Stream & Authoring" read your video size in KB and divide that by 60. That's your wanted sampler size :!:
Then just add the line "Sampler()" at the end of your script, and start TMPEG with a CQ of say ~60.
Change CQ accordingly until the size is ~1% close to the sampler size.
Remove your "Sampler()" line and encode your movie :)

-kwag

kwag do you load the script directly into TMPGEnc when using the sampler dll or can you still use ToK? I assume if you use ToK the encode will just stop after the sample runs?

Ahhh. The real old method. Now I remember.

:)

Racer99

You have to load it directly into TMPEG, because ToK will write it's own "video.avs" script which already has prediction with the "SelectRangeEvery()" function.

-kwag

Here's my result of "K-Pax" using "Sampler()".
Wanted final size: 714,303.19KB
Encoded final size: 732,612KB
For a difference of +2.5% ( 18MB difference ) :!:
I wish it was more on the ~2% mark, but 2.5 is fine for me (for the time being!).
Now back to encode "The Boondock Saints", which was the other extreme, which was way below the wanted size.

-kwag

Is there a way of configuring ToK to use Sampler() instead of SelectRangeEvery()? So we don't have to do manual prediction...

Just to followup on what audioslave asked... I'm trying to encode some xvid movies which are split into two avi's, so in my avs script I have to load the files in with two avisource() lines (separated by +). But Moviestacker only lets me load one avi file in, not both, so this current method doesn't let me find out my wanted sample size. Can I just change my video.avs file to use sampler()? Will this solve anything for me?

Results for "The Boondock Saints"

Wanted file size: 722,228KB
Final Encoded size: 702,440KB

For a -2.7% difference.

So that's it :!:
That's the best I can do for now. High action films ~-2.5% average, and low action films ~+2.5% average.
Now that's far better that 100MB off :D But still, I'm not completely pleased :twisted:
But I consider this is good enough to get back to encoding 8)

-kwag

I believe ToK version 0.0.5.2 (the previous version) used Sampler() instead of SelectRangeEvery(). Check it out :!: I'm going to try it too.

Edit: Yes indeed, ToK 0.0.5.2 does use sampler. Just make sure you set the path to sampler-2.5.dll in it's settings :)


-kwag

@kwag
Where can I download the older versions of ToK?