Ping-pong prediction vs Long-short prediction
 

Hi,
Here I would like to present the results from some testing in file size prediction.
Some details:
For ping-pong was used gl=2 and sa=5. Offset was the value which the "subtitle" script posted by incredible gave me.
For long-short prediction was used SelectRangeEvery(480,24) for 5% and SelectRangeEvery(2400,24) for 1% of the movie. This is the routine which is implemented in ToK, but different sample length.
Second movie
And third
And the winner is... Well, I would not say there is the winner, because I am using ping-pong for first time, and maybe I am doing something wrong. I will try to connect somehow both approaches, for example to run a pong determining the CQ with the 1%, or something similar. So, more testing...

Hi Abond,

nice to hear some one else is testing! :)
Well ... I never had that big differences between wanted and final outputsize! And never that oversized 8O

Whats the formula you use to calculate the reference size to match?

Yesterday I feeded the ping pong way with American Beauty, where in case of this movie I everytime had problems, no matter what prediction appl. used. The American beauty stream runs via avisynth and got Kwags new MA Routine!! ... and final output was very very close. about 8 MBs less gl set to 2 and sa also set to 5 which I everytime use.


Well when analyzing your encoding sizes I can see that at these movie examples the difference between the Ping and the pong output isn't that high! Which means that the offset turn didn't show big differences in scenes which wheren't inspected by the ping turn.

Because not every movie got such irritating differences between "conventional" sliced szenes.

But in my test I got 2-3 Movies where the Difference at 2% Movie-Sample-size was about 0,5-1,5MB! And thats where you see the advantage in doing this.

Borax's CQ Tester also uses Offsets but calculates the reference in a different way and other diff. things..

Here you can see how the Offset-way within my routine works:

http://www.digitalfaq.com/archives/error.gif

And to be safe to hit the middle of the ping slicing I tried to add the "offset recommendation". I hope its calculated well.

Yesterday I wrote the AVS Routine as a function, so you can safe it as .avsi and put it in your AVS Plugins Folder.

Just add below your script the following syntax

Slicer (clip c, int "percent", int "gop", int "off", int "gl", int "subs")

Explanation:

percent= percentage of the whole movie wich will be used prediction
(default=2 this means NOW logically 2%!!!)

gop= The length of you GOP you'll use to encode, so DVD(PAL) is 15 KVCD is 24, this has also to be understand as Samplelength!
(default=15 so set es default to std. PAL)

off= The offset where the sampling begins which is needed for Ping/pong prediction!
(default=00 so the sampling begins at Frame 0!)

gl= GOPlength multiplicator, this means by how much u want to enlarge the Samplelength which has to be GOP based!

subs= Gives you a Subtitle where you can find the recommendation for Offset settings! Use this the first time you just preview your script to see the recommended settings refering to "off"!
(default= 0 this means off, 1 = subtitel/recommendation active)


And here's the function which you can safe for example as "Slicer.avsi" and put in the Avisynth Plugins Folder so the Syntax as shown above will work:
Now you only see the recommendet Offset values as they are the only ones which are really needed to know.


How to do a prediction using this Command:

1. Encode your audio to mp2 and watch in the windowsexplorer how much Kbyte it takes

2. Add the Slicer() at the end of your existing script and configure it refering to your final encode I suggest 2% of total movie which has to be used for prediction, enter the lenght of your used GOP and set the GOP multiplikator to 2 (this means 2times GOPlenght as Samplerlength). The Offset has to be set to 0.

3. Preview your .avs and note the second offset value recommendation!

4. Now calculate:
Whole muxed size minus Muxing overhead in Kbyte, substract also the audiosize in Kbyte as you can see it in the WindowsExplorer.

So we finally want to have a 795MB muxed moviestream to burn on CDr80. In this case: 795-10=785 .... 785x1024= 803840 Kbyte ..... 803840 Kbyte - 119808 Kbyte (assuming for audio) = 684032 Kbyte!!!
Now the next calculation depends on what percentage you have set for prediction, we assume (and I reccomend) 2%, therefore 684032Kbyte / 100 *2 = 13.681 Kbyte !! That is the Reference Size we have to match cause its 2% of the Final Streamsize.

5. Configure your Encoder to CQ (or Q) (GOP settings set to the exact Gop Length as set in the Slicer()!!!) and import your .avs including the Slicer().

6. Assume a CQ to set in the encoding setting to start with.

7. RUN!

8. You see that the Sampler will be encoded, after encoding you have to watch the encoded size and compare it to the reference size.... already close?? Well if not just rise or lower the CQ value in the encoder and do a new encoding as much times as its needed to get very close to the reference size. If you are almost close (like 10% to the reference size) you do a "Pong". This means that this kind of Prediction uses a Ping-Pong Method and therefore an Offet of 0 is a Ping and a Offset set to the recommended value you noted when previewing you avs.. is a Pong!
So if you are close like 10% to the nedded size perform a pong by just setting the int "off" in the Slicer() to the recommended value. Safe your .avs again and run TmpgEnc again (no source reloading needed! and leave the CQ as used as in the last turn!!!!).
You will see that the Prediction now uses different Scenes than when performing the Ping Sampling! And thats also why we will get very accurate cause by performing an offset there will be more different scenes encoded.. and therefore we obtain a better average of scenes which will be calculated to the whole prediction.

9. Now ... do you see a difference between the last outputted size and this one? Regulary yes! So you can see how both predictions behave.

10. Now continue performing Pongs by rising/lowering the CQ and keep in mind the difference to the ping! Now when you are very close to the needed size and the ping size keeped in mind, you perform a last "safe"-Ping (offset=0) ... and if the average of Ping and pong outputs matches the reference size ... go into your avs script an deactivate the Slicer() line re-safe it and do your full encode!!

Here is a sample report where you can see how to approach to the needed CQ (I everytime mention the CQs and Offsets in my filenames so I will have a good overview):

http://www.digitalfaq.com/archives/error.gif

The first time it will a bit complicated or it will take some time, but you will very soon get a "feeling" for this and so you will be faster the next time ;-)

If you want you can add a safety margin to your calculation of the refered size above ... Well Im happy without but every user got different experiences so in your cases add or substract a factor to your reference size to match the needed final sizes .

The formula:
( Wanted video size/framecount)*frames in sample = refrence sample size
Slicer - very nice! New stuff to play with!
BTW after short examination of the ping-pongs I and the sizes they produce it seems that the difference between them for the same CQ is almost constant!
13996-12815=1181 and 13467-12291=1176 but this are KB, maybe in bytes they will match!
In my case (movie 3) I have 14328-14163=165
and 14165-13999=166
The other two are different though. Just a thought.
Currently I am testing long-short adjusting the reference with ping-pong for 1% sample. The reference file size increase. I think I just found a cure for permanently undersized files using long-short in the range 10-25 MB.
Well, will see after almost 4 hours.

Hi Abond ,

Yep, theorethically!
I discussed this with Krassi too. So theoretically you only would have to do one "pong" to see how the "other" slices do behave....

BUT! If Ping or Pong ... both are encodings don using CQ and we know that CQ is not linear ... so it could happen that if you figure out at a very beginning of the prediction turns, a difference of about 1024 Kbyte results .... at the end of your prediction there could result a difference of 512 Kbyte! So be safe and do a final "to be safe"-Pong at the end ... as I think you did.
I everytime do it like this:
I do a Ping and a Pong on a assumed CQ (fist turn)... ok. this gives me a ca. behaviour of bboth samples (ping & Pong).
Now I contiunue predicting using only pings ... until I get very very close to the Reference size BUT also the difference of the pong still in mind!
Finally I do the safe-pong and see if the average matches the reference. After this ... deleting the Slicer() in my Source Script - resaving the source.avs and let TmpgEnc do the whole Job.
I hope you did understand that you dont need at every turn to reload the source .avs in Tmpg! Just change the values of offset in the sclicer() in your source, resafe and hint again "start" in TmpgEnc ... you will see if he got the offset change!

You see! It could be a lot when finally getting the full encode

1024Kbyte Difference at 2% = would be a 50MBs failure on a final encoding ... and thats why the offset makes sense.

Maybe we all will figure out how much safety margin is needed or so on which has to be implementated in this manual way but I think the offset/movie-percentual/GOP - Based Prediction "could" give nice results on difficult to predict movies. But as Kwag said .... you will never be away of this tricky non-linear CQ .. until this will be fixed in TmpgEnc.

;-)

What I meant is a bit different. Well, maybe I am not right, but I assume the Explorer rounds the values from bytes to KB. So, for example, 13996 KB can be 13995700 bytes. Therefore, this difference in the difference :) is due to the rounding the bytes values to kilobytes :idea: . Well, I didn't calculate if with the current values quoted above it is at all posible :D . Maybe I will do it tomorrow.
Yes, I did it,but, sorry to say, in fact I'm using as default the long-short prediction. Of course I'm very thankful that you give me the idea for ping-pong method and it seems it will adjust very fine long-short approach. What I found in the method is that it save time and is almost so precise as any other method. :wink:

@incredible (or anyone else who can help ;) )

I added the line: "Slicer(clip c, int "percent", int "gop", int "off", int "gl", int "subs")" last in my script, but I get an error in Mediaplayer. It says "Expected a, or )". What does this mean? Could you please explain once again how to use this slicing routine?

This is my script:
And this is how the "Slicer.avs" looks:
And when I run my script in MediaPlayer it says: "Sample: Output frame count must be < input framecount" :cry:
What am I doing wrong? I'm really interested in trying this prediction method! :wink: Desperate for help... :banghead:

Audioslave,

Zaks in the german forum got exactly the same error and it was solved when he reinstalled AVS 2.53!

Another Point:
Don't use just Slicer()

You should set it to the needed parameters here an example as folowwed:

Slicer(2,24,00,2,1)

This means (2% of whole movie will be sliced, 24 (KVCD) frames in a GOP are used when encoding, offset is set to 00, one slice (GOP) will be multiplicated 2 times, and 1 stands for that subtitle for offset determination is on)

BTW the size of the subs are opimated for a view within a 480 x XXX size, so if you want to see the subs in a 352 x XXX you have to decrease the fontsize in the avsi doc.
There you see the "size=16", just correct it to 12 if needed

Cheers
Inc.

Thank you, Incredible. I installed AviSynth 2.5.3 and changed the Slicer parameters, and what do you know - it works! :D Thanks!

@incredible,

when i use 'slicer(2,15,00,2,1)' at the end of my script, i get the error-
message 'Sampler: output frame count must be < input frame count'...

mmh, any ideas??? - tnx in advance

regards, ***mfb***

@mfb

install avisynth 2.53!

gruß
zaks

Thanks zaks!

i installed avisynth 2.53 and it worked fine.
my first try with ping-pong prediction gave me quite accurate results :D
(~ 10 MB less than wanted filesize for the muxed videostream predicted
for a CDR-100 media)

regards, ***mfb***

Hell yeah! 8) :D
Just finished encoding Terminator 3 - 800,99 MB muxed! Amazing.
This is how I calculated:
I used the audio size I got from CalcuMatic.
Thank you Incredible! This "Slicer Routine" is great! Keep up the good work. :wink:

Close but no cigar
 

Incredible,

What am I missing here?

Here's my main AVS Script

And here's my Slicer.avs fileWhen I run it through Media Player I get the following error:

"There is no function named Sampler"
D:\Ripped\freq.avs line 25
D:\Ripped\Slicer.avs line 54

Any Ideas? :? :?:

missing sampler.dll for avisynth 2.52 /2.53

regards
zaks

Be shure you installed avisynth 2.53 and downloaded and installed the sampler version 2.5

www.incredible.de.tf/Downloads/Sampler-2.5.rar


BTW Vortech ...

You should delete the

MergeLuma(blur(0.1))

in your MA Script as the new MA routine now unfilters (blurs) linear without threshold settings, in combination with this MergeLuma(blur(0.1)) your image will end up to blurry even on almost no motion scenes!

Anybody can translate to Spanish this method ? :( It will be a great help for spanish Kdvders



Thanks in advance :P

Si te esperas, este fin de semana voy a escribir una explicación en español (bueno con mis pequeñas palabras en español que hasta ahora no me han perdido :) )

Yes, I agree :lol:

:D :D :D :D :D :D :D


YESSSSSSS PLEASEEEEE!!!!!!!

Hablas muy bien el español = You speak very well spanish :D

I will wait for it impatiently!!!!!!


THX very much

Incredible and Zacks,

Thanks for the heads up :!:

Mira, como es un asunto muy largo y complicado aqui te doy una translación hecho de "babelfish/altavista" solo para que se lo entiende mejor si uno sabe hablar español mejor que el inglés.

PERO solamente usas/copias el códico del Asunto original aqui!!!
Porque el Babelfish tambien modifica partes en el codico en la translación.

http://babelfish.altavista.com/babel...p=en_es&tt=url


Espero que te ajuda!

Muchos saludos!
Inc.

Hi Inc...

One little question... How does the Slicer Routine determine the second Offset value? What's the criteria? :roll:

Criteria??

Well the purpose of the pong offset is that the pong-slices will be in the position between the slices of the "ping" turn, so that effect. more movieinformation will be predicted in the same time than simply a normal prediction using more % of the movie. :wink:

I understand the theory... :wink:
I just wanted to know why the value for the second offset differs from movie to movie. I for instance got 30 sec. as offset value. And I saw a post where it's 50 seconds...

It mainly differs at diff. gop multipler settings as this results in larger slices with the same % of the whole movie = larger "holes" between the larger slices for example.

I really must thank you Incredible for making this "Slicer" routine :!: :wink: It has worked EVERY time for me so far. No more re-encoding movies that gets too big. :D

Also a nice advantage is that you can use it perfectly for mencoder or ffvfw prediction, .. cause its encoder independent ... just avisynth as frameserver is needed :)

MCE prediction "could" also be possible but the Q range of MCE is defenitely to small for acurate predictions.

Can you explain me that further. Maybe I can help you with MCE.
What would you need with MCE, for accurate predictions.

:)

Thanks but ....

I do know MCE more than well and "one" of the bad things when doing Q VBR is that the range is def. too small (as I remember something approx. about 1-32).

I tested MCE about for one week some month ago (actual release) and TmpgEnc does output better (and faster when doing a prediction in TmpgEnc compared to MCE at 2pass, if Q prediction in MCE is used to much failures accord. to filesize do occur cause of the reason expl. one line above).
Best quantisation on the same endfilesize does deliver mencoder at specific settings (2 Movies on one DVD-R) even better as QuEnc which I tested this evening ("still" too less options to set for optimizing... but that will change in the next release I think).

But ... thanx

Inc.

Here http://www.kvcd.net/forum/viewtopic.php?t=9832
there is a spanish traslation of this http://www.kvcd.net/forum/viewtopic.php?p=52657#52657 post.

But I don't speak about what is the better output. I just want to try your ping-pong with MCE. MCE can adjust the output file size very well, as you can see in the next screenshot. If you "know MCE more than well", you won't have problems for interpreting file names (related to settings).

http://www.digitalfaq.com/archives/error.gif

Why can't we use just Slicer(). Doesn't work it with default values in this way?

Hi:
@ Incredible
I was just playing maths with your function, and found out that
is equal than the simpler one
The difference in the results of these formulas are round functions related.

Said this, I think your function can be improved in the following way:
Only changing ping to pong, the script will get automatically the "off" value. (Nobody needs to see the subtitles before encoding)
Debug = 1 is only for that, you can set at 0 and you won't see any subtitle. (default = 0 this means inactive, 1 = Debug subtitle active)
Maybe you will need to adjust your factor 1.009 (test it please)
by the differences mentioned in the beginning (round functions related)

The other instructions just as in
http://www.kvcd.net/forum/viewtopic.php?p=52657#52657

See you.
Prodater64

NICE! :)

Thanks for taking the time to integrate that conditional auto off Method.

I didn't integrate it cause many users do make Offs like ping=5 and pong=calculated_pong+5 as predictions will come out more accurate if the first seconds of the beginning wont be sampled, means these Frames in 99% of cases are just black and this will stand in no good relation to the 2% prediction of the whole movie.

Also I leved it that way as this can be very nice tweaked to get diff. slices of test-encodes I did when testing FFvfw or Mencoder.... = better way to catch fast moving scenes for example.

But anyway, your auto off routine will seem less complicated to noobs!

According to the factor you discribed 1,009, ... well everybody gots his "own" perfect factor (as known from TOK) so evreybody can tweak here on his own.

BUT pleas do not change the order of the variables in the Command as many users now are "acostumbrado" in those settings

Slicer(percent, GOP, Offset, GOPmulti, Debug)!

So the variable for mutiply the GopFrameCount should be still right behind the Offset determination of the Command Syntax. Yours is def. more logic, but changing this and on the other hand refering to my explanation at the start of this therad will cause confusion of Users :wink:

To me it seems you're more in mathematics and if the "shrinked" calculation for sampler() will work the same, ok. Test this by using my latest and your new one on the same movie and watch the resulted framecounts in Vdub so you can compare.

According to "no just Slicer()", This command doesnt get its advantage of default usage, cause anyhow we have to alter that by changing offsets or in your case from "ping" to "pong". Just slicer() would cause big noob confusions :wink:

Thanks for participating in developement!

Inc.

Your welcome.

I already edited previous post.
Fixed. Users can make such Offs.

I think your tweaks don't changes to much.

Fixed.
Slicer(percent, GOP, Offset, GOPmulti, Type, Debug)!

It was tested before I post it.
I only found small differences round functions related.

See you.
Prodater64

Hi:
My first Ping-Pong Prediction (Incredible's Function) with Mainconcept Encoder. Not very easy, but thinking that is first time...

http://www.digitalfaq.com/archives/error.gif
Not so bad, isn't it? :)

Hi:
More improvements.
Change log
- Script can shows movie title on info screen.
- Script show more info on screen.
- Info screen don't show sample. Activate/Deactivate with Debug control.
- "Control Panel" in avs for: set movie name, CD number, CD time, Factor.
- Slicer Settings more comprehensibles, same order like Incredible original, plus Ping/Pong, Debug and Encode controls.
- Easy to understand for anybody.

This method has 1 avsi file, plus code to wright in your avs encoding file.

AVS Encoding Script

The statements after "Here your script" until "Not to modify" are just an example. You must to put your script there.
Copy this script and paste it in you AVS script. Trebuchet MS font, normal, 12, 1024*768 optimized.


AVSI file

Copy this script and paste it in a text file with avsi extension. Put it in your avisynth 2.5x folder.
It is self explanatory.

Where do you got all these CDs /CDtime parameters, and subtitel routines and so on ... is that inspired from the manual prediction thread??? As they also delaed with cd size and so on there. As I remember :)

Yes, it is, of course. I was learned almost all in this forum, of course. But not the same. I have adapted it to your method.