My test seems to confirm your conclusions. GOP length 8 turned out to be average 997kbps, Q=2,12. The length 15 was average 998kbps, Q=2,05. I haven't checked the graph thoroughly but as you've already examined the test results, it seems that there is no need to. Too bad, this could have been a revolution :twisted:

Kwag

Have you made some encodings with the x3 template??? Which were your results??? I only use this template because the x2 showed me blocks sometimes, i would like to go back if that gives me more quality...

Regards,
DaDe.

Hi DaDe,

The same GOP results with every template. :roll:
The x3 is still the best balance for quality/space ratio, specially on action scenes. Even viewed on a HDTV, the x3 is very close to DVD sharpness, and will exibit less blocks than the x2. If your movie is a drama or low action, the the x2 will look just near perfect.

-kwag

Hi All,

Must be some hot and heavy PM's between Christopher, SansGrip,
Kwag and other experts about GOP effecting file size. . :lol: It's
been very quite for at least 2-3 hours. I hope there's some sort of
answer to this issue. Either a new and revolutionary discovery will
emerge or this was all an exercise in futility :? Either way, it's fun to
see this much activity going on to solve problems. :D :D


-black prince

Hi black prince,

I think we're all crying now :cry: . Specially me, since I started the thread, I'm banging my head against the wall :twisted: :cry: :twisted: :cry:
Maybe more tests have to be conducted, and I hope my last test was wrong. I never give up :D , so I'm still looking for the jack pot :wink: . Still, it's great to brainstorm. I hope something great comes out of this :?

-kwag

Actually I quite enjoyed the flurry of activity :D. As for my silence, I've been doing dishes and cooking supper and so on ;).

I still think there's more room to tweak. I mean, with so many variables it's highly unlikely that we've hit upon the absolute best possible way of doing it.

For example, I've always wondered if "scene change detection" should be on or off (I imagine kwag's already tested that one), but also it's worth taking a look at the "force picture type" settings. I think we could squeeze some joy out of that dialog.

I too am looking over several ideas. I agree with SansGrip in that there is always room for improvement. Just a side note to let everyone know that the KVCDEncoder project is not dead. I have been going over several ideas from the FFmpeg and Xvid projects. And Kwag has been searching for a bug in your current codebase. I sure that things will pickup after the holidays.

Are you running on a generator down there or did they get the power back up yet? :)

I was one of the lucky ones, never lost power, but boy there was ice everywhere.

Tell me about it... My web server is located outside Raleigh and was down for about 5 days before the admin was able to find a couple of generators for sale :).

Nice to see I'm not the only "northerner" here .. :wink:

You're still north to me -- I'm in Kitchener ;).

Is that why you're always in the Kitchen cooking supper :mrgreen:

Ok, here we go again :D
I've been busy for the past 4 hours, because I just don't give up 8O
This is the GOP I've been testing, after trying out so many combinations, that it's driving me crazy.

Try this out: 1-12-2-1-24 against 1-36-3-1-36.
With KVCD Predictor, and using the 1-12-2-1-24, the CQ_VBR value is lower for the same target file size as with 1-36-3-1-36. However, the Q factor is lower, meaning higher quality, for the same file sizes. There are LESS visible artifacts with 1-12-2-1-24 than with 1-36-3-1-36. I've made several tests now, and with the same file size with both GOPs, the difference in quality is visible, specially around complex objects. The 3 B frames used in the current GOP, while giving higher compression, degrade the video quality slightly. So with 2 B frames and a MAX number of frames per GOP of 24, I've found that we get more punch (quality) for the same given file size as 1-36-3-1-36. Here's a clip of "Red Planet" with this new GOP: http://www.kvcd.net/clip3.m1v
The clip is a cut out from the file prediction sample, which is 11.83MB.
And THIS is the quality that will go on ONE CD-R at 704x480, as predicted with SansGrip's KVCD Predictor. Please try it out, and make some samples and compare to the the 1-36-3-1-36 GOP. I believe what I have found here is definitively a better GOP that what we had before. I only tried this with 704x480 PLUS template, but I assume it will work the same with the other templates, based on all tests made in this thread.
We need feedback on this!. This GOP could probably be optimized even further, but I'm tired now. And the result I see on the sample is so good for the resolution, that I don't know if it needs to be further optimized. :roll: I'll leave that up to anyone who would like to try and improve on this.

-kwag

Hey Kwag i s wondering if could see the avs script for red planet. That looks pretty damn good at 704X480. You da man kwag. So is compression still greater with 1-36-3-1-36 or the new gop using only 2 b frames.

Thanks Branden

HI,

Instead of searching and "optimal" GOP for each CQ/resolution/movie, and at the same time trying to do the file prediction, perhaps we can use a compromise solution and narrow the options, like a low-CQ GOP and a hi-CQ GOP.

With only a couple of GOPs the search space is reduced and it can be easier to find "sub-optimal" parameters that may not be as good as if we had exhaustively searched every possible GOP for every CQ for a given movie, but that are still better than always use the same GOP for every CQ.

Here's what I used for that sample:

I didn't use any "Sharpen" because I think at that resolution it's not needed. The complete encode came out to 756MB, and it was supposed to be ~713. So I guess it throws off the KVCD Predictor formula a little. Based on my original prediction sample size, 11.88MB, using the formula with a factor of 1.0, the result is 759MB. So it seems that for this GOP, the factor is 1.0 and not 0.95.

-kwag

Hi Kwag,

Here's my results:

Kx3 (528x480), FitCD=496x448, Movie Length= 8304 seconds

LoadPlugin("E:\DVD Backup\2 - DVD2SVCD\MPEG2DEC\MPEG2DEC.dll")
LoadPlugin("E:\DVD Backup\2 - DVD2SVCD\BlockBuster\BlockBuster.dll")
LoadPlugin("E:\DVD Backup\2 - DVD2SVCD\LegalClip\LegalClip.dll")
mpeg2source("D:\Temp\movie.d2v")
IL = Framecount / 100 # interval length in frames
SL = round(Framerate) # sample length in frames
SelectRangeEvery(IL,SL)

CQ_VBR=22.42 GOP=1-12-2-1-24 File size= 24,517,706
CQ_VBR=22.42 GOP=1-23-3-1-36 File size= 23,373,174

I used Tmpgenc to mask the borders and resize the video. :)
Blockbuster noise causes file size to fluctuate and sharp just
increases file size. Picture quality was great for both GOP's.
Will you still resize and mask borders with Tmpgenc or has that
changed :?: I like the fact that you're always searching for
better picture quality/smaller file size/faster process. It's what's
got us here and that's why I stick with this forum. :D Get some rest. :?

-black prince

Hi kwag!

For a 1CD with 704x480 I presume that the CQ_VBR should be low.
Is this gain in quality true for higher CQ_VBRs? (lets say 2CDs 704x480)
Or for higher CQ_VBRs the quality is high enough that we could use the other GOP and won't notice the degeneration in quality?

Cheers!

Hi black prince,

Rest, what's that :lol:
Your results are 100% consistent with mine :D . Now if you lower the CQ_VBR and encode again with the 1-12-2-1-24 until the file size matches the one encoded with 1-36-3-1-36, you'll still have better quality! Here is a screen shot of bit rate viewer showing my two sample encodes. The file size difference from one file to the other is only 5KB. So that's negligible.

Here's the first with a GOP of 1-36-3-1-36:
http://www.digitalfaq.com/archives/i.../2002/12/1.jpg
The peak Q level is 15.14, the average bit rate is 863KB and the average Q. level is 3.86.

And here's the one with 1-12-2-1-24:
http://www.digitalfaq.com/archives/i.../2002/12/2.jpg
Here, the peak Q. level is 13.93(better than above!), the average bit rate is 863KB (same) and the average Q. level is 3.53(Better than above!).

So you see, there is a difference, even though the file sizes are the same. This opens up a window of oportunity to try and continue to optimize the GOP until we can find the best Q factor while retaining the same file size.
Now I'm not crying anymore. I'm happy as a pig :lol:

-kwag

Hi muaddib,

What I like about this long GOP with the 2 B's, is that the visible artifacts that showed with 1-36-3-1-36 are almost gone :D
That is, comparing a target with identical file sizes, made with both GOP's. So for a 2 CD movie, the result must be just vomiting awesome :mrgreen:

Just try it with KVCD Predictor and make a sample target for 2 CD's. Remember to change the error margin % from 5 to 0, because this GOP is actually more close to the real formula. That is * 1.0, and not * 0.95.

-kwag

What I'm trying to do is target a file size that is identical to the other GOP ( 1-36-3-1-26 ), but gives a higher quality ( Q. Factor ). Right now, with ( 1-12-2-1-24 ) it seems to do just that. When used with Predictor, the CQ_VBR encoding value is lower than the same material than with the longer GOP. But when both samples are encoded with their calculated CQ_VBR value for the same file size, I can see clearly that with the 1-12-2-1-24 artifacts are almost gone and the picture looks cleaner. Viewed with bit rate viewer, you can see the quality curve is closer to the bit rate curve.

-kwag

So a higher CQ_VBR isn't necessarily better... Now you've really opened a can of worms :mrgreen:.

:mrgreen:
Now let's find the sweet spot :lol:
This is going to be fun 8)

-kwag

Well not much too tell. It live about 1.5 hours drive southwest of Raleigh. I would not have been such a big deal, but it caught everyone off guard. Never under estimate mother nature.

I just did American Pie (1h35m) at 704x480 with the above GOP structure. KVCDP overestimated yet again (I seem to have less luck with it than everyone else) with an error margin of 0%, ending up with a final size of 749mb compared to the predicted 797mb.

Anyway, I just sat and watched the first half and it looks incredible 8O! During regular viewing it's basically indistinguishable from a DVD, and it's only when you look closely at known problem areas (high contrast textures, etc.) that you notice worse Gibbs Effect than on a DVD.

But really, this is as good an encode as I've ever done. Not only are there nearly zero noticible DCT blocks but it's also super-sharp as you'd expect from that resolution... Now I have an encode I'm happy with, I can finally delete the vobs that have been on my HD for months!

:D :lol: :o 8) :mrgreen: :!:

Kwag, you're the man :).

Here's the script, in case you're curious (it's pretty standard):

Hey great SansGrip :lol:
So this thread wasn't a waste of time after all. Thought so there yesterday, for a moment :roll: , but I guess something good did come out of it :D
I'll have to run some tests on the other resolutions, and if I get the same results, we'll say "Bye Bye" to 1-36-3-1-36" and welcome 1-12-2-1-24 :mrgreen: and update all templates with those numbers.

-kwag

No way! I did try 720x480 once (long before I discovered your templates or even contemplated writing my own Avisynth filter) and was highly unimpressed with the results. If I remember right it was awfully blocky and I swiftly abandoned the idea of going over 352x240.

But it seems that your custom matrices and GOP structures are like a magic potion. I never would have believed that 1h35m on one disc at that resolution and NO BLOCKS would be possible if I hadn't seen it for myself :).

Hi Kwag,

My latest test results:

Movie Length = 8,712 seconds, Resolution (528x480), Resize=496x352

CQ_VBR=22.42 GOP=1-12-2-1-24 File size=22,225,134
CQ_VBR=22.42 GOP=1-36-3-1-36 File size=21,533,392

Lowering CQ_VBR to come close to file size=21,533,392
CQ_VBR=20.60 GOP=1-12-2-1-24 File size=21,608,937

Picture quality was actually better using GOP=1-12-2-1-24.
This was done with no filters. Adding Blockbuster and Flux
should increase picture quality greatly. Way to go Kwag :D
I guess this lowers the starting CQ for kvcd templates. For
KVCDx3 it’s 25 and now it could be 22 or 23. There is probably
still more room to improve picture quality/file size/process time. :D
These improvements are starting to add up. Keep it comming. :D :D


-black prince

:D Feels so good to hear those words :D
Some people "at other forums" keep saying NAH. just templates. There's nothing special. bla bla bla. And no matter how many times I try to explain that this is all about MPEG-1 optimizations, that they are put in templates ( thanks to TMPEG provisions ), they just keep saying "KVCD are just XVCD and are nothing more than templates"
Well, it has taken quite a lot of time to find these optimal parameters, and I think we're not done yet. So I would say to them, as I've said before in the "OTHER" forums, "You ain't seen nothing yet" :mrgreen:

Thanks SansGrip :wink:
-kwag

Hey Kwag,

Are you still using Tmpgenc to mask borders and resize or have
you dropped this process. :?:

-black prince

I'm still trying this with the mask.

-kwag

Anybody who is somebody that loves this craft of ours is really reading the posts here. Where else would the leeches get their ideas. You guys are head and shoulders ahead of them and ahead of me too. I'm trying to catch up.

Keep it up you guys. That goes for you too Black Prince, Sansgrip, Christopher, and any others I forgot to mention.

I'm currently encoding "Red Planet" one more time, with the file prediction adjusted, to take into account the new GOP difference. I'm 68% into the encode, and file size is ~545MB. Looks like it's going to be right on target. The CQ_VBR used is 11.1, much higher than the sample I posted yesterday, which was CQ_VBR 8.4 I recall. It's a HUGE difference in quality from 8.4 to 11.1. Artifacts are barely visible. So I'll let you know when the encode finishes, and hopefully validate the formula adjustments.

-kwag

:?: I thought with the new 2B GOP, the CQ_VBR should be lowered a little for getting the same file-size... how come you upped it to 11.1 ??

Because the final file size was under the predicted size on the movie I did. So there was a lot of space left. Now I'm trying to zero in on the offset ( error % ) to adjust file prediction with this new GOP. Not done yet.

-kwag

Here are my results and calculations. Based on the final size I got for "Red Planet", which when muxed with audio just made it with overburn, the new % error correction for file prediction formula is 0.89 and for KVCD Predictor it's 11
I'm going to run a couple of different movies today to see if the results are consistent. This is still a test phase, and these numbers are not final. Until we get to the ~2% to ~3% we had with the old GOP, I won't be happy :wink:

Edit:
Currently encoding "Red Planet" with adjusted CQ_VBR calculated from formula: MPEG size = ((Total frames/Framerate)/100) * (MPEG sample file size * .89)
In another computer, encoding "Bugs Life" Full screen to 352x240, with same calculations. What I'm doing is testing the formula with two different movies and the resolution extremes, 352x240 and 704x480. On 352x240 using standard .avs script with default skeleton provided by FitCD. No filters of any kind. All encodes using GOP of 1-12-2-1-24

-kwag

Hi Kwag,

I'm doing some testing for the file prediction factor (i.e. 0.95) and
learned that with Blockbuster noise there's a catch-22. Picture
quality is fantastic with Blockbuster noise but, file prediction is
constantly changing. Without it file prediction is much more accurate
but, I loose the quality. My solution is to use a smaller target file
(CD - audio = video target file). Instead of 128kb audio, I'm dropping
to 112kb and using the target for this. It's not what I want but, how
can we compensate for Blockbuster noise :?:

BTW, my encode time has increased from 6-7 hrs to 12-14 hrs using
Tmpgenc mask borders and resize. Why is Tmpgenc so slow :?:
Picture quality is supriseingly very good at these low CQ_VBR's.
CQ_VBR=11+ is much like the old CQ_VBR=25 with the old GOP. :D


-black prince

The filters shouldn't change the accuracy of the formula. Only the final file size. That's the reason I'm currently encoding one movie at 704x480 with Blockbuster noise, and the other one at 352x240 without any filters. The filters should only affect CQ_VBR value, but the formula should behave the same, no matter what the source is. If it does make a big drastic difference, then we have a serious problem! Then a new test should be done, probably with a wider sampling ( more that 24 frames per shot ) to accomodate GOP size and compression factors. Hopefully, this won't be needed :roll:. Right now, the .89 offset seems to be right on the nose 8) I'll know for sure when both movies are done.

-kwag

kwag,

I used your new file prediction methods and new technique to encode Moulin Rouge. I encoded it iwth the kvcdx3 template with fluxsmooth only and the mpeg ize came out to be 782 megs. Which is pretty close. Just wanted to let you know. Quality sure blew me off my feet. Nice......

Branden