To crap or not to crap =:O
 

Well, actually, to crop with a mask. Not to crap :D
How would you like to save now about another megabyte of space for every 90 seconds on your KVCDs :?: Well, here we go again 8)
Here are my findings, using KVCDx2 704x480 for test purpose, using three different techniques. The first one is the one we all use. That is, create a .avs script with FitCD, and encode with that.
My sample, using this method and a fixed CQ_VBR value of 12, came out to 15,948KB.

The second one is the same .avs script and same CQ_VBR, but before we encode, go into TMPEG/Settings/Advanced and check "Clip frame" and double click on it. Now scroll to some part of your movie where you can see a bright scene and the top, bottom, left and right are clearly visible.
Now check all four boxes "Top Mask" Bottom Mask" "Left Mask" and "Right Mask". Now change the top, bottom, left and right values until you see that you just started to clip into the picture. Back off to the point where you are masking your black area completely without cutting any part of the movie. You might ask, why do this if it's already black :?: Well, it is, but it also has stray noises that we cant see, but the encoder does see :!:
The sample with this method came out to 15,060KB.

The third sample involves commenting out the bilinear resize and add borders from the .avs script, and calculating the aspect to put it manually in TMPEG. Also, check that your masks are correct as in my second sample, because these will change once you remove the bilinear resize and add borders lines from your .avs script. This is what I did: My movie is 720x480, and my target is 704x480. So I use our friend FitCD, just to find out the values I need for my resize. When I open my .d2v, and I select XVCD (704x480) as destination with 2 blocks overscan, I see that the resize line is this: BilinearResize(672,352,16,0,688,480) so that means that I will use 672x352 in TMPEG under "Video arrange method: (Center custom size)" and use these values.
Result: 14,883KB

Sample #1: 15,948KB
Sample #2: 15,060KB
Sample #3: 14,883KB

So you see, there is a difference of 1,065KB per 90 seconds, and this means higher quality for the same size of media, or longer play time. Match this with the new GOP ( 1, 36, 3, 1, 36 ) and now you can put around 120 minutes with the KVCDx3 for a "never before seen" result :wink:

And of course, here's a small cut from sample #3: http://www.kvcd.net/red.m1v

Enjoy ( I know you will :lol: )
-kwag

Hi HKwag,

Kwag wrote:
It's helpful to change the color mask to a lighter color while cropping the
borders and changing back to black when done. The masked area is ignored
by Tmpgenc during encode, but uses a default color (black). This of
course decreases file size. I tried this and my test encode went from
9,811,498 kb to 9,647,271 for LBR. I'll try third process later today
after I get some sleep. ZZZzzzzzzzzz.

-black prince

Hi black prince,

Yes, indeed, with a lighter color you can zero in on the edges if you can't see the film. That's why I suggested to move to a bright frame, and then do the mask. But if the film is very dark, it's a good idea to change the color to a lighter one for the masking, and then change it back to black. :) Now go get some ZZZZZleeep :D

-kwag

Hi Kwag,

I'm not sure if I understood correctly, but the third sample means resizing in TMPGEnc? I thought that was of less quality than resizing with Avisynth.

8O

I don'y know if i did something wrong (i will give it another go) but here are my results (2 minute clip : resize to 544x576 PAL using long gop) Primal Fear:

Normal (avs script with resize/add borders) : 28,685 KB
AVS Resize + Clipped Borders : 28,724 KB
Resize in TMPEG + Clipped Borders etc : 28,707 KB :cry:

So as you can see, the normal method gives me best results, although there is hardly any difference to speak of. I don't know if your results have come from a fault in your version of avisynth, as i can't really imagine that there should be that much difference. Anyhow, i could have made a mistake so i'll try again and let you know.

Jim

That's what I thought too 8O , but the samples look basically the same, regardless if I resized with AviSynth or TMPEG. Maybe TMPEG has improved since the last time I resized with it. This was at 704x480, so maybe at 352x240 there's a difference :?:

@jamesp,

I'm using the latest version of AviSynth (2.07). I tried two times with two differenc movies, Red Planet and The Siege, and I got the same results. I will now try 352x240 and see if there's a difference. I'll post here my result.

-kwag

Ok, I just encoded the 100 second file prediction sample on K-Pax using the LBR (352x240) CQ_VBR=29, and here's my result:

Using this script:

LoadPlugin("C:\encoding\MPEG2DEC.dll")
LoadPlugin("C:\encoding\fluxsmooth.dll")
LoadPlugin("C:\encoding\blockbuster.dll")
LoadPlugin("C:\encoding\legalclip.dll")
mpeg2source("K:\KPAX\VIDEO_TS\movie.d2v")

LegalClip()
BilinearResize(336,192,45,0,630,480)
FluxSmooth()

BB_Resolution = 352*240

BB_StrengthConstant = 352 * 240 * 15 # Base strength
StrengthValue = round (BB_StrengthConstant / BB_Resolution)
Blockbuster( method="noise", detail_min=1, detail_max=10, variance=1, cache=1024 ) # Apply noise if complexity is <= 10%.
Blockbuster( method="sharpen", detail_min=20, detail_max=90, strength=StrengthValue) # Sharpen only if complexity is >= 20% AND <=90%.

AddBorders(8,24,8,24)
LegalClip()

IL = Framecount / 100 # interval length in frames
SL = round(Framerate) # sample length in frames
SelectRangeEvery(IL,SL)

Result = 9,250KB

Now resizing with TMPEG and masking black area using the same script above but removing the bilinear resize and the add border lines and using 336X192 as center custom size.

Result = 8,165KB.

Note: I can see that encoding is slower (way slower!) using TMPEG's internal resize, but I can also see that the increased amount of CQ_VBR pays off :wink:
I am currently encoding K-Pax with a CQ_VBR value of 50 on the LBR, for a single CD-R. Before, I had to use I believe a value of 30. So the difference in quality is gigantic.

-kwag

will you be updating the templates with the new smaller file size settings?

Yep!, later this evening.
The only change will be the GOP ( 1, 36, 3, 1, 36 ) on all templates, except the KDVD templates. Those can't use that GOP, because it's not part of DVD specifications, and that's a bummer :?

-kwag

Hi Kwag,

I don't get it - I did some tests for your crop-methods...

'Pearl Habour' 704x576plus CQ_VBR15 new GOP
all three 100 secs-clips (normal avs, clip frame/mask, clip frame/mask & TMPEG-resize) came out almost the same size* :?:

'Spiderman' 352x288plus CQ_VBR35 new GOP
all three 100 secs-clips (normal avs, clip frame/mask, clip frame/mask & TMPEG-resize) came out almost the same size* :?:

*means just a view bytes difference

any idea why i can't reproduce your results???

regards, ***mfb***

Hi mfb,

Strange :roll: , are you using any filters, Sansgrip's, etc., in your .avs script? Because I get a hell of a difference.

-kwag

I think I know what's happening. Using SansGrip filters, specially "Noise", the Blockbuster filter is sending noise to the complete frame, and that includes the black area. This is the reason why the file size is larger when we use the resizing with avisynth. If you don't use Blockbuster filters, then the file size drops. But when I use Blockbuster filters, and do the resize and mask with TMPEG, I'm cutting off the black area completely, so even if Blockbuster is used, it will only apply to the picture area and not to the black area, as I believe is what it's doing. That is why I get smaller files with this method. So before, when using Blockbuster sharpen and noise filters, the files were much larger. This way, we can use the filters, and the file size is dramatically reduced. If you are not using any filters, then there is barely any difference in file size.

-kwag

Kwag,

I think it must be something to do with sangrips filters. I tried some more tests and got the same results again. I don't use any filters other than temporalsmoother for DVD rips.

I've been doing a lot of testing with tv captures lately and i noticed that fluxsmooth(10,10) increased my encodes by about 1mb per minute compared to just using temporalsmoother(2,1), which fits in with the results your getting. I couldn't understand why, because the end results were looking loveley but the file size increase ment i ended up using a combination of Peachsmoother and temporalsmoother. I now wonder if fluxsmooth is adding noise somewhere arounf the edges of the video, and your cropping is cutting the noise out.

James

i used almost the same script as in your latest post (all the filters, same order). :!: The only thing i didn't use was the setting 'cache=1024' for blockbuster-noise...

regards, ***mfb***

Hi James,

I think we crossed posted :D
Yes, the thing is that for people using SansGrip filters, this is a benefit. Specially using the "Noise" method which kills the DCT blocks on dark scenes. Before, the file size would be much larger. But now, it's almost the same as if no filter is used, with the added bonus of making the filter work focused on the picture area. I also notice that the edge boundary of the pictures is somewhat a transition from black to "something" not to clear, and then to the actual pitcure area. This can be seen on the "Clip Frame" in TMPEG when doing the mask. And if we mask out this single fine line on all four edges, it makes a file reduction of several kilobytes on the final mpeg. This of course translates to some megabytes at the end of the encoding.

-kwag

Hi Kwag,

Here my results:

Tmpgenc Plus 2.59
Avisynth 2.07

LoadPlugin("E:\DVD Backup\2 - DVD2SVCD\MPEG2DEC\MPEG2DEC.dll")
LoadPlugin("E:\DVD Backup\2 - DVD2SVCD\BlockBuster\BlockBuster.dll")
LoadPlugin("E:\DVD Backup\2 - DVD2SVCD\FluxSmooth\FluxSmooth.dll")
LoadPlugin("E:\DVD Backup\2 - DVD2SVCD\LegalClip\LegalClip.dll")
mpeg2source("D:\Temp\movie.d2v")
#
LegalClip()
BilinearResize(336,224,0,0,720,480)
FluxSmooth()
Blockbuster( method="noise", detail_min=1, detail_max=10, variance=1, cache=1024 )
Blockbuster( method="sharpen", detail_min=20, detail_max=90, strength=15)
AddBorders(8,8,8,8 )
LegalClip()
#
IL = Framecount / 100 # interval length in frames
SL = round(Framerate) # sample length in frames
SelectRangeEvery(IL,SL)

Movie Length = 8465 seconds
CQ_VBR = 30

First Process:
Avisynth - BilinearResize(336,224,0,0,720,480)
Avisynth - AddBorders(8,8,8,8 )
Tmpgenc - Clip Frame (off)
Tmpgenc - Video Arrange Method - Full Screen (keep aspect ratio)
Time = 5:46 (mm:ss)
Test File Size = 9,899,721

Second Process:
Avisynth - BilinearResize(336,224,0,0,720,480)
Avisynth - AddBorders(8,8,8,8 )
Tmpgenc - Clip Frame (top=66,bot=55,right=11,left=10)
Tmpgenc - Video Arrange Method - Full Screen (keep aspect ratio)
Time = 5:10 (mm:ss)
Test File Size = 9,121,216

Third Process:
Avisynth - BilinearResize(none)
Avisynth - AddBorders(none )
Tmpgenc - Clip Frame (on)
Tmpgenc - Video Arrange Method - Center Custom (336x224)
Time = 8:54 (mm:ss)
Test File Size = 9,265,556

I expected the Third Process to produce a smaller test file size. :?
I ran these processes 4 times and came up with approximately the
same results. Finally, I ran avi scripts without SansGrip's filter's and the
results were similar to your findings. :) Not sure what to do next, since
my results are not dramatic enough in saving file size to use this process.
Maybe you can see something I am doing wrong. :)

-black prince

Hi black prince,

You should have "Full Screen", not "Full Screen (Keep aspect)".
I believe your first and second processes will look slightly smaller video than the third process, and that is the reason why the third file size is larger. Your third process aspect is the correct one. I can see clearly the second process file size is smaller than the first. So check your aspect on playback, and compare process one and process two and see if they look different to process three. I believe it will.

-kwag

I must admit i don't use Blockbuster myself, I tried it a few times with a 544x576 template (i don't use any other template) and i noticed no difference in quality - only a larger file size. I presumed it was only for lower resolution templates. In the cases i mentioned - i was only using Fluxsmooth.

I tried one more test (While you were sleeping), and I got pretty much the same results. Doing the resize in Tmpeg also takes a lot longer although it looked just as good. I suppose that really, it doesn't work for me so i'll carry on doing it the usual way. :(

Jim

Kwag,

You're right!! The Video Arrange Method should have been just "Full Screen"
for processes #1 and #2. The file sizes are 9,375,050 for #1 and
9,166,052 for #2 and 8,927,583 for #3. Seems at lower CQ_VBR there
is a greater file size difference between process #1 and #3. Now I have to
figure out how to automate this :D

thanks

-black prince

Hi Kwag,

Just finished my semi-automated process and the results are amazing :D
File size reduced enough so I could raise the CQ_VBR from 26.84 to
37.58. The picture quality has improved greatly and there is more room
for a higher audio bitrate. I paid special attention to FitCD resize of
336x192 and bilinearResize, Addborders are commented, since Tmpgenc
would now be resizing. I created a tmpgenc project file with all the fixed
settings and used "Autoit" to insert the variables that will change from
each encode (i.e. CQ_VBR, Clip Frame, and Video Arrange Method).
I tested clip frame for top, bot, left, and right masks and hardcoded the results
in my automated program. Now all I need to change is CQ_VBR and
when my file predictor recommends the new CQ_VBR. I also show the
percentage differance of Wanted vs Actual file size. I usually stop
within +-1% to 2%. I haven't tried KVCDx3, but that's next. :lol:
"Autoit" is very cool to work with once you learn to write macro scripts 8)

-black prince

Hey cool black prince :D
We're in sync today :lol: I just finished the movie "The Siege", a 116 minute film with the new KVCDx2 704x480 PLUS ( just for the hell of it :lol: ) to see how it came out. I was able to encode with a CQ_VBR of 8.5 which just barely made it quality wise. The video stream came out to 704,050KB, so I'm encoding now the audio at 128Kbps. The total will be 812MB. Looks pretty damn good for an action film and fitted on a single CD-R. Here's a cut-out of the actual .m1v : http://www.kvcd.net/clip.m1v
How do you like that for a 704x480 movie on a single CD-R :?:
Here's the script I used:

LoadPlugin("C:\encoding\MPEG2DEC.dll")
LoadPlugin("C:\encoding\fluxsmooth.dll")
LoadPlugin("C:\encoding\blockbuster.dll")
LoadPlugin("C:\encoding\legalclip.dll")
mpeg2source("K:\SIEGE\VIDEO_TS\siege.d2v")
LegalClip()
FluxSmooth()
Blockbuster( method="noise", detail_min=1, detail_max=10, variance=.5, cache=1024 ) # Apply noise if complexity is <= 10%.
LegalClip()

Masked and resized in TMPEG to 672 X 448 for the correct aspect.
This is way cool, I think I'm not going to do 352x240 resolutions anymore. If I can do this at 704x480, then a more complex movie I'll do at 528x480 (x3) or 352x480. :lol:

-kwag

Kwag,

The BlockBuster noise should not be present at the borders because the AddBorders line is AFTER the blockbuster stuff (in the script you posted).

The only thing after the AddBorders in your script is the LegalClip, maybe the TMPGEnc mask is an "illegal" black and that's what makes a difference.

Hi GFR,

But the TMPEG mask is already in the encoder, and that's applied after any external filters :!: , so that shouldn't be the case. If I remove Blockbuster "Noise" method, then the file size drops. So this is what causes the larger file sizes. If you don't apply the mask in TMPEG, with the Blockbuster "Noise" method the file is much larger, so the mask is cutting out something Blockbuster is feeding TMPEG.

-kwag

Hi Kwag,

Kwag wrote:
Try KVCDx3 (528x480) and post your results. I would love to fit most
movies on 1 CD using this template. :D It's seems to look better start-
ing at CQ_VBR 25. I got an extra 100MB with your process for the last
encode. 8)

Thanks

-black prince

sorry if you already mentioned this kwag, but which template do I use to get the quality you got with the m1v file?

That sample was encoded with the KVCDx2 704x480 PLUS and the methods mentioned here. As black prince said, with the x3, it will probably look better because the resolution is slightly lower so more bit rate is available to the movie. There won't be that much loss of sharpness dropping to 528x480, but for the same material, a higher CQ_VBR can be used and then the result is higher quality.

-kwag

Hi Kwag,

I've playing around with DCT-filter for compression without effecting
picture quality. Just wondered what your opinion was about using this
filter. :) Here the link for further discussion:

http://www.kvcd.net/forum/viewtopic....ht=compression

-black prince

Hi Kwag,

Is this a filter that works only with avisynth 2.5 only or can it work with
avisynth 2.07 :?:

-black prince

Hi Kwag,

For avisynth 2.07 which filter do I use? DCTFilter.zip or DCTFilter_YUY2.zip :?:

thanks

-black prince

Hey Kwag,

I used DCTFilter_YUY2 and the results were disappointing. :cry: The settings
used were DctFilter(1,1,1,1,1,1,.5,0). It had almost no effect in reducing
file size. FluxSmooth vs TemporalSmoother, FS() increased file size
while TS(1,2) reduced it. Blockbuster sharpen is an obvious no no. It
increased file size significantly. I was looking for a filter with good
compression (decrease file size) and has little or no effect on picture
quality. Do you have any other ideas to reduce file size :?:

-black prince

Hi black prince,

I haven't experimented with those filters yet. :roll: Will let you know as soon as I can test that.

Here's something else that's puzzling.
I need people to try something that doesn't make sense to me at all 8O
This is what I just tried. As you know, the new GOP is 1, 36, 3, 1, 36 which i theory, the file size will (should!) be smaller that using a smaller GOP. Right? WRONG 8O 8O How? Why?

I was tinkering with the GOP today, and I encoded a sample by accident with a GOP of 1, 26, 3, 1, 26 and the file size was smaller 8O
So I decided to do a test. I started to encode samples with max number of frames per GOPs from 6 to 24, and this is what I found. Using a CQ_VBR value of 11.34, the value I was using for a test encode with the x3, here are the results:

1,6,3,1,6 = 9,034KB
1,7,3,1,7 = 9,035KB
1,8,3,1,8 = 8,039KB
1,9,3,1,9 = 8,039KB
1,10,3,1,10 = 8,039KB
1,11,3,1,11 = 8,036KB
1,12,3,1,12 = 8,832KB
1,13,3,1,13 = 8,820KB
1,14,3,1,14 = 8,821KB
1,15,3,1,15 = 8,818KB <------- Optimal
1,16,3,1,16 = 9,442KB
1,18,3,1,18 = 9,444KB
1,24,3,1,24 = 9,539KB


So PLEASE, could some of you do some tests and see if you come up with the same results I have? If you do, and the results are the same, this means that the optimal GOP of 1,15,3,1,15 is the "Sweet Spot" for our Q. Matrix, which yields the smallest file size. Apparently because the number of B frames is 3, instead of the standard 2 used in the standard MPEG GOP. This is about 2MB smaller per 100 second sample, compared to the GOP we had before. This test was done with the masking method talked about in this thread and with the x3 template. I will try the same thing on the LBR and see if it works the same, or if there's another optimized GOP for that resolution. If these numbers are correct, this means a much higher CQ_VBR value than before, for the same amount of film.

Thanks,
-kwag

Are you sure Flux increased the file size? By definition a smoother should reduce it. Do you mean the file size reduction with Flux was less than with TS? That I could believe :).

Especially the borders, since they're as low-detail as you can get. I forgot to mention this -- you should always do AddBorders as one of the last filters in the script (definitely after Blockbuster).

I do it in this order:

Though you can smooth before resizing if you think that's better.

As far as TMPGEnc's internal resize producing smaller files, I'd say it must be because it results in a softer image even than Avisynth's bilinear. There's no other explanation that occurs to me right now.

Hi SansGrip,

I think it all had to do with the Blockbuster spraying noise on the black areas :lol:
When you mask the borders inside TMPEG, that makes black borders pitch black :D , and then the noise generation that is fed by Blockbuster via AviSynth is focused only inside the picure area. I can't really tell the difference between TMPEG's internal resize to AviSynth's bilinear resize as far as quality difference. At least on high resolutions like 704x480.

-kwag

I can't decide if this makes sense or not. Theoretically, of course, a big GOP means fewer I-frames which means a smaller final size. I have a hunch that these results might have something to do with the picture degradation introduced -- perhaps that many P- and B-frames has an adverse effect on motion estimation?

It will be very interesting to know if this is source- and/or resolution-dependent. I'm going to run some tests on American Pie right now :).

As does using AddBorders at the end of the script :). The advantage of cropping borders/resizing/filtering/adding borders all within Avisynth is that the filters only have to run on the actual picture area, which should speed things up.

Edit: How much a difference it makes depends on the source size and the final encoding size. If the source is 720x480 and the encode is 352x240 you'll notice a big speed drop if you don't resize before running smoothers, etc.. If the encode size is closer to the source, say 704x480, there won't be much difference at all.

Yes, that's absolutely right, but then why such a great file size difference when I use TMPEG's internal resize with masks :?: That's what I can't figure out. If it was 300 or 400KB, that fine, but the difference is between 1 to 1.5MB for a 100 second sample :?

Edit: Even at 704x480 there is a huge difference 8O

-kwag

Right now I'm running some max GOP tests. Could you try cropping the borders, resizing, filtering, and adding borders in Avisynth, and compare it to just cropping borders and filtering in Avisynth then resizing and adding borders in TMPGEnc?

I'd be interested to know the result of that specific test.

That really is enormous. You might have struck gold again here Kwag old boy :mrgreen:.

:mrgreen: :mrgreen: :mrgreen: Hey, you're the one who revolutionized the mpeg industry by pumping noise into the encoder to eliminate DCT blocks. I'm just giving your filters a haircut around the ears :lol: :lol: :lol:

Preliminary results
 

Encoding with KVCDx3 in TMPGEnc 2.58 non-plus. CQ_VBR 25. No masking/resizing in TMPGEnc.

Max GOP 15: 17.4mb
Max GOP 24: 16.3mb
Max GOP 36: 16.1mb

As you can see, with external resizing I obtain expected results, at least with this 1m25s clip. I'll try again with lower CQ_VBR and edit.

CQ_VBR 10:

Max GOP 15: 7.82mb
Max GOP 24: 7.91mb
Max GOP 36: 7.72mb

8O I'll leave analysis until later. Now I'm going to test internal resizing/letterboxing.

Max GOP 36, CQ_VBR 25:

External resize/letterbox: 16.1mb
Internal resize/letterbox: 16.3mb

:?:

Max GOP 36, CQ_VBR 10:

External: 7.72mb
Internal: 7.81mb

Can't duplicate your results here, kwag :?.