Likely something to do with the decompression algorithm used. WinDVD's might be more "correct" than Zoom's (or vice versa ;)). Either that or WinDVD applies some postprocessing to reduce artifacts.

:idea: A post processing filter. Call it "MpegSynth" filter. AviSynth -->> TMPEG -->> PostFilter( Sort of a system pipe, FIFO, etc. ) -->> Actual MPEG file :idea: :wink:

-kwag

I don't think that would be any simpler than writing our own encoder ;).

:mrgreen: :mrgreen: :mrgreen:

Ok, here's what I've been working on for the last five hours and the results are surprising. First, the relevent information:

x3 template
528x480
11.4mb clips

Here's the framegrab:

http://www.digitalfaq.com/archives/error.gif

First you'll notice six clips in the grab, labelled appropriately. On the top row are three encodes made with CQ_VBR, and on the bottom three made with CQ. The leftmost encodes were made with a Blockbuster variance of zero (i.e. no noise), the centre with one, and the rightmost with two.

While you can see a clear reduction in the number of visible DCT blocks as more noise is added to the CQ_VBR versions (with a corresponding reduction in CQ_VBR level), there is very little difference in the CQ versions (and very little difference in the CQ level). The CQ encode with noise variance of two shows as many blocks, almost as clearly defined, as both the CQ_VBR and CQ encodes with no noise added.

Based on these results, along with my encodes with Resident Evil before my "Christmas break" :), I conclude that Blockbuster is ineffective when using CQ mode, and that in general CQ mode with or without Blockuster is significantly blockier than CQ_VBR with Blockbuster.

One possible reason for this behaviour is the current Q matrix, which might be very suitable for CQ_VBR mode but too strong for CQ mode in the low-frequency components.

An additional observation I made is that at least with this source material adding noise (and reducing the CQ_VBR level) had very little or no noticible effect on the amount of Gibbs in the clip.

First of all "DANGER WILL ROBINSON". Here's a little teaser. I've been doing some analysis on the low frequency domain of the Q. Matrix. Here's my first attempt, with a different approach from any existing matrix. I've tried to tune a specific area, the lowest lit areas on MPEG files. This is mostly close to black (dark gray) areas and/or low frequency components, where you have your "Belly dancer" DCT blocks. What I'm doing is notching out this frequency domain with very low quantization values. So frequencies below and above this point will be transformed with the standard KVCD quantisizer values. The standard KVCD Q. Matrix has these values in the upper left corner, which represent the low frequency components.
And I am currently testing:
And here's the result "blown up" to enhance details and crap :D :

http://www.digitalfaq.com/archives/i.../2002/12/1.png

See how many things you can find in the top picture that are smoothed on the bottom :wink:
There's still a long way to go before this is fully optimized, but I can see the light at the end of the tunnel :wink:
This was encoded at 352x240 using this script:

LoadPlugin("C:\encoding\MPEG2DEC.dll")
LoadPlugin("C:\encoding\sampler.dll")
LoadPlugin("C:\encoding\fluxsmooth.dll")
LoadPlugin("C:\encoding\blockbuster.dll")
LoadPlugin("C:\encoding\legalclip.dll")
mpeg2source("K:\KPAX\VIDEO_TS\movie.d2v")
LegalClip()
BilinearResize(672,336,0,0,720,480)
FluxSmooth()
Blockbuster( method="noise", variance=.4, seed=1 )
AddBorders(16,72,16,72)
LegalClip()

-kwag

I can't really see any difference in those areas :?. However, I do notice a significant reduction in the number of "striped" DCT blocks with your experimental matrix. Can you post another grab, preferably one that's very static? I wouldn't want to get my macroblocks and DCT blocks mixed up :).

:lol: Ok.
The background is static, only she is moving on this scene.
I'm looking for another dark material to encode. Probably from a dark scene on "K-Pax". But take a look on the right side of the picture, there are some small vertical DCT blocks that are clear on top, and they don't show on the bottom. On top of the hole that looks like a handle to open a door. Sorry I didn't circle in red the areas, because the program I used ( Canvas 7 ), I don't know how to use :twisted: :oops:

-kwag

Yep, those are the ones I was talking about. When I save each image as a separate file and switch quickly between them I can see a nice overall smoothing of the background, with most "jarring" artefacts gone. You're definitely on the right track :).

But this is painful 8O Encode, change, encode, change, encode, change, encode, encode ..... :roll:
Eventually we'll get it right :D ( I hope 8O )
I wish I knew EXACTLY what fequency range corresponds to each matrix block. Then it would be much simpler :roll: Maybe if I encode a "White Screen" with "White Noise" using your blockbuster set to a high value so that I saturate the complete frequency spectrum and then change the matrix to all 0's except lower left top to 99 and keep rolling that "99" to each position, I can create a frequency histogram :idea: . Then we'll have a pretty good idea of what frequency corresponds to each block of the matrix :idea:
Maybe I should do that right now :mrgreen:

-kwag

i hate to interrupt the brainstorming but....
CQ or CQ_VBR?

SansGrip, the screenshots you posted show that CQ-VBR mode with Blockbuster variance=2 (with the current GOP and q. matrix) is the way to go! :) To me the CQ-VBR 10.2 var 2 sample looks much better than the others.

@kwag, I blew up the images you posted with Photoshop and the only difference I can see is the left side of her (Trinity?) neck- is more smooth. I can't see the block on the right side- could you circle it?

thanks a lot,
ren

When I learn to do a circle with this program, I will :D
Take a look at the hole to her left. Looks like a handle to open some cabinet. Right above the hole you see some artifacts that are not visible on the lower screenshot. This is still experimental, and I'm probably gettiing closer to the frequency domain of our target, but nowhere near yet. So you're just seeing only a very slight improvement of what I think can be accomplished once this is finaly tuned and tweaked :wink:

-kwag

I'm uploading the picture again at a higher resolution.

-kwag

I see this is going the right way now... :wink:
If I can help in any way, just let me know! I'm on holiday right now, so lots of time for testing 8)

Hey Kwag,

Kwag wrote:
If these screen shots are jpeg's or bmp's then use windows paint. :)
Choose ellipes with the box that's not shaded. Select red from the palet
on tool bar and hold down right mouse key and drag until the area you
want is selected. :)

-black prince

Me too, but the same may not apply at 704x480. I'd need to run the same test at that resolution before I could be certain.

As for the noise, remember that the variance will change depending on the source material. For this source I would say that the var 2 sample might have a little too much noise -- I think I'd probably try 1.5 as well.

You see black prince,

I feel like an idiot :D
Where's that little IDIOT icon so I can paste it to myself :lol:
All the time I'm thinking on Photoshop or Canvas etc.
And it was so easy to just use Windows Paint :P
I'll do that right now :!:
( And I might also upload a little idiot icon, so I can use it on these special occasions :idea: :lol: )

-kwag

I'm sure a search on "DCT quantization matrix" would turn up the info you require. It might make your head ache though ;).

Don't use Blockbuster for applications that require real random noise. Use my Noise Generators package. They're much slower but much more "random".

Not sure that noise would produce this. What you need is one of those test patterns that you get on TV calibration discs, where you have many vertical lines of differing widths and separated by differing amounts.

Hi All,

ALERT!! ALERT!!. 8O Be careful about using overburn with CQ encodes.
I just encoded a video for 2 CD's and used bbMPEG to multiplex. I set
the cut point at 794MB which produce an mpeg of 805MB and another
mpeg of 715MB. I set VCDEasy to overburn. The first disc (805MB)
played fine until it reached the overburn area and then the picture
broke-up and the movie just stopped playing. The second played fine
without any problems. The picture was perfect in both (same as DVD) but,
appears darker than DVD. I didn't use Blockbuster noise. Still
having trouble with file prediction for CQ >= 80. Tmpgenc seems to
increase file size dramatically for CQ above 80. For now I'm using a
range of 50 to 78 for CQ for 2 CD's and it still looks perfect :D

-black prince

Ok, go back to the previous page and look again. That's a darker sample, so you can see the results better. It's still only a very slight improvement, but it's there, and it can be optimized.

-kwag

It's definitely getting rid of most of those "striped" artifacts, and so is an improvement over the regular KVCD matrix for sure.

Do you think there is a way we could change the Q matrix so that Blockbuster has an effect on CQ mode? For me, CQ mode is not an option if it doesn't allow me to reduce blockiness, since those are the artifacts I find most annoying...

The ideal solution would be CQ mode with a modification to the Q matrix so that we can dither with Blockbuster if necessary, since CQ mode seems far better at keeping Gibbs under control than CQ_VBR.

Could it be that internally in TMPEG, it uses some sort of "High Pass" filter on CQ mode :?: . Then only frequencies above a treshold are beind encoded, and that's why Blockbuster doesn't have the same effect :?:
This is something I thought last night after I saw your samples :roll: . If this is the case, we're fried :!:. Have you tried your noise generator filter and see if you see the noise with CQ as it is seen with CQ_VBR :idea:

-kwag

I'm getting promising results with some Q matrix modifications. I'll let you know in a while :).

Looks like 2003 will bring us newer levels of MPEG-1 quality :mrgreen:

Ok, here are the Q matrices I've come up with through some testing of CQ mode with noise variance of 2:

The rest of each matrix remains the same. Here it is in TMPGEnc.ini format, which is probably more convenient:

Of course, this may not be your fifth entry so some care will need to be taken when pasting it into the INI file :).

With this Q matrix I notice a significant improvement in blockiness when used with CQ and Blockbuster variance=2, yet a slightly lower file size compared with CQ_VBR.

I'm not sure what other effects this change might have, for example when not adding noise at all. I'd be very grateful if people could test this one out and tell me what they think.

Starting to test right now :D

I'm on the edge of my seat. It's very uncomfortable. Hurry! :mrgreen:

Quick update: This matrix produces significantly higher file sizes than the KVCD matrix when used without noise. I'm starting to think that CQ mode has an optimal matrix depending on whether or not noise is added :?.

Edit: Though it is also less blocky. Less blocky == larger file size. :?

I'm currently trying something different...

Edit 2: At least, I was. Apparently I'm to go to the mall with my family, so I can "spend some time with them" 8O :? ;).

SansGrip Quote: "Do you think there is a way we could change the Q matrix so that Blockbuster has an effect on CQ mode? For me, CQ mode is not an option if it doesn't allow me to reduce blockiness, since those are the artifacts I find most annoying... "

:? I see the artifacts you are talking about in your samples. I'd highly recommend though that you do tests with higher resolutions for this purpose...

here's what you should look at SansGrip:

Try at 704x480:

CQ mode with and without blockbuster noise using a variance of one.
CQ_VBR mode with and without blockbuster noise using the standard value for variance.

compare these results. in my case they speak louder than words for CQ with blockbuster variance 1.
anyway I'm sure a new matrix will solve the little side effects for CQ...

CQ_VBR might be better for low resolutions though...

Sorry for the late reply :( but my DSL line ( F#%@ Verizon ) is down. Nice!, it had to go down today, Saturday. So we'll probably be down till monday! "DAMN" Here I am on a 56K dialup, and it hurts. So I won't be able to post any samples, and only limited browsing DAMN!
Sorry to keep you on the edge of the chair for so long :mrgreen:

Edit: Were back on DSL :wink:

-kwag

@SansGrip,

Well, I tested your changes, but my result is the other way around 8O
The top image is with "SG 1" matrix changes.
Here's what I mean:

http://www.digitalfaq.com/archives/i.../2002/12/2.png

Also note the grass(or weeds) just barely touching the shoulders. You'll see the bottom image is slightly more dithered.

Edit: Only used "LegalClip" on this one. No other filters were used.

-kwag

Here's another one, with even more tweaks on the matrix. This one 704x480. I'll let you guys do the red squares on this one :wink:

http://www.digitalfaq.com/archives/i.../2002/12/3.png

Here's the current test matrix:

I'm leaving Non Intra as default values. Only concentrated on Intra section. The file size is "barely" larger than the original KVCD matrix.

I'll try this now at 352x240 and post results.

-kwag

Here ya go, really blown up :mrgreen:

http://www.digitalfaq.com/archives/error.gif

Edit: The CQ used here, 25, is really low for a regular picture. I used 25 to force the low frequency areas to be blocky, and make the results and test more visible.

-kwag

Hey Kwag,

The KVCD test Matrix 704x480 Notched Filter picture is clearly better :?
I seems the lower resolutions 352x240 don't show as much improve-
ment as SansGrip suggested. I already get great encodes at this
resolution (704x480). I'd be curious to see how much improvement
this would show with a short clip of a fire, water and fast action scenes.
Gibbs effect has vanished at higher CQ's but would this test Matrix
improve it at lower CQ values. :?: I see you still have more tweak-
ing to do, but this looks very promising. All you needed was for
SansGrip to come back and encourage you on :mrgreen: I could
tell you missed his insight and testing prowess. :mrgreen:

-black prince

You all encourage us to push all this stuff to the limits. And the "NAH" sayers at other sites encourage us to go beyond the standards and over the limits ( which then pushes them to oblivion :mrgreen: )

-kwag

:D
really,really,really!
Kwag,we are with you!
:)

Thanks jorel :D
Hopefully we'll keep optimizing MPEG-1 to a point where we drive DivX nuts :mrgreen:
I'll post some screenshots on that tomorrow, where you can see MPEG-4's advantage ONLY on low bit rate and low action scenes, and you can see KVCD MPEG-1's CLEAR advantage on high motion scenes :wink: ( That is, with both files about the same size :wink: )
And when you jack up the bitrate on both, and target a movie for 2 CDs, I really can't tell the difference anymore :mrgreen: ( Only that I can play my KVCD in my standalone, but not a DivX 8) )

-kwag

:!: looking good :D

Hi Kwag,

I’m just teasing :mrgreen: Don’t malign Divx too much. File prediction is now an important
part of the KVCD process, but remember when the idea started with Ozzie’s very long
avs script using Trim, then he cames up with SelectRangeEvery(). You’ll be interested
to know he got it from Divx’s compression test. I believe you are raising the bar for
picture quality for all methods of video backup to reach and that’s what’s important to
me. I used Gknot for some time and created a lot of Divx’s only to be played on my
PC. I wanted more portability in playing movies with the best picture quality possible.
KVCD has not disappointed me and others. You are reaching for picture quality
that could very well become the standard for others to reach, not just Divx. I wish I had skills
like you, SansGrip, and other technical developers, but I know enough to realize what’s
happening at KVCD is very promising. :D

-black prince

Remember that I developed the matrix in conjunction with Blockbuster, variance of one or two. My reason for doing this was not so much to come up with good matrices for use with CQ mode and Blockbuster, but to show that it's going to be pretty difficult (if not impossible) to come up with matrices that are optimal for all circumstances, at least with CQ mode.

At the moment for me it's looking like CQ mode is better for high resolutions, provided by "better" you mean "less Gibbs" and not "fewer blocks". Perhaps the reason we're getting less Gibbs is because it's essentially ignoring that Blockbuster noise we're adding, and so can spend a lot more bits on the high-freq components?

It is somewhat discouraging that I see great improvements in low-freq quantization using my matrices providing I use Blockbuster, but no improvement (or even a reduction in quality) when not using it. This seems to make CQ mode much more unpredictable than CQ_VBR.