Gibbs effect - how to?
 

Hi guys,
I've been working around with 2 trilogies during the last couple of days.
One is Back to the Future that I had on one and a half KDVD and the other is Star Wars Eps IV, V and VI.
Actually Star Wars is not mine - it's my cousin's - but he has 2 little kids at home and he asked me to make a copy so that he can safely store the original pack.
That puts me working on a 3x movies PAL 16:9 Anamorphic 704x576 (with AC3 5.1 original audio) encodes to a 1x DVD-R.
My problem is that I'm catching a lot of - how shall I say - "ghost" effect or "ringing" effect on the edges of the characters and objects.
I'm not sure if you'll get what I mean with these words but that's about how I would describe the problem.
Appart from the resizing filters I have tried the regular Undot.Deen and Inc's ADS but no luck and scripting has never been my strong side :lol:.
So I was wondering if you guys cared to point me something new (or even old :)) that could give me some help on this matter.
I could go for a non-anamorphic 4:3 encodes but I would rather leave that as my last option.
TIA

Cheers

Can you post a snapshot ? I think it's necessary here.

@Rui

I think that the last Fupp's script version has a option for to treat with "ringing" effects ... Take a look at thread that announced the update of such script ... :D

See ya!

Hi Phil & Danpos,
Thanks for your suggestions.
@Danilo,
I had already forgotten about FuPP's script.
I'll search for it right away but I'dd like to go with something not as strong.

@Phil,
Since you ask for a picture, here's a picture of yourself :lol:.
I was trying other movies just to see if I could catch the same effect.
And here it is on Matrix 1.
Look at your head contour and also at your shoulders.
Can you see the ghosting that I was mentioning.
This is 704*576 only with resizing. No filters at all and a bitrate around 1500Kbits.
What do you think?
Cheers guys
http://www.digitalfaq.com/archives/i.../2005/05/1.png

This appears in hight frequencies (edges) and you can lower the effect with DCTFilter.

Note : this effect is due to too low (max ?) bitrate generally.

Hi Phil :),
I'll try DCTFilter and see what I get.
Never experienced this before which is rather strange.
I'm using HC 0.14 and I've tried HC 0.13 but the same happens.
But what even puzzles me the most is that I tried Tmpgenc and I still get the same results :roll:.
I am trying HC in OPV mode with 8000 set as my max bitrate and Q=7.100 and will incorporate DCTFilter after I read it's instructions.
I'll let you know how it goes after that.
TIA
Cheers

Use DCTFilters(1,1,1,1,1,1,0.5,0).

If this does not work, then proceed differently : raise the enthropy of the picture by adding noise (a common blockbuster line).

Or, new idea, use GausianResize (new in avs2.56) with a blur factor near 0.75 (I use it to replace lanczos4resize currently, with a factor at .90 but .75 is perhaps too blur).

Hi Phil :),
Unfortunately I still haven't found time to test your tips.
I have two questions, though:
1-What's the real name for the kind of effect I'm experiencing?
2-How in the name of the lord haven't I noticed this before?
For the 2nd question I think I have already found at least 1/2 an answer.
Last night I picked up my old KDVD (and even KVCD) encodings mostly done with Tmpgenc and NuEnc.
I raised the brightness on my TV to nearly the max and I could find something not as noticeable as what I've been experiencing but close enough to probably blame the colorspace issues I was noticing a few months ago.
That would mean I have always had this issues but never noticed them because the picture would be a little bit more darker than it should.
Nevertheless it's strange because I was quite sure I had never experienced colorspace issues at that time :roll:.
I'm sure you (Phil) can recall those pictures where you were saying that it was brighter and that I would say it was less brighter.
Well, tonight I'm installing AVS 2.5.6beta and I'll try all your tips to see where I can get.
Also I'll look at ColorMatrix() to check on the colorspace issues.
Thanks for your (and everybody else;-)) support on this matter.
Cheers

No name that I know. These are artifacts introduced by the rounding in math operation near the edges. Call them "artifacts" :-)

Your sources are "too clean". This artifacts come out when there is too much variation near the edges. The cleaner are the areas before and after the edge, the more you will see the artifact. That is what your noticed with your old KDVD.

That's Gibbs artifacts/noise. Appears in places where the contrast is very high (around sharp edges!) and the bitrate is not high enough. No filter can help you much, I'm afraid - the artifacts you see appear during compression.
There's your problem. You definitely won't have enough bits to keep the artifacts away. If you use CCE, you can try lowering the flat part priority value to something like 5-10 (quantizer characteristics in some version). However, it will probably make any flat parts look bad :?

Ok, now I know the name :).
But :arrow: I've been looking very closely at the sources and I can almost swear I can see the same artifacts but that's about it.
The key word is "almost".
So I'm starting to think that I shouldn't be surprised to see this effect on the encoded material too ;-).
Damn, I'm using HC so I guess I can't do anything else besides going for non-anamorphic, right?
Anyway, I'm still going to try Phil's hints and only then I'll take a decision.
Thanks for the help though.
Cheers

You can see the artifacts practically on every DVD if you look closely. It's the nature of the encoding process that causes it. Some commercial DVDs look like crap just because of that nasty ringing.
Yes, not much to do. Blurring the image reduces the effect (less sharp edges and also better compressibility) so bilinear resizing should help. Bilinear is usually a must when doing extreme compression as the difference to a sharp bicubic or Lanczos is huge.

I see you did not discover GausianRezise yet. Try it ;-) It's kinda resizer where you can change the blur strength.

And for Gibbs, I'm quite sure DCTFilters help also (too much filtering with this cause the edges to blur, so it should have an impact on these artifacts).

DCTFilter does pretty much the same as the quantizer matrix so in theory it might do the trick. With your settings, the higher frequencies are filtered and if I'm not entirely mistaken, that's where the Gibbs noise is. You'll just need to be careful, too strong DCTFilter will actually make the problem more apparent (easily tested with permanent subtitles on the video before DCTFilter).

I'll have to check the Gaussian resizing, sounds interesting. Funny how I didn't notice that it had been added, though I've been a bit busy lately.

Thats why I dropped DCTFilter, ...

It lowers the higher frequencies but anyway .... if high freqs will changed to block"itos" thes blocks also do appear to the encoder as edges. Pong.

So entering the Matrix would help. Like setting the last 2 collumns of notch to 99 at Intra and Inter. :)

But not exactly at that position :)
The problem is that the frequencies of Gibbs vary, and it will not be a fixed matrix position.
The ideal matrix would be a dynamic matrix, where it would take into consideration artifact ringing and other visual artifacts, and generate a correct value for the matrix weighting. That would be the perfect solution ;)

-kwag

Absolutely right Karl 8)
But I was refering to that dctfilter as its routines should be processed IN the encoding engine and NOT before using avisynth, and so if someone wants to kill high freqs in a static manner he better should cut them off using a "hi-freq-cut-off"-version of the used matrix.

But since I got my new Panasonic 32" CRT 16:9 TV Set, Im away from cutting high freqs. That baby is that sharp and detailed that I recognise a blurrier picture immediately. On the one hand :( but shurely on the other hand :D

OT on/
Mine costed about 600€ incl. shipping. A friend of mine got a Samsung 32" 16:9 LCD TV Set for about 2500€ and there's a clear difference. Sharpiness was the same, both where feeded by a non-composite RGB Signal Output of the SAP. But the Luma/Chroma depth was like if youre comparing a DIA Positive and a Photography. On the LCD Black wasnt really black and the difference from the darkest to less dark areas wasnt recognisable. And that Samsung got some very good reviews on that price range testet devices. So IF buying an LCD I think we still have to spend a lot of money IF quality in case of luma/Chroma will get into the range of a CRT ?!

OT off/

Here are my results keeping an avg of 1500Kbit/s.
DCTFilter with Phil's (and TRBarry's? ) recommended values: did a reasonable job on the Gibbs effect of the source but :arrow: it can still be clearly seen on the encoded picture.
GaussResize: tried severall p values from 70 to 50. On the source it did a reasonable job although it bluried too much the picture for my taste.
As for the encoded stream, it looked good with p=70 where the pict didn't look too bluried and the Gibbs was less noticeable but :arrow: still noticeable enough for my taste.
Encoded stream with <60 looked already toooo much bluried and Gibbs was hardly detectable.
This means that apparently, and unless I didn't try hard enough, Boulder was probably right when he said that I will probably have to live with Gibbs.
What I think he wasn't so much right on was when he said that if I raise the bitrate this effect will go away.
Well I tried an avg of 4500Kbit on a 704x576 anamorphic of The Matrix with simple script and I still can clearly see the Gibbs effect.
If nobody has anyother options I'm droping this issue as unsolved.
I only find this strange because noone else complains and I know many of you have similar taste as you also like to put 2 (or 3) 16:9 anamorphic movies in a DVD-R with original AC3 5.1 sound.
We could say that this is only happening because I'm low on bitrate but as I've said it happened with avg 4500Kbit's.
Anyway, it's been educational ;-)
Thanks for your hints.
Cheers

You have one last things to test :
This has few to do with your goal but with your sources. As you had see, even with a high bitrate you have the artifacts.

Note: want to remove them ? Do DVD -> Divx -> KVCD. Only for educational purposes. You will btw discover how and why divx codec screw up the picture.

Hi Phil :),
I'll add some noise with the blockbuster to see what I can get.
Nevertheless, if it's visible in the source (high quality source as The Matrix) then it's bound to be present in the encoded clip, right?
I have managed to minimize the problem using Andrej's ADS with very low sharpness and quite a high figure for the softening (1,8).
This way I still see the Gibbs but it is reasonably disguised ;-).
As for thr DVD -> DivX -> KVCD I don't think I'm prepared to do that :).
Even because afterwards I wouldn't be allowed to share the experience with the others here :mrgreen:.
Cheers

Rui,

if you want to get rid of ringing or gibbs artifacts in sources you could do the following ... (just for explaing I do use an "explaining-syntax")

input = YourSource(hagdjsgdjahgj.xxx)
mask1 = Edgemask(input, highthreshold)
mask2 = Edgemask(input, lowerthreshold)
maskaroundegdes = subtract the highthresh mask from the lowthreshmask
filtered = Spatialfilter(input)
MaskedMerge(input,filtered, maskaroundedges)

So what does that logic above do.

The "maskaroundegdes" should look like a bold silhouette around (not ON!) the edges where normally the gibbs/ringing does appear. This mask will be used to only put THAT by a spatial filter treatened Area back ONTO the input stream.

So u have to look into the documentation of masktools where Edgemasks can be produced using threshold and also these masks can be modded by using expand and inpand commands etc. Edgemask() can be used by "sobel", "Laplace" etc masks and its quite fast. But the best masks can be archived by using a "double soble" like didée uses them in his functions.
THAT DoubleSoble mask you can also find in the masking section af ADS() remember the HQ mask parameter? Where Logic() does generate the double sobel out of two DeEdgemask() generated Sobel 3x3 Convolution/Kernels.

Hi Andrej :),
Thank you so much for the explanation.
I am sorry to say that my scripting capabilities are not enough to understand what you explained :(.
When it comes to avs scripting I'm a complete noob :oops:.
I also did use the ADS with HQmask=true and I got similar results from those I obtained with HQmask=false plus treshold=8.
Unfortunately the HQ feature slows down the encoding a lot.
But I guess I could use it on the worst cases/sources that I find.
Also tried to add noise with Blockbuster but unfortunately it added too many grain to the picture in order to get rid of the Gibbs.
I would use the EdgeMask if I really knew how to use/tweak it.
In that case should I add it to the script after or before ADS?
Say, you're not planning on enhancing the ADS function to incorporate such a feature, are you :lol:?
Just kidding ;-).
But I'd like you to think about putting something similar on the "To do" list for ADS :).
Thanks for your help, though.

Cheers

ADS() does NOT mask areas AROUND edges but the edges theirselfes! and so also the plain areas. So the masks out of ADS() wont serve you as there are no two of them which will be subtracted.


The workaround/explanation above in pictures:

First the image:

http://www.digitalfaq.com/archives/i.../2005/05/6.jpg


Now we build a mask using a low Threshold

http://www.digitalfaq.com/archives/i.../2005/05/1.gif


... and a one using a high Threshold

http://www.digitalfaq.com/archives/i.../2005/05/2.gif


Now we do subtract the Low- from the High-mask

http://www.digitalfaq.com/archives/i.../2005/05/3.gif


.... and these White parts in the mask are "theoretically" the areas where the gibbs or ringings primary do appear. So we do let work the spatial- or whatever filter on THESE white masked parts only.

:)

Hi Andrej :)
This a fantastic and ilustrated demonstartion of what you had told me to do in your previous post.
This, I had more or less understood already.
But it is the scripting part that I lack knowledge on how to do what you just showed.
Could you show a "real" script for such an example?
Then I would just have to choose the spatial filter that I most like to do the dirty job.
You really don't have to do so.
By now I should already had learnt a little bit more than I have about avisynth filtering...
One odd thing is that you said ADS can't help me but in fact it can :roll:.
Well to be honest it's killing me a lot of details in the picture but Gibbs is hardly noticed that way.
One day, just one day, I'll learn to script and I won't be bugging you guys for help anymore :lol:.
Cheers

Excelent sample and explanation Andrej :D

-kwag

I do agree with you, Karl ! Andrej's explanations were very precise and technically perfect ! Kudos for him ! :D

See ya !!

Ahhhh ... now licking blood. :)
The meal is called "masktools", an avisynth dll from manao, also used by HybridFupp, ADS, iip, limitedsharpen, etc.

First: ADS() may help on the first look AS it keeps edges and denoises all around the edges (weel thats the purpose, like hybrid fupp) means the less detailed areas, so that would be logically done using the fine mask above. BUT we JUST WANT to treat the areas AROUND edges and not the plain areas (and shurely not edges/details). ADS() is very sensible in killing fewer recognisable details as these very fast do drop into the plain (black) area of the mask. So EVERYTHING depends on HOW you fine-tune the mask, like using levels() etc.

Second: (like explained in the last sentence of the point above) ... THESE explanation masks above are NO reference how a good mask should look!
These are builded in a fast/dirty way just for explaining. Good masks (like the double sobel approach from didée) are MORE detail keeping, in comparison the ones above, as these would kill too much details, means the man above would be totally shaved! :rotf:

:)

Now I really thinking to change my Phillipshave for this new Masktoolshave. 8O

Thats good Inc, thats really a good explanation!

How about a script example?

ok rui :)

The following script is a fast done, means no big testing and so on. The masks really need some optimizations, but youll see whats possible.
Caution: That script IS slow ... as a double sobel mask building via Logic() needs CPU power ;)

First well have a look at an original mpeg source using too much compression where ugly gibbs and artifacts do occur (that picture is scaled by 200%!!)

Original:

http://www.digitalfaq.com/archives/i.../2005/05/7.jpg


Now we do apply the degibbs routine:

http://www.digitalfaq.com/archives/i.../2005/05/8.jpg


And finally the degibbs routine plus a general Blindpp() BUT cpu=4 used (no deringing as it blurs too much the edges)

http://www.digitalfaq.com/archives/i.../2005/05/9.jpg



I think that fast approach already shows whats possible :)


The script:

Masktools is needed:
http://manao4.free.fr/MaskTools-v1.5.6.zip
Masktools explained:
http://manao4.free.fr/MaskTools.htm
Removegrain needed (in my filtering):
www.removegrain.de.tf

Great stuff Andrej :D
I will try this tomorrow and let you know how it goes.
I can't thank you enough for helping me with this Gibbs effect.
Cheers buddy.

@All

In this SITE there is comments and java applets about Gibbs Effect and topics related. Just for cultural general purposes ... 8)

Cheers,

Hi guys :),
Sorry for taking so long.
I've finished my tests.
It just took me a very long time because I wanted to test it under many different conditions and at least with more than one encoder.
Plus I wanted to "experiment" a little with it, and as I've emphasized here already I am very bad at scripting :lol:.
To cut to the point, Andrej's script is just like he presented it to us "slow and not at final stage!"
But it sure shows us a different approach to the problem from many other people that I've shared my problem and that have given their oppinion.
Not that other people gave me bad advise, on the contrary :!: ;-)
It's just that most have pointed in the same direction, which BTW is the one that I will use.
Well, at least untill Andrej doesn't work a bit more on some issues.
I'm not going to present you with a lot of info from all my testings but I surely will give you a pattern based on my last trials using HC 0.14.
The movie I used for this testing was The Matrix.
The script used was very similar to Andrej's DeGibbs script but I added a Slicer on it for 1% which gave me 1980 frames to encode.
As said I used HC 0.14 in OPV mode with a q=5.575 (don't ask me why :lol:).
Usually without the scrip it would take me 2 and a half minutes to encode the mini clip but with DeGibbs - what I nice name, huh? - it took me more than 30 minutes to encode.
But that's not the point here!
My problem was not about speed, it was about quality.
So I opened both outputs (with and without DeGibbs) in VDubMod and I noticed how it only touched the hard edges in the picture.
It is very nice that all the rest remains pretty much like the real thing because I checked without the script.
Then I noticed how the ringing was all gone - it works :!: :lol:
So I browsed the whole clip and then I started to notice some strange artifacts - would they be named microblocs?
Especially when a very dark color meets a medium bright color - e.g. a brown dark hair near the color of human skin.
There I noticed something very strange at 150%.
So I amplified it to 200% and there it was!
Tiny little blocks bluring the picture.
I mean I could hardly tell that I was looking at the hair contour near somebody's hears.
Without zooming in it looks strange but it isn't much noticable.
As said I played with the toy for a long time.
Used different encoders but the result was very similar.
To be honest I rather stay with some ringing on 10% of the movie but still have very nice and detailed picture.
But :arrow: I bet my money as Andrej can improove this baby a lot 8)
Maybe in some months I'll be telling a different story.
Oh, I almost forgot.
I've found that the Gibbs is intimately related to grainy/noisy sources that use high bitrates >=6000Kbit.
I have some movies where I notice it a lot and I have some (more softened) movies where it is almost impossible to notice it.
Bottom line1: keep using a small touch of DCTFilter which actually helps a bit to cut the gibbs and it slmost doesn't hurt the encoding speed.
Bottom line2: keep bugging Andrej so that he improoves DeGibbs over the next couple of months :lol:.
Thanks everybody for all your help.
Cheers

He he, .... u should test Didées ipp script ... THATs slow :lol: but effective!
If youre going into adaptive (masked) selections it could result in a very slow encoding, BUT as said above a use of "Logic(....)" makes things very slow ... but the positive sideeffect is that the mask is much more precise so you will loose less details. But maybe a simple mask generating is also ok, so Ill take a look at it these days. (maybe you also can take a look into Hybridfupp as fupp also generates masks in his script which can be very useful for our needs. So taking an edgemask from hybridfupp, expanding it and subtracting the edgemask from the expanded version of the same edgemask. Thats the approach :)


Hmmmmm .... I have no "Vision" ... I only can assume,
So please do upload via Vdub safed Images in jpeg (90% Quality - so no recognisable artifacts will be that added) using Imageshack.com

Refering to Artifacts like blocks: .... IF I do build a fading surface from dark to medium dark in Photoshop ... it even there happens that artifacts do occur no matter how much Colors your system is settet to display.
That is during incorrect math or whatever. These artifacts will be blown away IF you add a tiny minimal bit of noise ... just for the applications internal algorhythm.

And that has been so often done also in here ... applying a minimal blockbuster() so the encoding wont rise that much in its bits but less blocks on dark surfaces will be immideately the result - its worth!

:)

Hmmmm .... dont use DCTfilter on sources which do result in Problems like you mentioned in you last point! Applying that one results on grainy sources in minimal microblocks at high freqs - maybe not recognisable but if that output out of avisynth again apsses a DCT in your encoder ---- well Microblocks will get blocks and so on.
Im not shure if DCTfilter is coded properly as many issues have been mentioned in here ... maybe not comparable to your issue but ... maybe an indicate that DCTFilter has to be used with caution?!

DCTFIlter does not do anything proper! against an existing ringing in the source as DCTfilter touches ALL high freqs (depends on the integers you set to zero) .. and gibbs are mostly no highest details but up to 2-5px bold ugly trashborders around edges.

Unfortunately (is that unfortune!?) I've never had to deal with Gibbs'y sources except DVB-T captures, but these are that quantizated that such a function just improves 20% of 100% that have to be improved 8O :)
ipp from Didée often is used to build a good! looking anamorph source from a non-anamorph letterboxed 4:3 source .. IF the source is in a good state like DVB-S high bitrate ones or DVD ones (like my Titanic - a cheap mastering of a 2.35:1 non! anamorph *grrrrrrrrrr ) or you can use it just to improve DVD sources no matter wich AR. Also limitedsharpen brings out details that you cant beleive ;)



I also found this one on doom9 by Krieger .... I never tried it or get into it so I dont know what dlls are needed etc.

There was already an approach by mf, have a look at it ....

mf's functions can be found here:
http://mf.creations.nl/avs/functions/