digitalFAQ.com Forums [Archives]

digitalFAQ.com Forums [Archives] (http://www.digitalfaq.com/archives/)
-   Avisynth Scripting (http://www.digitalfaq.com/archives/avisynth/)
-   -   Avisynth: RemoveGrain + RemoveDirt (http://www.digitalfaq.com/archives/avisynth/12099-avisynth-removegrain-removedirt.html)

Dialhot 09-13-2004 04:01 AM

RemoveGrain + RemoveDirt
 
I finally did it !

"Little big" tests campagin on my PC this we. I decide to review my KDVD script and give a shot to RemoveDirt and RemoveGrain.

Fist, after having read the manuals, I decided to use them both at the same time because it is said the RemoveGrain is designed to be a pre-filter before RemoveDirt.

First of all, the test scripts :
Test1
Code:

Mpeg2Source("D:\Mes Ripps\ES_107252\Insurrection.d2v")
FieldDeinterlace()

Test2
Code:

Mpeg2Source("D:\Mes Ripps\ES_107252\Insurrection.d2v")
FieldDeinterlace()
Undot()
Deen()

Test3
Code:

Mpeg2Source("D:\Mes Ripps\ES_107252\Insurrection.d2v")
FieldDeinterlace()
DRemoveGrain()
RemoveDirt()

Test4
Code:

Mpeg2Source("D:\Mes Ripps\ES_107252\Insurrection.d2v")
FieldDeinterlace()
DRemoveGrain(mode=8)
RemoveDirt()

Test5
Code:

Mpeg2Source("D:\Mes Ripps\ES_107252\Insurrection.d2v",cpu=4,idct=7)
FieldDeinterlace()
DRemoveGrain(mode=8)
RemoveDirt()

Test6
Code:

Mpeg2Source("D:\Mes Ripps\ES_107252\Insurrection.d2v",idct=7)
FieldDeinterlace()
DRemoveGrain(mode=8)
RemoveDirt()

Note: DRemoveGrain is the SSE2 version of RemoveGrain.

All test were done using CCE. The source material is interlaced but I do not care about the deinterlace job, even if FielDeinterlace is probably not the best way.

The results (Final Video Size / Encoding speed) :

Test1 : 140343 / 1.16
Test2 : 130509 / 0.77
Test3 : 132065 / 0.93
Test4 : 130573 / 0.91
Test5 : 127537 / 0.78
Test6 : 130350 / 0.93

Comments :
- with Test2 and Test4 we can see that RemoveGrain/RemoveDirt is way faster than Undot/Deen for a final size identical.
- mode=2 of RemoveGrain is less efficient than mode=8 (Test3 and Test4)
- I did not test mode=4 (the most efficient according to the manual) because it made avisynth to crash anytime !

Quality results :
Test2 is definitely the worst of all results (except Test1 that does not use any filters). So I guess I will say "bye-bye" to undot/deen. Rip :lol:
I personnaly prefer Test5 but some will say that using Mpeg2Dec3 PP plus the filters is too much. I don't think so and we have a size gain of 3% !

Unfortunally, for usage with DVD-Rebuild only "Idct=7" is usable but not "cpu=4" so the results will be the ones of Test6.

Next step : (finally) change my V4 script for Divx sources :-)

incredible 09-13-2004 04:21 AM

Yep! The RemoveX series from Kassandro is a big shot!! :)

But watch out as A value of 8 in removegrain will "round" details!
You can see that in case of captures on the top at the station logo!

So a DVD often only needs a smaller value ;-)


OT/
Sorry to say that (and maybe to get annoying), but I do not understand why you are doing your tests on fieldblended NTSCtoPAL streams! As I also got the same DVD/Extras where you are doing your tests on.

The source is not only interlaced. It has been converted from NTSC Hybrid to PAL like in that way:

InputSource29.97
Bob()
ConvertFPS(50)
Separatefields().Selectevery(4,1,2).weave()

So even a restore24() script to recover the 23.976 progressive fps wont work as the source is HYBRID (mixed video interviews and film movie parts).

Why do I say that? Because on fieldblended streams you wont see the clear power of a progressive intended filter! And as we both do our tests on relaworld conditions, I could assume that we got the same intention.
The correct temporal filtering estimations of the filter wont work correct as almost every second frame gots a very different Frame Picture architecture information (you know what I mean by saying that) as almost every second frame is BLENDED, means softened and edge distorted. ;-)

Why not taking exactly that main movie and doing a parts selected trimmed/slice/SelectrangeEvery or whatever method to obtain a got mixture of a testing sample

/OT off

My intention of these sentences is that (maybe) I can directly participate these tests as you can remember we tried to find a way to find equal sources for "online" testing workouts ---- And Id like to ! :)

Dialhot 09-13-2004 04:39 AM

Quote:

Originally Posted by incredible
Sorry to say that (and maybe to get annoying), but I do not understand why you are doing your tests on fieldblended NTSCtoPAL streams!

Because, as I explained : "I do not care about the deinterlace job, even if FielDeinterlace is probably not the best way" (I typed the sentence espacially for you :-)).
But if I can remember it, I will use your suggestion for my future tests.
(even if I don't think that can modify the order of merit of tested scripts as they are all working of the same "badly deinterlaced" source).

jorel 09-13-2004 04:51 AM

magnific post Phil! 8)

yesterday ink give me recomendations to test
RemoveGrain and RemoveDirt
posting the links to read and download that files!

now i see that test 5 is your choice and confirm the ink's recomendations.
i don't had time to do my own tests!

now i know what i have to use.
what was good yesterday is bad today.
( i'm talking about undot/deen and my ex-wife) :lol:

thanks so much Phil, ink and to Boulder for suggested to use it first.
:D

Dialhot 09-13-2004 05:01 AM

Quote:

Originally Posted by jorel
now i see that test 5 is your choice and confirm the ink's recomendations.

Note that in his answer, Inc suggested than "mode=8" that I used in my test5 script is not always the best idea.
Maybe you should use "mode=2" anytime (in the manual that is said that mode=8 can cause artifacts)

incredible 09-13-2004 05:15 AM

Quote:

Originally Posted by Dialhot
Because, as I explained : "I do not care about the deinterlace job, even if FielDeinterlace is probably not the best way" (I typed the sentence espacially for you :-)).

I repeat the explanation of why blended frames do harm on temporal correct filtering :) :
Quote:

The correct temporal filtering estimations of the filter wont work correct as almost every second frame gots a very different Frame Picture architecture information (you know what I mean by saying that) as almost every second frame is BLENDED, means softened and edge distorted.
Quote:

Originally Posted by Phil
But if I can remember it, I will use your suggestion for my future tests.

Thats the way we should do it ;-)

Related to Removegrain Modes: I did read that mode1 or 2 is the same like undot() .... hmmm ... I think Ive to go into the readms again 8O :)

stephanV 09-13-2004 06:02 AM

If the source is interlaced and you are using dgdecode then test 5 is performed in the wrong way. If you use pp on a interlaced source you have to set "iPP = true" as it is "false" by default. You will throw off a proper de-interlacing this way. This is wrong practice even if you don't care about deinterlacing that much.

Dialhot 09-13-2004 06:08 AM

Quote:

Originally Posted by stephanV
If you use pp on a interlaced source you have to set "iPP = true" as it is "false" by default.

Nice advice (even if I do not use DgdDecode but DVD2AVI). I'll try to see the impact.

incredible 09-13-2004 08:22 AM

The postprocessors core of the DGdecode equivalent mpeg2source generally is taken from MarcFDs mpeg2dec3.dll, so also in the past that ipp=true was present ;-)

BTW: At gleitz.de/forum we where comparing that ipp true/false on interlaced shitty DVB streams and the result wont change "that" much, so dont expect wonders.

*Idea*!

Try using fielddeinterlace(blend=false, full=false) .... but risky if source is fieldblended.
If that ends up in a approx. 1sek jerky-interval then try at least fielddeinterlace(full=false) which at least wont smooth up your source that much in general as only combed scenes will be deinterlaced.

Dialhot 09-13-2004 08:25 AM

For the moment my tests on RemoveDirt are hung to the problem I report there (where I use ipp=True, BTW ;-)) :
http://www.kvcd.net/forum/viewtopic.php?t=13990

Uruk-hai 09-15-2004 01:54 AM

Re: RemoveGrain + RemoveDirt
 
Hi Guru & inmates,
Quote:

Originally Posted by Dialhot
Next step : (finally) change my V4 script for Divx sources :-)

Before the new (fine tuned) V4 script comes, do the following codes make sense:
Code:

AviSource("PATH\NAME.avi",false)
BlindPP(cpu=4)
DRemoveGrain(mode=5)
GripCrop(WIDTH, HEIGHT, overscan=1, source_anamorphic=false)
Removedirt()
GripSize(resizer="LanczosResize")
DCTFilter(1,1,1,1,1,1,0.5,0)
Blockbuster(method="noise",detail_min=1,detail_max=10,variance=0.3,seed=5623)
GripBorders()

Cheers

incredible 09-15-2004 02:29 AM

It doesnt make sense to put a DCTfilter which does kill high details first and then applying fine details again by using blockbuster.

Also DCTfilter isnt recommended for streams with a smaller width than 480 (just in case of) ;-)

Dialhot 09-15-2004 03:51 AM

Quote:

Originally Posted by incredible
It doesnt make sense to put a DCTfilter which does kill high details first and then applying fine details again by using blockbuster.

Inc, the last Blockbuster line is to avoid DCT blocks while encoding. If you put it before the DCTFilters line you will lower the its efficiency.

@Uruk-hai
That is the idea I have for next script. Except I don't know why you choosed mode=5 (I have to test it). But for sure, all this has to be tested a lot.

Uruk-hai 09-15-2004 03:52 AM

Quote:

Originally Posted by incredible
It doesnt make sense to put a DCTfilter which does kill high details first and then applying fine details again by using blockbuster.

I copied the DCTFilter and Blockbuster lines from V4 script :?
I think the intention of Blockbuster is to add noise to areas without too much detail such that they are not washed out.

incredible 09-15-2004 05:13 AM

Quote:

Inc, the last Blockbuster line is to avoid DCT blocks while encoding. If you put it before the DCTFilters line you will lower the its efficiency.
I know DCTfilter AND Blockbuster more than well as you know (still in doubt?)

So the resumee is, if Blockbuster finally is used, then get rid generally of DCTfilter as THEN its purpose ist killed.
Because then you could also use a blur(x) instead of DCTfilter as it also smoothes for further compression but on the other hand you dont have to worry about the known risks of DCTfilter. Because the main purpose to PRE-filter to 8x8 specs is trashed when adding a spray afterwards.

@ Uruk-hai
Quote:

I think the intention of Blockbuster is to add noise to areas without too much detail such that they are not washed out.
Exactly and we still want to keep that blockbuster, nobody denies that.
The problem here was the combination of both filters.

Do search in here for "DCTfilter" and youll see many explanations of mine where also graphs are included which do explain the 8x8dct prefiltering.

2nd is that many users did report line-errors on their results (some others not) ... depends on the width resoultion as dctfilter filters on horizontal DCT collumns.

Dialhot 09-15-2004 05:43 AM

Quote:

Originally Posted by incredible
So the resumee is, if Blockbuster finally is used, then get rid generally of DCTfilter as THEN its purpose ist killed.

Okay : your are saying than "DCT + Blockbuster" kills DCT effects and I'm saying "Blockbuster + DCT" kill blockbuster effect :-)
Hummm... as I removed DCTFilter for my personnal usages, I think I won't hurt my brain on that too long :-)

Note: I did some test on only one sources and after a dozen of script it seems the best results are obtained with :

Code:

AviSource("PATH\NAME.avi",false)
BlindPP(cpu=4)
GripCrop(WIDTH, HEIGHT, overscan=1, source_anamorphic=false)
GripSize(resizer="LanczosResize")
Blockbuster(method="noise",detail_min=1,detail_max=3,variance=0.1,seed=1)
DRemoveGrain()
Deen()
GripBorders()

Be carreful : this MUST be verified on sources with DCT blocks (already in the source + created in the encoded video). I don't have enought source kind on my office PC to test this completly.
Quote:

Because then you could also use a blur(x) instead of DCTfilter as it also smoothes for further compression
Sorry by I can agree with you ! DCTFilter does not blur the picture (except on low resolution as you proved it). Blur affects equally all the visual range while DCTFilters is focused only on the high (less visible) frequencies. The two filters can't be assimilated !

incredible 09-15-2004 07:11 AM

Quote:

Originally Posted by Dialhot
Quote:

Originally Posted by incredible
So the resumee is, if Blockbuster finally is used, then get rid generally of DCTfilter as THEN its purpose ist killed.

Okay : your are saying than "DCT + Blockbuster" kills DCT effects and I'm saying "Blockbuster + DCT" kill blockbuster effect :-)

Exactly! Thats why I say "the combination of both is senseless" No matter if one before the other or vice versa ;-)
Quote:

Note: I did some test on only one sources and after a dozen of script it seems the best results are obtained with :

Code:

AviSource("PATH\NAME.avi",false)
BlindPP(cpu=4)
GripCrop(WIDTH, HEIGHT, overscan=1, source_anamorphic=false)
GripSize(resizer="LanczosResize")
Blockbuster(method="noise",detail_min=1,detail_max=3,variance=0.1,seed=1)
DRemoveGrain()
Deen()
GripBorders()

Be carreful : this MUST be verified on sources with DCT blocks (already in the source + created in the encoded video). I don't have enought source kind on my office PC to test this completly.
Ill check that :)
For me no problem ... taking a good DVD Source and I just use a standard "normal" peg4 profile in XVID and doing one encode where the Quality reperesents the average u use for your testings. Means no real trash but DCT blocks and rings sometimes present.

Quote:

Quote:

Because then you could also use a blur(x) instead of DCTfilter as it also smoothes for further compression
Sorry by I can agree with you ! DCTFilter does not blur the picture (except on low resolution as you proved it). Blur affects equally all the visual range while DCTFilters is focused only on the high (less visible) frequencies. The two filters can't be assimilated !
Ok, but if DCTfilter is killed by adding new complexitivity then the ONLY compress. advantage is the occuring of more blur at low resolutions.
At high resolutions, then no more compr. gain is existent, and IF yes then its cause of a minimal blurring you wont recognise visually but your encoder does. :)
And that minimal gain does not stand in relation to the risk that some users could end up in these know handycaps of DCTfilter.

I hope I could explain it well in english, but I think you catch my point.

Ill test your script this week as today Ill be at work till late evening :(

J-Wo 09-15-2004 10:12 AM

I'm so thankful for your tests Phil! I too have been experimenting with RemoveGrain, but not so much RemoveDirt. The problem I had with the latter was my encoding of sci fi DVDs. Where there are panning starfield shots, RemoveDirt thinks that some of those stars are dirt when it sees their position move from frame to frame. The result is some of those stars disappear in certain frames, so instead of a smooth pan of stars, you get a pan of twinkling stars!!! So keep that in mind, your source material matters! I'm guessing that it's a good combination tho for any "earth" based live action movies. :D

Oh one question: can you tell me why you've removed DCTFilter and BlindPP from your dvd scripts?

edit: another question. Which resize would you recommend? Bicubic precise, standard, or lanczos?

Dialhot 09-15-2004 12:02 PM

Quote:

Originally Posted by J-Wo
Oh one question: can you tell me why you've removed DCTFilter and BlindPP from your dvd scripts?

BlindPP is not removed, it is included in the MPEG2DEC3 line (the cpu=4 parameter)

DCTFilter : as Inc shows me that this filter is not so harmless, and because the script is efficient enought without it, I decided to not use it. It's not necessary to add filters if the result you obtained is good enought.

Quote:

edit: another question. Which resize would you recommend? Bicubic precise, standard, or lanczos?
Did you ever see other things than Lanczos in all the scripts I provide ? ;-)

(I never tested Lanczos4 that is now internal to avisynth if I'm not wrong. You can use Bicubic precise insteed of Lanczos if you prefer).

J-Wo 09-15-2004 12:08 PM

okay, here are the results of my own tests. I agree with phil that undot+deen is now terrible, and according to my own tastes I prefer a combo of removegrain(mode=2) + removedirt() + dctfilter. Let me explain:

I am of the train of thought that LESS filtering/post-processing the better. I'm using sources which have a lot of film grain, they are the latest DVDs of the TV shows Alias and Six Feet Under. These have a lot more grain in them than your typical hollywood movie, most evident in smooth background surfaces e.g. walls. My desire was to only lightly remove the grain, smooth out the background a little as these grains liked to dance around. RemoveGrain + RemoveDirt does a fantastic job of this, much better than Undot + Deen. However if I increase my filtering too much, e.g. RemoveGrain(mode=8), the smoothing out of my backgrounds become more unrealistic. I now get a "halo" effect, like ripples in a pond, bands of one colour each slightly different in hue than the other. The filtering is doing TOO MUCH to remove the grains, and this makes my particular source look worse.

Now what I discovered is that adding DCTFilter back at the end of this helps to alleviate some of this rippling effect. But mode=8 already does too much, so the damage is done. Any Mpeg2source("",cpu=x) or BlindPP post processing is also much too strong. So at the end of the day, I've settled on the following combo:

Code:

mpeg2source("f:\alias\301.d2v")
BicubicResize(688, 448, 1/3, 1/3, 0, 8, 720, 464)

removegrain(mode=2)
removedirt()
DCTFilter(1,1,1,1,1,1,0.5,0)

AddBorders(16, 16, 16, 16)


Dialhot 09-15-2004 12:27 PM

Quote:

Originally Posted by J-Wo
Any Mpeg2source("",cpu=x) or BlindPP post processing is also much too strong

As I mentioned in my first post, I was wondering about the fact that this PP can be "too much" in this script. As I did all my tests on only one source, the answer was "no, it is not too much". But for sure, as you told, the source does matter.

For the mode=8, I revised my opinion : in the tests I did for avi sources, mode=8 was too bad also. I returned back to default (mode=2).

Quote:

BicubicResize(688, 448, 1/3, 1/3, 0, 8, 720, 464)
I'm just telling my opinion here : I find bicubic standard too soft for my eyes. That is why I always use Lanczos.

J-Wo 09-15-2004 12:45 PM

Quote:

Originally Posted by Dialhot
For the mode=8, I revised my opinion : in the tests I did for avi sources, mode=8 was too bad also. I returned back to default (mode=2).

Then we are in agreement! :D I think the default is just fine, as the docs say mode=2 is simply a "smarter" undot, and that's true. It compresses a bit more but is also a lot faster than undot.

I think for DVD sources you want to preserve as much detail as possible. So mode=2 is best for me. But maybe for some other material (tv caps?) mode=8 will be just fine. I suppose one will just have to test and see for one's self.

Quote:

Originally Posted by Dialhot
I'm just telling my opinion here : I find bicubic standard too soft for my eyes. That is why I always use Lanczos.

heheh, ya I do know you've mentioned that several times. ;) But to my eyes, I really can't see a difference between bicubic standard/precise vs. lanczos. And this is on a 53" HDTV. Unless more people tell me I'm dead wrong, I'll stick with bicubic standard for now for the higher compression.

J-Wo 09-16-2004 01:00 PM

Quote:

Originally Posted by Dialhot
Code:

AviSource("PATH\NAME.avi",false)
BlindPP(cpu=4)
GripCrop(WIDTH, HEIGHT, overscan=1, source_anamorphic=false)
GripSize(resizer="LanczosResize")
Blockbuster(method="noise",detail_min=1,detail_max=3,variance=0.1,seed=1)
DRemoveGrain()
Deen()
GripBorders()


Hey Dialhot, just wondering if you've made any progress in your new avi script. Any particular reason you're using a RemoveGrain + Deen combo instead of REmoveGrain + RemoveDirt? I'm going to try the latter as it should make the encode run faster (might not compress as well though)

edit: okay I did a little minitest on a 2000 frame clip using the above script. It's a Deen vs RemoveDirt test.

Test 1:
RemoveGrain() + Deen()
size = 21,036 kb
speed = 1.74

Test 2:
RemoveGrain + RemoveDirt()
size = 21,693 kb
speed = 1.96

So in my eyes, RemoveDirt compresses less than Deen only by 3%, but is 12% faster. Results look the same on my computer but I'll burn and test on my standalone (it always shows faults more readily).

incredible 09-16-2004 03:36 PM

Well J-WO the problem "when altering just not too much the noise" in your script is that you do a double spatial filter job, thats why your compression gain rises ;-)

Removedirt is a spatial denoiser only.
Deen() at default is a spatio-temporal, means 3d denoiser!

So two times spatial smoothed means more details killed - but your eyes will tell you the truth as this now is theoretical.
Removegrain was primary build to give removedirt() a hand on its job.

Removedirt does its fantastic (at right settings individual to the source) job as it seems that its working a bit motioncompensated (but Im not shure).

To get rid out of theese temporal filter artifacts you should look out for depan.dll as its a motioncompensation filter which avoids such temporal artifacts in combination with a temporal denoiser.

Code:

avisource("progressivePALfilm.avi")
#converttoyv12()
data=depanestimate(last,pixaspect=1.094,zoommax=20  ,improve=true)
depaninterleave(last,data,pixaspect=1.094)
YOUR TEMPORAL SMOOTHER HERE()
selectevery(3,1)


Code:

avisource("progressiveNTSCfilm.avi")
#converttoyv12()
data=depanestimate(last,pixaspect=0.911,zoommax=20  ,improve=true)
depaninterleave(last,data,pixaspect=0.911)
YOUR TEMPORAL SMOOTHER HERE()
selectevery(3,1) # not shure if NTSC needs here other values


Dialhot 09-17-2004 03:43 AM

Quote:

Originally Posted by J-Wo
Results look the same on my computer but I'll burn and test on my standalone (it always shows faults more readily).

And then you will probably understand why I choosed RemoveGrain+Deen and not Removedirt ;-)
(even if for DVD sources, it's the opposite).

Quote:

Originally Posted by incredible
Removedirt is a spatial denoiser only.
Deen() at default is a spatio-temporal, means 3d denoiser!

That is exactly that.
My original script add a "temporalsoften" after the Undot+Deen lines. I removed this, and that is probably why RemoveDirt is not enought and Deen is better.

Note: I didn't test RemoveGrain+RemoveDirt+TemporalSoften.

Uruk-hai 09-17-2004 04:00 AM

Quote:

Originally Posted by incredible
Removedirt is a spatial denoiser only.

Are you sure? It was said in the very first line of the help file Removedirt.htm that RemoveDirt is temporal:
Quote:

RemoveDirt is a so called temporal cleaner for Avisynth 2.5x.

Dialhot 09-17-2004 04:10 AM

Quote:

Originally Posted by Uruk-hai
Are you sure? It was said in the very first line of the help file Removedirt.htm that RemoveDirt is temporal:
Quote:

RemoveDirt is a so called temporal cleaner for Avisynth 2.5x.

:-D

It seems I'm as tired as Inc is :-). Of course RemoveDirt is purely temporal. (but deen is still more efficient as it is spatio/temporal). So forget my line about testing wiht temporalsoften :-)

incredible 09-17-2004 04:55 AM

Yep I AM TIRED :lol:

To clearify this (after having 8 cups of coffee *lol ):

RemoveGrain = Spatial Only

RemoveDirt = Temporal Only

J-Wo 09-17-2004 06:46 PM

okay, i'm hoping someone might post an optimal dvd or avi script soon, as i'm getting mighty confused! I just tried a removegrain + removedirt + dctfilter on my Alias DVDs, but I had the same problem as with sci fi sources like Babylon 5. During certain panning shots, you can see pixels "twinkling" in and out. Like if it's a night sky pan, stars will disappear randomly, or if you see a skyscraper in the distance then certain windows will have their lights randomly turning on or off. It's RemoveDirt at fault, because as mentioned it is temporal only. In the past I've experimented with scripts like this:
Code:

(resize here)
removegrain(mode=2)
TemporalSoften(2,4,6,6,2)
DCTFilter(1,1,1,1,1,1,0.5,0)

I'm really hoping someone can provide some more tips!

Dialhot 09-20-2004 04:16 AM

Quote:

Originally Posted by J-Wo
I'm really hoping someone can provide some more tips!

Kinda answer :
http://www.kvcd.net/forum/viewtopic.php?t=14141


All times are GMT -5. The time now is 01:03 PM  —  vBulletin © Jelsoft Enterprises Ltd

Site design, images and content © 2002-2024 The Digital FAQ, www.digitalFAQ.com
Forum Software by vBulletin · Copyright © 2024 Jelsoft Enterprises Ltd.