Quantcast Avisynth: Removegrain + Removedirt - digitalFAQ.com Forums [Archives]
  #1  
09-13-2004, 04:01 AM
Dialhot Dialhot is offline
Free Member
 
Join Date: May 2003
Posts: 10,463
Thanks: 0
Thanked 0 Times in 0 Posts
I finally did it !

"Little big" tests campagin on my PC this we. I decide to review my KDVD script and give a shot to RemoveDirt and RemoveGrain.

Fist, after having read the manuals, I decided to use them both at the same time because it is said the RemoveGrain is designed to be a pre-filter before RemoveDirt.

First of all, the test scripts :
Test1
Code:
Mpeg2Source("D:\Mes Ripps\ES_107252\Insurrection.d2v")
FieldDeinterlace()
Test2
Code:
Mpeg2Source("D:\Mes Ripps\ES_107252\Insurrection.d2v")
FieldDeinterlace()
Undot()
Deen()
Test3
Code:
Mpeg2Source("D:\Mes Ripps\ES_107252\Insurrection.d2v")
FieldDeinterlace()
DRemoveGrain()
RemoveDirt()
Test4
Code:
Mpeg2Source("D:\Mes Ripps\ES_107252\Insurrection.d2v")
FieldDeinterlace()
DRemoveGrain(mode=8)
RemoveDirt()
Test5
Code:
Mpeg2Source("D:\Mes Ripps\ES_107252\Insurrection.d2v",cpu=4,idct=7)
FieldDeinterlace()
DRemoveGrain(mode=8)
RemoveDirt()
Test6
Code:
Mpeg2Source("D:\Mes Ripps\ES_107252\Insurrection.d2v",idct=7)
FieldDeinterlace()
DRemoveGrain(mode=8)
RemoveDirt()
Note: DRemoveGrain is the SSE2 version of RemoveGrain.

All test were done using CCE. The source material is interlaced but I do not care about the deinterlace job, even if FielDeinterlace is probably not the best way.

The results (Final Video Size / Encoding speed) :

Test1 : 140343 / 1.16
Test2 : 130509 / 0.77
Test3 : 132065 / 0.93
Test4 : 130573 / 0.91
Test5 : 127537 / 0.78
Test6 : 130350 / 0.93

Comments :
- with Test2 and Test4 we can see that RemoveGrain/RemoveDirt is way faster than Undot/Deen for a final size identical.
- mode=2 of RemoveGrain is less efficient than mode=8 (Test3 and Test4)
- I did not test mode=4 (the most efficient according to the manual) because it made avisynth to crash anytime !

Quality results :
Test2 is definitely the worst of all results (except Test1 that does not use any filters). So I guess I will say "bye-bye" to undot/deen. Rip
I personnaly prefer Test5 but some will say that using Mpeg2Dec3 PP plus the filters is too much. I don't think so and we have a size gain of 3% !

Unfortunally, for usage with DVD-Rebuild only "Idct=7" is usable but not "cpu=4" so the results will be the ones of Test6.

Next step : (finally) change my V4 script for Divx sources
Reply With Quote
Someday, 12:01 PM
admin's Avatar
Site Staff / Ad Manager
 
Join Date: Dec 2002
Posts: 42
Thanks: ∞
Thanked 42 Times in 42 Posts
  #2  
09-13-2004, 04:21 AM
incredible incredible is offline
Free Member
 
Join Date: May 2003
Location: Germany
Posts: 3,189
Thanks: 0
Thanked 0 Times in 0 Posts
Send a message via ICQ to incredible
Yep! The RemoveX series from Kassandro is a big shot!!

But watch out as A value of 8 in removegrain will "round" details!
You can see that in case of captures on the top at the station logo!

So a DVD often only needs a smaller value


OT/
Sorry to say that (and maybe to get annoying), but I do not understand why you are doing your tests on fieldblended NTSCtoPAL streams! As I also got the same DVD/Extras where you are doing your tests on.

The source is not only interlaced. It has been converted from NTSC Hybrid to PAL like in that way:

InputSource29.97
Bob()
ConvertFPS(50)
Separatefields().Selectevery(4,1,2).weave()

So even a restore24() script to recover the 23.976 progressive fps wont work as the source is HYBRID (mixed video interviews and film movie parts).

Why do I say that? Because on fieldblended streams you wont see the clear power of a progressive intended filter! And as we both do our tests on relaworld conditions, I could assume that we got the same intention.
The correct temporal filtering estimations of the filter wont work correct as almost every second frame gots a very different Frame Picture architecture information (you know what I mean by saying that) as almost every second frame is BLENDED, means softened and edge distorted.

Why not taking exactly that main movie and doing a parts selected trimmed/slice/SelectrangeEvery or whatever method to obtain a got mixture of a testing sample

/OT off

My intention of these sentences is that (maybe) I can directly participate these tests as you can remember we tried to find a way to find equal sources for "online" testing workouts ---- And Id like to !
Reply With Quote
  #3  
09-13-2004, 04:39 AM
Dialhot Dialhot is offline
Free Member
 
Join Date: May 2003
Posts: 10,463
Thanks: 0
Thanked 0 Times in 0 Posts
Quote:
Originally Posted by incredible
Sorry to say that (and maybe to get annoying), but I do not understand why you are doing your tests on fieldblended NTSCtoPAL streams!
Because, as I explained : "I do not care about the deinterlace job, even if FielDeinterlace is probably not the best way" (I typed the sentence espacially for you ).
But if I can remember it, I will use your suggestion for my future tests.
(even if I don't think that can modify the order of merit of tested scripts as they are all working of the same "badly deinterlaced" source).
Reply With Quote
  #4  
09-13-2004, 04:51 AM
jorel jorel is offline
Invalid Email / Banned / Spammer
 
Join Date: Aug 2002
Location: Brasil - MG - third stone from the sun
Posts: 5,570
Thanks: 0
Thanked 0 Times in 0 Posts
magnific post Phil!

yesterday ink give me recomendations to test
RemoveGrain and RemoveDirt
posting the links to read and download that files!

now i see that test 5 is your choice and confirm the ink's recomendations.
i don't had time to do my own tests!

now i know what i have to use.
what was good yesterday is bad today.
( i'm talking about undot/deen and my ex-wife)

thanks so much Phil, ink and to Boulder for suggested to use it first.
Reply With Quote
  #5  
09-13-2004, 05:01 AM
Dialhot Dialhot is offline
Free Member
 
Join Date: May 2003
Posts: 10,463
Thanks: 0
Thanked 0 Times in 0 Posts
Quote:
Originally Posted by jorel
now i see that test 5 is your choice and confirm the ink's recomendations.
Note that in his answer, Inc suggested than "mode=8" that I used in my test5 script is not always the best idea.
Maybe you should use "mode=2" anytime (in the manual that is said that mode=8 can cause artifacts)
Reply With Quote
  #6  
09-13-2004, 05:15 AM
incredible incredible is offline
Free Member
 
Join Date: May 2003
Location: Germany
Posts: 3,189
Thanks: 0
Thanked 0 Times in 0 Posts
Send a message via ICQ to incredible
Quote:
Originally Posted by Dialhot
Because, as I explained : "I do not care about the deinterlace job, even if FielDeinterlace is probably not the best way" (I typed the sentence espacially for you ).
I repeat the explanation of why blended frames do harm on temporal correct filtering :
Quote:
The correct temporal filtering estimations of the filter wont work correct as almost every second frame gots a very different Frame Picture architecture information (you know what I mean by saying that) as almost every second frame is BLENDED, means softened and edge distorted.
Quote:
Originally Posted by Phil
But if I can remember it, I will use your suggestion for my future tests.
Thats the way we should do it

Related to Removegrain Modes: I did read that mode1 or 2 is the same like undot() .... hmmm ... I think Ive to go into the readms again
Reply With Quote
  #7  
09-13-2004, 06:02 AM
stephanV stephanV is offline
Free Member
 
Join Date: May 2004
Posts: 43
Thanks: 0
Thanked 0 Times in 0 Posts
If the source is interlaced and you are using dgdecode then test 5 is performed in the wrong way. If you use pp on a interlaced source you have to set "iPP = true" as it is "false" by default. You will throw off a proper de-interlacing this way. This is wrong practice even if you don't care about deinterlacing that much.
Reply With Quote
  #8  
09-13-2004, 06:08 AM
Dialhot Dialhot is offline
Free Member
 
Join Date: May 2003
Posts: 10,463
Thanks: 0
Thanked 0 Times in 0 Posts
Quote:
Originally Posted by stephanV
If you use pp on a interlaced source you have to set "iPP = true" as it is "false" by default.
Nice advice (even if I do not use DgdDecode but DVD2AVI). I'll try to see the impact.
Reply With Quote
  #9  
09-13-2004, 08:22 AM
incredible incredible is offline
Free Member
 
Join Date: May 2003
Location: Germany
Posts: 3,189
Thanks: 0
Thanked 0 Times in 0 Posts
Send a message via ICQ to incredible
The postprocessors core of the DGdecode equivalent mpeg2source generally is taken from MarcFDs mpeg2dec3.dll, so also in the past that ipp=true was present

BTW: At gleitz.de/forum we where comparing that ipp true/false on interlaced shitty DVB streams and the result wont change "that" much, so dont expect wonders.

*Idea*!

Try using fielddeinterlace(blend=false, full=false) .... but risky if source is fieldblended.
If that ends up in a approx. 1sek jerky-interval then try at least fielddeinterlace(full=false) which at least wont smooth up your source that much in general as only combed scenes will be deinterlaced.
Reply With Quote
  #10  
09-13-2004, 08:25 AM
Dialhot Dialhot is offline
Free Member
 
Join Date: May 2003
Posts: 10,463
Thanks: 0
Thanked 0 Times in 0 Posts
For the moment my tests on RemoveDirt are hung to the problem I report there (where I use ipp=True, BTW ) :
http://www.kvcd.net/forum/viewtopic.php?t=13990
Reply With Quote
  #11  
09-15-2004, 01:54 AM
Uruk-hai Uruk-hai is offline
Free Member
 
Join Date: Apr 2004
Location: Middle Earth
Posts: 30
Thanks: 0
Thanked 0 Times in 0 Posts
Hi Guru & inmates,
Quote:
Originally Posted by Dialhot
Next step : (finally) change my V4 script for Divx sources
Before the new (fine tuned) V4 script comes, do the following codes make sense:
Code:
AviSource("PATH\NAME.avi",false) 
BlindPP(cpu=4) 
DRemoveGrain(mode=5)
GripCrop(WIDTH, HEIGHT, overscan=1, source_anamorphic=false) 
Removedirt()
GripSize(resizer="LanczosResize") 
DCTFilter(1,1,1,1,1,1,0.5,0) 
Blockbuster(method="noise",detail_min=1,detail_max=10,variance=0.3,seed=5623) 
GripBorders()
Cheers
Reply With Quote
  #12  
09-15-2004, 02:29 AM
incredible incredible is offline
Free Member
 
Join Date: May 2003
Location: Germany
Posts: 3,189
Thanks: 0
Thanked 0 Times in 0 Posts
Send a message via ICQ to incredible
It doesnt make sense to put a DCTfilter which does kill high details first and then applying fine details again by using blockbuster.

Also DCTfilter isnt recommended for streams with a smaller width than 480 (just in case of)
Reply With Quote
  #13  
09-15-2004, 03:51 AM
Dialhot Dialhot is offline
Free Member
 
Join Date: May 2003
Posts: 10,463
Thanks: 0
Thanked 0 Times in 0 Posts
Quote:
Originally Posted by incredible
It doesnt make sense to put a DCTfilter which does kill high details first and then applying fine details again by using blockbuster.
Inc, the last Blockbuster line is to avoid DCT blocks while encoding. If you put it before the DCTFilters line you will lower the its efficiency.

@Uruk-hai
That is the idea I have for next script. Except I don't know why you choosed mode=5 (I have to test it). But for sure, all this has to be tested a lot.
Reply With Quote
  #14  
09-15-2004, 03:52 AM
Uruk-hai Uruk-hai is offline
Free Member
 
Join Date: Apr 2004
Location: Middle Earth
Posts: 30
Thanks: 0
Thanked 0 Times in 0 Posts
Quote:
Originally Posted by incredible
It doesnt make sense to put a DCTfilter which does kill high details first and then applying fine details again by using blockbuster.
I copied the DCTFilter and Blockbuster lines from V4 script
I think the intention of Blockbuster is to add noise to areas without too much detail such that they are not washed out.
Reply With Quote
  #15  
09-15-2004, 05:13 AM
incredible incredible is offline
Free Member
 
Join Date: May 2003
Location: Germany
Posts: 3,189
Thanks: 0
Thanked 0 Times in 0 Posts
Send a message via ICQ to incredible
Quote:
Inc, the last Blockbuster line is to avoid DCT blocks while encoding. If you put it before the DCTFilters line you will lower the its efficiency.
I know DCTfilter AND Blockbuster more than well as you know (still in doubt?)

So the resumee is, if Blockbuster finally is used, then get rid generally of DCTfilter as THEN its purpose ist killed.
Because then you could also use a blur(x) instead of DCTfilter as it also smoothes for further compression but on the other hand you dont have to worry about the known risks of DCTfilter. Because the main purpose to PRE-filter to 8x8 specs is trashed when adding a spray afterwards.

@ Uruk-hai
Quote:
I think the intention of Blockbuster is to add noise to areas without too much detail such that they are not washed out.
Exactly and we still want to keep that blockbuster, nobody denies that.
The problem here was the combination of both filters.

Do search in here for "DCTfilter" and youll see many explanations of mine where also graphs are included which do explain the 8x8dct prefiltering.

2nd is that many users did report line-errors on their results (some others not) ... depends on the width resoultion as dctfilter filters on horizontal DCT collumns.
Reply With Quote
  #16  
09-15-2004, 05:43 AM
Dialhot Dialhot is offline
Free Member
 
Join Date: May 2003
Posts: 10,463
Thanks: 0
Thanked 0 Times in 0 Posts
Quote:
Originally Posted by incredible
So the resumee is, if Blockbuster finally is used, then get rid generally of DCTfilter as THEN its purpose ist killed.
Okay : your are saying than "DCT + Blockbuster" kills DCT effects and I'm saying "Blockbuster + DCT" kill blockbuster effect
Hummm... as I removed DCTFilter for my personnal usages, I think I won't hurt my brain on that too long

Note: I did some test on only one sources and after a dozen of script it seems the best results are obtained with :

Code:
AviSource("PATH\NAME.avi",false)
BlindPP(cpu=4)
GripCrop(WIDTH, HEIGHT, overscan=1, source_anamorphic=false)
GripSize(resizer="LanczosResize")
Blockbuster(method="noise",detail_min=1,detail_max=3,variance=0.1,seed=1)
DRemoveGrain()
Deen()
GripBorders()
Be carreful : this MUST be verified on sources with DCT blocks (already in the source + created in the encoded video). I don't have enought source kind on my office PC to test this completly.
Quote:
Because then you could also use a blur(x) instead of DCTfilter as it also smoothes for further compression
Sorry by I can agree with you ! DCTFilter does not blur the picture (except on low resolution as you proved it). Blur affects equally all the visual range while DCTFilters is focused only on the high (less visible) frequencies. The two filters can't be assimilated !
Reply With Quote
  #17  
09-15-2004, 07:11 AM
incredible incredible is offline
Free Member
 
Join Date: May 2003
Location: Germany
Posts: 3,189
Thanks: 0
Thanked 0 Times in 0 Posts
Send a message via ICQ to incredible
Quote:
Originally Posted by Dialhot
Quote:
Originally Posted by incredible
So the resumee is, if Blockbuster finally is used, then get rid generally of DCTfilter as THEN its purpose ist killed.
Okay : your are saying than "DCT + Blockbuster" kills DCT effects and I'm saying "Blockbuster + DCT" kill blockbuster effect
Exactly! Thats why I say "the combination of both is senseless" No matter if one before the other or vice versa
Quote:
Note: I did some test on only one sources and after a dozen of script it seems the best results are obtained with :

Code:
AviSource("PATH\NAME.avi",false)
BlindPP(cpu=4)
GripCrop(WIDTH, HEIGHT, overscan=1, source_anamorphic=false)
GripSize(resizer="LanczosResize")
Blockbuster(method="noise",detail_min=1,detail_max=3,variance=0.1,seed=1)
DRemoveGrain()
Deen()
GripBorders()
Be carreful : this MUST be verified on sources with DCT blocks (already in the source + created in the encoded video). I don't have enought source kind on my office PC to test this completly.
Ill check that
For me no problem ... taking a good DVD Source and I just use a standard "normal" peg4 profile in XVID and doing one encode where the Quality reperesents the average u use for your testings. Means no real trash but DCT blocks and rings sometimes present.

Quote:
Quote:
Because then you could also use a blur(x) instead of DCTfilter as it also smoothes for further compression
Sorry by I can agree with you ! DCTFilter does not blur the picture (except on low resolution as you proved it). Blur affects equally all the visual range while DCTFilters is focused only on the high (less visible) frequencies. The two filters can't be assimilated !
Ok, but if DCTfilter is killed by adding new complexitivity then the ONLY compress. advantage is the occuring of more blur at low resolutions.
At high resolutions, then no more compr. gain is existent, and IF yes then its cause of a minimal blurring you wont recognise visually but your encoder does.
And that minimal gain does not stand in relation to the risk that some users could end up in these know handycaps of DCTfilter.

I hope I could explain it well in english, but I think you catch my point.

Ill test your script this week as today Ill be at work till late evening
Reply With Quote
  #18  
09-15-2004, 10:12 AM
J-Wo J-Wo is offline
Free Member
 
Join Date: Nov 2002
Location: Toronto, Canada
Posts: 454
Thanks: 0
Thanked 0 Times in 0 Posts
I'm so thankful for your tests Phil! I too have been experimenting with RemoveGrain, but not so much RemoveDirt. The problem I had with the latter was my encoding of sci fi DVDs. Where there are panning starfield shots, RemoveDirt thinks that some of those stars are dirt when it sees their position move from frame to frame. The result is some of those stars disappear in certain frames, so instead of a smooth pan of stars, you get a pan of twinkling stars!!! So keep that in mind, your source material matters! I'm guessing that it's a good combination tho for any "earth" based live action movies.

Oh one question: can you tell me why you've removed DCTFilter and BlindPP from your dvd scripts?

edit: another question. Which resize would you recommend? Bicubic precise, standard, or lanczos?
Reply With Quote
  #19  
09-15-2004, 12:02 PM
Dialhot Dialhot is offline
Free Member
 
Join Date: May 2003
Posts: 10,463
Thanks: 0
Thanked 0 Times in 0 Posts
Quote:
Originally Posted by J-Wo
Oh one question: can you tell me why you've removed DCTFilter and BlindPP from your dvd scripts?
BlindPP is not removed, it is included in the MPEG2DEC3 line (the cpu=4 parameter)

DCTFilter : as Inc shows me that this filter is not so harmless, and because the script is efficient enought without it, I decided to not use it. It's not necessary to add filters if the result you obtained is good enought.

Quote:
edit: another question. Which resize would you recommend? Bicubic precise, standard, or lanczos?
Did you ever see other things than Lanczos in all the scripts I provide ?

(I never tested Lanczos4 that is now internal to avisynth if I'm not wrong. You can use Bicubic precise insteed of Lanczos if you prefer).
Reply With Quote
  #20  
09-15-2004, 12:08 PM
J-Wo J-Wo is offline
Free Member
 
Join Date: Nov 2002
Location: Toronto, Canada
Posts: 454
Thanks: 0
Thanked 0 Times in 0 Posts
okay, here are the results of my own tests. I agree with phil that undot+deen is now terrible, and according to my own tastes I prefer a combo of removegrain(mode=2) + removedirt() + dctfilter. Let me explain:

I am of the train of thought that LESS filtering/post-processing the better. I'm using sources which have a lot of film grain, they are the latest DVDs of the TV shows Alias and Six Feet Under. These have a lot more grain in them than your typical hollywood movie, most evident in smooth background surfaces e.g. walls. My desire was to only lightly remove the grain, smooth out the background a little as these grains liked to dance around. RemoveGrain + RemoveDirt does a fantastic job of this, much better than Undot + Deen. However if I increase my filtering too much, e.g. RemoveGrain(mode=, the smoothing out of my backgrounds become more unrealistic. I now get a "halo" effect, like ripples in a pond, bands of one colour each slightly different in hue than the other. The filtering is doing TOO MUCH to remove the grains, and this makes my particular source look worse.

Now what I discovered is that adding DCTFilter back at the end of this helps to alleviate some of this rippling effect. But mode=8 already does too much, so the damage is done. Any Mpeg2source("",cpu=x) or BlindPP post processing is also much too strong. So at the end of the day, I've settled on the following combo:

Code:
mpeg2source("f:\alias\301.d2v")
BicubicResize(688, 448, 1/3, 1/3, 0, 8, 720, 464)

removegrain(mode=2)
removedirt()
DCTFilter(1,1,1,1,1,1,0.5,0) 

AddBorders(16, 16, 16, 16)
Reply With Quote
Reply




Similar Threads
Thread Thread Starter Forum Replies Last Post
Avisynth: Removegrain 1.0 - New plugins supermule Avisynth Scripting 25 03-15-2006 07:06 AM
Avisynth: RemoveGrain maurus Avisynth Scripting 3 05-16-2005 06:27 AM
KDVD: Resizing, croping and RemoveGrain digitall.doc Video Encoding and Conversion 1 01-12-2005 05:00 PM
Avisynth: RemoveDirt stopped? Dialhot Avisynth Scripting 16 09-22-2004 03:19 PM
Avisynth: Concern with RemoveDirt Dialhot Avisynth Scripting 13 09-13-2004 12:28 PM

Thread Tools



 
All times are GMT -5. The time now is 02:42 AM  —  vBulletin Jelsoft Enterprises Ltd