Quantcast Sansgrip Filters: Improve Picture Quality Using "Addnoise" Filter - digitalFAQ.com Forums [Archives]
  #1  
11-07-2002, 03:51 PM
black prince black prince is offline
Free Member
 
Join Date: Jul 2002
Posts: 1,224
Thanks: 0
Thanked 0 Times in 0 Posts
Hi All,

Here's an interesting post from Doom9. A new filter "AddNoise"
to improve picture encode quality by fooling the encoder program
(i.e. Tmpgenc) by sending noise to cause more bitrates to be applied
and correcting a problem (DCT, blockiness in dark areas) that
it would normally ignor. Here the link:

http://forum.doom9.org/showthread.ph...light=AddNoise

I have tried "AddNoise" and it certainly improved the background
blockiness for mpeg-1 352x240 or low resolution. The file did not
increase greatly with this improvement. Could it be possible to fool
Tmpgenc to pay more attention to motion search and vastly improve
the "mosquito effect", rather than pour more bitrates via CQ_VBR over
every frame to make it go away. The idea here is to test Tmpgenc
with the "Addnoise" and see what is effected. SansGrip of Doom9
has written "NoiseGenerator" which should create noise in many areas.
I am going to try his next with his recommended setting as a start.
This could be one solution to a better picture quality for low or very
low bitrate and resolutions. With "BitrateViewer" I hope to find
a noise that effects "mosquito effect". Even if it makes it worse, that
means some setting could improve it.


-black prince
Reply With Quote
Someday, 12:01 PM
admin's Avatar
Site Staff / Ad Manager
 
Join Date: Dec 2002
Posts: 42
Thanks: ∞
Thanked 42 Times in 42 Posts
  #2  
11-07-2002, 06:17 PM
TKS TKS is offline
Free Member
 
Join Date: May 2002
Posts: 118
Thanks: 0
Thanked 0 Times in 0 Posts
This sounds amazing... geez reading some of stuff on Doom9's forums is quite scary.. After a while though.. i start to understand it (i hope)

tks

BTW how does one implement this into ones .avs file?
Reply With Quote
  #3  
11-07-2002, 08:34 PM
black prince black prince is offline
Free Member
 
Join Date: Jul 2002
Posts: 1,224
Thanks: 0
Thanked 0 Times in 0 Posts
Hi TKS,

TKS wrote:
Quote:
TKS Posted: Thu Nov 07, 2002 7:17 pm Post subject: cool!

--------------------------------------------------------------------------------

This sounds amazing... geez reading some of stuff on Doom9's forums is quite scary.. After a while though.. i start to understand it (i hope)

tks

BTW how does one implement this into ones .avs file?
Go the link in my first post and download "AddNoise" filter. Follow
the sample script used by SansGrip and test a small clip of say
30 seconds with "AddNoise" and without it. Pay attention to
background of video and look for blocky areas to compare. I
think you'll be suprised at the results.

#----- AddNoise Filter Test ------
LoadPlugin("E:\bbbbbbbb\MPEG2DEC2.dll")
LoadPlugin("E:\bbbbbbbb\AddNoise.dll")
mpeg2source("D:\Temp\movie.d2v")
Trim(0,720) # <-- ~ 30 seconds of video play
AddNoise(yv=1,loop=4,mode="round")# <-- test yv = 1.5,2,3 etc.
BilinearResize(336,224)
TemporalSmoother(1,2)
AddBorders(8,8,8,8 )
# ------ End of Script ------



-black prince
Reply With Quote
  #4  
11-07-2002, 08:40 PM
kwag kwag is offline
Free Member
 
Join Date: Apr 2002
Location: Puerto Rico, USA
Posts: 13,537
Thanks: 0
Thanked 0 Times in 0 Posts
How does this affect the file size? What's the difference on both samples. I would assume that the sample with the "AddNoise" filter would be larger, because it's actually increasing the noise floor ( My guess ) and this would force a higher CQ/bit rate factor. So the resulting file size would be larger. Is my guess correct

-kwag
Reply With Quote
  #5  
11-07-2002, 09:51 PM
TKS TKS is offline
Free Member
 
Join Date: May 2002
Posts: 118
Thanks: 0
Thanked 0 Times in 0 Posts
From what i read really quickly on doom9s forum is that it didnt really change the file size all that much.. havent tried it myself yet...

tks
Reply With Quote
  #6  
11-07-2002, 10:17 PM
TKS TKS is offline
Free Member
 
Join Date: May 2002
Posts: 118
Thanks: 0
Thanked 0 Times in 0 Posts
I made some quick 1 min samples with Good Will Hunting DVD and here is my .avs script:

LoadPlugin("F:\util\video\mpgdecdll\MPEG2DEC.dll")
LoadPlugin("F:\Program Files\GordianKnot\AddNoise.dll")
mpeg2source("F:\rips\good1.d2v")
BilinearResize(336,192,45,0,630,480)
TemporalSmoother(1,2)
AddBorders(8,24,8,24)
#Trim(0,182097).FadeOut(150)
AddNoise(yv=1,loop=4,mode="round")
#ConvertToRGB24 # For TMPGEnc or VFAPI
#ResampleAudio(44100) # CCE 2.5 'crashfix' for Athlons
#== If you want this 'fix' permanently, edit the INI-file:
#== Under [AVSscript] set CCEcrashfix=1

Now the file size of the first sample done with kvcd LBR NTSC in Tmpgenc without Addnoise was 5.2 megs. The one with Addnoise was 5.6, not much of a difference at all.
Now this is hoping that I did everything right and placed the Addnoise statement in the right place within the .avs script

Quality diference:

Well i noticed that there were a lot of blackboard scenes and this really showed the difference between the two files. The Addnoise filter really helped there, it was much better, alot less mosquito effect going on with the black and white or the blackboard/chalk.

tks
Reply With Quote
  #7  
11-07-2002, 10:25 PM
kwag kwag is offline
Free Member
 
Join Date: Apr 2002
Location: Puerto Rico, USA
Posts: 13,537
Thanks: 0
Thanked 0 Times in 0 Posts
Hi TKS,

That's a ~50MB difference right there on a two hour movie. Can you try just increasing the CQ_VBR just a little, so that you get the same file size you got with the noise filter. But encode without the filter. See if you get the same "less" mosquito effect.

-kwag
Reply With Quote
  #8  
11-08-2002, 07:59 AM
TKS TKS is offline
Free Member
 
Join Date: May 2002
Posts: 118
Thanks: 0
Thanked 0 Times in 0 Posts
I did the entire movie and to fill the entire disc the audio was 128 with HeadAChe. Without the audio the movie was 678 megs.. So far it looks really good.. See Good Will Hunting isnt a really quick action scene type of movie..

But ill give that a try Kwag..


tks
Reply With Quote
  #9  
11-08-2002, 08:30 AM
black prince black prince is offline
Free Member
 
Join Date: Jul 2002
Posts: 1,224
Thanks: 0
Thanked 0 Times in 0 Posts
Hi Kwag,

Kwag wrote:
Quote:
kwag Posted: Thu Nov 07, 2002 9:40 pm Post subject:

--------------------------------------------------------------------------------

How does this affect the file size? What's the difference on both samples. I would assume that the sample with the "AddNoise" filter would be larger, because it's actually increasing the noise floor ( My guess ) and this would force a higher CQ/bit rate factor. So the resulting file size would be larger. Is my guess correct

-kwag
Yes it does increase file size, but blockiness in dark areas does disappear.
I tried a higher CQ without "AddNoise" and blockiness in dark areas was
still there. My understanding after reading the link was this noise only
caused more lumina noise and had no effect on "mosquitos effect". The
increase of CQ seemed to improve "mosquitos effect". The question is
picture quality vs small increase in file size. Tmpgenc sees macro block
and block changes during motion search, but pixel block changes are not
detected and the "mosquitos effect" occurs unless CQ is highest enough
to overcome this in every frame. Sort of an over-kill in CQ to remove
"mosquitos effect". Another question is what would make Tmpgenc look
at pixel block. Could a noise do this or an enhancement to the motion
search formula? Would there be a price for this refinement (i.e. very slow
encode times, larger file size, etc.). "AddNoise" is just a start to stimulate
more discussion on how to improve picture quality for low resolution (352x
240) and low bitrate videos.

-black prince
Reply With Quote
  #10  
11-08-2002, 08:45 AM
black prince black prince is offline
Free Member
 
Join Date: Jul 2002
Posts: 1,224
Thanks: 0
Thanked 0 Times in 0 Posts
Hi TKS,

TKS wrote:
Quote:
Quality diference:

Well i noticed that there were a lot of blackboard scenes and this really showed the difference between the two files. The Addnoise filter really helped there, it was much better, alot less mosquito effect going on with the black and white or the blackboard/chalk.

tks
Try yv=1.5 or 2. This really showed a bigger differance between using
"AddNoise" and without it. Be careful not to be too agressive with yv. It
will start to blurr if you use higher values.

-black prince
Reply With Quote
  #11  
11-08-2002, 09:52 AM
kwag kwag is offline
Free Member
 
Join Date: Apr 2002
Location: Puerto Rico, USA
Posts: 13,537
Thanks: 0
Thanked 0 Times in 0 Posts
This is VERY interesting . Starting to read about it in depth

-kwag
Reply With Quote
  #12  
11-08-2002, 12:51 PM
GFR GFR is offline
Free Member
 
Join Date: May 2002
Posts: 438
Thanks: 0
Thanked 0 Times in 0 Posts
Adding noise before quantizing a signal is a technique called "dither". It gives the "impression" that you have a finer quantization than you have indeed.

Imagine you have a smooth gradient (degradée). When you quantize it, it is not smooth anymore, it looks like a "staircase", with each "step" corresponding to a quantizing step.

If you add a little noise to this gradient, the points where you change from one step to the next are not in line anymore, it's like if the step is "fuzzy". If you look at a sufficient distance the effect is that the gradient is smooth again.

In MPEG, suppose you have a block that's almost at a constant colour and gets coded as a solid colour square. The block next to it is also almost at a constant colour and gets coded as a solid colour square, but at a slightly different colour than its neighbour. The effect is that the transistion from one block to the other is not smooth, but you have a step.

If you add some noise to these blocks then each block will hopefully not be a solid colour square but will have some variance and the transition from one block to the other will be smoother.

The ideal is to select the noise level such that quantization will kill it for real solid colours but slow changing colours that cause ugly blocks are "enhanced". This way you don't get blocks and the file size does not increases too much.

I think increasing CQ doesn't help very much, but you can try tweaking the quantizing matrix for slow frequencies and if you're using mpeg-2 maybe trying more bits for the DC.

As for mosquitoes, the Doom9 thread suggests that you use this filter with method=blur and invert=true - this will do a selective blur just on the edges.

I really hope this helps, cause the sources I'm most interested in converting are VHS camcorder shootings and anime, both suffer a lot from mosquitoes!!!
Reply With Quote
  #13  
11-08-2002, 02:46 PM
black prince black prince is offline
Free Member
 
Join Date: Jul 2002
Posts: 1,224
Thanks: 0
Thanked 0 Times in 0 Posts
Hi GFR,

Thanks for the technical info. Could you be more specific about Doom9's
solution to improve the "mosquito effect". I encoded "Enough" using
"AddNoise". Here the LBR script below:

LoadPlugin("E:\DVD Backup\2 - DVD2SVCD\MPEG2DEC\MPEG2DEC2.dll")
LoadPlugin("E:\DVD Backup\2 - DVD2SVCD\AddNoise\Addnoise.dll")
mpeg2source("D:\Temp\movie.d2v")
#
LanczosResize(336,224)
#
opening=Trim(0,550)
opening=opening.TemporalSoften(3,8,30)
#
movie=Trim(551,158936)
movie=movie.AddNoise(yv=1.5,loop=4,mode="round")
#
credits=Trim(158937,0)
credits=credits.Greyscale().TemporalSoften(3,8,30)
#
last=opening+movie+credits
#
AddBorders(8,8,8,8 )
#
#LegalClip()
ConvertToRGB()
#
# ---------- End of Script ----------

Tmpgenc MPEG-1 352x240 500-2564kbps CQ-100


Not sure what LegalClip() is so it's commented.
The results played back on standard 27" TV was WOW
The picture quality was excellent even up close. Again WOW
Of course file size was large (2 CD's), but I had to see how
good this could get. The real test was viewing this movie in
a dark room. This would show blockiness in dark areas very
easily. WOW No blockiness in dark areas. With a CQ = 100
their was no "mosquito effect", but I will test lower CQ's to try
and get the file size to fit one CD. I hope GFR's solution for this
works without impacting file size. This is a certainly a manual
process, but it's well worth it.

-black prince
Reply With Quote
  #14  
11-08-2002, 04:10 PM
kwag kwag is offline
Free Member
 
Join Date: Apr 2002
Location: Puerto Rico, USA
Posts: 13,537
Thanks: 0
Thanked 0 Times in 0 Posts
The noise filter is now called "Blockbuster"
Check here: http://www.jungleweb.net/~sansgrip/avisynth/

-kwag
Reply With Quote
  #15  
11-08-2002, 04:14 PM
black prince black prince is offline
Free Member
 
Join Date: Jul 2002
Posts: 1,224
Thanks: 0
Thanked 0 Times in 0 Posts
Hi GFR and Kwag,

I found a filter that effects "mosquitoe noise" called UnDot, BUT it
can only work with avisynth 2.5 alpha release. Bummer!! This release
is unstable and for developers. Here's the doc file:

File: Readme_UnDot.txt

// UnDot - Simple Deringing Dot Remover
// Copyright (C) 2002 Tom Barry - trbarry@trbarry.com
//
// This program is free software; you can redistribute it and/or modify
// it under the terms of the GNU General Public License as published by
// the Free Software Foundation; either version 2 of the License, or
// (at your option) any later version.
//
// This program is distributed in the hope that it will be useful,
// but WITHOUT ANY WARRANTY; without even the implied warranty of
// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
// GNU General Public License for more details.
//
// You should have received a copy of the GNU General Public License
// along with this program; if not, write to the Free Software
// Foundation, Inc., 675 Mass Ave, Cambridge, MA 02139, USA.

Also, this program is "Philanthropy-Ware". That is, if you like it and feel
the need to reward or inspire the author then please feel free (but not obligated)
to consider joining or donating to the Electronic Frontier Foundation. This will
help keep cyber space free of barbed wire and bullsh*t.

See their web page at www.eff.org

***************

Okay, on to business.

WARNING - This version only runs on the Avisynth 2.5 alpha release.

UnDot is a simple median filter for removing dots, that is stray orphan pixels and
mosquito noise. It basicly just clips each pixel value to stay within min and max
of its eight surrounding neigbors.


USAGE - To use it just:

1) Place the UnDot.dll in a directory somewhere. You can get it from
www.trbarry.com/UnDot.zip

2) In your Avisynth file use commands similar to

LoadPlugin("F:\UnDot\UnDot.dll")
Avisource("D:\wherever\myfile.avi")
UnDot()

Of course replace the file and directory names with your own. There are no parameters.


KNOWN ISSUES AND LIMITATIONS

1) Requires either YUY2 or YV12 input.

2) Sorry, currently requires a P-III, Athlon, or higher. Needs SSEMMX support.

3) So far it has only been tested on SSEMMX (P3 & P4) machines.

4) In YV12 format it will filter both luma and chroma. In YUY2 format it will
only filter luma.

FILE LOCATIONS

For now, both source, this readme, and DLL should be at:

www.trbarry.com/UnDot.zip

A copy of this Readme_UnDot.txt file should be at:

www.trbarry.com/Readme_UnDot.txt

***************

Change Log:


2002/11/03 V 0.0.1.0 Initial test release for Avisynth 2.5 alpha only


If only the author could have written this for avisynth 2.05 or 2.06.
Bummer!!
However, Tom Barry may allow his source code to be used for avisynth
2.05 or 2.06. Maybe

-black prince
Reply With Quote
  #16  
11-08-2002, 06:06 PM
TKS TKS is offline
Free Member
 
Join Date: May 2002
Posts: 118
Thanks: 0
Thanked 0 Times in 0 Posts
In Good Wil Hunting there are also some darker bar scenes and these are much better then the average daylight scenes. Pratically no mosquito effect that is really noticable..

tks
Reply With Quote
  #17  
11-09-2002, 08:43 AM
kwag kwag is offline
Free Member
 
Join Date: Apr 2002
Location: Puerto Rico, USA
Posts: 13,537
Thanks: 0
Thanked 0 Times in 0 Posts
I'm trying something with the new Blockbuster filters ( the old AddNoise filters ), but can't get it to work. http://www.jungleweb.net/~sansgrip/avisynth/
I can use Blockbuster(method="show") or Blockbuster(method="noise") or Blockbuster(method="blur")
But I can't make Blockbuster(method="noise, 8, 10, False, 0, 1, 0, 1, 256 ) as in the documentation, or any different value. I also tried Blockbuster(method="noise", 8, 10, False, lm=0, lv=1, cm=0, cv=1, cache=256)
I've tried Blockbuster(method="noise") but with the default values, I don't see any difference in test encodes with or without the filter.
So if anyone has been able to make the filters work, passing arguments, please let me know.

-kwag
Reply With Quote
  #18  
11-09-2002, 10:45 AM
SansGrip SansGrip is offline
Free Member
 
Join Date: Nov 2002
Location: Ontario, Canada
Posts: 1,135
Thanks: 0
Thanked 0 Times in 0 Posts
Hi Karl, I tried to reply to your email but it was rejected by the server so I registered here instead . Here is my reply:

It's definitely working with non-default parameters, or at least it is here.

If you try it with a very high variance (say, lv=20) and a zero threshold (meaning do the whole frame) you'll see it working for sure. Personally I always use named arguments, and that may be why it's not working for you. Try something like:

Blockbuster(method="noise", detail_threshold=0, lv=20)

and you should see very strong noise all over the frame.

Thanks for being open-minded and trying a little noise .
Reply With Quote
  #19  
11-09-2002, 11:47 AM
SansGrip SansGrip is offline
Free Member
 
Join Date: Nov 2002
Location: Ontario, Canada
Posts: 1,135
Thanks: 0
Thanked 0 Times in 0 Posts
I just read the whole thread and am very glad you guys are finding it useful.

As far as the mosquito noise problem goes, it's obviously a high-frequency (high detail level) phenomena, which means theoretically the "invert" mode should be able to target it.

I implemented blur because this gigantic Doom9 thread mentioned that a combination of offsetting the luma by -2 and then using UnFilter either sharpening or blurring had an effect on DCT blocks in dark areas (incidentally, it was this thread that gave me the idea for the filter in the first place).

While I am still dubious about the effectiveness of blurring in already low-frequency areas, it's possible that in combination with invert mode it might work to reduce mosquito noise. If it does this'll be a totally unintended use for the filter .

If I get good reports about blur/invert for mosquito noise I'll have a look at the blur method again to see if I can improve it. Right now it just does a simple 3x3 average, and one possible improvement would be to make the radius configurable. Another option might be to incorporate temporal smoothing.

Anyway, I'm glad kwag emailed me with this thread. It's always nice to hear that what started out as an experiment is actually turning out to be useful .
Reply With Quote
  #20  
11-09-2002, 12:04 PM
SansGrip SansGrip is offline
Free Member
 
Join Date: Nov 2002
Location: Ontario, Canada
Posts: 1,135
Thanks: 0
Thanked 0 Times in 0 Posts
black prince: Not sure what LegalClip() is so it's commented.

I apologise for being off-topic, but I thought I'd just address this quickly.

LegalClip is unrelated to DCT blocks. Without going into too much detail, there's a standard called CCIR-601 which defines various elements of video signals destined for TV (all formats: NTSC, PAL, SECAM etc.). It specifies that luma pixel values must be in the range of 16-235 and chroma pixel values must be in the range of 16-240. Colours outside this range are known as "superblack" and "superwhite" and can cause problems on some TVs.

Since Avisynth can produce pixels outside these ranges, I wrote this filter to clamp to the legal ranges. In other words, luma pixels in the range 0-15 become 16 and 236-255 become 235. The same is done in chroma except the upper limit is 240.

If you look at the output of mpeg2dec you'll notice a lot of out-of-range pixels, since they seem to be a side-effect of MPEG compression. In this case these pixels convey no actual information -- they are just artifacts -- so removing them is both The Right Thing To Do For TVs (tm) and doesn't force the encoder to use up bits describing useless information that will get removed by the DVD player anyway.

Edit: When encoding from an MPEG-2 source I put one instance of LegalClip directly after Mpeg2Source so that the out-of-range artifacts don't interfer with the results of other filters in the script, and then one at the very end to remove any that might have been generated during processing.
Reply With Quote
Reply




Similar Threads
Thread Thread Starter Forum Replies Last Post
""""""Por favor me ajude" ficoboy Conversão e Codificação de Vídeo (Português) 49 08-24-2004 05:35 PM
Quais os codec's? """"""" ficoboy Conversão e Codificação de Vídeo (Português) 1 05-11-2004 10:14 AM
SansGrip's Filters: "I stopped working on the filters." SansGrip Avisynth Scripting 8 04-28-2004 07:38 PM
CQ con Tok "automatico" o CQ_VBR "manual" r1racing1 Convertir y Codificar Video (Español) 4 11-27-2003 05:14 PM
SansGrip Filters: How To Use "Overscan" together with Letterbox Jellygoose Avisynth Scripting 7 02-11-2003 05:57 AM

Thread Tools



 
All times are GMT -5. The time now is 09:40 PM  —  vBulletin © Jelsoft Enterprises Ltd