#1  
04-20-2020, 01:45 AM
latreche34 latreche34 is online now
Free Member
 
Join Date: Dec 2015
Location: USA
Posts: 3,257
Thanked 537 Times in 497 Posts
I have a file that I captured from a very old VHS tape (possible early 70's), I use BM MediaExpress to capture lossless AVI 4:2:2 @ 10bit video and 24Bit audio, I then use Vdub to crop off 2 lines from top and 4 lines from the bottom of the frame to get 480 lines. This particular tape has some consistent noise that I want to slightly reduce, What filter should I use in QTGMC or other applications?

The 40 seconds sample is over 1GB, So I down converted it to 8 bit but it is still 279 MB so I attach it to an external cloud storage here, I however did include a small encoded version but compression smoothed out the noise so it may not show the problem accurately.

But I would really appreciate it if someone takes a look at the 10 bit original sample here with the true parameters and help me with a working script (I suck at scripting), Thanks a lot.


Attached Files
File Type: mp4 LineTBCOn.mp4 (53.38 MB, 17 downloads)
Reply With Quote
Someday, 12:01 PM
admin's Avatar
Ads / Sponsors
 
Join Date: ∞
Posts: 42
Thanks: ∞
Thanked 42 Times in 42 Posts
  #2  
04-20-2020, 05:07 AM
lordsmurf's Avatar
lordsmurf lordsmurf is offline
Site Staff | Video
 
Join Date: Dec 2002
Posts: 13,508
Thanked 2,449 Times in 2,081 Posts
It still needs some slight masking around all sides, maybe 2px at top/bottom, 10px total to recenter left/right.

The image seems fine. I'd hesitate to remove any more grain.

I do know film flecks, little white/black dust, boogers and hairs. For that, various dirt strips for film are probably needed. And for those, videofred's and johnmeyer's would be most helpful. (Neither of them post here, but I wish they would -- and I've asked.) Fred doesn't do much anywhere anymore, but John does at VH and Doom9.

BTW, that script is here, though I'm not sure if it's the latest, or unmolested (as it was shared via another user).
I've PM'd him about it.

Code:
/*
Film restoration script by videoFred.
Denoising, resizing, stabilising, sharpening, auto-levels and auto-white balance.
 
Modified by John Meyer, October 7, 2012
Modified by John Meyer, July 7, 2015 (remove errors)
Modified by John Meyer, September 13, 2017
 
Changes made by Meyer (June 8, 2010):
    1. Replaced the MVDegrainMulti function, which was part of the original MVTools
       with the equivalent function using the newer and faster MVTools2.
    2. Added RemoveDirt function prior to Degrain in order to eliminate large dust spots. This
       substantially improves the dirt removal capabilities of this script. The user will need to 
       download this plugin at:
           http://www.removedirt.de.tf/
    3. Eliminated a great deal of sharpening. The original script did sharpening in at least four places.
       With grainy film stock, this sometimes created objectionable grain. Also, the limitedSharpenFaster
       function, while excellent for video, is uncessarily slow, and isn't really needed for this low 
       resolution source. Too much sharpening can make the film look too much like video.
    4. Added multithreading. I was able to roughly triple the script performance. It may be possible to
       substantially increase this, perhaps as much as 12x instead of 3x. However, the autolevels function
       would have to be replaced.
    5. Fixed several things I found. The result3 option didn't have the manual color correction code, so I 
       added that back in. All the numbered "stab" variables (stab1, stab2, etc.) aren't needed except 
       for providing a test function for stabilization. I eliminated all of this to streamline the script.
    6. Deflicker seemed redundant, given all the averaging that takes place with MDegrain, and also the averaging
       that is done when the autolevels outputs are selected. 
    7. I reduced the number of frames used for averaging autolevels from the default (which is 5) to 2.
       I probably should add a variable in the header so the user can change this. Something else to do
       in the future ...
    8. I added yet another set of crop parameters. I did this because both my capture and my output are 
       done using NTSC DV AVI which is 720x480. However, 8mm film is almost exactly square, so the captured
       720x480 video has black bars on the side. These need to be cropped off prior to doing motion 
       stabilization, but then added back prior to the final output, which must still be 720x480 with the
       black bars on the side.
    9. I reduced the default depan settings to 20, which is what was recommended in the original script. I
       also reduced the post-depan cropping. I did this so I could keep as much of the original frame
       as possible.
   10. I removed the second denoising and sharpening function. It just seemed to be too much, and made the 
       result too artificial
   11. I removed the MVFLowFPS interpolation. I did this during one of dozens and dozens of attempts
       to improve the speed of the script. I should probably add this back, but if I do so, I also 
       need to make it work correctly for interlaced output. If the goal is to show this on an NTSC or PAL
       television set, then it is not correct to convert from the film fps to 25 fps progressive (PAL) or
       29.97 fps progressive (NTSC). Instead, this should be done as follows (example given is NTSC):
 
          MFlowFPS(source,super,backward_vec, forward_vec, num=60000, den=1001,ml=200)
          SeparateFields()
          SelectEvery(4, 0, 3)
          Weave()
 
       This yields interlaced 29.97, which has twice the temporal resolution as 29.97 progressive, and will
       therefore look correct on a TV set. I have done a lot of this as part of my Kinescope to video
       conversion scripts. For those scripts, the goal is to make the filmed version of a TV show look
       like it was actually videotaped. However, for something that originated on film, this "does violence"
       to the original feel of the media. It is true that it can make horizontal pans less "juddery," but
       it won't feel like film anymore. Also, this technique does break down, espcially with fast motion
       in the foreground.
   12. Removed the unecessary "coloryuv(off_U=blue,off_V=red)" statement from the denoising section.
   13. Added killaudio() statement to prevent lockups when using SetMTMode().
 
Changes made on October 7, 2012
 
    1. Modified RemoveDirtMC function to do estimation in two steps. This improves performance and quality.
    2. Modified MDegrain to do estimation in two steps. This provides roughly the same quality with blocksize=8
       as what I was getting with blocksize=4, but with virtually no performance penalty
    3. Updated to use special build of MVTools2 that works with SVP.
    4. Moved Autolevels inside special multi-threading section in order to improve performance with autolevels.
    5. Removed all trim commands because I do all editing outside the script (in Vegas).
    6. Included two alternative ways to do frame interpolation: SVP and InterFrame. The InterFrame script is probably better
       but I included the SVP code because it may provide a starting point for more customized approaches. Neither is enabled by default.
 
Changes made on September 13, 2017
 
    1. Replaced old useless color correction with GamMac.
    2. Removed never-used manual controls.
    3. Moved deflicker to end of chain because GamMac introduces flicker.
    4. Moved RemoveDirtMC to happen AFTER denoising.
*/
#====================================================================================


#VIDEO FILE
#----------------------------------------------------------------------------------------------------------------------------
#Change the following line to point to your video file
film="01 Film 01.avi"


#GENERAL PARAMETERS
#----------------------------------------------------------------------------------------------------------------------------
result="result2"                                 #specify the wanted output here 
play_speed=24                            #play speed (8mm=16; Super8=18; 16mm sound=24)


#COLOR AND LEVELS PARAMATERS
#----------------------------------------------------------------------------------------------------------------------------
saturation = 1.0                                  #for all outputs
gamma      = 1.0                                  #for all outputs 

#GamMac Parameters
LockChan   = 1                                     #(0=red channel)
LockVal    = 128.0                                 #default 128 -- Used when LockChan = -1 (for flicker)
Scale      = 2                                     #Fred recommended 2 instead of 1
RedMul     = 1.0
GrnMul     = 1.0
BluMul     = 1.0
Th         = 0.1
GMx        = 0
GMy        = 0
GMw        = 0
GMh        = 0
LOTH = 0.2
HITH = 0.2
OMIN =   0                                         #limiting the output a little bit makes it a little 'softer' to look at
OMAX = 255
Al2  =  20
autolev_bord1 = 50
borderV=10 borderH=10    

#SIZE, CROP AND BORDERS PARAMETERS
#----------------------------------------------------------------------------------------------------------------------------
CLeft=0  CTop=0  CRight=0  CBottom=0                             #crop values after Depan and before final resizing 
W=1920  H=1080                                                   #final size after cropping 
bord_left=0        bord_top=0     bord_right=0     bord_bot=0    #720p= borders 150
in_bord_left=204  in_bord_top=8  in_bord_right=204  in_bord_bot=8    #Borders around input that must be removed

#STABILISING PARAMETERS
#----------------------------------------------------------------------------------------------------------------------------
maxstabH=20                                                      #maximum values for the stabiliser (in pixels) 20 is a good start value 
maxstabV=20
est_left=40  est_top=40  est_right=40  est_bottom=40             #crop and contast values for special Estimate clip
est_cont=1.1                                                     #Too large a value defeats stabilization


#DENOISING PARAMETERS
#----------------------------------------------------------------------------------------------------------------------------
#denoising_strength=600                    #denoising level of first denoiser: MVDegrain() 
#block_size=   16                           #block size of MVDegrain
#block_size_v= 16
#block_over=    8                           #block overlapping of MVDegrainMulti()

#Alternate denoising using smaller block size
#denoising_strength= 600                   #denoising level of first denoiser: MDegrain() 
#block_size=   8                           #block size of MVDegrain
#block_size_v= 8
#block_over=   4                           #block overlapping of MVDegrainMulti()

#Alternate denoising using smalles block size
denoising_strength= 600                   #denoising level of first denoiser: MDegrain() 
block_size=   4                           #block size of MVDegrain
block_size_v= 4
block_over=   2                           #block overlapping of MVDegrainMulti()

dirt_strength=54                           #sets amount of dirt removal (big spots)


#FOUR STEP SHARPENING PARAMETERS
#--------------------------------------------------------------------------------------------------------------------------------
#120 is pretty aggressive; 85 seems more subtle
#PRE_sharp_ness= 120   PRE_radi_us= 3            #presharpening (UnsharpMask) just after first denoising
 PRE_sharp_ness= 85    PRE_radi_us= 3            #presharpening (UnsharpMask) just after first denoising

Sharp_Strength = 0.05                             #0.05 is often better (avoid too much sharpening)


#AUTO LEVELS PARAMETER
#--------------------------------------------------------------------------------------------------------------------------------
X=4   #X  parameter for reducing the autolevels effect on the whites
X2=4  #X2 parameter for reducing the autolevels effect on the blacks 


#NUMBER OF THREADS
#--------------------------------------------------------------------------------------------------------------------------------
threads=3


# END VARIABLES, BEGIN SCRIPT
#=================================================================================================================================
SetMemoryMax(512)

#AddAutoloadDir("E:\Video\plugins\",toFront=true)

/*
Here is a list of plugins used
Loadplugin("mvtools2.dll")
LoadPlugin("mvtools2.dll")        #Version 2.5.11.9    2/24/2012
LoadPlugin("autolevels.dll")       #Version 0.6.0.0     1/09/2011
LoadPlugin("Deflicker.dll")        #Version 0.4.0.0     8/16/2004
Loadplugin("Depan.dll")            #Version 1.10.0.0    4/09/2007
LoadPlugin("DepanEstimate.dll")    #Version 1.9.2.0     3/25/2007
LoadPlugin("fft3dfilter.dll")      #Version 2.1.1.0     2/20/2007
Loadplugin("mt_masktools.dll")     #Version 2.0.23.0    3/14/2008
loadplugin("RemoveDirtSSE2.dll")   #Version 0.9         5/05/2005    
Loadplugin("RemoveGrainSSE2.dll")  #Version 0.9         5/01/2005
Loadplugin("removegrain.dll")     #Version 0.9         5/01/2005
Loadplugin("warpsharp.dll")        #                    4/05/2010
*/

source1=AviSource(film).killaudio().assumefps(play_speed).converttoYV12()
cropped_source=source1.crop(in_bord_left,in_bord_top,-in_bord_right,-in_bord_bot)               #remove any black borders on input video
#cropped_source=filldrops(cropped_source)     #Use this when removing bad frames that have been removed by duplicating previous frame


#STABILIZING
#....................................................................................................................................................................
stab_reference= cropped_source.crop(est_left,est_top,-est_right,-est_bottom).tweak(cont=est_cont).MT_binarize(threshold=80).greyscale().invert()
mdata=DePanEstimate(stab_reference,trust=1.0,dxmax=maxstabH,dymax=maxstabV)
#stab=DePanStabilize(cropped_source,data=mdata,cutoff=0.5,dxmax=maxstabH,dymax=maxstabV,method=1,mirror=15).deflicker()
stab=DePanStabilize(cropped_source,data=mdata,cutoff=0.5,dxmax=maxstabH,dymax=maxstabV,method=1,mirror=15)

#DENOISING
#...................................................................................................................................................................
input_to_removedirt=stab.crop(CLeft,CTop,-CRight,-CBottom)
stabcrop=RemoveDirtMC(input_to_removedirt,dirt_strength,false)

  prefiltered = RemoveGrain(stabcrop,2)
  superfilt =   MSuper(prefiltered, hpad=32, vpad=32,pel=2)
  super=        MSuper(stabcrop, hpad=32, vpad=32,pel=2)

  halfblksize= (block_size>4)   ? block_size/2 : 4
  halfoverlap= (block_over>2)   ? block_over/2 : 2

  bvec1 =  MAnalyse(superfilt, isb = true,  delta = 1, blksize=block_size, overlap=block_over,dct=0)
  bvec1 =  MRecalculate(super, bvec1, blksize=halfblksize, overlap=halfoverlap,thSAD=100) 

  fvec1 =  MAnalyse(super, isb = false, delta = 1, blksize=block_size, overlap=block_over,dct=0)
  fvec1 =  MRecalculate(super, fvec1, blksize=halfblksize, overlap=halfoverlap,thSAD=100)

  bvec2 =  MAnalyse(super, isb = true,  delta = 2, blksize=block_size, overlap=block_over,dct=0)
  bvec2 =  MRecalculate(super, bvec2, blksize=halfblksize, overlap=halfoverlap,thSAD=100)

  fvec2 =  MAnalyse(super, isb = false, delta = 2, blksize=block_size, overlap=block_over,dct=0)
  fvec2 =  MRecalculate(super, fvec2, blksize=halfblksize, overlap=halfoverlap,thSAD=100)

  denoised=stabcrop.MDegrain2(super, bvec1,fvec1,bvec2,fvec2,thSAD=denoising_strength).levels(0,gamma,255,0,255).tweak(sat=saturation).unsharpmask(PRE_sharp_ness,PRE_radi_us,0)


#SHARPENING
#...................................................................................................................................................................
sharp1=denoised.sharpen(Sharp_Strength)    
PreBorderFrame = sharp1.Lanczos4Resize(W - bord_left - in_bord_left - bord_right - in_bord_right, H - bord_top - in_bord_top - bord_bot - in_bord_bot)

#FRAME INTERPOLATION (optional)
#...................................................................................................................................................................
#Everything in this section is commented out because I usually choose NOT to do frame interpolation because 
#it causes too many unexpected artifacts on certain frames. It also makes the film look like video which
#aesthetically is not always pleasing.

/*
prefiltered = RemoveGrain(PreBorderFrame,2)
super = MSuper(PreBorderFrame,hpad=16, vpad=16, levels=1) # one level is enough for MRecalculate
superfilt = MSuper(prefiltered, hpad=16, vpad=16) # all levels for MAnalyse
backward = MAnalyse(superfilt, isb = true, blksize=16,overlap=8,search=3,dct=0)
forward = MAnalyse(superfilt, isb = false, blksize=16,overlap=8,search=3,dct=0)
forward_re = MRecalculate(super, forward, blksize=8, thSAD=100)
backward_re = MRecalculate(super, backward, blksize=8, thSAD=100)
PreBorderFrame = MFlowFps(PreBorderFrame,super, backward_re, forward_re, num=30000, den=1001,ml=200,mask=2)
*/

#Alternative interpolation
/*
superfps= MSuper(PreBorderFrame,pel=2)
backward_vec2 = MAnalyse(superfps, isb = true,blksize=16)
forward_vec2  = MAnalyse(superfps, isb = false,blksize=16)
PreBorderFrame = MFlowFps(PreBorderFrame,superfps, backward_vec2, forward_vec2, num=60000, den=1001, ml=200).SeparateFields().SelectEvery(4, 0, 3).Weave()
*/

#Yet another way to do interpolation
#The following alternative can provide synthesized frames for hi-def material that causes MSuper to choke
#////// Begin alternative to MFlowFPS ////
#SmoothFPS2(PreBorderFrame,threads)
#////// End alternative to MFlowFPS ////

#A final way to do interpolation, although Interframe is just a front-end to SmoothFPS, so I don't think the results are any different.
#The following is another alternative that can provide synthesized frames for hi-def material that causes MSuper to choke. Probably better than the one above.
#////// Begin alternative to MFlowFPS ////
#PreBorderFrame=InterFrame(NewNum=30000,NewDen=1001,PreBorderFrame,GPU=true,Cores=threads)
#////// End alternative to MFlowFPS ////

#RESULT1: AUTOLEVELS,AUTOWHITE
#......................................................................................................................................................................

Baseclip = PreBorderFrame.crop(borderV,borderH,-borderV,-borderH,align=true).bicubicresize(W,H)
blank_black = Blankclip(baseclip, width=autolev_bord1,height=autolev_bord1)
blank_white=  Blankclip(baseclip, width=autolev_bord1,height=autolev_bord1, color=$FFFFFF)
Average= baseclip.FredAverage().invert()
over1 = overlay(baseclip,blank_black, x=40,y=300) 
over2 = overlay (over1,blank_white, x=160, y=300) \
        .bicubicresize(width(baseclip)-(al2)*2,height(baseclip)-(al2)*2)
Detect = (al2 >1) ? overlay (Average, over2,x=al2,y=al2) \
         .converttoRGB24(matrix="rec709") : over2.converttoRGB24(matrix="rec709")

result1= PreBorderFrame.ConvertToRGB24.GamMac(verbosity=4,DC=Detect,Show=False, \
         LockChan=LockChan, Th=Th, LockVal=LockVal, Scale=Scale, RedMul=RedMul,\
         GrnMul=GrnMul, BluMul=BluMul, loTh=LOTH,hiTh=HITH,oMin=OMIN,oMax=OMAX,\
         x=GMx,y=GMy,w=GMw,h=GMh).converttoYV12().deflicker().addborders(X,0,0,0,$FFFFFF) \
         .addborders(0,0,X2,0,$000000).autolevels(filterRadius=2).crop(X,0,-X2,-0) \
         .addborders(bord_left+in_bord_left, bord_top+in_bord_top, \
         bord_right+in_bord_right, bord_bot+in_bord_bot)

#RESULT2: AUTOWHITE
#......................................................................................................................................................................
result2= PreBorderFrame.ConvertToRGB24.GamMac(DC=Detect,Show=False, Dither=TRUE,verbosity=5,LockChan=LockChan, \
         Th=Th, LockVal=LockVal, Scale=Scale, RedMul=RedMul, GrnMul=GrnMul, BluMul=BluMul, \
         loTh=LOTH,hiTh=HITH,oMin=OMIN,oMax=OMAX,x=GMx,y=GMy,w=GMw,h=GMh).converttoYV12() \
         .deflicker().addborders(bord_left+in_bord_left, bord_top+in_bord_top, \
         bord_right+in_bord_right, bord_bot+in_bord_bot)

#RESULT3: AUTOLEVELS
#.....................................................................................................................................................................
result3= PreBorderFrame.addborders(X,0,0,0,$FFFFFF).addborders(0,0,X2,0,$000000).autolevels(filterRadius=2).crop(X,0,-X2,-0).addborders(bord_left+in_bord_left, bord_top+in_bord_top, bord_right+in_bord_right, bord_bot+in_bord_bot)

#RESULT4: NO CORRECTIONS
#.....................................................................................................................................................................
result4= PreBorderFrame.addborders(bord_left+in_bord_left, bord_top+in_bord_top, bord_right+in_bord_right, bord_bot+in_bord_bot)

#PARAMETERS FOR THE COMPARISONS
#.....................................................................................................................................................................
W2= W+bord_left+bord_right
H2= H+bord_top+bord_bot
source4=Lanczos4Resize(source1,W2,H2)


#COMPARISONS: ORIGINAL VS RESULTS
#......................................................................................................................................................................
resultS1= stackhorizontal(subtitle(source4,"original",size=28,align=2),subtitle(result1,"autolevels, GamMac",size=28,align=2))
resultS2= stackhorizontal(subtitle(source4,"original",size=28,align=2),subtitle(result2,"GamMac",size=28,align=2))
resultS3= stackhorizontal(subtitle(source4,"original",size=28,align=2),subtitle(result3,"autolevels",size=28,align=2))
resultS4= stackhorizontal(subtitle(source4,"original",size=28,align=2),subtitle(result4,"no level or color corrections",size=28,align=2))

Eval(result)
#return Detect
# Enable MT!
Prefetch(threads)


# END SCRIPT, BEGIN FUNCTIONS

#=================================================================================================================================
#REMOVE DIRT FUNCTION
#......................................................................................................................................................................
function RemoveDirt(clip input, int "limit", bool "_grey")
{
  clensed=input.Clense(grey=_grey, cache=4)
  alt=input.RemoveGrain(2)
  return RestoreMotionBlocks(clensed,input,alternative=alt,pthreshold=6,cthreshold=8, gmthreshold=40,dist=3,dmode=2,debug=false,noise=limit,noisy=4, grey=_grey)

  # Alternative settings
  # return RestoreMotionBlocks(clensed,input,alternative=alt,pthreshold=4,cthreshold=6, gmthreshold=40,dist=1,dmode=2,debug=false,noise=limit,noisy=12,grey=_grey,show=true)
  # return RestoreMotionBlocks(clensed,input,alternative=alt,pthreshold=6,cthreshold=8, gmthreshold=40,dist=3,tolerance= 12,dmode=2,debug=false,noise=limit,noisy=12,grey=_grey,show=false)
}

function RemoveDirtMC(clip,int "limit", bool "_grey")
{
  _grey=default(_grey, false)
  limit = default(limit,6)

# Alternative way to do pre-filtering.
#  prefiltered = fft3dfilter(clip,sigma=1,sigma2=2,sigma3=3,sigma4=5,bw=64,bh=64)
  
  prefiltered = RemoveGrain(clip,2)
  superfilt = MSuper(prefiltered, hpad=32, vpad=32,pel=2)

  super=MSuper(clip, hpad=32, vpad=32,pel=2)

  bvec = MAnalyse(superfilt,isb=true,  blksize=16, overlap=2,delta=1, truemotion=true)
  fvec = MAnalyse(superfilt,isb=false, blksize=16, overlap=2,delta=1, truemotion=true)

# Increase thSAD if moving objects are being removed
  bvec_re = Mrecalculate(super,bvec,blksize=8, overlap=0,thSAD=100)
  fvec_re = Mrecalculate(super,fvec,blksize=8, overlap=0,thSAD=100)

  backw = MFlow(clip,super,bvec_re)
  forw  = MFlow(clip,super,fvec_re)

  clp=interleave(forw,clip,backw)
  clp=clp.RemoveDirt(limit,_grey)
  clp=clp.SelectEvery(3,1)
  return clp
}

- Did my advice help you? Then become a Premium Member and support this site.
- For sale in the marketplace: TBCs, workflows, capture cards, VCRs
Reply With Quote
  #3  
04-20-2020, 01:40 PM
latreche34 latreche34 is online now
Free Member
 
Join Date: Dec 2015
Location: USA
Posts: 3,257
Thanked 537 Times in 497 Posts
Oh wow that's a lot to digest, a giant multi function script, Thanks Lordsmurf.
Just to be clear, if I don't want a filter I just delete its line or group of lines, right?
Reply With Quote
Reply




Similar Threads
Thread Thread Starter Forum Replies Last Post
How to remove grain, how to fix film shot in low light? RyfromNY Restore, Filter, Improve Quality 28 09-16-2018 09:56 PM
Capture video, then filter in MPEG for DVD? rocko Restore, Filter, Improve Quality 4 02-08-2018 09:09 PM
What is the best video resize filter/method? lordsmurf Restore, Filter, Improve Quality 0 02-02-2018 11:06 PM
Can't remove video noise/artifacting (grain in dark areas?)) DeeSeven Restore, Filter, Improve Quality 8 07-23-2014 08:19 AM
Help restoring dot crawl, dropouts, and grain on VHS videos? zack82 Restore, Filter, Improve Quality 18 04-29-2014 08:35 PM

Thread Tools



 
All times are GMT -5. The time now is 02:25 AM