How to replace frames with duplicate frames?

[bmichaelb]

I have a tape which has some vertical jitters, and unfortunately, they aren't limited to a single field, or I'd simply discard the bad field and interpolate into a good frame. Luckily they're all limited to a single frame at a time though, so I was hoping to replace the bad frames with a duplicate of their previous frames, then run a script from johnmeyer that replaces the duplicate frames with new frames interpolated from the previous and following frames. I can manually scan the clip and copy the frame numbers to be fixed in virtualdub, I just need to figure out how to replace them with duplicates of their previous frames in avisynth before I can begin.

Any ideas appreciated.

Edit: Forgot to mention that I'll be dumping the wav file before inserting the duplicate frames, then I'll weave it back in after the frames are fixed.

-- merged --

Figured it out, or at least figured out 'my' way.

clip1=AviSource("G:\video.avi").Trim(0,500)
clip2=AviSource("G:\video.avi").Trim(500,500)
clip3=AviSource("G:\video.avi").Trim(502,112233)
clip1+clip2+clip3

This should also work for frames with stubborn spots or streaks, as long as it's a single frame issue. Just replace the frame with a duplicate, then run the FillDropsI script by johnmeyer. But for issues spanning more than a single frame, I came across a video on YouTube suggesting After Effects FrameRestorer can handle multiple duplicate frames in a row, so maybe if it's a streak running 2 or 3 frames in succession, this may work as well. After Effects seems to be a monthly subscription now, so it wouldn't be as expensive as a full purchase.

[lollo2]

Not sure if this helps, but the following is what I normally do for vertical jitter caused by field shift and for bad frames.

If you experience a shift in a field there is no need to interpolate, you can use this function to correct the shifted field:

function shift_fields_GMa(clip c, int frame_number, line_shift_even, line_shift_odd)
{
# separate fields tff
c_tff_sep=c.AssumeTFF().separateFields()

# separate fields tff even
c_tff_sep_even=c_tff_sep.SelectEven()

# separate fields tff odd
c_tff_sep_odd=c_tff_sep.SelectOdd()

# shift field even
c_tff_sep_even_rep = (line_shift_even > 0) ?\
	c_tff_sep_even.trim(0,frame_number-1)\
	++c_tff_sep_even.trim(frame_number,frame_number).crop(0,0,0,-line_shift_even).addborders(0,line_shift_even,0,0)\
	++c_tff_sep_even.trim(frame_number+1,0)\
:\
	c_tff_sep_even.trim(0,frame_number-1)\
	++c_tff_sep_even.trim(frame_number,frame_number).crop(0,-line_shift_even,0,0).addborders(0,0,0,-line_shift_even)\
	++c_tff_sep_even.trim(frame_number+1,0)

# shift field odd
c_tff_sep_odd_rep = (line_shift_odd > 0) ?\
	c_tff_sep_odd.trim(0,frame_number-1)\
	++c_tff_sep_odd.trim(frame_number,frame_number).crop(0,0,0,-line_shift_odd).addborders(0,line_shift_odd,0,0)\
	++c_tff_sep_odd.trim(frame_number+1,0)\
:\
	c_tff_sep_odd.trim(0,frame_number-1)\
	++c_tff_sep_odd.trim(frame_number,frame_number).crop(0,-line_shift_odd,0,0).addborders(0,0,0,-line_shift_odd)\
	++c_tff_sep_odd.trim(frame_number+1,0)

# repaired video
c_rep=interleave(c_tff_sep_even_rep,c_tff_sep_odd_rep).Weave()

return(c_rep)
}

If you see in the frame X a shift of Y line up in the even field you use shift_fields_GMa(X, Y, 0)
If you see in the frame X a shift of Y line down in the even field you use shift_fields_GMa(X, -Y, 0)
If you see in the frame X a shift of Y line up in the odd field you use shift_fields_GMa(X, 0, Y)
If you see in the frame X a shift of Y line down in the odd field you use shift_fields_GMa(X, 0, -Y)

If you need to replace a bad frame with an interpolated frame from frame-1 and frame+1 you can use this function

function interpolate_frame_GMa_1(clip c, int "frame_number")
{
	# interpolate bad frame
sup = c.MSuper()\
bv = MAnalyse(sup, isb=true, delta=2)
fv = MAnalyse(sup, isb=false, delta=2)
interpolated = MFlowInter(c, sup, bv, fv)

#stackhorizontal(c, interpolated)
#return(interpolated)

	# replace bad frame with interpolated frame
video_repaired = \
	c.trim(0,frame_number-1)++\
	interpolated.trim(frame_number-1,-1)++\
	c.trim(frame_number+1,0)

	# video repaired
return(video_repaired)
}

if you see a bad frame number X you use interpolate_frame_GMa_1(frame_number=X)


When dealing with multiple consecutive bad frames you need to replace the additional bad frames with a duplicate of the good frames and then interpolate one by one; in this example how to proceed for 7 consecutives bad frames

# interpolate frame GMa_1 2921-2927
# a x      x     x    x    x    x     x      b
# a a      a     a    x    b    b     b      b
# a a      a     a    iab  b    b     b      b
# a a      a     iaab iab  b    b     b      b
# a a      iaaab iaab iab  b    b     b      b
# a iaaaab iaaab iaab iab  b    b     b      b
# a iaaaab iaaab iaab iab  iabb b     b      b
# a iaaaab iaaab iaab iab  iabb iabbb b      b
# a iaaaab iaaab iaab iab  iabb iabbb iabbbb b

# replace frames 2921, 2922 and 2923 with frame 2920 and frames 2925 and 2926, 2927 with frame 2928
video_org_rep1=video_org.trim(0,2920)++video_org.trim(2920,2920)++video_org.trim(2920,2920)\
++video_org.trim(2920,2920)++video_org.trim(2924,2924)++video_org.trim(2928,2928)\
++video_org.trim(2928,2928)++video_org.trim(2928,2928)++video_org.trim(2928,0)

# interpolate bad frames
video_org_rep2a=video_org_rep1.interpolate_frame_GMa_1(frame_number=2924)
video_org_rep2b=video_org_rep2a.interpolate_frame_GMa_1(frame_number=2923)
video_org_rep2c=video_org_rep2b.interpolate_frame_GMa_1(frame_number=2922)
video_org_rep2d=video_org_rep2c.interpolate_frame_GMa_1(frame_number=2921)
video_org_rep2e=video_org_rep2d.interpolate_frame_GMa_1(frame_number=2925)
video_org_rep2f=video_org_rep2e.interpolate_frame_GMa_1(frame_number=2926)
video_org_rep2=video_org_rep2f.interpolate_frame_GMa_1(frame_number=2927)

My interpolate_frame_GMa_1 function is based on http://avisynth.org.ru/mvtools/mvtools2.html. In doom9's forum you can find alternative procedures written by several users for frame interpolation.

edit: to avoid audio problems with AlignedSplice ++ AviSynth command, better to dub with the original audio after the shifting and the replacement:

# use audio from original clip
video_restored=audioDub(video_org_rep2, video_org)

[hodgey]

Quote:

Originally Posted by bmichaelb

Figured it out, or at least figured out 'my' way.

FreezeFrame can be used to use duplicates it a bit easier. There are also some plugins to blend/interpolate a bit though trying to correct shifted fields might be better if the fields are just shifted.

[traal]

My own, similar solution to this problem that LordSmurf calls "vertical bounce" is here.

I would like it to be able to apply the same correction to multiple frames in a row, but I haven't figured out how to script that yet.

[keaton]

Similar to FreezeFrame, Avisynth also has DeleteFrame and DuplicateFrame. Neither of which effect the audio. So long as you have a Duplicate for every Delete, you won't change the audio/video sync.

If you call DeleteFrame(101), frame 102 now becomes 101. If you want to duplicate the frame after, you would call DuplicateFrame(101). Likewise if you wanted to duplicate the frame before, you would call DuplicateFrame(100). FreezeFrame may be easier. But both accomplish the same thing.

Similar to what others have posted, I have a custom function I use to fix jitter. It's very tedious if you have a lot of frames to fix. See http://www.digitalfaq.com/forum/vide...html#post64626 What I don't mention in the post is you can get blank rows of video at the top of the frame if you move the frame down more than you moved a field up. So you then need to find a way to interpolate that blank space. What I often do is use a function like this to patch up things.

function blockreplace(clip c, int targetframe, int refframe, int left, int top, int right, int bottom, int xoff, int yoff, int "cols", int "rows")
{
  numrows = Default(rows,480) # Default 480 rows if not specified
  numcols = Default(cols,720) # Default 720 columns if not specified
  f1=c.trim(targetframe,targetframe)
  f2=c.trim(refframe,refframe)
  f2b=f2.crop(left,top,(right-numcols),(bottom-numrows))
  newf=Overlay(f1,f2b,x=(left+xoff),y=(top+yoff),opacity=1.0)
  result=c.trim(0,targetframe-1)++newf++c.trim(targetframe+1,0)
  return result
}

Assuming a 720 x 480 video, to copy the top 10 rows of frame 100 onto frame 101, you would call blockreplace(101,100,0,0,720,10,0,0). If there's a bit of motion between the 2 frames, you could shift it left or right with negative or positive values in the 2nd to last number, and vertically with the last number.

This is pretty meticulous frame by frame type work, which I would understand you trying to avoid.

You seem to have a script already in mind for the cleanup of bad frames. So I won't go into this much, but I will mention ReplaceFramesMC.avsi. It usually works very well on short numbers of frames when there is not much change. Sometimes I'm amazed at what it can do. But when it doesn't work, the results can look rather freaky. You can use it to replace the entire frame, or you can get into functions like crop and Overlay to fix only the parts of the frame that need it if the rest of the frame is good and there's too much motion in the rest. It's been discussed on the forum and requires other plugins and all of the Microsoft Visual C++ libraries installed. If you have QTGMC setup, then you likely have the prerequisites installed. Here's a post with a link to the avsi file http://www.digitalfaq.com/forum/vide...html#post57869 You can search the forum for more. sanlyn posted about it a few times. Here's one example where it's used to fix a small part of a frame. http://www.digitalfaq.com/forum/vide...html#post49761

I believe this filter does effect audio if used directly. If you used Overlay to put results from ReplaceFramesMC over the original frame(s), then you would not effect the audio. Or you can use AudioDubEx (http://avisynth.nl/index.php/AudioDub) to use the audio of the original, untouched clip with the video of the final clip. Example:

myVid=AVISource("input.avi")
AssumeTFF(myVid)
audio = last # save a copy of file before I make changes, last is a special variable that holds the latest copy up to this point
<Insert work here>
video = last
AudioDubEx(video,audio)

[bmichaelb]

Thanks lollo2, greatly appreciated. But with my approach, I wouldn't need to figure out which line was the bad line by separating fields, I'd just replace the entire frame after seeing it jump. But your section on interpolating the frame is definitely interesting. I'll be testing it out this weekend. I know for a fact I have a frame in one of my tapes that has a pretty big black streak/dropout that even RemoveDirtMC would have issues with, so simply replacing the frame altogether would be the easiest approach.

Thanks hodgey, I'll take a look at that this weekend.Cheers.

Thanks traal, I'll check it out. Cheers.

Thanks keaton. I came across ReMapFrames and ReplaceFramesSimple, but it didn't seem to be what I was looking for. Or at least I didn't understand it enough.

Cheers.

[lollo2]

Quote:

But with my approach, I wouldn't need to figure out which line was the bad line by separating fields, I'd just replace the entire frame after seeing it jump.

Yes, much much easier, but in this case you "alter" the original video, even if the interpolate functions of MVTools are excellent. In addition you cannot interpolate a shifted frame at a scene change.

Quote:

But your section on interpolating the frame is definitely interesting.

As keaton properly suggested (I would also add ReplaceFramesSVPFlow from https://forum.doom9.org/showthread.p...05#post1839605, there are existing AviSynth functions to interpolate 1 or more bad frame. I found sometimes on my videos that the final result was not good for several consecutive frames, and then used a customized approach when needed.
On top of that, most of the existing functions may have incorrect management of backward and forward vectors:
https://forum.doom9.org/showthread.p...47#post1409147
https://forum.doom9.org/showthread.php?p=1838775

[traal]

Quote:

Originally Posted by keaton

This is pretty meticulous frame by frame type work, which I would understand you trying to avoid.

It's meticulous but requires little to no creativity, making it an ideal task for crowdsourcing, much like Project Gutenberg Distributed Proofreaders.

Or maybe find some neighborhood kids who need a job and don't want to be outside doing yard work in this heat.

[bmichaelb]

Quote:

Originally Posted by lollo2

As keaton properly suggested (I would also add ReplaceFramesSVPFlow from https://forum.doom9.org/showthread.p...05#post1839605, there are existing AviSynth functions to interpolate 1 or more bad frame. I found sometimes on my videos that the final result was not good for several consecutive frames, and then used a customized approach when needed.
On top of that, most of the existing functions may have incorrect management of backward and forward vectors:
https://forum.doom9.org/showthread.p...47#post1409147
https://forum.doom9.org/showthread.php?p=1838775

I understand that it's a guesstimation from the available information between the previous and following frames, but with my limited use of interpolation for replacing duplicate frames, I haven't seen any real issues. That's just a single frame though, not multiples in a row. That's why I suggested the After Effects FrameRestorer, as the video I watched had it doing 5 frames in a row, using AI. Even then I realize it may not be perfect in some situations, but for just 2 or 3 frames, it should be pretty good.

[nvrsk]

Quote:

Originally Posted by lollo2

use this function to correct the shifted field

I have recently faced some random jitter problems and made a fixing script.
hope that someone finds it helpful. =)

my script uses FrameSel() plugin to run:

LoadCPlugin("ffms2.dll")
LoadCPlugin("FrameSel_x64.dll")

global OPT_Enable_V210 = true
REJECT=False

FFVideoSource("capture.avi").KillAudio().SeparateFields()
#.ScriptClip(" Subtitle(String(current_frame)) ")

JumpFrames1="478;930;1324;1938;7685;9012;10174;10922;19863;25242"
JumpFrames2="9015;9214;10434;18651;19860"

SELECTED=FrameSel(scmd=JumpFrames1,ordered=TRUE,reject=REJECT,debug=true)
SELECTED=crop(SELECTED,0,0,0,-1).addborders(0,1,0,0)
FrameRep(Last,SELECTED,scmd=JumpFrames1,reject=REJECT,debug=true)

SELECTED=FrameSel(scmd=JumpFrames2,ordered=TRUE,reject=REJECT,debug=true)
SELECTED=crop(SELECTED,0,0,0,-2).addborders(0,2,0,0)
FrameRep(Last,SELECTED,scmd=JumpFrames2,reject=REJECT,debug=true)

Weave()

the source video has 1px and 2px random field jitter, so I've made two lists - JumpFrames1 and JumpFrames2, which holds the lists of problem fields (as frames after SeparateFields), so that I don't need to bother with complex script constructions and do just these two lists population and a very compact structure.

I have also made a portion of code to inject edited by hand frames as PNG-images (with the same way) as manual repair, but it needs optimizing. if anyone interested I can share it also.