Averaging multiple captures? (test samples included)

[juhok]

Inspired by jmac698's VHS averaging I made some averaging tests with BetacamSP material. I have many SP decks with worn heads which provide good samples. This tape/deck combo played out pretty good and I didn't have the huge amount of dropouts I hoped for, but there's something. And it's also interesting to see if it helps with noise.

Tools: Avisynth, Average (newer versions towards the end of the thread).

Samples (ffdshow huffyuv codec):
Capture 1 - straight from deck
3 cap average
Difference between the above (Avisynth, Subtract)

As anticipated, there's a bit less noise. Nothing spectacular. I guess the format's pretty noise free already. A bit of softening can be seen. What I didn't expect is that horizontal line 1/3 down from the top which is visible in the subtracted video. Looks like a mild physical damage in the tape, maybe.

[admin]

That source is really clean -- I see nothing wrong in either version.
Maybe I blinked?

I'm also not entirely sure I understand what this filter is supposed to be doing here. Is this an automatic method to blend several captures, and take the best version from each? The usage notes in the thread mean nothing to me -- no idea what "alpha" and "gamma", etc is supposed to mean in plain words. My problem with Avisynth is those who write instructions for it often assume you already know lots of other things.

Avisynth plugin filter "Average" attached below...

[juhok]

The source is almost perfect. It's a bad example in this regard but I didn't have patience to dig another 30kg machine for redoing it. I'll try it on some VHS tapes later.

Alfa, Beta etc. are your input. They could be called clip1, clip2, etc. Here's my script if it helps to understand the principle:

v1 = AVISource("E:\avetest\1.avi").trim(16, 1016)
v2 = AVISource("E:\avetest\2.avi").trim(24, 1024)
v3 = AVISource("E:\avetest\3.avi").trim(20, 1020)

Average(v1, 1.0/3, v2, 1.0/3, v3, 1.0/3)

If I were to have 4 input clips, it'd look like:

v1 = AVISource("E:\avetest\1.avi").trim(16, 1016)
v2 = AVISource("E:\avetest\2.avi").trim(24, 1024)
v3 = AVISource("E:\avetest\3.avi").trim(20, 1020)
v4 = AVISource("E:\avetest\4.avi").trim(20, 1020)

Average(v1, 0.25, v2, 0.25, v3, 0.25, v4, 0.25)

So it's "input clip", then it's "weight", next input and so on. Input must be trimmed so that they match temporally on every frame.

I'm in a habit of "using, not thinking about how it works", but definition for weighted average is:

Quote:

An average in which each quantity to be averaged is assigned a weight. These weightings determine the relative importance of each quantity on the average. Weightings are the equivalent of having that many like items with the same value involved in the average.

If not mistaken the principle is that majority pixels rule. Odd errors get "voted out".

Sample of an error frame(number 27) included. I'd have missed this one in normal workflow. It makes me laugh a bit to think about the "sales talk" of offering this kinda service to people. I have very hard time in the present just presenting that quality hardware and postprocessing is worth the tiny difference in costs Vs some of my "competitors" who use VHS-combo decks and such.

[jmac698]

I think I can explain this another way. Your original recording is x. The tape playback added some noise, n. So you have x+n in each capture. Let's say your original pixel was 10. Your noise is random, let's say -2,3,-1 each time you capture it. So let's just average them together:
10-2=8 capture 1
10+3=13 capture 2
10-1=9 capture 3
-------
(8+13+9)=30
30/3=10 average of 3 captures, in this case exactly the original.

This works because noise always averages to 0. If you added a huge amount of noise, with enough captures and averaging, it has to come out less noisy.

Neat story: The amazing thing is, Shannon's theorem says that you can always find the tiniest signal is the biggest noise. We've come very close to this with cellphones which perform close to the theoretical limits. Instead of averaging, we use Viterbi decoding - but that's another story

Another technique is called median. This is even simpler! Take the pixel values, sort them, and pick the middle of the list. That will also be close to the original - and better yet, it automatically throws out any extreme values like black or white lines.

Example: 8,13,9 -> sorted to 8,9,13 -> middle is 9.

I think this is superior because, in his capture above, one of the caps had a big line in the middle. Instead of includiing this line in the average, with median, it's completely ignored, because it will be like the '8' above, and thrown out as an extreme value.

Where does the blurring come from in averaging? I believe this is because of slight jitter left in the video. Here's why:

Let's say my original pixels are 0,50,90 ... this is a sharp edge going from black to white. Now lets say in capture 2 it's shifted 1 pixel to the left: 50,90.... now look what happens when I average the two captures: (0+50)/2,(50+90)/2,(90+?)/2 = 25,70.... my edge isn't so sharp anymore! It also starts sooner. So the 'energy' is being spread out over more horizontal area.

Median avoids this problem as well, it will be less blurry.

I'd like to see the original 3 captures for download.

[jmac698]

Basics of scripting: If you learn just these two commands, it's all you need to use a lot of plugins.
v1 = AVISource("E:\avetest\1.avi").trim(16, 1016)
AVISource opens a video file. We're going to refer to that file by some name, so we'll call it v1 (for video 1). The period is just a shortcut to put another command on the same line. Trim is an video edit command. It means to keep (inclusively) frames 16 through 1016. The use here is to edit all 3 videos so that they start on the same frame. They start on different frames depending on the moment you clicked record in your capture software.
So we end up with 3 videos, edited and named, v1-v3. The next command comes from the average plugin. You will find the details in it's manual. It takes two or more videos and averages them.
Average(v1, 1.0/3, v2, 1.0/3, v3, 1.0/3)
Is like saying: v1/3+v2/3+v3/3 which is mathematically the same as: (v1+v2+v3)/3, which is an average.

There is no = sign here... that's because there is always a 'floating' value called last which sticks around after each line. What is really happening is: last=Average(v1, 1.0/3, v2, 1.0/3, v3, 1.0/3) and when I end the script suddenly, the result returned is the value of last, which is the averaged video.

Now I can explain the other way of writing this. Instead of using period to put things on one line, I'm going to use last and put things on two lines:
AVISource("E:\avetest\1.avi")
trim(16, 1016)
v1 = last
After the first line, last is my (unnamed) video. Then last gets edited. Finally, I assign v1 to last and it's now the equivalent of the all in one version.

Only one more thing I can mention here. There's limits to using 8 bit video. Let's say I had 3 pixels of 100 each and I want to average them. If I do it this way: (100+100+100)/3, look what we get: 300/3. But you can't store 300 in a video, it only goes up to 255 (and 235 is supposed to be white). That's why average does it a different way: 100/3+100/3+100/3=33+33+33=99. Notice we got a roundoff error, because we can't store .333 or 1/3 as part of a pixel either.
It would be possible to write a better average plugin but it would have to deal with fractional arithmetic and do things like reduce fractions, use GCD, and so on.

Neat story: Avisynth *does* use fractional arithmetic in one case, for storing the framerate of a video. It has to! There's annoying problems that occur when 29.970... is not the same as 29.97003... anyhow, the exact fraction is 30000/1001. This is the only way to compare framerates that are exactly the same, you need it to join two videos together, or else you'd get a "frame rates don't match" error.

[juhok]

Quote:

Originally Posted by jmac698

I'd like to see the original 3 captures for download.

I don't remember the cut points and don't feel spending time to find them, so here's completely new set of caps:

http://www.siluriformes.net/mm/1229_cap1.avi
http://www.siluriformes.net/mm/1229_cap2.avi
http://www.siluriformes.net/mm/1229_cap3.avi

[jmac698]

cap1 and cap2 seem to be exactly the same.
Very good sample, but cap3 has one moment of noise.
There's some chroma noise.

[juhok]

Oh, then I've saved the same file twice. Maybe new batch of files later again

[magillagorilla]

I haven't tried this so I'm operating on the theory only. First off, isn't a pain to try and synch the frames?

My thoughts are this:
If I have 3 versions of a video, each have a bad pixel (this is our noise) in a different place on the same frame. When I average the frames together haven't I effectively created a composite with 3 incorrect pixels where any of the individual frams only had 1. Shure I have diluted the 3 error pixels, but did 1frame's error make another frame's non-error 33% incorrect?

This method seems odd to me. It's like adding noise to waterdown other noise isn't it?

It would seem more effective if you could use a regular noise filter to hihglight what is considers noise and use this as a mask. You could then use the mask to replace the detected noise using pixels from another clip. Replacing rather than blending could correct pixels 100%.

.... I'm sure someone already tried this. I'm pretty late to the video restoration party.

[jmac698]

Good point! And you are right. That's why I use median. If a dropout is in one of the videos, you get pixels like this: 45, 52, 200 and median will take the middle value, 52. It works very well!
Random noise is different. It is always there and everywhere on the frame, so average will work in that case.
I have recently wrote a script to simulate a bad video tape, complete with noise, jitter, and dropouts, and I can simulate 3 captures and show that it works too.

As for syncing frames, theoretically you just trim the start of each capture until they start on the same frame. In reality I found VirtudalDub drops frames by default and shows no indication of it. Then I did tests with every setting until I found one that works. To prove I fixed it I had to write a script to generate my own timecode and then automatically analyze that for missing frames. I'm also working on a way to automatically detect missing frames. There's a 3rd problem, where I don't have a TBC so the videos don't merge well, but I've almost got that solved too.

[Bugs.Bunny]

Hi jmac698, what avisynth filter do you use (instead of average) to apply median to a number of input clips?

Link to the average filter (those who want to try the average version)
http://forum.doom9.org/showthread.ph...19#post1129919

Got a reply at doom9 - for this type there is no filter but rather avisynth scripts. If anyone's interested look here:
http://forum.doom9.org/showthread.ph...90#post1169990
http://forum.doom9.org/showthread.ph...76#post1331776

Tried it a bit with the 3 captures version - seems to give very good results.

[jmac698]

Function Median1(clip input_1, clip input_2, clip input_3, string "chroma")
{# median of 3 clips from Helpers.avs by G-force

chroma = Default(chroma,"process") #default is "process". Alternates: "copy first" or "copy second"

Interleave(input_1,input_2,input_3)
chroma == "process" ? Clense(reduceflicker=false) : Clense(reduceflicker=false,grey=true)
SelectEvery(3,1)

chroma == "copy first" ? last.MergeChroma(input_1) : chroma == "copy second" ? last.MergeChroma(input_2) : last

Return(last)
}

Clense is a temporal median, so we interleave the clips to do a spatial median. It's from RemoveGrain I think. Another way to do this is with an inline sort, min(max(a,b),max(b,c)) I think

[lordsmurf]

I'd like to see a median filter that works on interlaced video without messing it up.

Or even something that can average two interlaced sources.

Or let's go one further -- averaging a PAL and an NTSC, to take the best of each. I know this is a severe issue among fan restorers of rare movies, which often involves worldwide sources. The Superman II and Superman IV movies come to mind.

[Bugs.Bunny]

Quote:

Originally Posted by lordsmurf

I'd like to see a median filter that works on interlaced video without messing it up.

Or even something that can average two interlaced sources.

I've recorded in YUV2 where the vertical chroma resolution is the same as the luma resolution. I had no problems averaging / median my interlaced sources.

I'm not 100% sure but shouldn't the average filter / median script work on a per pixel basis?

What I would be interested in, is a median script / filter for an even number of sources.
http://en.wikipedia.org/wiki/Median
Because it would perform an average of the two "middle" values.
The result would be a combination of median and average.
The median eliminates spikes and the likes that are only present in one or two of the sources while the averaging afterwards reduces temporary playback noise. Maybe Fizick can adapt the average filter he posted?

[admin]

Median chews up the interlacing, both VirtualDub and Avisynth versions.
I think I tried Avidemux, too -- same thing (and likely the same filter from Vdub or Avs anyway).

[juhok]

How about

SeparateFields()
Your average filter here
Weave()

[admin]

First thing I tried. No effect. Same mess.

I've been saving all the clips for a future guide, post or editorial (something that appears on the new not-yet-live site). Just backed up with projects at the moment, can't breathe long enough to write it up -- it's long and ugly.

With 3-4 people now interested, I'll have to carve out time next week for it.

[Bugs.Bunny]

Quote:

Originally Posted by jmac698

Another way to do this is with an inline sort, min(max(a,b),max(b,c)) I think

It probably has to be something like that:
min(min(max(a,b),max(b,c)),max(a,c))

@ admin: does the average filter work for you, or does it mess up as well?

With the median filters one has to take care a lot. When I was searching for median filters, these are build to do a median on a single frame, but not a median of three frames that come from three different captures of the same video. At least that's what I think.
I did not find a median filter that can process three or more sources. Thanks to forum members I was pointed to scripts that should do the trick.
I'm not very into avisynth scripts. Maybe the script is not perfect? Maybe it could be enhanced. Or someone could make a dedicated filter for it.
At least my idea (median+average with 4 sources) does not seem very complex. I do not know anything about programming an avisynth filter, but code one like this should be rather easy.

[jmac698]

What you're saying doesn't make sense. I can see a spatial median on each frame causing problems, but the median we are talking about is completely different. Show me the exact script that screwed up your interlacing.
As far as averaging ntsc and pal, that's easy:

ntsc=avisource("...").telecide.decimate.assumefps(25)
pal=avisource("..")
average(pal,.5,ntsc,.5)

Of course it depends how your sources were converted, if there's been film->pal->ntsc blending, you would need srestore.

bugs: I have a script for median5.

[admin]

Quote:

Originally Posted by jmac698

spatial median

This is what I was doing. Maybe I'm off-topic for this conversation?