Success with Avisynth Median filter, it works wonders!

[MetalSlug3]

Just wanted to share how impressed I am with the median filter for Avisynth. So I though I'd give back to the community with a little write-up.

But first, an introduction. I'm capturing PAL VHS tapes. It's only recently that I started experiencing the joys (and pains) of Avisynth scripting. My capture workflow is as follows:

• Panasonic NV-HS1000 VCR: Great deck. I picked this over a JVC. No regrets but I kinda want to try a JVC too now. lol
• Panasonic DMR-ES15 in passthrough: this was CRUCIAL for some tapes. On some tapes, Virtuadub would insert a bunch of frames (tens of frame inserts per minute) on some specific segments, but not consistently (i.e. it would not always insert the same set of frames). the ES15 stabilized the video a lot. No frame inserts reported in VDub, although I later found out that the ES15 had dropped a handful of frame per hour of capture across my captures.
• August VGB100 USB capture card: I picked this one because it has the same Conexant chipset as the Hauppauge USB2Live and I knew it would work with VDub. I can't compare it to other capture cards but I tested it by capturing a DVD being played over S-video and I didn't see any egregious artifacts or video quality loss. So I'm happy with this stick.

The video flowed as follows:
NV-HS1000 (output toggled to S-Video) -> SCART CABLE -> DMR-ES15 -> S-Video cable -> VGB100

Captured in Virtualdub2 and encoded losslessly using the UTVideo 422 BT610 codec.

I captured my VHS 3 times (the minimum needed to use the median filter).
I loaded them up on Avisynth and I trimmed them at the start so that the first non-black frame was aligned.

clip1= AVISource("D:\capture.ES15.take1.avi").trim(243,0)
clip2= AVISource("D:\capture.ES15.take2.avi").trim(118,0)
clip3= AVISource("D:\capture.ES15.take3.avi").trim(103,0)

I then had to compare the clips side-by side to verify the clips were staying aligned and spot the exact frames that were dropped in each clip. (supposedly, Median has a "sync" parameter that allows you to auto-align the frames, but I've had issues with it not working or working unexpectedly)

clip1f=clip1.crop(0,0,-360,0)
clip2f=clip2.crop(360,0,0,0)
StackHorizontal(clip1f,clip2f)

This shows the left half of clip1 and the right half of clip2 side-by-side in the AVSpmod preview. I then changed the script to compare clip 1 to clip 3, and clip 2 to clip 3. This visual side-by-side view helped me identify which of the clips was "behind" at any given point, i.e. it dropped a frame.

Of course, sometimes it's hard to spot the exact moment when a frame is dropped in one of the 2 clips, especially if the frame is dropped in a low-motion sequence (I was literally giving myself headaches staring at single pixels). So the Subtract() function (which shows you just the difference between 2 clips) came in VERY useful:

Subtract(clip1,clip2)

I commented out the StackHorizontal call and uncommented the Subtract call to switch between these two modes. When the clips are frame-aligned, the difference between them looks like a pretty uniform noise (by the way, this is the very noise the Median filter will help remove):
SubtractNoise.Aligned007879.png
However, when a frame is dropped from one of the two clips, the difference highlights the motion area very blatantly:

You must be logged in to view this content; either login or register for the forum. The attached screen shots, before/after images, photos and graphics are created/posted for the benefit of site members. And you are invited to join our digital media community.

Every time I found a frame that was missing from one of the 2 clips, I noted down the frame number and inserted it from the other clip using code like this:

insertinto2_1 = clip1.trim(7880,7880) 
clip2= clip2.trim(0,7879)++insertinto2_1++clip2.trim(7880,0)

The example above simply inserts frame n.7880 in clip1 intro the same position in clip2 leaving all the other frames untouched.

In total, I "patched" 22 frame drops across three 90-minute captures. Not bad, considering that without the ES15, I had hundreds of frame inserts over a single capture.

I now had three clips perfectly aligned to the frame. LET'S MERGE THEM INTO A BETTER QUALITY VIDEO

Here is a screenshot from one of the 3 captures:
SubtractNoise.misAligned007879.png

You can see the noise towards the top. this same frame was clean in the other 2 captures. so the median filter will know to discard the noisy pixels here.

Here is the clip built by simply calling:

median(clip1,clip2,clip3)

You must be logged in to view this content; either login or register for the forum. The attached screen shots, before/after images, photos and graphics are created/posted for the benefit of site members. And you are invited to join our digital media community.

Look at them side by side! It's incredible how much I could clean up the picture with a single filter. And the great thing is that I haven't introduced anything new into the picture, The filter hasn't degraded the video quality: in fact, it restored the video quality by extracting every last bit of detail from the VHS!

I could, if I wanted, capture the same VHS another couple of times so I can median-filter 5 clips instead of 3. Not sure if the quality will improve a little bit more, or if I'll hit diminishing returns immediately. Might be worth a try.

and finally, this is the median-filtered screenshot after going through QTGMC(Preset="faster",inputtype=1)
videocap.CapA.096550.png

Personally, I'm very happy with the results. I recommend this process to anyone who's trying to get the best possible quality out of their tapes.

[lollo2]

I tried the median of 3 captures (there is some saturation in the process, so 5 is not giving a significant difference). I noticed that the noise is somehow slightly reduced, so it was worth the effort, even if in your sample the problem on the top is more a defect than a noise problem.

However, using avisynth filters like TemporalDegrain or MCTemporalDenoise or SMDegrain or embedded QTGMC denoise on the single clip I obtain a better result.

Concerning the pure noise, IMHO if you do not plan any avisynth restoration Median is good; if you plan avisynth denoise filtering is not necessary. But you can experiment yourself: apply TemporalDegrain with a soft setting on the first clip and compare. If you do, let us know the result!

Concerning the defect on the top, Median did great, but you can also try some avisynth script for defect removal (decomet, destripe, much more complicated techniques, etc...), and again compare with Median.

Jerry Calà

-- merged --

This is an example of applying twice FixRipsp2 within a single clip on a defect similar to yours. In my case it is reduced but is not perfect.

Maybe if you use it after Median() that already almost eliminated your problem, the final result will be perfect (or maybe worst)

FixRipsp2.jpg

[MetalSlug3]

Quote:

Originally Posted by lollo2

I tried the median of 3 captures (there is some saturation in the process, so 5 is not giving a significant difference). I noticed that the noise is somehow slightly reduced, so it was worth the effort, even if in your sample the problem on the top is more a defect than a noise problem.

However, using avisynth filters like TemporalDegrain or MCTemporalDenoise or SMDegrain or embedded QTGMC denoise on the single clip I obtain a better result.

Concerning the pure noise, IMHO if you do not plan any avisynth restoration Median is good; if you plan avisynth denoise filtering is not necessary. But you can experiment yourself: apply TemporalDegrain with a soft setting on the first clip and compare. If you do, let us know the result!

Concerning the defect on the top, Median did great, but you can also try some avisynth script for defect removal (decomet, destripe, much more complicated techniques, etc...), and again compare with Median.

Jerry Calà

Ciao! Bello vedere un altro italiano qui!

First of all, the defect in the frame I showed is actually only happening in 1 capture out of the 3, so that's why median worked so well to remove it. There were other frames where I can see dropouts in all 3 captures, and median didn't do much.

I think the only problem median can have is when a defect is present in the majority of the captures. In that case, median will pick the bad pixels and discard good detail that's got the misfortune to be in a minority of the captures.

Still, I think Median is great for extracting the maximum quality from the tape. If the video in the tape is itself noisy, Median will retain the noise that's actually recorded in the tape, while removing the noise that was added during a each single playback and capture of the tape.

I can still use any filter after Median to try and fix what Median couldn't. e.g. QTGMC after median is better than QTGMC by itself.

[lollo2]

Quote:

Ciao! Bello vedere un altro italiano qui!

Median will retain the noise that's actually recorded in the tape,

Absolutely! But concerning the noise, as I said, if you plan to restore later you will remove (most of) it, so having a less noisy source by using Median does not change the final result in a significant way.
Try yourself applying, for example, TemporalDegrain on 1 clip and on the median of 3 clip. (I do not have anymore a comparison to post, sorry)

If you do not plan denoising for whatever reason, Median is excellent to remove non repetitive noise or defect!

[MetalSlug3]

Here's the comparison (done on a different frame):

TemporalDegrain2 on a single capture
videocap.CapA.TD2.053996.png

TemporalDegrain2 on a 3-capture median

You must be logged in to view this content; either login or register for the forum. The attached screen shots, before/after images, photos and graphics are created/posted for the benefit of site members. And you are invited to join our digital media community.

Seems to me that TD2 has created fewer artifacts on the median compared to the single capture.

Personally, if I encode a capture to x264, I don't like to completely wipe out the noise. That would also wipe out the grain, and make the picture look washed out and waxy -- like a low quality, low bitrate video. So if I can get a more "accurate" noise, I'll take it

[lollo2]

Quote:

eems to me that TD2 has created fewer artifacts on the median compared to the single capture.

To my eyes there are very similar and I do not see additional artifacts.

cfr.jpg

You must be logged in to view this content; either login or register for the forum. The attached screen shots, before/after images, photos and graphics are created/posted for the benefit of site members. And you are invited to join our digital media community.

Let see what the metric says:

Subtract and Compare
UtoY_VtoY.jpg

SSIM analysis

You must be logged in to view this content; either login or register for the forum. The attached screen shots, before/after images, photos and graphics are created/posted for the benefit of site members. And you are invited to join our digital media community.

All this is based on the images, with colorspace conversion; if you wish to go deeper, better redo it with the video.

Quote:

Personally, if I encode a capture to x264, I don't like to completely wipe out the noise. That would also wipe out the grain, and make the picture look washed out and waxy -- like a low quality, low bitrate video. So if I can get a more "accurate" noise, I'll take it

Well, we are entering in another world here:

- washed out and waxy: after a denoising stage a sharpening is required, I generally use LSFmod. Try it and you'll see how the filtering improves!

- I don't like to completely wipe out the noise: me neither, but the encoders do not like noise, and are less efficient in its presence. I am one of the few watching my captures in lossless format (I never encode except for youtube/facebook or forums) and I am not a x264 expert, but if you are going to encode noise could be a problem.

- I agree that sometime a bit of noise is welcome to make the digital videos less "digital", but noise and grain are not exactly the same: I think is better to reinject some of the "noise" with AddGrain or similar or some parameters available for the denoisers, because they knew what they did. An example here https://forum.doom9.org/showthread.php?t=147796

Thanks for the enjoyable discussion!

[MetalSlug3]

Hey, thank you for engaging!

Let me just first point out what the 3-capture median workflow allowed me to achieve with this clip:

• Allowed me to spot and completely restore 8 dropped frames from a clip: and by restoring I mean actually taking the missing frames from the other 2 captures and inserting them into each clip. Admittedly this is just a preparation step and not the work of the median filter, but I couldn't have done it with a single capture.
• removed all the noise that's present in a single capture: as I mentioned before. If a frame in a capture has dropouts, but the other 2 are fine, the dropouts will not be present in the median-filtered clip. You could have multiple seconds worth of pure white noise during a capture, but if the other two captures show data, then you're fine and median will put good video data into your clip.
• removed a lot of the noise that's introduced during tape playback: each frame in a capture will be ever-so-slightly different from the same frame in another capture. See the first picture in my original post above -- that's the difference between two distinct captures on a single frame. you can see there's a lot of random color noise, and finer noise around edges. The median filter smoothed out this noise and returned cleaner color and more solid edges

Quote:

To my eyes there are very similar and I do not see additional artifacts.

Are you sure?
I see you posted shrunk down JPEGs. I can only see JPEG artifacts there, plus the screenshots are tiny and we lost all detail

Let's compare the screenshots again, zoomed in and saved as PNG this time:

videocap.CapATD2.MedianTD2.sidebyside.053996000000.jpg

You must be logged in to view this content; either login or register for the forum. The attached screen shots, before/after images, photos and graphics are created/posted for the benefit of site members. And you are invited to join our digital media community.

Notice the jagged, fuzzy edges, and the overall noise in the single capture compared to the median. Sure, more filters could further improve the single capture, but the same filter will certainly give a better result on the median-filtered clip.

Quote:

I think is better to reinject some of the "noise" with AddGrain or similar

Why would I inject made-up algorithm-generated noise if I have the original analog tape noise in all of its glory?

[lollo2]

Fisrt of all let me clarify that I love multiple captures technique as you do, but I think that in the case I explained (clean capture, further restoration) is not really necessary because it requires a lot of effort.
To remove occasional defects or defect showing themself in a differente way across the capture is excellent , but in general needs to be followed by a restoration anyhow (I experimented that a "trace" of my defects will stay in all captures in most of the case, YMMV).

Quote:

I see you posted shrunk down JPEGs

Yes, sorry, for some reason I lost the .png original attachements while posting my reply; they should be still there but I ignore how to recover them.

Quote:

Notice the jagged, fuzzy edges, and the overall noise in the single capture compared to the median.

I don't, but it can be my 50 years old eyes. That's why I used some metrics. Subtract shows almost no difference, numbers in Compare partially confirmed, and SSIM (used to judge the quality of the resizers and of the different encoding methods) gave a score of 96%.
There is indeed some small difference (for sure Compare does not say 100dB overall), but very very marginal and it will not be seen in a standard play.
All this is based on the images you posted, surely not a "orthodox" method.

Quote:

Why would I inject made-up algorithm-generated noise if I have the original analog tape noise in all of its glory?

Because we do not want noise and AviSynth and its filters were invented to improve a video, and the main item is "remove noise". Adding grain based on the removed noise is an nice technique (we do not reinject removed noise).

Quote:

Allowed me to spot and completely restore 8 dropped frames from a clip: and by restoring I mean actually taking the missing frames from the other 2 captures and inserting them into each clip

I accept 0 drop/inserted frame in all of my captures. Actually, the reason I use sometimes Median() in my scripts is because if a have even 1 single dropped/inserted frame I redo the whole capture from scracth. I am obsessed, I know , but i try to do not align few frames from another capture to avoid audio glitches I experienced sometimes.

Quote:

removed all the noise that's present in a single capture

That's not possible using only Median; by the way, I thought you loved it

Quote:

removed a lot of the noise that's introduced during tape playback

Absolutely. Median is good in this, but again if you are going to restore later...

Quote:

each frame in a capture will be ever-so-slightly different from the same frame in another capture. See the first picture in my original post above

The major difference is because your tape has probably a defect (or your VCR is behaving differently), that is showing in a different way across different captures. Again, Median() is excellent in this field...

[timtape]

A probable reason for the noise band in the upper part of the first still is a piece of dirt on the tape. The effect is way beyond the capabilities of the DOC to mask it. But the problem with multiple captures is there's no guarantee that on subsequent passes the piece of dirt will obediently remove itself. Playing the tape two or more times seems a lot of time and effort hoping that luck will favour us. It seems kind of hit or miss.

Here's a link to a Specs Bros White Paper on the benefits of tape cleaning.

http://www.specsbros.com/white-paper-the-benefits-of-tape-cleaning.html

The great thing about tape cleaning is that it cleans off dirt and other contaminants regardless of whether we've seen or heard its effects on picture and sound, and in a tape which might run to hours. There's less need to monitor the entire program second by second looking for possible faults due to a dirty tape. The artifacts - and there could be many of them in many frames - could have been avoided in the first place, before capture, by proper tape cleaning. As the paper says, the cleaning applies just as much to the player deck as to the tape.

[MetalSlug3]

Quote:

Originally Posted by timtape

A probable reason for the noise band in the upper part of the first still is a piece of dirt on the tape. The effect is way beyond the capabilities of the DOC to mask it. But the problem with multiple captures is there's no guarantee that on subsequent passes the piece of dirt will obediently remove itself. Playing the tape two or more times seems a lot of time and effort hoping that luck will favour us. It seems kind of hit or miss.

Here's a link to a Specs Bros White Paper on the benefits of tape cleaning.

You're right, there's no guarantee. But in my limited experience I can honestly say there's quite some noise that happens sporadically due to random playback issues or just due to random nature of analog tape media itself. And multiple captures+Median filtering fixes this kind of noise amazingly well.

Sure there's also noise that happens in the majority of captures, and they will defeat median filtering. But they're fewer and easy to address later, if needed, once you've done multiple captures.

Tape cleaning... that looks hardcore. There's always something extra we can do to squeeze a little bit of extra clarity or less noise from a tape