The two scenes following the D shot have low-light noise. That brings me back to the "B" scene bypassed earlier. The noise in these low-light and night shots is CCD/CMOS noise. It's from using a camera that's unsuitable for low light shooting. The signal strength of the CMOS residual noise is stronger than the signal strength of the low-light objects. Think of it as audio tape where the hiss is louder than the music. CMOS noise is clumpy, thick, very coarse grain. But it doesn't look like it these scenes. Below, a 200% blowup from the upper right section of the 94th frame from scene "B":
The noise isn't heavy grain any more. It's been hit with a simple blurring filter that converted it from distinct grain into a swirl of simmering sludge. Sludge is tougher to remove than grain. It doesn't help, either, that this is another shot with high IRE (black levels) that looks filtered and sharpened while still interlaced, or perhaps during earlier playback or recording.
Avisynth has a handful of filters to address CMOS stuff, but they don't work so well after it's turned to mush. Having seen this before, IMO it's best to use two heavy-hitters but with more moderate settings, rather than one big brute that tears everything apart. So I made this a two-step process. Two steps, because running both sets of these plugins in the same script slows the script to a crawl.
The first step uses a modification of a favorite cleaner called MDegrain, which is part of the MVTools v2 plugin (Motion Vector Tools). MDegrain comes in two flavors, 2 and 3. Flavor 3 is used here. The function is a separate avs script that can be saved as a plugin. It's modded to use interlaced video without deinterlacing it, using SeparateFields() instead. Use MDegrain before deinterlacing, in this example. Why? To prevent that ugly swirling junk from being interpolated across multiple frames during deinterlace, thus smearing it even more.
I attached a copy of the function as a script called "MDegrain2i2.avs". Save that avs attachment as-is in your Avisynth plugins folder. To use an avs scripted function in your own script, you have to import it into your script using the Import() function as shown in the previous post, and as shown in the script below for STEP #1:
Code:
# ----------------------------
# FIRST STEP #1 --------------
# ----------------------------
Import("D:\Avisynth 2.5\plugins\MDegrain2i2.avs") ## <<-- Change path to your plugins folder !!
qtInput("E:\forum\pinheadlarry\Aug09\example.mov",audio=1) ## <<-- Change path to your video's folder !!
Trim(131,265)
ColorYUV(off_y=-7,cont_y=30)
TComb()
ConvertToYV12(interlaced=true)
SmoothTweak(saturation=1.5)
MDegrain2i2(last,8,4,0)
In VirtualDub I saved the output of Step #1 using YV12 color, Lagarith YV12 lossless compression, and the "fast recompress" video processing mode. This file will be the input for STEP #2.
STEP #2 uses some familiar filters you saw earlier. Notice that the video isn't opened with qtsource(), but uses AviSource() instead:
The second step should output a clip that looks like an appropriately dark but fairly clear night shot instead of a washed-out foggy what-is-this. The result is attached as "mov_B00mv3_Q.mpg". Yes, you do give up some dirt in the process. But, then, you can always leave it as-is or omit MDgrain.
I took another step (call it STEP 3), running NeatVideo at about 1/3 power on the output of step 2. NV made things just a little more visibly smoother. No script there. NeatVideo is a paid plugin, often abused and bad-mouthed by people who don't read its extensive user guide. Clip attached as "mov_B00mv3_Q_NV.mpg".
I think you can see how much trouble it is to clean up careless work, and how much it's possible to lose in the process.
The cardinal rule is this: "garbage in, garbage out". A good VHS original can look pretty good. But watch out for careless analog work. It's a real pain.
This is awesome so far and i'm not even half way through. Before i really get started i have to download all of these plugins. I'm having trouble finding a couple of these, such as maa(). Not sure if i should use maa2 or not. Are there any resourceful plugin packs?
maa2 is a special version of maa that requires a bunch of AVisynth_MT plugins. It won't run in other versions of Avisynth. It's aggressive, not recommended for soft or over processed video.
The plugins package for QTGMC contains a lot of support files used by many of these filters, including the versions of masktool and mvtools called by many filters. It also includes support files for FFT3dFilter. http://forum.doom9.org/showthread.php?t=156028. You want the "Plugins Package" offered, not the modded package.
Attached:
-LimitedSharpenFaster.zip
-Santiag.zip
-maa.avs
-SangNom.zip (this might come with QTGMC. Used by several anti-alias plugins)
My content will primarily be seen on a computer screen. In that case, should i ignore the restore interlace and interlace=true steps?
Also a workflow question that may sound noobish, but there aren't many resources out there. I assume you skim through frames throughout the video and find frames to fix. You add filters to one frame, but what do you do when you approach a frame that needs more filtering? Do you add filters on the existing script? Or is there a different way?
I just figure if you keep adding filters on the script for different frames it would do more damage.
My content will primarily be seen on a computer screen. In that case, should i ignore the restore interlace and interlace=true steps?
What does "primarily" mean? If you mean that all this work is being given only to people who watch videos on a PC, you don't have to reinterlace. The results will be larger files with twice as many frames and will run at 59.94fps. You can make it 29.97fps by discarding every second frame, but you'll have to decide for yourself whether the inferior motion handling and lower resolution suits you. One thing's for certain: it wouldn't play smoothly on TV. This is fast-action video, not still photographs.
Since you don't see some of these videos that well, you probably haven't noticed that some of the shots in your samples are not interlaced.
This scene is not interlaced:
This scene isn't interlaced, either:
The scene in the upper image is slightly but visibly underexposed. The scene in the lower image has the opposite levels problem: it's over-exposed. Would you apply the same filter to both scenes?
If you deinterlace both scenes and leave them deinterlaced, you will have duplicates of every frame. If you discard duplicate frames, you will have scenes of effectively lower-resolution frames.
For a specific problem that affects only a few specific frames, below is a 2X blowup of part of the the 9th frame in one of the scenes. It affects 6 consecutive frames. The glitch can be fixed, but you'd need about 30 more lines of code and two more plugins. You can't use that same code on the entire video. You'd also have to save the audio as a separate file and rejoin the audio after repairing those frames, because the code will affect audio sync. When working with this scene, I decided it's probably better to just leave the problem as-is and live with it. I see several bugs in the scene pictured below, some of which don't exist in some other shots. The image is not interlaced. See if you can spot the problems in the image below:
Quote:
Originally Posted by pinheadlarry
Also a workflow question that may sound noobish, but there aren't many resources out there.
I'm not sure what you mean by resources. The restoration threads of this and several other forums have thousands of examples of problem videos and how they were repaired.
Quote:
Originally Posted by pinheadlarry
I assume you skim through frames throughout the video and find frames to fix. You add filters to one frame, but what do you do when you approach a frame that needs more filtering? Do you add filters on the existing script? Or is there a different way?
I just figure if you keep adding filters on the script for different frames it would do more damage.
Are you referring to "frames", or to "scenes"? If a histogram or sample frame is displayed, it's usually a sample of how things look in general in a scene, not how they look in a specific frame unless that specific frame has a specific problem such as a bad dropout or one-time glitch. The scripts posted are run as separate scripts that address defects throughout a scene and that result in separately filtered video clips, not lines of code added into one huge script and run all at once on the entire video. The scenes that were cut out and filtered are rejoined into a final product, which can be done with FCP or any editor.
Yes, there is a different way. Keep the videos intact, run them through FCP and set up three or four filters for the entire video. Or use FCP to cut and join scenes into a different arrangement, then use the same filters for that entire compilation. Of course, the scenes joined in that way will come from different sources with different problems, but isn't that the way your sample videos were originally produced?
Noise and visual defects are usually more obvious on a computer screen than on TV. That aside, if you see no apparent problem with any of the scenes in your samples, you're wasting your time with Avisynth and VirtualDub. Almost all NLE's in every price range have the same or similar cut and join features as another NLE, similar timeline and audio features, similar simplified color balance and denoisers, and similar scaled-down encoders and authoring tools. If differences between the original scenes aren't a problem in your view, and if basically they all seem to look alike, then why allow others to program everything you're doing? That wouldn't be learning anything, it would just be following rote procedure.
I don't skim through frames looking for something to fix. All you have to do to is watch. The boo-boo's jump right off the screen, no searching required. But as I say, if you don't see problems or differences in an assemblage of badly processed video clips, you should be feeding everything intact through a single NLE and not worrying about it. If the friend that you mentioned earlier claims that he uses a single universal Avisynth script to fix all his videos and doesn't need to do any specific repair work, then realize that you, too, have lossless media and can run his script or any other's script just as well as they can.
You might want to rethink the purpose of fixing up several really bad videos. Or leave them as-is and do a simpler and quicker re-edit into custom scenarios using a single set of a few NLE filters to make them look the way you want.
8 segments from the first example.mov, A thru H, filtered separately and rejoined. Encoder TMPGEnc Plus 2.5, with edit/join/audio sync/AC3 resampling in TMPGEnc MPEG Editor V3. Auditioned with MPC-BE and VLC players (*I hate VLC). Could use more tweaks.
should i ignore the restore interlace and interlace=true steps?
Sorry, I forgot to address this.
interlaced=true and interlaced=false are relevant for colorspace conversation depending on whether or not the video is in an interlaced state when the conversion statement is run. Remember that different colorspaces store chroma information differently. Doing the conversion properly requires that avisynth is informed about the interlace state. Doing it the wrong way screws up chroma.
Colorspace and interlace factors also affect the Crop() function. Be carefulo how you crop. The following rule is quoted from the Avisynth online help concerning Crop():
Quote:
In order to preserve the data structure of the different colorspaces, the
following mods should be used. You will not get an error message if they
are not obeyed, but it may create strange artifacts.
In RGB:
width no restriction
height no restriction if video is progressive
height mod-2 if video is interlaced
In YUY2:
width mod-2
height no restriction if video is progressive
height mod-2 if video is interlaced
In YV12:
width mod-2
height mod-2 if video is progressive
height mod-4 if video is interlaced
mod-2 means that a number must be evenly divisible by 2.
mod-4 means that a number must be evenly divisible by 4. http://avisynth.nl/index.php/Crop
I always like Avisynth discussions. I see an item or two I never use.
Although themaster1 never attaches scripts to the forum (WHY?), he has some good ones too. He gave me an idea last week, and the script turned out very nice indeed. It's something to add to repertoire, and onto an "advanced multiscript" when I finish the typical one.
What does "primarily" mean? If you mean that all this work is being given only to people who watch videos on a PC, you don't have to reinterlace.
These videos will be for youtube and for download. For the most part they will only be seen on a PC. Unless they download the file and watch it on a tv. But i'm focusing on the PC viewers.
Quote:
Originally Posted by sanlyn
The scenes that were cut out and filtered are rejoined into a final product, which can be done with FCP or any editor.
This makes much more sense now lol. But i'm not sure if i want to spend that much time with these considering the amount of scenes that need to be addressed. I'd much rather fix the video as a whole. I understand that will now limit me to what i can fix.
The thing with other programs like FCP is that you need to buy additional software filters. And with avisynth or virutaldub, you get powerful software for free. There's a high learning curve, but i think it would be worth it incase i want to go scene by scene in the future.
Also, I got ahold of the 3rd party i mentioned before and he sent me his script. Thoughts?
Code:
#SetMemoryMax(1200) # Optional line. See below for value M
SetMTMode(3, 6) # See below for value X, could try 5 instead of 3 for non-standard source-filter/avisynth combinations
dgdecode_MPEG2Source("C:\Users\Tim\Desktop\Hoax 2\VTS_01_1.d2v")
SetMTMode(2)
crop(8, 0, -8,-8)
QTGMC( Preset="Very Slow", EdiThreads=1 ).SelectEven()
Okay, fellas, have it your way. All the script does is create an invalid frame height for anything except PC playback or YouTube (invalid altogether for BluRay), some degraining, frame-decimation deinterlace, plays back at one-half it's original vertical resolution, and ends up as pretty much what you started with except for jerky playback on TV -- if you can find a way to play it on TV, anyway. Definitely amateur work. Try it. You'll need Avisynth_MT or the 2.6 mod downloaded from the Doom9 QTGMC thread.
After all, it does work. It just do very much and makes a couple of mistakes. On the other hand, one can improve most of those shots only minimally, while a handful or so could just stay as-is, and perhaps another handful are so bad that even a small effort could make them look, well, noticeably better. But for your vids you'd have to make some adjustments. For one thing, the crop() was obviously designed for 720x480 mpeg's but doesn't give a 3:4 image back, leaves the video at a nonstandard height that's invalid for several formats, and doesn't clean the upper border twitter.
You can leave your videos at 640x480 for PC playback or YouTube, but that frame size can't be used for DVD, etc., and DVD/BluRay are usually interlaced formats or at least encoded that way. So you could use that QTGMC statement but reinterlace afterwards to keep your original resolution and to be able to resize for other formats (which should be either 704x480 or 720x480). So, with Avisynth 2.5.8 you could do it this way and customize for your particular set of 640x480 captures:
First, you can't open with dgdecode, you have open your .mov files with qtinput. Dropping the Avisynth 2.6 requirements (since QTGMC won't run much faster in 2.6 "MT" anyway):
Then just leave it that way, deinterlaced at double frame rate (59.974) and double frame length, compress losslessly at YV12 with Lagarith. That's a very slow script that does some basic denoising, debanding, de-halo, and reinterlace. This would be your cleaned-up archive version from which you could go in multiple directions. For progressive display for PC-only or YouTube, all you'd need to do is run this statement on your cleaned-up version:
You could also use that deinterlaced cleaned-up version for anything, including DVD or BluRay. For encoding to those formats, use this on the clean-up deinterlace version:
Encode that with a 4:3 display aspect ratio. For standard def BluRay you can go up to 8 mbps variable bitrate and Max 12Mbps, with a GOP of size of 30 or even 15 for better fast action control, encoded to HD MPEG or h264. For plain old DVD you could use a bitrate of about 6500 VBR for fast action, 2-pass encoding. To be safe, you'd set the max VBR to 8000 and for action DVD you'd use a GOP of 12 (it can't be GOP bigger than 18 frames for DVD).
Whatever you do, don't use low bitrates on noisy action video, and don't use huge GOP sizes if you want clean motion encoding. Noise, fast action, and swift and jumpy camera pans require higher bitrates than clean static scenes with steady cameras and normal motion or less.
OOoops. Yes, it's TR2. But that's what I get for being in a rush, because you don't even need to specify it. At "Very slow", TR2 is already set to 2 by default. So this line:
Code:
QTGMC(Preset="Very Slow", RT2=2,EZDenoise=2)
can be:
Code:
QTGMC(Preset="Very Slow", EZDenoise=2)
The access violation message comes from Windows. Try clearing memory by shutting down and rebooting. I get that Windows message now and then if I run a lot of memory intensive stuff for a long time without shutting down occasionally.
Few notes from me:
- I prefer to do all my cropping in VirtualDub.
- I don't much care for the multi-threaded version on Avisynth. It never made a difference in my encoding times, and just made the scripting longer.
NOTE: Remember to use the [code][/code] bbcode to display the scripts.
The # symbol in the quick reply or advanced reply make it easy.
Otherwise forum smiley bbcode can mess up the script. The smileys don't work inside the CODE blocks, however.
Ok. so after i export to the deinterlaced archive version (first script). I run the second script which is just SelectEven.
Now, what would be the proper way to get the video into FCP. That program doesn't accept the lagarith AVI. I'm not sure if it's the compression or the container, that FCP doesn't recognize
The only solution i know would be to use handbrake on the progressive Youtube copy. Then pull the handbrake version into FCP. Is that acceptable or will that kill the quality?
Last edited by pinheadlarry; 08-25-2014 at 12:20 PM.
I'm not familar with FCP but in many pro softwares (like sony vegas) you can use a little app called avs2avi (google it) which create a pseudo .avi (only a few kb) and you can import it thereafter.
This way you avoid a compression step [always better ], the downside is if you use a complex script it'll be slow as hell on your timeline to move back/forth
i realized it doesn't matter if they are compressed because they're going on youtube anyway.
I used the scripts on a full VHS capture and i think it came out well. I still have to compare scenes, but i can definitely noticed less interlacing.
But thank you sanlyn, you really came through big time helping me out and explaining all this. I'm going to use the scripts you provided and continue to reference this thread and all the filters you have been mentioning. I'm slowly starting to understand avisynth and hopefully i can start to figure out what all these filters actually do lol.
Thanks again!
edit- Actually one last question..
When i pull the script into VirtualDub and save as AVI. I'm confused on the input and output video it shows in VirtualDub. It doesn't show the changes between the original video and avisynth filters, correct? These two screens only show the differences in VirtualDub filters?
When i pull the script into VirtualDub and save as AVI. I'm confused on the input and output video it shows in VirtualDub. It doesn't show the changes between the original video and avisynth filters, correct?
The left-hand "Input" windows shows what it's name says it does: the input, which came from Avisynth. VirtualDub has no way of knowing what the signal looked like before it was processed by Avisynth.
Quote:
Originally Posted by pinheadlarry
These two screens only show the differences in VirtualDub filters?
The right hand window ("Output") would be different from the input only if you used some VirtualDub filtering. However, if you saw both windows "playing" the video while it was being saved to AVI, then you probably set VirtualDub processing to "full processing mode". If so, your YUV input from Avisynth was converted to RGB, then on output to a new AVI it was reconverted to YV12 if that's the colorspace you specified. If the right-hand Output window did not change at all while the new file was being saved, then no conversion occurred and you probably had VirtualDub set to "fast recompress", which is where it should have been.
Unless you plan to do some complicated processing, why would you need FCP? I thought you were going to encode the results, a progressive encode for mp4 or something for PC-only or web, and then an interlaced version for DVD, BluRay, or AVCHD. Those two steps can be done in Windows. along with some simple cut/join, using free software.
], the downside is if you use a complex script it'll be slow as hell on your timeline to move back/forth
