Go Back    Forum > Digital Video > Video Project Help > Restore, Filter, Improve Quality

Reply
 
LinkBack Thread Tools
  #21  
03-31-2020, 02:47 PM
sanlyn sanlyn is offline
Premium Member
 
Join Date: Aug 2009
Location: N. Carolina and NY, USA
Posts: 3,648
Thanked 1,307 Times in 982 Posts
Quote:
Originally Posted by billct97 View Post
Taking things very slowly as I had to work today. Can you please fix the IVTC line in Post #11 as it tripped me up for a few minutes. If someone who is new finds this later on I suppose it would be good to have it corrected.

TFM().(Tdecimate) should be TFM().Tdecimate()
Holy smokes! How did that typo sneak by? Anyway, your version is correct. Unfortunately posts can't be edited 1 hour after they post. Shucks. I'll try sending a note to a moderator. The other scripts shouldn't have this problem, they were all cut and paste from the originals.



Meanwhile I tried to post some pics and results for Firestone_1, but the cable company is working in my neighborhood and the internet keeps popping on and off. I'll wait until they're finished and post later tonight.
Reply With Quote
Someday, 12:01 PM
admin's Avatar
Ads / Sponsors
 
Join Date: ∞
Posts: 42
Thanks: ∞
Thanked 42 Times in 42 Posts
  #22  
04-01-2020, 06:28 AM
sanlyn sanlyn is offline
Premium Member
 
Join Date: Aug 2009
Location: N. Carolina and NY, USA
Posts: 3,648
Thanked 1,307 Times in 982 Posts
This post on Firestone_1.avi introduces a couple of filters.

There's a thread from a few years back that goes into detail on scripts, histograms, and VirtualDub controls. A post entitled "Information Overload #3" demonstrates the VirtualDub Levels, ColorMill, and gradation curves adjustment controls, which I used extensively working with the Firestone caps.

Another essential tool is the CSamp pixel reader (http://www.digitalfaq.com/forum/atta...-time-csampzip) -- a free desktop standalone gizmo that comes in very handy. Using it is covered with pictures in post #39 in another, older thread. The later thread also discusses the very important ColorTools histogram. There's a link to it, but it's the old version that won't work in Win7 or later. The updated ColorTools is at http://www.digitalfaq.com/forum/atta...1&d=1487006540.

I had a fairly tough time with Firestone_1.avi. Every camera shot in that ad has different problems. Some of the glitches are common to every shot, such as the bad edge ghosts. All have pretty much the same moderate noise signature which isn't nearly as noisy as my own caps posted earlier. You controlled signal levels pretty well. Because each shot was so different I had to split the video into separate shots; the scripts for each are similar but with minor differences, and the color was -- well, it's no surprise from the days of analog broadcasting that colors were a mess. Your decisions about the color might differ from mine, but hopefully the details I post will be helpful.

The results of the work were encoded with restored 3:2 pulldown for DVD and is attached as "Firestone1_cleanup_DVD_pulldown.mpg". It was encoded with my old standby, TMPGenc 2.5 plus, and video and audio were remuxed with Dolby Ac3 in TMPGenc Smart Renderer (the old 2.5-Plus is so old it was never updated for Dolby audio!).

The first step was to remove telecine from the entire video, then work with individual scenes. First, I inverse telecined Firestone_1.avi and saved it as Firestone1_ivtc.avi:

Code:
AssumeTFF()
TFM().TDecimate()
Firestone1_ivtc.avi was saved out of VirtualDub as Lagarith YUY2. I then used 5 different scripts to cut Firestone1_ivtc into 5 different scenes, or working files, named sc1 thru sc5. This is not unusual especially for analog broadcasts, which contain many wild variations.

The image above contains two color schematics based on the original unfiltered frame 32 of scene 1 in inverse telecined Firestone1_ivtc. The chart on the left is a 4-band RGB histogram of color values in the frame, showing brightness (white), Red, Green, and Blue. The chart to the right is an RGB vectorscope which is basically a picture of color saturation in the frame.

The pic below is unfiltered frame 32 of scene 1. In the RGB histogram above you'd expect lots of yellow, but the strongest color is the Red band with slightly less Green. Blue is easily the shortest band, indicating a deficit of bright Blue. In the guy's hand and face, skin looks a little dim and pumpkin-colored. Most of the "black" in the car's front grill is actually blackish green. In the vectorscope, the biggest color blob is red. The vertical line that extends upward and slanted to the left is where flesh tones ideally belong but there's just a muddy mix on that line. The other dots asnd squiggles are disorganized remnants of green, cyan, and blue. In the frame you might also note dot crawl on the right-hand border.



The image above is frame 32, filtered and color-corrected. The script and the settings for the VDub color controls are described below. You've seen most of the Avisynth filters in previous scripts:

Code:
Import("D:\Avisynth 2.5\plugins\ContrastMask.avs")
AVISource("I:\forum\faq\billct97\X1\F1\Firestone1_ivtc.avi")
Trim(0,36)
### --- Next 4 lines are for test only --- ###
#Crop(20,0,-8,-8)
#ConvertToYV12(interlaced=false)
#Histogram("Levels")
#return last

FixChromaBleeding()
ConvertToYV12(interlaced=false)
contrastmask(enhance=2.0)
source=last
  super = source.MSuper(pel=2, sharp=1)
  backward_vec2 = MAnalyse(super, isb = true, delta = 2, blksize=8, overlap=4, dct=0)
  backward_vec1 = MAnalyse(super, isb = true, delta = 1, blksize=8, overlap=4, dct=0)
  forward_vec1 = MAnalyse(super, isb = false, delta = 1, blksize=8, overlap=4, dct=0)
  forward_vec2 = MAnalyse(super, isb = false, delta = 2, blksize=8, overlap=4, dct=0)
  MDegrain2(source,super, backward_vec1,forward_vec1,backward_vec2,forward_vec2,thSAD=400) 
MergeChroma(aWarpSharp2(depth=30).aWarpSharp2(depth=10))
LimitedSharpenFaster()
AddGrainC(2.0,2.0)
ConvertToRGB32(matrix="Rec601",interlaced=false)
Crop(20,0,-8,-6).AddBorders(14,2,14,4)
return last
AVISource("I:\forum\faq\billct97\X1\F1\Firestone1_ ivtc.avi")
Trim(0,36)

AviSource() opens and decodes Firestone1_ivtc.avi and uses only the first 37 frames, which is Scene 1.

Import("D:\Avisynth 2.5\plugins\ContrastMask.avs")
contrastmask(enhance=2.0)

ContrastMask.avs is a specialized filter for correcting poor luminance in shadow areas without affecting the rest of the spectrum. In practise it needs some fixing in the upper midrange around RGB 64 to 128, which it sometimes over-brightens. Generally this video seems to have overly dense darks. Because contrast mask is an .avs scripted filter, it has to be added to the script at runtime with the Import() command. The default "enhance" value is 10, which is almost always too bright. The .avs is at http://www.digitalfaq.com/forum/atta...ontrastmaskavs. It requires the MaskTools plugin that comes with QTGMC. It also requires VariableBlur.dll, which in turn requires a Microsoft VC+ runtime and the fftw3 library, which are linked on Variable Blur's web page at http://avisynth.nl/index.php/VariableBlur. The fftw3 library also comes with the QTGMC package.

The conversion to RGB32 is for VirtualDub's filters. The filters used were ColorCamcorderDenoise, ColorMill, exorcist, and gradation curves. The main idea behind color adjustment was to add a visible amount of midrange blue for the skin tones, then reduce dark blue and dark green for the front grill. A small amount of dark red was needed, too. ColorMill made further tweaks in Blue, Green, and saturation. The color settings for Scene 1 are in "scene1.vcf", which will be inside the attached "Firestone1_vcf_settings.zip".

Don't download .vfc's into VirtualDub plugins. Rather, keep .vcf's in the same area as your source videos or other project files.

Frame 28 of the final Scene 1 has a "hiccup" in the broadcast, looking like a break in the telecine stream that has sawtooth edges on objects. It appears to be a fault in the original broadcast, and the man's arm movement is distorted. In the original telecined clip, the frame is #35. Unfortunately there is too much motion to allow frame interpolation for ReplaceFramesMC; the resulting distortion would look worse than the problem.

Scene 2 consists of the Firestone outdoor signage and the red-yellow-green stoplight sequence. Color balance seems convincing enough as-is but it could use a little more "snap" with brighter highlights. It has edge ghosts and halos along with chroma displacement and smearing.

Code:
AVISource("I:\forum\faq\billct97\X1\F1\Firestone1_ivtc.avi")
Trim(37,89)
### --- Next 4 lines are for test only --- ###
#Crop(20,0,-8,-8)
#ConvertToYV12(interlaced=false)
#Histogram("Levels")
#return last

FixChromaBleeding()
ChromaShift(c=6)
ConvertToYV12(interlaced=false)
source=last
  super = source.MSuper(pel=2, sharp=1)
  backward_vec2 = MAnalyse(super, isb = true, delta = 2, blksize=8, overlap=4, dct=0)
  backward_vec1 = MAnalyse(super, isb = true, delta = 1, blksize=8, overlap=4, dct=0)
  forward_vec1 = MAnalyse(super, isb = false, delta = 1, blksize=8, overlap=4, dct=0)
  forward_vec2 = MAnalyse(super, isb = false, delta = 2, blksize=8, overlap=4, dct=0)
  MDegrain2(source,super, backward_vec1,forward_vec1,backward_vec2,forward_vec2,thSAD=400) 
dfttest(sigma=8)
DeHalo_Alpha()
MergeChroma(aWarpSharp2(depth=30).aWarpSharp2(depth=10))
LimitedSharpenFaster()

Crop(20,0,-8,-6).AddBorders(14,2,14,4)
AddGrainC(2.0,2.0)
ConvertToRGB32(matrix="rec601",interlaced=false)
return last
AVISource("I:\forum\faq\billct97\X1\F1\Firestone1_ ivtc.avi")
Trim(37,89)

AviSource opens and decodes Firestone1_ivtc, using only frames 37 to 89 for Scene 2.

dfttest(sigma=8)
DeHalo_Alpha()

dfttest() is a denoiser noted for smoothing noise rather than removing grain. It's one of the support files that comes with QTGMC. It's used here at half power. DeHalo_Alpha() attacks either bright or dark edge halos (bright halos by default). It's pixel width can be adjusted, but wider pixel width softens video more. Use sparingly (http://www.digitalfaq.com/forum/atta...dehaloalphazip). Its support files MVTools2 and RGTools are part of QTGMC's package.

VirtualDub filters used were ColorCamcorderDenoiose, exorcist, and the built-in VDub Levels control. VDub Levels is used to restore brightness and contrast after using the exorcist filter. The settings are in "Scene2.vcf" which is inside "Firestone1_vcf_settings.zip".

The images below are from scene 3. Top image = original unfiltered frame #3 with YUV histogram showing dominance of Blue (U channel) and deficit of Red (V channel). Bottom image = frame 3 after YUV adjustment with Blue reduceed and red added. Did it matter? Too much Blue looked overdone to me.



Code:
AVISource("I:\forum\faq\billct97\X1\F1\Firestone1_ivtc.avi")
Trim(90,105)

ColorYUV(off_u=-10,off_v=5)
### --- Next 4 lines are for testing only --- ###
#Crop(20,0,-8,-8)
#ConvertToYV12(interlaced=false)
#Histogram("Levels")
#return last

FixChromaBleeding()
ChromaShift(c=6)
ConvertToYV12(interlaced=false)
source=last
  super = source.MSuper(pel=2, sharp=1)
  backward_vec2 = MAnalyse(super, isb = true, delta = 2, blksize=8, overlap=4, dct=0)
  backward_vec1 = MAnalyse(super, isb = true, delta = 1, blksize=8, overlap=4, dct=0)
  forward_vec1 = MAnalyse(super, isb = false, delta = 1, blksize=8, overlap=4, dct=0)
  forward_vec2 = MAnalyse(super, isb = false, delta = 2, blksize=8, overlap=4, dct=0)
  MDegrain2(source,super, backward_vec1,forward_vec1,backward_vec2,forward_vec2,thSAD=400) 
DeHalo_Alpha()
MergeChroma(aWarpSharp2(depth=30).aWarpSharp2(depth=10))
LimitedSharpenFaster()

Crop(20,0,-8,-6).AddBorders(14,2,14,4)
AddGrainC(2.0,2.0)
return last
AVISource("I:\forum\faq\billct97\X1\F1")
Trim(90,105)

AviSource() opens and decodes Firestone1_ivtc.avi, and Trim(90,105) uses only frames 90 thru 105 for Scene 3.

No VirtualDub filters were used on output, so there is no .vcf file for Scene 3.

Scene 4: color and gamma levels were a pain in the neck. The scene looks dim, with lame color. The original histogram is all twisted up: midtones are too red and are missing blue, darks are bluish, other colors look too green or too red, the salesman's skin tone looks putrid and heaven knows what shade the shirt is. Shadow areas look underexposed and there's an overall gritty look to dark details; the young kid's face is obscured and the car batteries on display are the wrong shade of white. These problems look like sloppy film mastering. It took a while of making RGB adjustments and then studying the effect for a long time, then coming back later to check it again. I must have changed everything 40 times. Some colors are like that, especially with commercials.

Code:
Import("D:\Avisynth 2.5\plugins\ContrastMask.avs")
Import("D:\Avisynth 2.5\plugins\ReplaceFramesMC.avs")

AVISource("I:\forum\faq\billct97\X1\F1\Firestone1_ivtc.avi")
Trim(106,155)
a=last
Save_audio=a
a

ColorYUV(off_u=9,off_v=-5)
Levels(12,1.0,255,16,250,dither=true,coring=false)
###--- Next 4 lines are for testing only ---###
#Crop(20,0,-8,-8)
#ConvertToYV12(interlaced=false)
#Histogram("Levels")
#return last

FixChromaBleeding()
ChromaShift(c=6)
ConvertToYV12(interlaced=false)
ReplaceFramesMC(30,1)
contrastmask(enhance=2.0)
source=last
  super = source.MSuper(pel=2, sharp=1)
  backward_vec2 = MAnalyse(super, isb = true, delta = 2, blksize=8, overlap=4, dct=0)
  backward_vec1 = MAnalyse(super, isb = true, delta = 1, blksize=8, overlap=4, dct=0)
  forward_vec1 = MAnalyse(super, isb = false, delta = 1, blksize=8, overlap=4, dct=0)
  forward_vec2 = MAnalyse(super, isb = false, delta = 2, blksize=8, overlap=4, dct=0)
  MDegrain2(source,super, backward_vec1,forward_vec1,backward_vec2,forward_vec2,thSAD=400) 
dfttest(sigma=8)
DeHalo_Alpha()
MergeChroma(aWarpSharp2(depth=30).aWarpSharp2(depth=10))
LimitedSharpenFaster()

Crop(20,0,-8,-6).AddBorders(14,2,14,4)
AddGrainC(2.0,2.0)
ConvertToRGB32(matrix="rec601",interlaced=false)
AudioDub(last,Save_audio)
return last
AVISource("I:\forum\faq\billct97\X1\F1\Firestone1_ ivtc.avi")
Trim(106,155)

AviSource() opens and decodes Firestone1_ivtc.avi, and Trim() uses only frames 106 thru 155 for Scene 4.

ColorYUV(off_u=9,off_v=-5)
Levels(12,1.0,255,16,250,dither=true,coring=false)

ColorYUV() increases Blue and decreases Red. Avisynth's Levels() looks at values starting with "12" and raises them to 16, to get some of the dark grime out of deep shadows. The value "250" slightly tames the brightest specular highlights. Look at the Levels function in online Help to see what those first 5 numbers mean (http://avisynth.nl/index.php/Levels). You can adjust those values to massage the behavior of dark and bright extremes. Use the Avisynth YUV histogram to check the results.

While scrolling in VirtualDub frame by frame when checking a script, you can turn script or filter changes on or off in the script, then check results instantly by clicking "File..." -> "Reopen video file" (or press F2), which reopens the video to the current frame. Sometimes there is a slight pause while Avisynth repeats the processing. You can repeat that again and again until memory clogs up and VirtualDub freezes, in which case you just have to reopen the script from scratch.

ReplaceFramesMC(30,1)
There is a very dark horizontal ripple in frame #30 that flashes across the screen just above the customer's forearm, in the horizontal stripes of his shirt. Because there's very little motion, ReplaceFramesMC interpolates a new frame successfully. In the final version of the DVD with pulldown, the black ripple is in frame 170 and is also partly visible in the stripes on the customer's tummy.

VirtualDub filters used were ColorCamcorderDenoiose, exorcist, gradation curves, and ColorMill. Gradation curves added brighter values of Red and Green, which brightened the top end of the spectrum, while Blue was added mostly to the midrange. After using exorcist, the overall brightness values were restored with the gradation curves "RGB" setting. The image below has a pink upward arrow that shows how the diagonal "RGB" line was altered to boost the brights:


Be careful when adding contrast or highlight brightening. When bright detail starts to fog out or disappear, the brightening is too strong.

The VDub filter settings are in "Scene4.vcf" which is inside "Firestone1_vcf_settings.zip".

Scene 5:

Code:
Import("D:\Avisynth 2.5\plugins\ContrastMask.avs")
AVISource("I:\forum\faq\billct97\X1\F1\Firestone1_ivtc.avi")
Trim(156,0)

Levels(16,1.0,255,16,245,dither=true,coring=false)
###--- Next 4 lines are for testing only ---###
#Crop(20,0,-8,-8)
#ConvertToYV12(interlaced=false)
#Histogram("Levels")
#return last

FixChromaBleeding()
ChromaShift(c=6)
ConvertToYV12(interlaced=false)
checkmate(thr=10, max=20, tthr2=2)
contrastmask(enhance=2.0)
source=last
  super = source.MSuper(pel=2, sharp=1)
  backward_vec2 = MAnalyse(super, isb = true, delta = 2, blksize=8, overlap=4, dct=0)
  backward_vec1 = MAnalyse(super, isb = true, delta = 1, blksize=8, overlap=4, dct=0)
  forward_vec1 = MAnalyse(super, isb = false, delta = 1, blksize=8, overlap=4, dct=0)
  forward_vec2 = MAnalyse(super, isb = false, delta = 2, blksize=8, overlap=4, dct=0)
  MDegrain2(source,super, backward_vec1,forward_vec1,backward_vec2,forward_vec2,thSAD=400) 
dfttest(sigma=8)
DeHalo_Alpha()
MergeChroma(aWarpSharp2(depth=30).aWarpSharp2(depth=10))
LimitedSharpenFaster()

Crop(20,0,-8,-6).AddBorders(14,2,14,4)
AddGrainC(2.0,2.0)
ConvertToRGB32(matrix="rec601",interlaced=false)
return last
AVISource("I:\forum\faq\billct97\X1\F1\Firestone1_ ivtc.avi")
Trim(156,0)

AviSource opens and decodes Firestone1_ivtc.avi and uses only frames 156 to the end for Scene 5.

Levels(16,1.0,255,16,245,dither=true,coring=false)
The blue lettering is just a tad overly bright at greater than y=235. Setting the bright limit to "245" tames that problem and makes it possible to spiff up the bright end in RGB.

checkmate(thr=10, max=20, tthr2=2)
Checkmate() is a dot crawl filter. Its wiki download page describes it and its settings (http://avisynth.nl/index.php/Checkmate). If you play your original Firestone_1.avi, look at the edges of the big Blue $20.00 characters. You'll see a noisy edge "buzz" that looks like bad blue screening in the mastering lab. Checkmate is used to calm that buzz, using settings widely recommended for this sort of thing. You can also use the filter at its weaker default settings by just typing "Checkmate()". Stronger settings create softer effects.

The VDub filters used were CamcorderColortDenoise, two (2) instances of the exorcist filter, and gradation curves to tweak color and restore highlights. The exorcist filter is run twice because of the very heavy edge ghosting on those blue letters. The VDub filter settings are in "Scene5.vcf" which is inside "Firestone1_vcf_settings.zip".

The 5 intermediate files were all saved out of Virtualdub as Lagarith YV12. I joined them in Avisynth and encoded them with 3:2 pulldown in TMPGenc as "Firestone1_cleanup_DVD_pulldown.mpg" (attached).

That was a toughie. Most of the problems appeared to be in the original broadcast.
On to Firestone_2....


Attached Images
File Type: png sc1 frame 32 RGB histogram and vectorscope.png (60.8 KB, 54 downloads)
File Type: jpg sc1 frame 32 original.jpg (124.5 KB, 55 downloads)
File Type: jpg sc1 frame 32 after cleanup.jpg (137.9 KB, 53 downloads)
File Type: jpg sc3 frame 3 original with YUV.jpg (82.2 KB, 51 downloads)
File Type: jpg sc3 frame 3 YUV adjusted.jpg (81.7 KB, 53 downloads)
File Type: png sc4 curves brightness adjust.png (54.3 KB, 56 downloads)
Attached Files
File Type: mpg Firestone1_cleanup_DVD_pulldown.mpg (7.65 MB, 7 downloads)
File Type: zip Firestone1_vcf_settings.zip (3.7 KB, 1 downloads)
Reply With Quote
  #23  
04-01-2020, 08:26 AM
billct97 billct97 is offline
Free Member
 
Join Date: Jan 2018
Posts: 40
Thanked 3 Times in 3 Posts
sanlyn, just a quick reply to say thank you. So much in this first clip that looks like I knew it should, wasn't sure how to make it happen and now it does. The car looking... yellow, the face tones, the detail in the black grille, the burgundy car, the battery which now looks white and the character edges that look so much tighter. I look forward to part 2 if you have time but more so look forward to playing with all of the examples in this forum thread and actually learning all I can. I expect this will be slow going as I have a lot to learn. Again, a quick thank you in case I don't post anything while I'm learning.
Reply With Quote
The following users thank billct97 for this useful post: sanlyn (04-01-2020)
  #24  
04-02-2020, 09:35 PM
sanlyn sanlyn is offline
Premium Member
 
Join Date: Aug 2009
Location: N. Carolina and NY, USA
Posts: 3,648
Thanked 1,307 Times in 982 Posts
A slowdown problem here, too. Doing COVID19 chores in our gated neighborhood, lots of houses and apts. Helping to check on the elderly and make sure they're OK. Had a pulmonologist appointment today at a hospital only 10 minutes from here and we live in a small town, but there was so much chaos at the clinic I ended up waiting for 2.5 hours, ate up the rest of the afternoon. Bummer.

Oh, well. Tomorrow is another day. I think that's a famous line I heard in a movie somewhere.
Reply With Quote
  #25  
04-04-2020, 10:57 AM
msgohan msgohan is offline
Free Member
 
Join Date: Feb 2011
Location: Vancouver, Canada
Posts: 1,323
Thanked 334 Times in 276 Posts
Quote:
Originally Posted by sanlyn View Post
As usual, ivtc came first as a previous file used for input here.
...
The final 23.976 fps 4:3 encoded results are attached as "Dentist_AG1980_cleanup_480p.mp4".
I think you got your wires crossed from too many crappy tapes! This one was shot on film, but at 30fps. Your decimated version is choppy since you took out every 5th frame.

After the dissolve at the 0:11, the rest of the ad is 29.97p already. No deinterlacing required.

The "black & white talking kids" segment that precedes this looks interlaced but is really just phase-shifted. SeparateFields().Trim(1,0).Weave() turns the shots themselves into proper progressive, but the edits are out-of-phase, so now those few frames are interlaced (i.e. one frame at each scene change).

TFM works "okay" here, but picks a lot of wrong matches because of the small amount of motion and high amount of noise. I created a filter years ago to automatically handle content like this with better results, but I couldn't nail down why it would crash sometimes, so I never released it.

Last edited by msgohan; 04-04-2020 at 11:20 AM.
Reply With Quote
  #26  
04-04-2020, 12:13 PM
sanlyn sanlyn is offline
Premium Member
 
Join Date: Aug 2009
Location: N. Carolina and NY, USA
Posts: 3,648
Thanked 1,307 Times in 982 Posts
Quote:
Originally Posted by msgohan View Post
I think you got your wires crossed from too many crappy tapes! This one was shot on film, but at 30fps. Your decimated version is choppy since you took out every 5th frame.

After the dissolve at the 0:11, the rest of the ad is 29.97p already. No deinterlacing required.

The "black & white talking kids" segment that precedes this looks interlaced but is really just phase-shifted. SeparateFields().Trim(1,0).Weave() turns the shots themselves into proper progressive, but the edits are out-of-phase, so now those few frames are interlaced (i.e. one frame at each scene change).

TFM works "okay" here, but picks a lot of wrong matches because of the small amount of motion and high amount of noise. I created a filter years ago to automatically handle content like this with better results, but I couldn't nail down why it would crash sometimes, so I never released it.

Yes, I noticed those shortcomings on working with newer filters for the clips I wasn't satisfied with, but I've not had time to address it and post corrections. However, I didn't catch the phase shifted part. Thanks for catching it. I'll have to take a closer look tonight.
Reply With Quote
  #27  
04-05-2020, 09:30 AM
sanlyn sanlyn is offline
Premium Member
 
Join Date: Aug 2009
Location: N. Carolina and NY, USA
Posts: 3,648
Thanked 1,307 Times in 982 Posts
Quote:
Originally Posted by msgohan View Post
The "black & white talking kids" segment that precedes this looks interlaced but is really just phase-shifted. SeparateFields().Trim(1,0).Weave() turns the shots themselves into proper progressive, but the edits are out-of-phase, so now those few frames are interlaced (i.e. one frame at each scene change).
I get most of what you're saying. But SeparateFields().Trim(1,0).Weave didn't make anything look progressive. However the script I used below did so. Since there is such a mix of frame structures I decided to make it all progressive.

The video in question was a sample capture encoded to high bitrate MPEG and attached to post #12 as "Dentist_AG1980_sample.mpg". It's original title as lossless Lagarith YUY2 was "GI1_e_dentist_Ag1980.avi". It's linked to a pro MediaFire account at https://www.mediafire.com/file/jogh2...g1980.avi/file. Pro accounts do not show popup ads or web pages.

So the "dentist" clip has both interlaced and progressive sequences. It's not telecined. That's what I get for making assumptions and rushing in without looking around first. (wagging finger at myself) Let that be a lesson to me! And it's not the first time, either.

The content of the original clip has two parts, beginning with a black and white sequence that dissolves into two or three panning shots of dental tools, toothpaste, and a model tooth. It looks interlaced but, according to msgohan (who happens to be correct), it's really fields out of phase. The second part begins at frame 652. It is progressive and in color.

I remade this clip in three parts. Part 1 assumes that the first part up to frame 651 looks interlaced ut has out of phase fields. The script for Part 1 discards the first field in the clip, moveing the others one field forward. Then QTGMC deinterlaces the result and discards alternate frames, which are duplicates. Part 2 takes the results of part 1 and applies the same denoising and filters that were used in the first version of this clip in post #12.

Part 3 then takes the clip from the original's progressive sequence in frame 652 to the end. It denoises and corrects color, mainly skin tones. The beginning and end sequences are then joined together and encoded.

The attached "Dentist_Remake_29_97_480i.mpg" is encopded as interlaced, although physically it's progressive 29.97fps. Why encode it that wa7? if you wanted to join this clip into a series of other videos that are either interlaced or telecined (the latter is treated as interlaced in authoring programs), you would have to encode the dentist clip as interlaced. And for standard definition BluRay, progressive isn't allowed. If you wantesd to mount itg on the internet, you would encode he progressive version as progressive in something like h.264/mp4. But remember that you can't mix progressive and interlaced or mix different frame rates in the same video.

Here's the script I used (Part 1) to make the front end of the dentist clip entirely progressive:

Code:
AVISource(I:\forum\faq\billct97\dentist\GI1_e_dentist Ag1980.avi")
Trim(0,651)

SeparateFields().Trim(1,0)
FixChromaBleeding()
ChromaShift(c=-6)
Weave()
ConvertToYV12(interlaced=false)
hqdering()
qtgmc(preset="medium",EZDenoise=2,denoiser="dfttest",GrainRestore=0.3,\
  border=true,ChromaNoise=true,FPSdivisor=2)
SeparateFields().Trim(1,0)
Weave()

SeparateFields() breaks full-sized interlaced frames into half-size fields, using alternating scanlines to build in field (http://avisynth.nl/index.php/SeparateFields). Trim(1,0) starts with takes the second field in the stream of fields (field numbers start with zero) and makes a new stream of fields using only field 1 to the end of the clip (http://avisynth.nl/index.php/Trim). Weave() re-weaves the stream of fields into top and bottom fields in interlaced frames (http://avisynth.nl/index.php/Weave).

"FPSdivisore=2" in the QTGMC line drops every other field in the deinterlaced video, thus keeping the frame rate at 29.97fps instead of 59.94. The fields being dropped are out of phase duplicates.

Below is the script for Part 2, which denoises Part 1. It's mostly grayscale but with some color that already looks OK, so no complicated color correction is needed.

Code:
AVISource("I:\forum\faq\billct97\dentist\DentistA_i_to_p.avi")   #<- Part 1 saved as YV12
###--- Next 3 lines are for test only ###---
#Crop(4,6,-20,-4)
#ConvertToYV12(interlaced=false)
#Histogram("Levels")
#return last

###--- INPUT IS YV12 ---#
ChromaShift(C=-4)
source=last
  super = source.MSuper(pel=2, sharp=1)
  backward_vec2 = MAnalyse(super, isb = true, delta = 2, blksize=8, overlap=4, dct=0)
  backward_vec1 = MAnalyse(super, isb = true, delta = 1, blksize=8, overlap=4, dct=0)
  forward_vec1 = MAnalyse(super, isb = false, delta = 1, blksize=8, overlap=4, dct=0)
  forward_vec2 = MAnalyse(super, isb = false, delta = 2, blksize=8, overlap=4, dct=0)
  MDegrain2(source,super, backward_vec1,forward_vec1,backward_vec2,forward_vec2,thSAD=400) 
dfttest(sigma=8)
MergeChroma(aWarpSharp2(depth=30).aWarpSharp2(depth=10))
LimitedSharpenFaster(edgemode=2)
grainfactory3(g1str=1, g2str=1, g3str=1)
AddGrainC(2.0,2.0)

Crop(4,6,-20,-4).AddBorders(12,4,12,6)
Dither_convert_8_to_16 ()
Dither_resize16 (720, 480, kernel="spline36", cplace="MPEG2")
Dither_convert_yuv_to_rgb(matrix="601",interlaced=false,tv_range=true,cplace="MPEG2",\
   lsb_in=true,output="rgb32")
return last
The conversion to RGB32 is for the NeatVideo filter and ColorCamcorderDenoise for the color frames of this sequence. If you don't have NeatVideo, increase the value of sigma in the dfttest line to "dfttest(sigma=16)". The VDub filter settings are in "Dentist_remake_Part2.vcf" inside the attached "Dentist_Remake_VDub_Settings.zip". The NeatVideo noise test patches are attached in post #12.

Also, If you don't have NeatVideo the .vcf file will give an error. In that case, make a backup copy of the .vcf. Open the vcf in Notepad or other plain text editor. Delete the next to last 2 lines in the .vcf that begin with these words:
Code:
VirtualDub.video.filters.Add("Neat Video");
VirtualDub.video.filters.instance[4].Config("I:\\forum\\faq\\billct97....." and so on
Leave the last line intact in the .vcf. The last line will read:

Code:
VirtualDub.audio.filters.Clear();
Don't leave Word Wrap turned on in Notepad when editing scripts.

Part 3 is only one script. It takes line 652 to the end of the clip and works with the color sequence that is already progressive. Again, I'm using QTGMC in progressive mode ("InputType=2") for smoothing and denoising of this crummy video, along with MGrain2.

Code:
AVISource("D:\Avisynth 2.5\plugins\AVS26\dither\GI1_e_dentist Ag1980.avi")
Trim(652,0)

###--- Next 4 lines are for test only ---###
#Crop(4,6,-20,-4)
#ConvertToYV12(interlaced=false)
#Histogram("Levels")
#return last

###--- Input is the original YUY2 ---###
FixChromaBleeding()
ChromaShift(l=-2,c=-6)
ConvertToYV12(interlaced=false)
hqdering()
qtgmc(InputType=2,preset="medium",EZDenoise=2,denoiser="dfttest",GrainRestore=0.3,\
  border=true,ChromaNoise=true)
source=last
  super = source.MSuper(pel=2, sharp=1)
  backward_vec2 = MAnalyse(super, isb = true, delta = 2, blksize=8, overlap=4, dct=0)
  backward_vec1 = MAnalyse(super, isb = true, delta = 1, blksize=8, overlap=4, dct=0)
  forward_vec1 = MAnalyse(super, isb = false, delta = 1, blksize=8, overlap=4, dct=0)
  forward_vec2 = MAnalyse(super, isb = false, delta = 2, blksize=8, overlap=4, dct=0)
  MDegrain2(source,super, backward_vec1,forward_vec1,backward_vec2,forward_vec2,thSAD=400) 
MergeChroma(aWarpSharp2(depth=30).aWarpSharp2(depth=10))
LimitedSharpenFaster(edgemode=2)
grainfactory3(g1str=1, g2str=1, g3str=1)
AddGrainC(2.0,2.0)
Crop(4,6,-20,-4).AddBorders(12,4,12,6)

Dither_convert_8_to_16 ()
Dither_resize16 (720, 480, kernel="spline36", cplace="MPEG2")
Dither_convert_yuv_to_rgb(matrix="601",interlaced=false,tv_range=true,cplace="MPEG2",\
   lsb_in=true,output="rgb32")
return last
RGB32 conversion is for NeatVideo and the color correction filters, which were ColorCamcorderDenoise v1.7, ColorMill, gradation curves, and hue/saturation/intensity (download Hue.vdf at http://www.digitalfaq.com/forum/atta...9&d=1540679970). The latter filter was used to reduce saturation in Red and Magenta. VDub settings are in "Dentist_remake_Part3.vcf" inside of the attached "Dentist_Remake_VDub_Settings.zip".

As mentioned earlier, if you don't have NeatVideo use "dfttest(sigma=16)" in the line following the MDegrain2 routine and before the "MergeChroma" line. And again, if you don't have NeatVideo use the .vcf file changes suggested earlier for Script 2.

The finished and attached .mpg "Dentist_Remake_29_97_480i.mpg" is encoded for DVD with 3:2 pulldown restored.


Attached Files
File Type: mpg Dentist_Remake_29_97_480i.mpg (26.21 MB, 4 downloads)
File Type: zip Dentist_Remake_VDub_Settings.zip (1.8 KB, 1 downloads)
Reply With Quote
  #28  
04-05-2020, 10:46 AM
msgohan msgohan is offline
Free Member
 
Join Date: Feb 2011
Location: Vancouver, Canada
Posts: 1,323
Thanked 334 Times in 276 Posts
Quote:
Originally Posted by sanlyn View Post
SeparateFields().Trim(1,0).Weave didn't make anything look progressive.
You have to use AssumeTFF() before any field-based operation in Avisynth, like SeparateFields, Bob, etc., since the default operation of AVISource is to assume BFF.

When I was testing it I didn't have to do this since the MPEG-2 you posted is flagged TFF, and I hadn't downloaded the lossless version earlier.

Here:
Code:
AVISource("GI1_e_dentist Ag1980.avi").AssumeTFF()

Crop(0,6,-18,-0) # Cropping now so that the deinterlacer doesn't need to consider the garbage on the side in its decisions

# Split into two segments, because the B&W part is out-of-phase and the rest is in-phase progressive.
split = 322
Phase1 = Trim(0,split)
Phase2 = Trim(split+1,0)

# Process Phase1
Phase1
SeparateFields().Trim(1,0).Weave() # Swap phase
Phase1 = last

Phase1+Phase2

# Deinterlace the left-over combing at "Phase1" scene changes, the cross-dissolve, parts of the CGI tooth animation, and the fades from & to black
# *** NOTE: REMOVE MAP=2 FOR FINAL ENCODE. ***
TDeint(full=false,cthresh=8,map=2)
# Wanted to use tryWeave=true so that the scene changes are weaved rather than deint'd, but it screws up the CGI tooth
Quote:
Originally Posted by sanlyn View Post
The finished and attached .mpg "Dentist_Remake_29_97_480i.mpg" is encoded for DVD with 3:2 pulldown restored.
Not 3:2 pulldown!


Attached Files
File Type: avi Dentist_AG1980_MSG_29p_H264-in-AVI.avi (9.35 MB, 1 downloads)

Last edited by msgohan; 04-05-2020 at 11:35 AM. Reason: Changed AVS script a little; added test encode
Reply With Quote
The following users thank msgohan for this useful post: sanlyn (04-05-2020)
  #29  
04-05-2020, 02:43 PM
sanlyn sanlyn is offline
Premium Member
 
Join Date: Aug 2009
Location: N. Carolina and NY, USA
Posts: 3,648
Thanked 1,307 Times in 982 Posts
Quote:
Originally Posted by msgohan View Post
You have to use AssumeTFF() before any field-based operation in Avisynth, like SeparateFields, Bob, etc., since the default operation of AVISource is to assume BFF.
I did TFF and BFF. Finally took the default and most looked interlaced anyway. Also tried fancy tricks around the dissolves, but it always looked clunky at those points so I just kept it consistent all the way through to the color sequence. When I run into clips with oddball mastering it's always a hassle and never comes off smoothly. I end up keeping it simple or consistent.

Quote:
Originally Posted by msgohan View Post
Here:
Code:
AVISource("GI1_e_dentist Ag1980.avi").AssumeTFF()

Crop(0,6,-18,-0) # Cropping now so that the deinterlacer doesn't need to consider the garbage on the side in its decisions

# Split into two segments, because the B&W part is out-of-phase and the rest is in-phase progressive.
split = 322
Phase1 = Trim(0,split)
Phase2 = Trim(split+1,0)

# Process Phase1
Phase1
SeparateFields().Trim(1,0).Weave() # Swap phase
Phase1 = last

Phase1+Phase2

# Deinterlace the left-over combing at "Phase1" scene changes, the cross-dissolve, parts of the CGI tooth animation, and the fades from & to black
# *** NOTE: REMOVE MAP=2 FOR FINAL ENCODE. ***
TDeint(full=false,cthresh=8,map=2)
# Wanted to use tryWeave=true so that the scene changes are weaved rather than deint'd, but it screws up the CGI tooth
I tried something similar and gave up. I'll try your exact wording and see what happens, but I hate it when mixed-bag dissolves come along. I have a long history of just eventually plowing through that stuff.

Quote:
Originally Posted by msgohan View Post
Not 3:2 pulldown!
No, it doesn't have 3:2 pulldown but when I caught the typo it was too late to change it. One hour isn't enough lag time IMO.
Reply With Quote
  #30  
04-05-2020, 08:07 PM
sanlyn sanlyn is offline
Premium Member
 
Join Date: Aug 2009
Location: N. Carolina and NY, USA
Posts: 3,648
Thanked 1,307 Times in 982 Posts
Quote:
Originally Posted by billct97 View Post
I have recaptured the Firestone commercial from 1986 if anyone might care to take a look.
Thanks again for the better samples. I worked over Firestone-1.avi (or it worked me over!) in an earlier post.

The Firestone-2.avi sample demonstrates how VHS and old analog commercials demand patience. Every analog commercial and scene has different requirements. It seems the farther back you go in time with analog TV, the more things differ scene by scene. I cut the sample into 6 scenes this time. Filters were similar to those used in Firestone-1, with exorcist and gradation curves as usual to handle those edge ghosts.

The first script removes telecine from the incoming Firestone-2.avi and was saved as Lagarith YUY2.
Code:
AVISource("I:\forum\faq\billct97\X1\F2\Firestone-2.avi")
AssumeTFF()
TFM().TDecimate()
Scene 1 is certainly dark with very little shadow detail, and overly red. Brightening doesn't help (nothing but noise and distortion down there). Correcting color for real-life accuracy isn't possible -- the colors really turn to garbage if you try for realism. But it does look a little better with whiter whites on those knobs and the gear shift numerals. That's as far as one can get. It's obvious that the creators wanted macho darks and red-hot color, for whatever reason. The scene does need smoothing: the grain on the exposed parts of the hand could turn to schmutz when encoded.

scene 1 color/ Before (left), after (right)

Code:
AVISource("I:\forum\faq\billct97\X1\F2\Firestone2_ivtc.avi")
Trim(0,18)

ColorYUV(off_y=-3)
###--- Next 4 lines for test only ---###
#Crop(20,0,-8,-8)
#ConvertToYV12(interlaced=false)
#Histogram("Levels")
#return last

FixChromaBleeding()
ConvertToYV12(interlaced=false)
source=last
  super = source.MSuper(pel=2, sharp=1)
  backward_vec2 = MAnalyse(super, isb = true, delta = 2, blksize=8, overlap=4, dct=0)
  backward_vec1 = MAnalyse(super, isb = true, delta = 1, blksize=8, overlap=4, dct=0)
  forward_vec1 = MAnalyse(super, isb = false, delta = 1, blksize=8, overlap=4, dct=0)
  forward_vec2 = MAnalyse(super, isb = false, delta = 2, blksize=8, overlap=4, dct=0)
  MDegrain2(source,super, backward_vec1,forward_vec1,backward_vec2,forward_vec2,thSAD=400) 
dfttest()
MergeChroma(aWarpSharp2(depth=30).aWarpSharp2(depth=10))
LimitedSharpenFaster()
AddGrainC(2.0,2.0)
ConvertToRGB32(matrix="rec601",interlaced=false)
Crop(20,0,-8,-8).AddBorders(14,4,14,4)
return last
dfttest() is used at its default sigma value of 16 for a little more smoothing of the remaining grain. The conversion to RGB32 is for the VirtualDub filters loaded on output, which were ColorCamcorderDenoise, exorcist, and gradation curves. Frankly the original color is probably closer to what the creators wanted. To keep it, just don't use the VirtualDub filters. The VDub settings are in "scene1.vcf" inside the attached zip file "Firestone2_VDub_Settings.zip".

"Firestone2_VDub_Settings.zip"

A note about gradation curves in this project: VDub's gradation curve filter isn't used only for color correction. It's also used to blacken the final frame borders, which often take on a color cast after correction work. If you look at some of the curves applied you'll see some with a short corrective "hook" at the lower left zero point. If you don't use curves then you can't use this method, so you would have to replace the frame's borders with new black ones by using VDub's "Border Control.vdf" filter (https://sourceforge.net/projects/bor...atest/download). Unfortunately this .vdf can't be saved in a .vcf. A vcf that contains text for the filter will not load it properly.


The image above is the RGB panel of a curves control that adjusts brightness after the exorcist filter darkens the image. At the lower left corner of the graph the slanted line has several "o"-rings to hold the curve in place, then along the lower border the lowest o-ring on the slanted line is placed last and slid slightly to the right. Just below the arrow, the caption over the "Input Value" reads "4", while the caption over the "Output Value" reads "0". This indicates that the bottom o-ring has set all values 4 and darker to 0. The other o-rings keep the setting from affecting other parts of the image. This image of a curves control is from Scene 6.

Scene 2 with the three guys doesn't look bad, but it's rather greenish. There is also some blue lighting off and dark blue in the "black:" background>

Code:
AVISource("I:\forum\faq\billct97\X1\F2\Firestone2_ivtc.avi")
Trim(19,44)
###--- Next 4 lines are for test only ---###
#Crop(20,0,-8,-8)
#ConvertToYV12(interlaced=false)
#Histogram("Levels")
#return last

ConvertToYV12(interlaced=false)
source=last
  super = source.MSuper(pel=2, sharp=1)
  backward_vec2 = MAnalyse(super, isb = true, delta = 2, blksize=8, overlap=4, dct=0)
  backward_vec1 = MAnalyse(super, isb = true, delta = 1, blksize=8, overlap=4, dct=0)
  forward_vec1 = MAnalyse(super, isb = false, delta = 1, blksize=8, overlap=4, dct=0)
  forward_vec2 = MAnalyse(super, isb = false, delta = 2, blksize=8, overlap=4, dct=0)
  MDegrain2(source,super, backward_vec1,forward_vec1,backward_vec2,forward_vec2,thSAD=400) 
dfttest()
MergeChroma(aWarpSharp2(depth=30).aWarpSharp2(depth=10))
LimitedSharpenFaster()
Crop(20,0,-8,-8).AddBorders(14,4,14,4)
AddGrainC(2.0,2.0)
ConvertToRGB32(matrix="rec601",interlaced=false)
return last
The VirtualDub filters used are ColorCamcorderDenoise, exorcist, and gradation curves. The VDub settings are in "scene2.vcf" inside the attached zip file "Firestone2_VDub_Settings.zip".

Scene 3 script:

Code:
AVISource("I:\forum\faq\billct97\X1\F2\Firestone2_ivtc.avi")
Trim(45,75)
###--- Next 4 lines are for test only ---###
#Crop(20,0,-8,-8)
#ConvertToYV12(interlaced=false)
#Histogram("Levels")
#return last

ConvertToYV12(interlaced=false)
source=last
  super = source.MSuper(pel=2, sharp=1)
  backward_vec2 = MAnalyse(super, isb = true, delta = 2, blksize=8, overlap=4, dct=0)
  backward_vec1 = MAnalyse(super, isb = true, delta = 1, blksize=8, overlap=4, dct=0)
  forward_vec1 = MAnalyse(super, isb = false, delta = 1, blksize=8, overlap=4, dct=0)
  forward_vec2 = MAnalyse(super, isb = false, delta = 2, blksize=8, overlap=4, dct=0)
  MDegrain2(source,super, backward_vec1,forward_vec1,backward_vec2,forward_vec2,thSAD=400) 
dfttest()
MergeChroma(aWarpSharp2(depth=30).aWarpSharp2(depth=10))
LimitedSharpenFaster()
AddGrainC(2.0,2.0)
ConvertToRGB32(matrix="rec601",interlaced=false)
Crop(20,0,-8,-8).AddBorders(14,4,14,4)
return last
The VirtualDub filters used are ColorCamcorderDenoise, exorcist, and gradation curves. The VDub settings are in "scene3.vcf" inside the attached zip file "Firestone2_VDub_Settings.zip".

Scene 3 has the same color problems as a previous salesroom scene in Firestone-1. The yellowish problem seems to be from interior lighting. Fortunately there is white, gray and black in the tires on display, which is helpful even if the skin tones are still a little yellow and the whitewalls are not pure white. I didn't try corrections in YUV, which turned everything blue, especially in the blacks. Its another case of trying an RGB adjustment, letting it bake for a few hours, and checking later for another round of adjustments.

How do you know when the whites are white, and so forth? Basic color correction attempts to find a white balance and, if available, a gray balance and a black balance. All shades of white from pure white thru gray to black are formed with RGB colors Red, Green, and Blue (which is where wwe get the acronym "RGB"). The RGB numbers for those colors are often stated as RGB RRR-GGG-BBB, as below:

super white = Red 255, green 255, B 255, or = RGB 255-255-255
video white = RGB 235-235-235
light gray = RGB 192-192-192
middle gray = RGB 128-128-128
dark gray = RGB 64-64-64
digital black = RGB 16-16-16
super black = RGB 0-0-0

You'll see that each shade of white from super white to super black consists of equal chunks of each color, R, G, and B. If you can find a white or gray object in an image and make it look correct for that shade, then all RGB colors are in balance at that point and the other mixed colors will fall into place. Achieving that balance is called finding the white point, or the gray point or, when needed, a proper black point.

A pure red will have only Red, with no Green or Blue in the mix. But yolu're not likely to fin d pure color in nature, mnot even in a "blue" sky (which will likely contain some slight red and green).,We would say that the RGB colors in pure primary RGB terms are:
Red = RGB 255-000-000
Green = RGB 000-255-000
Blue = RGB 000-000-255

The secondary colors mix 2 basic RGB colors together:
Yellow = RGB 255-255-000
Cyan = RGB 000-255-255
Magenta = RGB 255-000-255

All other colors such as orange, hot pink, scarlet, brown, indigo, etc., are just various mixes of R, G, and B.

How do you measure those pixel values? With pixel readers like those provided with big-name video and photo editors, or with freebies like the CSamp pixel reader control mentioned earlier or the ColorPic at
http://www.iconico.com/colorpic/. Take a reading and then use ColorMill, gradation curves or other RGB controls to set those colors to the ideal combo. It will seldom be perfect but at least you'll get close. And you'll also find out just how bad VHS color really is.

Scene 4 script:

Code:
AVISource("I:\forum\faq\billct97\X1\F2\Firestone2_ivtc.avi")
Trim(76,105)

ColorYUV (off_u=-2)
###---  Next 4 lines are for test only ---###
#Crop(20,0,-8,-8)
#ConvertToYV12(interlaced=false)
#Histogram("Levels")
#return last

FixChromaBleeding()
ConvertToYV12(interlaced=false)
source=last
  super = source.MSuper(pel=2, sharp=1)
  backward_vec2 = MAnalyse(super, isb = true, delta = 2, blksize=8, overlap=4, dct=0)
  backward_vec1 = MAnalyse(super, isb = true, delta = 1, blksize=8, overlap=4, dct=0)
  forward_vec1 = MAnalyse(super, isb = false, delta = 1, blksize=8, overlap=4, dct=0)
  forward_vec2 = MAnalyse(super, isb = false, delta = 2, blksize=8, overlap=4, dct=0)
  MDegrain2(source,super, backward_vec1,forward_vec1,backward_vec2,forward_vec2,thSAD=400) 
dfttest()
MergeChroma(aWarpSharp2(depth=30).aWarpSharp2(depth=10))
LimitedSharpenFaster()
AddGrainC(2.0,2.0)
Crop(20,0,-8,-8).AddBorders(14,4,14,4)
ConvertToRGB32(matrix="rec601",interlaced=false)
return last
FixChromaBleed() helps tame the magenta lighting from the side and in the background. But practically speaking the VirtualDub RGB color changes don't help much. The black tire reflects colors from everywhere and will never be consistently black everywhere. Nevertheless the filters used were ColorCamcorderDenoise, exorcist, and gradation curves. The only part of the curves filter you really need is the "RGB" panel to correct luminance after exorcist is applied. The VDub settings are in "scene4.vcf" inside the attached zip file "Firestone2_VDub_Settings.zip".

Scene 5 with the bold blue characters had the usual problems, like the similar shot in Firestone-1:

Code:
ppath="D:\Avisynth 2.5\plugins\"
Import(ppath+"ContrastMask.avs")
Import(ppath+"ReplaceFramesMC.avs")
AVISource("I:\forum\faq\billct97\X1\F2\Firestone2_ivtc.avi")
Trim(106,194)
a=last
Save_audio=a
a

Levels(16,1.0,255,16,245,dither=true,coring=false)
###--- Next 4 lines are for test only ---###
#Crop(20,0,-8,-8)
#ConvertToYV12(interlaced=false)
#Histogram("Levels")
#return last

FixChromaBleeding()
ConvertToYV12(interlaced=false)
contrastmask(enhance=2.0)
ReplaceFramesMC(43,1)
source=last
  super = source.MSuper(pel=2, sharp=1)
  backward_vec2 = MAnalyse(super, isb = true, delta = 2, blksize=8, overlap=4, dct=0)
  backward_vec1 = MAnalyse(super, isb = true, delta = 1, blksize=8, overlap=4, dct=0)
  forward_vec1 = MAnalyse(super, isb = false, delta = 1, blksize=8, overlap=4, dct=0)
  forward_vec2 = MAnalyse(super, isb = false, delta = 2, blksize=8, overlap=4, dct=0)
  MDegrain2(source,super, backward_vec1,forward_vec1,backward_vec2,forward_vec2,thSAD=400) 
dfttest()
DeHalo_Alpha(brightstr=2.5)
MergeChroma(aWarpSharp2(depth=30).aWarpSharp2(depth=10))
LimitedSharpenFaster()

Spline36Resize(width/2,height)
Spline36resize(width*2,height)

Crop(20,0,-8,-8).AddBorders(14,4,14,4)
AddGrainC(2.0,2.0)
AudioDub(last,Save_audio)
ConvertToRGB32(matrix="rec601",interlaced=false)
return last
ppath=Import(ppath+"ContrastMask.avs")"
Import(ppath+"ContrastMask.avs")
Import(ppath+"ReplaceFramesMC.avs")

These lines are really tricks to help avoid some typing. "ppath" is a name I invented that means "plugin path", which becomes a user variable that contains a string with the name of the Avisynth plugin path in my system, which is "D:\Avisynth 2.5\plugins". Don't forget the dividing "" on the end of the string. Import(ppath+"ContrastMask.avs") uses the concatenate character "+" to join the ppath string "D:\Avisynth 2.5\plugins" to the filter name string, "ContrastMask.avs". So the line Import(ppath+"ContrastMask.avs") actually gets read as Import"D:\Avisynth 2.5\plugins\ContrastMask.avs"). You don't have to use that trick if you don't want it, otherwise you have to type the path entirely from scratch on each Import line.

a=last
Save_audio=a
a

You've seen this before. A frame replacement routine cause a break in the audio, so a copy of the incoming clip called "a" and another copy is saved as "Save_Audio" for later use to restore the original audio near the end of the script. The clip "a" is named explicitly to return processing focus to the incoming clip. At the end of the script, Audio-Dub() uses Save-Audio to restore the original sound.

contrastmask(enhance=2.0)
ReplaceFramesMC(43,1)

ContrastMask() at a low enhance value is used to bring up some deep shadow detail. ReplaceFramesMC fixes a thin horizontal ripple (dropout) in frame #43 that obscures some of the white lettering under the blue characters. In the original Firestone-1.avi the frame is #186.

DeHalo_Alpha(brightstr=2.5)
The brightstring parameter in DeHalo_Alpha is strengthened a bit to calm the bright right-edge halo on the pile of tiles. VDub's exorcist filter later helps with the edge ghosts.

Spline36Resize(width/2,height)
Spline36resize(width*2,height)

The firt line resizes the frames to 360x480, the second line returns the frame to 720x480. This reduction in horizontal size helps to thin some of the heavy whitish ghost lines in the blue letters and also helps thin out the white characters a little. This routine affects sharpness only slightly.

The VDub filters used were ColorCamcorderDenoise, two (2) consecutive copies of exorcist for the heavy ghosting, gradation curves, and the built-in Levels to give specular highlights a little more snap after using exorcist. The VDub settings are in "scene5.vcf" inside the attached zip file "Firestone2_VDub_Settings.zip".

Scene 6 script, with a few more problems:

Code:
Import("D:\Avisynth 2.5\plugins\RemoveDirtMC.avs")
AVISource("I:\forum\faq\billct97\X1\F2\Firestone2_ivtc.avi")
Trim(195,0)
###--- Next 4 lines are for test only ---###
#Crop(20,0,-8,-8)
#ConvertToYV12(interlaced=false)
#Histogram("Levels")
#return last

FixChromaBleeding()
ChromaShift(c=6)
ConvertToYV12(interlaced=false)
source=last
  super = source.MSuper(pel=2, sharp=1)
  backward_vec2 = MAnalyse(super, isb = true, delta = 2, blksize=8, overlap=4, dct=0)
  backward_vec1 = MAnalyse(super, isb = true, delta = 1, blksize=8, overlap=4, dct=0)
  forward_vec1 = MAnalyse(super, isb = false, delta = 1, blksize=8, overlap=4, dct=0)
  forward_vec2 = MAnalyse(super, isb = false, delta = 2, blksize=8, overlap=4, dct=0)
  MDegrain2(source,super, backward_vec1,forward_vec1,backward_vec2,forward_vec2,thSAD=400) 
RemoveDirtMC(40,false)
dfttest()
DeHalo_Alpha(brightstr=2.0)
MergeChroma(aWarpSharp2(depth=30).aWarpSharp2(depth=10))
LimitedSharpenFaster()
Crop(20,0,-8,-8).AddBorders(14,4,14,4)
AddGrainC(2.0,2.0)
ConvertToRGB32(matrix="rec601",interlaced=false)
return last
RemoveDirtMC(40,false) is used to quell some grainy noise as wwell as to clean up 3 horizontal rips in frames #12, 20, and 30 (in the original Firestone-2.avi the frames are #249, 259, and 269). These are short ripples across the white letters at the bottom of the frame. They're easier to see full screen or on TV. RemoveDirtMC(40,false) is used here at strength 40 which can clean up many forms of small dropouts and white streaks, Stronger settings work but can sometimes make moving objects disappear. The usual strength for denoising is 20. The "false" parameter means you're not working with monochrome.

The VirtualDub filters used were ColorCamcorderDenoise, exorcist, gradation curves, built-in Levels, and FadeFX to mask the very dirty fade-out in frames 31 to 35. The Levels filter and curves were used to clean up all the colorful noise in the so-called "black" background which isn't a very clean black without some help. Remember to use controls like FadeFX at the end of the filter chain, or your other filters will affect FadeFX's action. The VDub settings are in "scene6.vcf" inside the attached zip file "Firestone2_VDub_Settings.zip".

Hope there weren't too many weird typos this time. The result of these scripts and filters is attached as "Firestone2_cleanup_DVD_pulldown.mpg".


Attached Images
File Type: jpg Scene 1 color before - after.jpg (62.0 KB, 46 downloads)
File Type: png curves filter zero black for borders.png (53.7 KB, 43 downloads)
Attached Files
File Type: zip Firestone2_VDub_Settings.zip (6.2 KB, 1 downloads)
File Type: mpg Firestone2_cleanup_DVD_pulldown.mpg (7.71 MB, 2 downloads)
Reply With Quote
  #31  
04-05-2020, 11:09 PM
billct97 billct97 is offline
Free Member
 
Join Date: Jan 2018
Posts: 40
Thanked 3 Times in 3 Posts
This is a fantastic education! I spent some time this weekend comparing scripts in this thread as well as a few older threads and built a script to play with. First to prove that I can get Virtual Dub and Avisynth loaded with all the plugins that seem to be commonly used here (and I did!) and second to see the effects of using different plugins. Entirely useless for actual repair work without logical selection but confidence building just the same. This is so much farther than I got 2 years ago before giving up.

I also have come to the realization that doing this kind of repair for everything I have will need to be prioritized. I mapped out in a one hour TV show about 25 commercials, news breaks, etc... quite likely an impossible task to take on given that I have 40 tapes each with 6 hours of TV shows. More or less. The plan for now is to capture everything with lossless compression (lagarith has worked well so far) while being careful to watch for proper levels and any cases where the capture itself can be improved (AG-1980 vs. JVC HR-S9800U, TBC-3000 adjustments, tape problems, etc...). The ones that are worth the time and effort of post-process will get it as time permits while the others are at least available to enjoy in their raw form.

One other thing that I've come to realize (and I believe Lord Smurf has extolled) is that the imperfections in some cases might be worth leaving as-is. This one hour that I mysteriously have from 1986 (the rest are later 80's and 90's) is so much fun to step through frame by frame. I worked at a CBS affiliate for a short time around this time and seeing "the crud" in the capture brings back memories. Particularly seeing the loss of sync between commercials - I would guess because, as we used, the stations TBC had to switch between two commercial cart machines. Technology was good then but not perfect. And video levels at the start of a commercial fluctuate, I'd guess, because that 2" Ampex cart had to go from zero to full speed pretty quick. Also the same video level fluctuation in the Firestone spot when the price is superimposed likely showing that two video sources again had to be combined and the technology to do that, well... it was 1986. The one that surprised me was the show (a Miami Vice episode) was obviously on film as you've told me many times. I recorded it on the air date and I assume it was satellite fed (that's what we did for network shows) yet the film copy, especially the show open is horribly laden with all kinds of black spots you can see quite clearly frame by frame. Best of all, the first frame when the camera is pointing at the sun, the sun itself is magenta-purple-orange another sign, I'd assume that it's film. Things I never would have seen playing back this VHS tape on an old TV. That frame is attached for your enjoyment.

Sorry, got a bit off topic... To wrap up, again I so very much appreciate everyone's help - sanlyn, lordsmurf, themaster, msgohan - and if I don't post anything for a bit it is because I'm deep into learning mode.


Attached Images
File Type: jpg miami-vice-frame2.JPG (76.1 KB, 7 downloads)
Reply With Quote
The following users thank billct97 for this useful post: sanlyn (04-06-2020)
  #32  
04-05-2020, 11:57 PM
Winsordawson Winsordawson is offline
Free Member
 
Join Date: Sep 2010
Location: Behind you
Posts: 473
Thanked 29 Times in 25 Posts
I second billct97's praise for Sanlyn's tutorials here and elsewhere on this site--they are possibly the best we have for newcomers on all of the interwebs. Sanlyn wrote this a few posts ago:

Quote:
Also, the last time I heard about YouTube submissions, video intended for 4:3 display aspect ratio has to be submitted in a certain frame size, or YouTube will reformat it for you using the quickest and dirtiest means possible. They prefer submissions in 720p format. 720x480 can't be used on YouTube. The 4:3 video has to be resized to 960x720, then 160 black border pixels are added to each side to make a 4:3 video enclosed in a 1280x720 frame. Anamorphic frames (720x480) can't be used.
However, Youtube notes that it will also accept 480p at 854x480 (https://support.google.com/youtube/answer/6375112). Is it better to avoid the resizing and add a border to make the width 854 (66 and 68 on the sides, due to the mod-2)?
Reply With Quote
  #33  
04-06-2020, 12:37 AM
msgohan msgohan is offline
Free Member
 
Join Date: Feb 2011
Location: Vancouver, Canada
Posts: 1,323
Thanked 334 Times in 276 Posts
No. YouTube uploads look the best the more you upscale them, because they use higher quality compression/processing for the larger frame sizes. The other day I resized a video I was posting to YT from 352x240 to 2880x2160 ("4K" but in 4:3 aspect ratio) because their processing is so awful at 240p.

Sanlyn is incorrect when it comes to YouTube's current aspect ratio requirements. Nowadays, you can upload 4:3, 16:9, 2.35:1, etc. and it handles them appropriately without baking in black bars.
Reply With Quote
The following users thank msgohan for this useful post: sanlyn (04-06-2020)
  #34  
04-06-2020, 01:14 AM
Winsordawson Winsordawson is offline
Free Member
 
Join Date: Sep 2010
Location: Behind you
Posts: 473
Thanked 29 Times in 25 Posts
Quote:
Originally Posted by msgohan View Post
No. YouTube uploads look the best the more you upscale them, because they use higher quality compression/processing for the larger frame sizes. The other day I resized a video I was posting to YT from 352x240 to 2880x2160 ("4K" but in 4:3 aspect ratio) because their processing is so awful at 240p.

Sanlyn is incorrect when it comes to YouTube's current aspect ratio requirements. Nowadays, you can upload 4:3, 16:9, 2.35:1, etc. and it handles them appropriately without baking in black bars.
Thanks. Do you think it is a safer compromise to upscale to 1080p because of the effect of upscaling on video that is not lossless, assuming noise removal has been attempted? 4K is twice the vertical resolution, but is YouTube's 4K codec really twice as good as it's 2K version?
Reply With Quote
  #35  
04-06-2020, 03:56 AM
sanlyn sanlyn is offline
Premium Member
 
Join Date: Aug 2009
Location: N. Carolina and NY, USA
Posts: 3,648
Thanked 1,307 Times in 982 Posts
Quote:
Originally Posted by msgohan View Post
No. YouTube uploads look the best the more you upscale them, because they use higher quality compression/processing for the larger frame sizes. The other day I resized a video I was posting to YT from 352x240 to 2880x2160 ("4K" but in 4:3 aspect ratio) because their processing is so awful at 240p.

Sanlyn is incorrect when it comes to YouTube's current aspect ratio requirements. Nowadays, you can upload 4:3, 16:9, 2.35:1, etc. and it handles them appropriately without baking in black bars.
My familiarity with YouTube policy is obviously dated but I never post there. Thanks for the corrections.
Reply With Quote
  #36  
04-06-2020, 05:18 AM
msgohan msgohan is offline
Free Member
 
Join Date: Feb 2011
Location: Vancouver, Canada
Posts: 1,323
Thanked 334 Times in 276 Posts
Quote:
Originally Posted by Winsordawson View Post
Do you think it is a safer compromise to upscale to 1080p because of the effect of upscaling on video that is not lossless, assuming noise removal has been attempted? 4K is twice the vertical resolution, but is YouTube's 4K codec really twice as good as it's 2K version?
I haven't compared directly between the two myself, but I did do a test a few years ago of 720p video game footage where I uploaded original, upscaled to 1080p, and upscaled to 1440p. (Because 2160p wasn't an option back then.) Note that this game applies an artificial "grain" filter over the rendered image, which makes it much harder to compress than a clean-looking video game.

https://forum.videohelp.com/threads/...be#post2353139

Upscaling to 2160p is closer to an integer scale of 480 and 576 than 1080p is, so that is an added benefit.
Reply With Quote
  #37  
04-06-2020, 11:34 AM
Winsordawson Winsordawson is offline
Free Member
 
Join Date: Sep 2010
Location: Behind you
Posts: 473
Thanked 29 Times in 25 Posts
Quote:
Originally Posted by msgohan View Post
I haven't compared directly between the two myself, but I did do a test a few years ago of 720p video game footage where I uploaded original, upscaled to 1080p, and upscaled to 1440p. (Because 2160p wasn't an option back then.) Note that this game applies an artificial "grain" filter over the rendered image, which makes it much harder to compress than a clean-looking video game.

https://forum.videohelp.com/threads/...be#post2353139

Upscaling to 2160p is closer to an integer scale of 480 and 576 than 1080p is, so that is an added benefit.
In that videohelp thread there are conflicting opinions as to whether upscaling does help. If it does help, the difference may not be worth the time.

In this more recent thread, the OP's analysis concludes that it is a waste of time, although down the page I did see more quality in a 4K upscale from the 1080p version. The thread starter counters that the difference in quality is just due to sharpening filters in the browser.

https://www.magiclantern.fm/forum/in...?topic=22681.0
Reply With Quote
  #38  
04-07-2020, 11:05 AM
billct97 billct97 is offline
Free Member
 
Join Date: Jan 2018
Posts: 40
Thanked 3 Times in 3 Posts
I'm feeling exceptionally lazy today... 2 years ago my Dad wanted a VHS tape put on DVD. My usual workflow captured at 720x480 with lagarith lossless compression has given me a 2 hour 50GB AVI file. The original video is from 1993, is in 4:3 format and except for some B&W footage at the beginning that is 3:2 telecine (easily seen when stepping frame by frame) the rest is not telecine, just interlaced. I don't intend to do anything besides trimming the warped tape garbage at the start and cropping out the edge noise and adding borders back to 720x480.

My question, can someone share with me a high level plan for taking this video and getting it onto DVD? I have a DVD burner, plenty of writable DVD's and probably software that will do the burning. I don't care about menus or anything more than a DVD that can be put in a stand-alone DVD player (not BluRay) and plays with decent quality. If breaking the video into 2 or more DVD's will yield better quality I'm all for that.

Pointing me to a past forum discussion and RTFM is also acceptable
Reply With Quote
Reply




Similar Threads
Thread Thread Starter Forum Replies Last Post
AviSynth Scripts/Guide dnppro Restore, Filter, Improve Quality 6 10-24-2020 07:19 PM
Are AviSynth scripts, VirtualDub filters run as separate processes? stevevid Restore, Filter, Improve Quality 7 09-25-2018 04:35 PM
Best scripts for de-telecine ragu0012 Restore, Filter, Improve Quality 8 03-17-2018 08:36 PM
VCD to DVD conversions-question? MagnificentMarcus Encode, Convert for discs 11 09-05-2006 02:38 PM

Thread Tools



 
All times are GMT -5. The time now is 06:04 AM