Flickering colours on VHS tape? - Page 4

koberulz · #61 10-08-2016, 03:40 AM

Don't have time to look at any of that right now, but I just checked and my estimated total encode time for 2406 frames of video using sanlyn's first script is now up to 14.5...days.

Yeah.

sanlyn · #62 10-08-2016, 10:00 AM

Quote:

Originally Posted by koberulz

Don't have time to look at any of that right now, but I just checked and my estimated total encode time for 2406 frames of video using sanlyn's first script is now up to 14.5...days.

Yeah.

Why put up with that? Obviously something's not right, so just shut that script down. Tell us about your computer system generally: processor, OS, hard drives. Then open your copy of script #1 in Notepad, and copy and paste that script into a post.

Whenever you have a problem with Avisynth or VirtualDub running filters, or if you have error messages, you should give more detail (with errors, if any, the exact message is needed). We;ll have to study what went wrong. Sorry you're having a problem, but my script #1 should not take more than a minute or two, even on my slow old AMD 2.2GHz.

If my script #1 is a problem, you'll definitely have problems with lordsmurf's script #2.

koberulz · #63 10-08-2016, 10:20 AM

Well, I figured at 18 hours I might as well let it go, see how it ended up. At 14 days...yeah, no point. It got out to 21 just now (earlier I literally just checked it, posted, then headed out), so it's obviously completely locked up at some point. Hit abort, got this in VDub:
---------------------------
VirtualDub Internal Error
---------------------------
Something appears to be stuck while trying to stop (thread deadlock). Abort operation and exit program?
---------------------------
OK Cancel
---------------------------

I dropped MCTemporalDenoise from 'very high' to 'medium' and I'm now getting an encode at 0.41fps. Still really slow, but significantly faster.

Intel(R) Core(TM) i7 CPU 920 @ 2.67GHz 2.66GHz
Windows Vista Home Premium SP2
Not sure what info you want on my hard drives? I'm running everything off a USB drive, which AFAIK will slow things down compared to an internal. Not sure how much, though, and it certainly shouldn't be a matter of weeks.

Code:

AVISource("..\Captures\Capture Panasonic.avi")
ConverttoYV12(matrix="rec601",interlaced=true)
ColorYUV(cont_v=-40)
AssumeTFF().QTGMC(preset="medium",Border=true,Ezdenoise=10.0,denoiser="dfttest",shownoise=false)
Dehalo_alpha()
StabMod()
FixChromaBleeding()
MergeChroma(MCTemporalDenoise(settings="medium"))
ChromaShift(C=10,L=-4)
MergeChroma(awarpsharp2(depth=30))
SmoothUV()
LimitedSharpenFaster(edgemode=2)
AddGrainC(2.0,2.5)
SeparateFields().SelectEvery(4,0,3).Weave()
Trim(1313,2147) ++ Trim(81511,82004) ++ Trim(152304,153380)

EDIT: MCTemporalDenoise was the only one of those filters I didn't already have, so I had to find and download it and a couple of the required plugins via the wiki. Could be something to do with it, I may have screwed up somewhere there?

sanlyn · #64 10-08-2016, 12:18 PM

Thank you for the info and script copy. I see a couple of bottlenecks but I'm on the road driving for a few hours. Your last step is a logic problem. More later when i return home. Meanwhile you should be working with smaller samples you want to keep, not pro cessing the entire capture at once. Move your Trim statement to the line after you open the file with Avisource. That alone will shorten the total time.

More later...

Quote:

Originally Posted by koberulz

I dropped MCTemporalDenoise from 'very high' to 'medium' and I'm now getting an encode at 0.41fps. Still really slow, but significantly faster.

Faster, yes, but not as effective. That's bad flicker you have in these vids. I re-wrote a simpler script using "High".

An Intel i7 920 CPU should be OK for mthis kind of processing. Indeed, using an exernal 5400rpm USB drive will slow you down, but not down to below 0.1 fps processing. You can try it without QTGMC and just use dfttest alone. Not as effective, but there's no sense running 1/3 filtering strength for everything.

Try it this way:

Code:

AVISource("..\Captures\Capture Panasonic.avi")
Trim(1313,2147) ++ Trim(81511,82004) ++ Trim(152304,153380)
ConverttoYV12(matrix="rec601",interlaced=true)
ColorYUV(cont_v=-40)
AssumeTFF()
SeparateFields()
dfttest()
FixChromaBleeding()
MergeChroma(MCTemporalDenoise(settings="high"))
ChromaShift(C=10,L=-2)
MergeChroma(awarpsharp2(depth=30))
LimitedSharpenFaster(edgemode=2)
AddGrainC(2.0,2.5)
Weave()

Quote:

Originally Posted by koberulz

EDIT: MCTemporalDenoise was the only one of those filters I didn't already have, so I had to find and download it and a couple of the required plugins via the wiki. Could be something to do with it, I may have screwed up somewhere there?

I wouldn't think so. MCTD is a heavy hitter but it's just a filter, like all the others. You'll need it in chroma mode to attack that particular kind of chroma flashing.

I'll take another look at that Trim() statement later tonight, but it's definitely in the wrong line sequence in your copy of the script.

koberulz · #65 10-09-2016, 12:26 AM

I really just dropped it as a troubleshooting step. I'm assuming the lower the setting, the less processing power it needs? Or is that not the case?

I do remember one of the required plugins had a couple of different versions and I grabbed the one I thought was right. Not sure what effect that might've had though.

With Trim() moved up and MCTemporalDenoise on 'very low', I'm getting 1.2fps.

Ran this:

Code:

AVISource("..\Captures\Capture Panasonic.avi")
ConvertToYV12(interlaced=true)
MCTemporalDenoise(settings="very high")

2.90fps.

Tried the non-QTGMC script you posted above. 2.06fps. Down to 1.88 if I set MCTemporalDenoise back to 'very high'.

The MCTemporalDenoise step was the one where AvsPmod stopped refreshing the video window at any sort of decent speed. With the two non-QTGMC scripts I'm still getting an hourglass cursor and '(Not Responding)' in the title bar for a few seconds when opening the file in VDub.

sanlyn · #66 10-09-2016, 03:52 AM

Quote:

Originally Posted by koberulz

Ran this:

Code:

AVISource("..\Captures\Capture Panasonic.avi")
ConvertToYV12(interlaced=true) 
MCTemporalDenoise(settings="very high")

2.90fps.

2.89 is about half the rate it runs on my i5 Intel (4.75 average), although in some really noisy spots on your samples my i5 went down to 2.0fps for a few seconds on the full script. If I eliminated MCTD the same script would run at about 14 fps with QTGMC. Some other system factor is slowing things down, because a 920 CPU isn't reputed to be a slouch. So it would seem that you'd do what I had to do and still occasionally do, and that's to cut complex scripts into 2 steps. People do this all he time, so it's common with heavy-duty work.

But working a dull-scale filtering job on an entire capture at once, I don't know of anyone who completely filters that way. The main problem is that most VHS captures require different filters or settings for different segments. The only time I scrub a capture 100% is with a common step, such as inverse telecine on an old movie capture -- it would take a couple of hours for a 30GB capture, but I usually run something like that overnight.

That MCTD tet script has two errors, by the way, that screw things up:

Code:

ConvertToYV12(interlaced=true)

This isn't exactly a mistake because you've just opened an avi, but in the main script the video was already deinterlaced. But it's often the case that people sometimes forget the current state of the video.

Code:

MCTemporalDenoise(settings="very high")

The previous statement assumed your video is interlaced, but MCTD is designed for progressive video and definitely proves ineffective or makes a sloppy mess on interlace. But it's one of few filters that has an interlace mode, so for interlaced video you can say:

Code:

MCTemporalDenoise(settings="very high", interlaced=true)

, which will run slightly faster. And don't forget, MNTD is petty strong and will damage a very soft video like Studio.avi, so it was used for chroma only. So for this context, you could use MCTD this way:

Code:

MergeChroma(MCTemporalDenoise(settings="very high", interlaced=true))

So you have a somewhat slow system and a rather dirty and problematic video to clean up, so you have to adapt your methods. Fortunately not every video needs an industrial strength cleanup, but it happens that this is one is one of those restoration bad dreams you often hear about. Everyone has been through this. But it does look as if some segments of the tape are cleaner than others.

I had a similar experience with a nephew's championship football game during his college senior year. It took a month of cleanup, short segments at a stretch, and that game with half time ceremonies lasted a total of 3 hours -- not counting the frequent TV network commercials, which alone totalled almost another hour, for a total of 4 hours of tape recorded at slow. detail-killing 6-hour speed. Not only that, but those college boys really know how to wreck a tape, forcing me to learn some motion interpolation and motion cleanup techniques. A sample of the input tape here: http://www.digitalfaq.com/forum/atta...sample2_badmpg. That tape wouldn't track in my expensive tbc player, I had to temporarily borrow a Toshiba DVD recorder for its line tbc and use it as a pass-thru device, then recapture more than a third of the video, and process all of it on a 2-core 2GHz AMD PC with 1.5GB of RAM. Working in small segments all day, it took 5 weeks. That's the last time I took on a VHS restoration without first looking at the tape. And I do this without pay for relatives and friends. You'd think they would repay by at least naming their first male child after me, LOL!

Anyway, a slow PC and a bad tape are what you're dealing with. Fortunately later portions of the tape don't seem to look as poorly as earlier parts.

lordsmurf · #67 10-09-2016, 04:05 AM

Quote:

Originally Posted by koberulz

Ran this:

Code:

MCTemporalDenoise(settings="very high")

2.90fps.

broken.jpg

Post what I need, or I can't help.

The Avisynth wiki is screwed up, tired of hunting for it.

koberulz · #68 10-09-2016, 04:30 AM

So I had a look at the two lordsmurf scripts. First one is real easy. Second one...I have no idea what I'm looking at.

Also downloaded the sanlyn mp4 featuring the intro and game clips. Other than being redder than previous shots, I'm not seeing what's wrong with the last shot?

Quote:

Originally Posted by sanlyn

Some other system factor is slowing things down, because a 920 CPU isn't reputed to be a slouch.

Well you'd hope not; this was specifically built as a video editing machine.

Any idea as to what it is? If I cut the script down, where's the best place to cut it?

Quote:

This isn't exactly a mistake because you've just opened an avi, but in the main script the video was already deinterlaced. But it's often the case that people sometimes forget the current state of the video.

Again, really just troubleshooting. I tried to run it without the colourspace conversion at all initially, but obviously that doesn't work. Just didn't want any other filter at all possibly contributing to any drop in speed.

Quote:

Fortunately later portions of the tape don't seem to look as poorly as earlier parts.

Not sure if it's that, or the game being in better shape than the studio segments. It goes absolutely crazy almost every time it cuts between the two.

sanlyn · #69 10-09-2016, 07:31 AM

Quote:

Originally Posted by lordsmurf

Post what I need, or I can't help.

The Avisynth wiki is screwed up, tired of hunting for it.

Yeah, that GradFun2db page is certainly messed up. Gradfun2db and short ReadMe attached.

Running MCTD in the manner shown is the best (and likely only) way I've sen to calm bad chroma flicker and I've used it for years. The two active ingredients seem to be TTempSmooth and FFT3DFilter. If you run it as SeparateFields it doesn't seem to work, so bad interlacing on that flicker must have something to do with it.

MCTD calls GradFun2DBmod, but if you look at GradDun2DBmod it wants the older GrandFun2db also. PITA.

sanlyn · #70 10-09-2016, 07:38 AM

Quote:

Originally Posted by koberulz

Also downloaded the sanlyn mp4 featuring the intro and game clips. Other than being redder than previous shots, I'm not seeing what's wrong with the last shot?

It's not just weird red, it's also a lot of noise that isn't in the previous shots. Run it without filters and yo'll see a red haze in the left side and blue blotches in the background shadow areas. I had to add CCD in VirtualDub for that shot, which I did separately. Welcome to VHS restoration.

Quote:

Originally Posted by koberulz

If I cut the script down, where's the best place to cut it?

I'll rearrange some things, Will post soon.

sanlyn · #71 10-09-2016, 11:17 AM

Quote:

Originally Posted by koberulz

If I cut the script down, where's the best place to cut it?

The script cut in two, below.

Of the three extra samples (Intro, Game1, Game2) exactly the same filters can't be used, or at lest shouldn't be. Each had its own problems. Intro has over staurated yellow as well as red, but other colors look OK. Each shot in the mp4 I posted was processed separately. That method is definitely not unusual for VHS, which varies horribly from scene to scene and even during scenes.

Below is a new script I tested for Intro1. Scripts for other shots are very similar, they juust have different saturation points. You can split scripts for the other vids in the same way:

Script1

Code:

AviSource("Intro.avi")
Levels(16,0.95,255,16,235, coring=false, dither=true)
TweakColor(sat=0.85, startHue=80, endHue=140)    #red
TweakColor(sat=0.80, startHue=140, endHue=190)   #yellow
AssumeTFF()
ConvertToYV12(interlaced=true)
QTGMC(preset="medium",border=true)
FixChromaBleeding()
ChromaShift(C=10,L=-4)
TemporalSoften(4,4,8,15,2)
FluxSmoothT()
return last

I saved this as Intro_Script1, Lagarith YV12, and it is deinterlaced.
"Levels" and "TemporalSoften" are Avisynth built-ins. "TweakColor" and "FluxSmooth" are attached below, with docs.
My OS & PC: XP, Intel i5 3570 (3.4GHz)
Run time for this script1 and clip: 18 seconds (17.5 fps average)

Script2

Code:

AviSource("Intro_Script1.avi")
MergeChroma(MCTemporalDenoise(settings="very High"))
MergeChroma(awarpsharp2(depth=30))
LimitedSharpenFaster()
AddGrainC(2.0,2.5)
Crop(18,6,-22,-10).AddBorders(20,8,20,8)
SeparateFields().SelectEvery(4,0,3).Weave()

Run time for this script2 and clip: 1m 30sec (1.72 fps average)

There is still a little shimmer in the red letters after this script and no blinking, but the MP4 used another step that reduced even the mild shimmer. Or you might try adding VDub's TemporalSmoother at about a value of 3 or 4 if you wish, instead of running another script. TemporalSmoother starts to cause ghosting at values higher than 4.

Script3 (optional)
runs in less than 2 seconds:

Code:

AviSource(vidpath+"Intro_Script2.avi")
AssumeTFF()
SeparateFields()
a=last
e=a.SelectEven().RemoveSpotsMC2x()
o=a.SelectOdd().RemoveSpotsMC2x()
Interleave(e,o)
Weave()
return last

The "RemoveSpotsMC" avs plugin is attached. It contains three functions, any one of which can be called in a script: RemoveSpots, RemoveSpotsMC, and RemoveSpotsMC2x. The 2X version is often used to clean up horizontal dropouts. These are all just a few of many versions of RemoveSpots and RemoveDirt, so they are in .avs form instead of .avsi because the same functions are called in many other RemoveSpots scripts.

The original 3-part mp4 used added Virtualdub color filters. The attached Script3 mp4 didn't use VDub filters.

koberulz · #72 10-10-2016, 12:49 AM

I feel like I'm just being given fish again.

Still haven't gone through your second script from last time, either, given I couldn't get the first to work.

I already have FluxSmooth, but it's different - 4KB smaller. Replace it?

TweakColor's documentation is just confusing; not really sure what it does.

Quote:

Originally Posted by sanlyn

The script cut in two, below.

What's the logic behind the cut points? Are there good places to cut, bad places to cut, places you can't cut?

AVIs should be output in whatever colourspace the script is working in at that current time, yes?

Quote:

Script1

TweakColor is basically doing what ColorYUV did in the Studio script, yes? What's the reason for using one or the other?

What effect does a levels number greater than 255 have? I have no idea what the second paragraph on the Wiki means (about gamma-correcting and RGB and such). What difference does it make running this now rather than running it in VDub, or running it after the noise is handled?

I still don't really get the difference between adjusting input and output. They seem to have the same effect when dragged in opposite directions.

TemporalSoften and FluxSmooth replace EzDenoise in QTGMC?

Quote:

Run time for this script1 and clip: 18 seconds (17.5 fps average)

It took me 1:34, averaging 3.25fps. I moved everything onto an internal drive, just to eliminate that variable, and it took 1:30.

WTF is going on?

Quote:

These are all just a few of many versions of RemoveSpots and RemoveDirt, so they are in .avs form instead of .avsi because the same functions are called in many other RemoveSpots scripts.

I don't really understand this? RemoveSpots also doesn't even have a Wiki page.

Regarding the Hanover bars and red haze on the left of that shot, they're also present in other shots from that camera. So it at least appears consistent in that respect. When I had my first run at this I did three versions, one for the half-court camera, one for the left-corner camera, and one for the right-corner camera.

msgohan · #73 10-10-2016, 04:21 AM

Quote:

It took me 1:34, averaging 3.25fps. I moved everything onto an internal drive, just to eliminate that variable, and it took 1:30.

WTF is going on?

AVSMeter is a tool purpose-built for checking script speeds. I've attached it along with two scripts in a ZIP file.

Extract this ZIP file to some folder on your internal drive. To make sure we're truly comparing apples to apples, download your own Intro.avi and add it to the same folder.

Make sure your CPU usage from other programs is around 0% before running the test. Drag and drop Intro_AVISource_x10.avs onto the BAT file and let it complete. The window will close once it's done. Then do the same for Intro_Script1.avs. ZIP up the two log files and attach them so we can begin to attempt troubleshooting.

Quote:

Originally Posted by koberulz

AVIs should be output in whatever colourspace the script is working in at that current time, yes?

Yes.

Quote:

Regarding the Hanover bars and red haze on the left of that shot, they're also present in other shots from that camera. So it at least appears consistent in that respect. When I had my first run at this I did three versions, one for the half-court camera, one for the left-corner camera, and one for the right-corner camera.

I don't think that's truly the Hanover artifact, as Hanover bars appear in chroma. Here, even the "black" background shows alternating bands. It looks like some issue with the camera's electronics or whatever equipment that angle passed through prior to meeting at the video mixer.

Quote:

Originally Posted by koberulz

Is there a way to turn the luma off and just look at chroma to see what the denoiser and later sharpener are doing? Even copying the frames to the clipboard, pasting them into a Photoshop document and toggling the top layer I could barely see a difference.

View chroma channels in isolation (as a greyscale image): StackHorizontal(UtoY(),VtoY())

Quote:

Originally Posted by koberulz

I'm not using VirtualDub's capture histogram, I'm using AviSynth's Histogram() function. Which is still luma only I know, but I wanted to clarify. Is avoiding chroma clipping a concern at the capture stage? I've only ever worried about luma clipping previously.

Yes, but it should only be an issue if the video is extremely saturated.

Quote:

But if it has clipped values, why is nothing even extending into the 'unsafe' areas on the YUV histograms?

As a color model or color space, YUV is larger than RGB. Values that can fit into YUV cannot be represented in RGB because they would map to colors exceeding 255, or even negative colors. The result is that these pixels "clip" when digital conversions are used. As I recall, the precise effects will vary depending on the actual algorithm implemented to convert from YUV to RGB, so Avisynth may show a different result from VDub which may be different than your graphics card which may differ from your hardware player or HDTV. The differences should be minor, though.

Analog conversions like a CRT or VCR might use internally will have other undesired and potentially unpredictable effects that vary with the exact hardware used.

http://forum.doom9.org/showthread.ph...26#post1402326

http://forum.videohelp.com/threads/3...ighlightbadrgb
http://forum.videohelp.com/threads/3...gb#post2450356
http://forum.videohelp.com/threads/3...gb#post2447194

http://forum.videohelp.com/threads/3...87#post2302087
http://forum.videohelp.com/threads/3...51#post2353351

lordsmurf · #74 10-10-2016, 05:21 AM

Quote:

Originally Posted by sanlyn

PITA.

Thanks for attaching the filters.

I honestly hate Avisynth discussions on Doom and VH, because of the versioning, mods, and forks of plugins. Hunting Google is stupid when we can just attach things to threads. Those files are tiny, and I have no issue attaching tiny files to posts over and over again. I'd rather have too many than not enough.

Many things are now only fond at this forum, because we archived from day 1, some 10+ years ago.

Quote:

Originally Posted by koberulz

I feel like I'm just being given fish again.

If it makes you feel better, I'm not sure what's going anymore either.

I plan to look next weekend. (sanlyn and msgohan are giving you some great attention, so I can to spend some time on site dev and other posts. My work around here is never done!)

koberulz · #75 10-10-2016, 06:02 AM

msgohan: Cheers, I'll look into that later. I'm encoding something else right now. What difference are you expecting to eliminate by having me download the Intro clip, rather than use the copy I already have?

That U/VtoY thing is really helpful. Throwing exactly the same sharpener at it that sanlyn's used goes from 'it looks a little different, I guess' to making a really obvious difference.

Quote:

If it makes you feel better, I'm not sure what's going anymore either.

I think you may have misunderstood: I meant that as a reference to the 'give a man a fish and he'll eat for a day' thing. I have thousands of these to do (thankfully most in much better nick than this), so the more I can learn in a way that will help me apply it to future tapes the better. I'd rather not keep annoying you lot every five minutes.

Although I will say your earlier AviSynth script makes absolutely no sense to me. At least with all the others I can read through them, spend some time on the Wiki and have an idea of what's going on, even if I have to work more to understand the logic behind it all. I have no idea what yours is even doing.

lordsmurf · #76 10-10-2016, 06:32 AM

It's advanced NR with both forward and backward lookups. It's not for Avisynth newbies, that's for sure!

In time, I'll try to explain it all. That comes later. Again, it's a beta script, not entirely stable either. But it works almost like magic for removing bad tracking and dropout noise from a VHS/tape capture.

sanlyn · #77 10-10-2016, 09:54 AM

Quote:

Originally Posted by koberulz

Still haven't gone through your second script from last time, either, given I couldn't get the first to work.

The first scripts I posted were customized for the Studio.avi clip by itself. It wasn't designed to process your entire 2-hour capture. None of the scripts were designed for that. There's no single script that works effectovely for your entire capture. The second group of scripts was for the "Intro.avi" shot, which has its own problems. Each of the 4 samples you posted had common problems (flashing) as well as problems of their own. This is par for VHS sources.

Quote:

Originally Posted by koberulz

I already have FluxSmooth, but it's different - 4KB smaller. Replace it?

Yes. The current version is 1.1b, which I attached. I think you had 1.1a. The two are very similar, as far as I can tell.

Quote:

Originally Posted by koberulz

TweakColor's documentation is just confusing; not really sure what it does.

Most of the documentation on what it does it OK, but then the author screws around with your head with those confusing test examples. TweakColor can target specific saturation levels in specific color ranges, which is similar to what Hue/Sat/Intensity does in VirtualDub. Unfortunately, if you don't know how a circle is divided into 360 degrees, most explanations won't help much.

Quote:

Originally Posted by koberulz

What's the logic behind the cut points? Are there good places to cut, bad places to cut, places you can't

That qustion made it suddenly occur to me that you might not understand how AVisynth script or other such procedures operate. Avisynth executes line 1, then those results are fed to line 2 for execution, then line 3, and so forth, until Avisynth gets to the end of the script.

Let's say that a procedure consists of 10 lines. The procedure on line 3 takes a really long time to execute. The procedure on Line 6 also takes a really long time. All of the other lines will execute in turn, but they all take only a small average time and fewer CPU cycles to do their their task.

The statements that take a long time to execute are using large amounts of memory and CPU activity, while the faster lines are less CPU- or memory-intensive. Slow execution that is CPU and/or memory intensive involves a lot of memory swapping and congestion. You can avoid some of that congestion buildup by clearing memory and starting afresh. One way to clear things out is to stop the procedures by ending the script. Then start a new script that takes up where the other script stopped. So, find a spot that uses the first of the slow statements and place that in one script, then start the new script with the next slowpoke statement.

Quote:

Originally Posted by koberulz

TweakColor is basically doing what ColorYUV did in the Studio script, yes?

No. The earlier ColorYUV line reduced contrast/saturation in the V channel, which affects red and part of the green color range but doesn't affect deeper yellows. The first scripts were for "Studio.avi", which didn't have a bad yellow problem. The later samples have problems with red and yellow and part of the magenta range, which TweakColor can handle more precisely.

Quote:

Originally Posted by koberulz

What effect does a levels number greater than 255 have?

I think msgohan's comments and earlier discussions covered that. In RGB, the results of out of range YUV values would be clipping.

Quote:

Originally Posted by koberulz

I have no idea what the second paragraph on the Wiki means (about gamma-correcting and RGB and such). What difference does it make running this now rather than running it in VDub, or running it after the noise is handled?

Because once clipping occurs in RGB, most of the clipped data can't be recovered.

Quote:

Originally Posted by koberulz

I still don't really get the difference between adjusting input and output. They seem to have the same effect when dragged in opposite directions.

Not sure what you mean. Are you referring to AvsPmod?

Quote:

Originally Posted by koberulz

TemporalSoften and FluxSmooth replace EzDenoise in QTGMC?

EZDenoise isn't a filter, it's an instruction that tells QTGMC to apply extra noise cleanup. It specifies the filter asdfttest, which isn't very fast. TemporalSoften and FluxSmooth replace the extra dfttest filtering. It's not quite as effective as dfttest but it works OK here and is faster.

Quote:

Originally Posted by koberulz

Quote:

Originally Posted by sanlyn

Run time for this script1 and clip: 18 seconds (17.5 fps average)

It took me 1:34, averaging 3.25fps.

I can understand taking twice as long, as would my old AMD X2, or even 3 times longer with single-core chips. But more than 500% longer with an i7 doesn't make sense, unless you're having RAM problems.

Quote:

Originally Posted by koberulz

RemoveSpots also doesn't even have a Wiki page.

It probably won't have its own page until developers at Doom9 stop conniving umpteen different versions and finally declare at least one, two or three known stable performers. One example: RemoveSpotsMC3X is in a big avs script titled RemoveSpotsMC5 that has code for a 2x version that doesn't work and tries to use an obsolete VirtualDub filter that no one uses any more. And RemoveSpots is really a hybrid version of RemoveDirt, which itself comes in more flavors than ice cream. Most of this mix appears in Doom9 posts, where a lot of stuff looks cool but works only for the developer and no one else.

Quote:

Originally Posted by koberulz

Regarding the Hanover bars and red haze on the left of that shot, they're also present in other shots from that camera. So it at least appears consistent in that respect. When I had my first run at this I did three versions, one for the half-court camera, one for the left-corner camera, and one for the right-corner camera.

Possibly those waves aren't genuine Hanover bars, as msgohan noted. They looked that way at first until I focused on them last night and tried some Hanover bar filters, to which they didn't respond one iota. Rather than destroy what remains in the image, I'd just live with it.

If you know thgere were two cameras (both of which were set for visually obvious different black levels) and a third camera with operating problems, it would be easier to use sets of filters designed to address three different problems, after pulling off sections of video that you want in your final product. You can't use exactly the same filter settings for all three variances, they look too different for a single filter setup to work well. What worked for Game1 wouldn't work well at all for the last shot in Game2. I processed the two shots in Game1 separately: the first shot is too bright, the second is too dark, yet the second shot is obviously a continuation of the first. For both shots I used the same filters but with different levels settings, then joined them later.

I've had these variances even in retail tapes and TV broadcast tapes I've captured -- not to mention wildly different exposures from my sister's camera, many of which are unusable except as archives.

Meanwhile you're actually doing very well for someone who never tried most of this earlier. I agree, it's a hassle the first time around. Now if we can just help figure out what's going on with those slow run times......

koberulz · #78 10-10-2016, 11:18 AM

Quote:

Originally Posted by sanlyn

The first scripts I posted were customized for the Studio.avi clip by itself. It wasn't designed to process your entire 2-hour capture. None of the scripts were designed for that. There's no single script that works effectovely for your entire capture. The second group of scripts was for the "Intro.avi" shot, which has its own problems. Each of the 4 samples you posted had common problems (flashing) as well as problems of their own. This is par for VHS sources.

Yes? I'm aware of all this, I'm not really sure if you've misconstrued something I've said?

Quote:

Yes. The current version is 1.1b, which I attached. I think you had 1.1a. The two are very similar, as far as I can tell.

Will do then.

I also have a RemoveSpotsMC.avs, which I opened in AvsPmod, along with your RemoveSpotsMC.avsi, and they seem identical.

Quote:

TweakColor can target specific saturation levels in specific color ranges, which is similar to what Hue/Sat/Intensity does in VirtualDub.

It's the specific saturation thing that gets me most, I think. I get the circle, you give it a point in that circle and it works within that circle, I'm across that (actually figuring out which numbers go where is another story, but not necessary for understanding the theory). Is the saturation command 'set the saturation of everything within this colour range to this saturation'? Looking at it fresh it seems like it must be, but referring to it as 'targeting' that saturation makes it sound like it only affects pixels with that saturation value in the first place.

The thing it's really lacking is the clear syntax/parameters thing. For example, on the FluxSmooth page on the Wiki:

Code:

FluxSmoothT (clip, int "temporal_threshold") 
FluxSmoothST (clip, int "temporal_threshold", int "spatial_threshold")

So for TweakColor, is this right?:

Code:

TweakColor (clip, "hue", "sat", "bright", "cont", "startHue", "endHue", "smooth")

I'm just not sure if I'm missing any parameters there, it muddles off into the paragraph about colour placement.

And while doing that, it occurred to me, I have no idea what 'int' and 'float' are, they both seem to be numbers. Bool I know.

EDIT: Do you remember the other basketball game I was working on, with the green areas that needed to be bluer? Would TweakColor have handled that better?

Quote:

So, find a spot that uses the first of the slow statements and place that in one script, then start the new script with the next slowpoke statement.

Excellent, thanks.

Quote:

No. The earlier ColorYUV line...TweakColor can handle more precisely.

But in the sense of 'handles oversaturation', yes?

Quote:

I think msgohan's comments and earlier discussions covered that. In RGB, the results of out of range YUV values would be clipping.

I'll rephrase: Why would you ever want to use values higher than 255?

Quote:

Not sure what you mean. Are you referring to AvsPmod?

VirtualDub, Photoshop...anywhere with a 'levels' filter and a GUI.

Quote:

It probably won't have its own page until developers at Doom9 stop conniving umpteen different versions and finally declare at least one, two or three known stable performers. One example: RemoveSpotsMC3X is in a big avs script titled RemoveSpotsMC5 that has code for a 2x version that doesn't work and tries to use an obsolete VirtualDub filter that no one uses any more. And RemoveSpots is really a hybrid version of RemoveDirt, which itself comes in more flavors than ice cream. Most of this mix appears in Doom9 posts, where a lot of stuff looks cool but works only for the developer and no one else.

So...is there a way to figure out how to use it?

Quote:

If you know thgere were two cameras (both of which were set for visually obvious different black levels) and a third camera with operating problems, it would be easier to use sets of filters designed to address three different problems, after pulling off sections of video that you want in your final product.

I actually have no idea how many cameras there were - this game took place long before I was even born - but there are definitely at least those three. I think there's probably a fourth, but I can't quite tell if there's a fifth.

And yeah, as I said I originally ran three sets of filters, one for each obvious camera. The existence of the fourth didn't become noticeable until I was putting it all together...and then there's the fun part where one camera angle wipes to another camera angle so they're both on screen at once and I HATE THIS TAPE.

Quote:

Meanwhile you're actually doing very well for someone who never tried most of this earlier. I agree, it's a hassle the first time around. Now if we can just help figure out what's going on with those slow run times......

I have played around in VirtualDub quite a bit over the last few years, and written a few (very basic) AVS scripts, so I'm not completely new to it all, but this is definitely significantly deeper than anything I've done in the past. I also have some experience with Photoshop, HTML, PHP, JavaScript etc, which helps in terms of understanding things like levels and curves adjustments, coding syntax and so on.

I've attached the two AVSMeter logs msgohan requested.

sanlyn · #79 10-10-2016, 02:35 PM

Quote:

Originally Posted by koberulz

Quote:

Originally Posted by sanlyn

The first scripts I posted were customized for the Studio.avi clip by itself. It wasn't designed to process your entire 2-hour capture.

Yes? I'm aware of all this, I'm not really sure if you've misconstrued something I've said?

Perhaps I misconstrued. It looked as if you were applying the same filters to sundry and various pieces that might require different methods each. Sorry if I misunderstood. My captures are very old tapes with all kinds of problems that change minute to minute. I tend to work them one at a time, in sequence. I do get to parts that kook as if an earlier part used the same technique, so I just hop back to an earlier script and re-use the settings. But for the most part I operate in sequence.

Quote:

Originally Posted by koberulz

I also have a RemoveSpotsMC.avs, which I opened in AvsPmod, along with your RemoveSpotsMC.avsi, and they seem identical.

They are. You don't need both. The .avsi loads autmatically. I have several RemoveSpots pluginbs, which I keep as .avs files and import as needed. Some of them just don't work, so I import the .avs files that I know will do the job without crashing. The others are kept so I can fiddle with them and try to figure out what's wrong -- as if I knee what they doing half the time, LOL!!

Quote:

Originally Posted by koberulz

Quote:

TweakColor can target specific saturation levels in specific color ranges, which is similar to what Hue/Sat/Intensity does in VirtualDub.

It's the specific saturation thing that gets me most, I think. I get the circle, you give it a point in that circle and it works within that circle, I'm across that (actually figuring out which numbers go where is another story, but not necessary for understanding the theory). Is the saturation command 'set the saturation of everything within this colour range to this saturation'? Looking at it fresh it seems like it must be, but referring to it as 'targeting' that saturation makes it sound like it only affects pixels with that saturation value in the first place.

TweakCcolor can work on specific colors or color ranges without affecting other colors. Compare that to lowering saturation everywhere (Tweak's "sat" parameter), which with this video wouldn't be desired. TweakColor is a more specific version of Tweak, which has fallen out of favor because of its limitations and lack of precision. Adjusting "Contrast" of a color effectively modifies its saturation. Imagine that if you desaturate everything, red might look normal but every other color would start graying out. Lowering contrast on, say, the V channel also affects some green. Often the overrun doesn't matter, but with the video at hand it can be a problem. Just why red and yellow are so saturated over the other colors here, I don't know (processing? Hardware glitch?).

The color "Red" can vary from pure red to red that contains some other color. In the YUV
color wheel, notice that there aren't distinct borders between shades of red and pure red. Red at the beginning of the color band contains some blue (red + blue = magenta, or purple or pink, depending on how much blue and red). Near the end of its range Red contains some green (which tends toward yellow -- red+green = yellow). So with TweakColor you can use min and max hue numbers to specifically hone in on a shade of red between purply red, pure red, and yellowy red. It seems in this case that the problem is in the full red spectrum and deeply into yellow. I don't know how that happened.

Quote:

Originally Posted by koberulz

The thing it's really lacking is the clear syntax/parameters thing. For example, on the FluxSmooth page on the Wiki:

Code:

FluxSmoothT (clip, int "temporal_threshold")
FluxSmoothST (clip, int "temporal_threshold", int "spatial_threshold")

FluxYUV can be used as either a temporal filter (bases its decision on how noise changes over multiple frames) or a spatialfilter, which bases its decisions on the supposed noise level of a single frame. I used the "T" or temporal mode by specifying FluxSmoothT and took the default values for threshold.

By the way, an "int" is a whole number without decimals, like 2, 2, 500, etc. A float is a floating decimal number, like 1.4, 6.75, 0.85.

Quote:

Originally Posted by koberulz

So for TweakColor, is this right?:

Code:

TweakColor (clip, "hue", "sat", "bright", "cont", "startHue", "endHue", "smooth")

I'm just not sure if I'm missing any parameters there, it muddles off into the paragraph about colour placement.

Yes listing every param is confounding. I've always used the simpler setup of "sat, startHue, EndHue", the first number being understood yo be the default "sat" position. The min and max sat doesn't seem to work the same way and sends me off into la-la land. "Smooth" can be set to extend or creep a little farther into adjacent color bands and avoid a sharp cutoff. I think the numreic value for sat is easier to picture, since it's only a number between 0 and 10, which includes decimal values like 0.8, 0.75, and so on. Min/max just compicates things, and I have no idea why you'd want to set a minumum sat value for a color. What if some objects need something lower than the minimum?

Quote:

Originally Posted by koberulz

Quote:

No. The earlier ColorYUV line...TweakColor can handle more precisely.

But in the sense of 'handles oversaturation', yes?

Both can adjust contrast, i.e, saturation, but ColorYUV is less specific as to total color range.

Quote:

Originally Posted by koberulz

I'll rephrase: Why would you ever want to use values higher than 255?

With normal video operations, you wouldn't. If you're into wide-gamut video, it's useful. But we mere mortals aren't working with wide-gamut color.

Quote:

Originally Posted by koberulz

Quote:

Not sure what you mean. Are you referring to AvsPmod?

VirtualDub, Photoshop...anywhere with a 'levels' filter and a GUI.

So...is there a way to figure out how to use it?

I don't have a problem seeing level changes with those controls in a GUI. But, yes, there is plenty of info about using the three-pointer Levels controls in most graphics programs. http://www.cambridgeincolour.com/tutorials/levels.htm

Visualizing saturation levels and the TweakColor control, or similar:

The image below is a 640x480 frame from Game1.avi. It's no problem seeing the glaring bloom of yellows and reds here, and magenta to a lesser extent. If you lower saturation everywhere, colors in the crowd will tend toward gray. It almost looks as if the players and some people are in front of an orange photolamp. Also, black levels are a little dark. But some colors look correct (like white, for instance), so not everything is over saturated.

Below, The YUV color wheel in the colors2 vectorscope is arranged in a different orientation than the RGB color wheel, but you get the idea. On the left is the way the color data is stored in YUV. On the right is the way those sme color levels get translated into RGB. High Brights are expanded, so the RGB vectorscpe (right) shows a sharp "wedge" cutoff (clipping at high chroma values).

Below, the YUV vectorscope demonstrates lowering saturation of red and yellow with Tweakcolor and FixChromaBleeding, low enbough to help the other plugins reduce chroma flickering. It's a tad dark that way, but RGB filters can restore more vibrant values after the flicker is tamed. At right, RGB shows valid levels at about RGB 240 or 245, which is pretty bright. Note there's no clipping "wedge" in RGB here, so some clipped YUV detail is being restored.

Below, after filtering in YUV and RGB, colors look more natural.

msgohan · #80 10-10-2016, 05:39 PM

Quote:

Originally Posted by koberulz

What difference are you expecting to eliminate by having me download the Intro clip, rather than use the copy I already have?

If you kept the exact sample file on your hard drive, none. I just wanted to be sure that you didn't use the original full capture or some other file that may differ in some aspect. And I tend to delete samples after posting them, personally, so I figured you didn't keep it either.

Take a look at the first line of the final section of your Intro_Script1.log. It took 1.18 seconds just to render the first frame of the video, which is 2.5x the time it took to render any other frame (frame 6 @ 0.47s is the next-worst) and somewhere around 6x the general time/frame.

I suspect one of the more complex filters is causing this delay. Run AVSMeter with the script below. It's just the first 6 lines of sanlyn's script: everything from QTGMC onwards has been removed.

Code:

AVISource("Intro.avi")
Levels(16,0.95,255,16,235, coring=false, dither=true)
TweakColor(sat=0.85, startHue=80, endHue=140)    #red
TweakColor(sat=0.80, startHue=140, endHue=190)   #yellow
AssumeTFF()
ConvertToYV12(interlaced=true)

Next, run it again with this script. Post both log files again, please.

Code:

AVISource("Intro.avi")
Levels(16,0.95,255,16,235, coring=false, dither=true)
TweakColor(sat=0.85, startHue=80, endHue=140)    #red
TweakColor(sat=0.80, startHue=140, endHue=190)   #yellow
AssumeTFF()
ConvertToYV12(interlaced=true)
QTGMC(preset="medium",border=true)

Your CPU usage also looks bizarrely low to me, though this may simply be due to Hyper-Threading...? Highest is 22%. My quad-core AMD hovers around 25% with some jumps to the mid-30s. I can probably run it on my i5-4670 later, but that one doesn't have HT either.

[EDIT: 1-thread AVISource is 12% instead of 25%. Disable power saving.]

Sanlyn: Please run the AVSMeter test as well, if you don't mind. It would be useful to have another data point.

Quote:

Originally Posted by lordsmurf

I honestly hate Avisynth discussions on Doom and VH, because of the versioning, mods, and forks of plugins. Hunting Google is stupid when we can just attach things to threads. Those files are tiny, and I have no issue attaching tiny files to posts over and over again.

Worth noting that some of our redistributions here can violate the free software license, if source code does not accompany the DLL binaries.