Avisynth: Anime filters?
 

I've been encoding anime for the past few months using fluxsmooth and sometimes with and without blockbuster. divx sources & gop 12

352*240 Plus KVCD
LoadPlugin("C:\WINDOWS\system32\fluxsmooth.dll")
LoadPlugin("C:\WINDOWS\system32\blockbuster.dll")
AviSource("D:\Share\Anime\witch.avi")
BicubicResize(336,224,0,0.6,0,0,480,360)
ConvertToYUY2()
FluxSmooth()
Blockbuster(method="noise", variance=0.7, seed=1)

Now, I just want to know which set of filters should I use now for my new template since we are using CQ. This is my new template that I'm using right now and testing. Should I take out blockbuster or use fluxsmooth instead of temporalcleaner? (I don't use 352*480 because it would take me 1hr45min for a 20min film)

352*240 Plus KVCD
LoadPlugin("C:\WINDOWS\system32\temporalcleanerold .dll")
LoadPlugin("C:\WINDOWS\system32\msharpen.dll")
LoadPlugin("C:\WINDOWS\system32\blockbuster.dll")

AviSource("D:\Share\Anime\witch.avi")
BicubicResize(336,224,0,0.6,0,0,480,360)
ConvertToYUY2()
MSharpen(15,100,true,false,false)
TemporalCleaner()
Blockbuster(method="noise", variance=0.7, seed=1)

Hi TemplateAVS,

I would add SpaceDust() right after ConvertToYUY2() :wink:

-kwag

Thanks, I'll try out SpaceDust().

I've also been working on anime KVCDs, and KDVDs over the past few months.

This is what I've used most recently, can't complain with the output either:

:)

Try also fluxsmooth(15,10). After testing with many smoothers i find this is the best you can use with Animes.
Try also Peachsmoother in addition (after resize):

Decided to try out MSharpen, but am not sure it's working. Default, "MSharpen()" opens fine in vdub, but if I change the strength to say 24, I get an error saying the data has to be in RGB format only.. so I add "ConvertToRGB", that takes care of that error. but then legalclip needs it to be YUY2..

So I wonder, can this filter and legalclip be used in conjunction?

MSharpen() is actually a good filter. It is suppose to work with YUY2 and RGB.

I think this is the latest version (Version 1.0 beta 1). But I don't doubt it, it looks very useful.

is it really necessary to use LEgalClip.dll? cause I put it in the plugin folder of AVisynth and it keeps saying it is not a Avisynth 2.5 plugin.. and I've looked everywhere and I can't find it.

Legalclip was a 2.0 filter. It is now replaced by the command "Limiter()" that is internal to avisynth 2.5.

But you are trying to use a too old script (look at the threa year : 2003 !). SpaceDust does not work with 2.5 either.

Use latest optimal script you can find on the "optimal script" section of the forum.

Also I would NEVER use Limiter() as Limiter just cuts the luma range where ColorYUV() in pctoTv mode scales the luma range and this results in visible more maintained "depth's" in very dark an white areas.

Do a avisynth script test using the Contrast Templates from THX Optimizer and you'll see ;)

Hi inc!
Where can I find them?

On almost all last THX labellized DVD released.

http://www.thx.com/mod/products/dvd/brightness.html

http://www.thx.com/mod/products/dvd/contrast.html

Where did you get this information Inc :?:
Limiter is designed for "TV" purposes, and not for PC monitors.
I always use Limiter() before filters and after filters, as a safety factor, so the color space is always bracketed to TV colorspace.
Or, has technology changed since all the discussions we had with SansGrip and sh0dan, about this issue :?:

-kwag

It has changed at least since the time I am on this forum, and I try to explain to anyone that this clipping is already done by the encoders so using it in the script is a nonsense.
And we (you and me) once concluded that it can be still used but only as any other filter used to reduce the filesize and for nothing else ! This day you removed the lines from the optimal scripts :!:
(see the history log of the optimal script).

Take a full luma range 0-255 capture (just an example) and add to your AVS script at the very end a histogram("levels") .. stream btw has to be YV12.

Now do move to a frame where the levels right above show a clear 0-16 luma clipping, these clipping values in the graph will be colored as yellow.

IF you simply add limiter() these fine tones/details within the black will be cut. but if you add Coloryuv(Levels"PC->TV") these details will be still there as the luma range will be scaled. And for me these details are important as in relation the value 16 on TV is black.
I just had that issue a few days ago where my capture cards driver was not manually optimal set to a proper 16-235 Luma range.

Now as you can set the encoder to force a 16-235 Luma range we should figure out (or search for threads) IF the encoder scales or crops the luma range. AFAIK CCE only affects the stream by these settings IF the Input is RGB ... just AFAIK. ;)

But even if you do a test using your ncoded mp2 afterwards, and check that with avisynth etc. it has been known that DVD2AVI or DGindex resulted d2v files do got wrong cooefficients from the mpeg (if I did understand that right) so an adding of colormatrix() to the script seems to solve that.

And even if not encoded at 16-235, many DVDplayers do got a luma "cropper" in their Hardware, means on my Cyberhome505 the shadow behind the "THX" isnt present and the bright Quadra will be shown as a full white. On my Mediencom DVD Player the visuals are correct and NOT "cutted". ;)

IMHO the THX optimizer isnt that accurate or well produced as I for example found out if you use the AR Template to exactly "center" the image provided by the THX Optimizer, the result is an image generally too much left, proofed by a too much left positioned 4:3 image o my 16:9 and also a wrong image position at other Movies or even reference DVDs.

When using ColorYUV(Levels="PC->TV")

http://www.digitalfaq.com/archives/i.../2005/01/4.jpg



When using just limiter()

http://www.digitalfaq.com/archives/i.../2005/01/5.jpg

You can clearly see that a) the deep black quadras in the stripes are erased and also the shadow behind the THX Logo. For me the situation is clear and the Producers of the STARWARS Box did a mistake by encoding including super black areas (0-16). On TV thats a mess, on a Beamer not.

In the script finally at the very end I did made the black a touch brighter to simulate the TV as the TV in general is brighter but the levels modification was done on the orig untouched luma range directly after the mpeg2source line!

It's not nonsense at all :!:
What about if I want to encode an MPEG for PC and NOT for TV :?:
If the encoder clips the levels, the result is not acceptable on a PC monitor.

@Inc,
I'll have to make some tests to verify this.

-kwag

Karl, watch my THX examples.

also if doing this ...

mpeg2source("xxxxxxxxxxxxx.d2v")
Converttorgb32()
Converttoyv12()

... also here the Luma range will be cropped to 16-235 due to rgb and restore back to yv12 conversion .... Means a conversation out of a rgb source into a YUV source will be done incl. a luma cropping and not scaling or even still 0-255 kept. Thats why you should use the correct coefficent flag in the converttoXXX() command as explained in the avisynth manual.

Also when using TmpgEnc forced CCIR601 encoding the luma will be cropped! Means if luma cropping for you doesnt matter, then follow Phil's recommendation to kick out the limiter() at the very end of your scripts, as not needed.
So the best would be to fix the luma within the script to exact 16-235 (even on very well known mastered sources as you see above ;) ) and finally doing a conversation at the very end of the script to the encoders native colorspace where in the encoder an untouched luma encoding (non!-CCIR601) will be done, To me that results in an encoding in a FULL luma/chroma detail encoding where still 16-235 CCIR601 Luma TV Specs are guaranteed.

AFAIK but still I not proofed that .... CCE only affects the Luma range IF the input is RGB, means at YUV inputs no affection will be done (even if set to 16-235) .... but as said I "heared" about that. So that has to be proofed also ;)

A final encoder-native colorspace conversion in the script avoids the system to use a Dshow or vfw decoder where its not proofed HOW a conversion to RGB (like which is requested by TmpgEnc) by that decoder is done, means kept, cuttet or scaled ;)

When using a Cyberhome player IMHO thats a MUST as the player itself does "crop" to 16-235, means on my Cyberhome no THX shadow is shown, but on my Mediencom its present. 8O

#1 - I did not say it is not an option. Tmpgenc and CCE let you the choice but the default is to encode in TV scale.
#2 - even if this was not an option, using limiter in a clip is still a nonsense, moreover if you want to encode for PC monitor.

For sure not !
Since when having numbers in entry that are 16-235 gives you number in output that are 16-235 also ???
If you do not tell to the encoder to round (or clip, or scale - it depends on the tool) its internal maths to 16-235 then whatever you give to it, it generates numbers between 0-255 !

Not if you're encoding for a "Streaming" target over the net, where the destination is a PC monitor ;)

-kwag

At least that's what the manual says so I would expect it to be that way. The range is used for converting from RGB to YUY2 in which the encoder works internally.

The luma range would be padded in its super values. But the movie treatment informations would be within 16-235.
What you see is what you get and thats why my re-encodings (especially captures .. but the logic is the same) where never clipped, too dark, too bright or even in their details cropped.
Basic YCbCr komponent output regulary means "do encode what you get and dont touch it" thats th logic, but some people say so or so according to TmpgEncs internal routines and that has to be proofed.
IF the non-CCIR setting in tmoegenc results in a blowed up luma range from whatever to 0-255 RGB than this also could happen via the decoder of the system as tmpgenc requests rgb24 thats why I said, to convert in the script to the native colorspace of the encoder.
also its a thing of the mpeg stream provided coefficents of color luma as "it could be" that a real 16-235 luma ONLY data would be encoded, means less luma/chroma data, means smaller encoding sizes?

We should do a real test, means a clear table where all steps are mentioned and the final image output compared to the orig or even optimal state.



http://www.digvid.info/tmpgenc/settings_quantize.php
To me that means not "forcing a convers. of the ACTIVE Luma to a 0-255 range" but taking whats happning within the whole 0-255, means to me "untouched".

But anyhow, if you go out in the script using perfect 16-235 CCIR Video Data, then no matter if let YCrCb raw untouched or CCIR "cropped"!, the result would be a full ACTIVE detail range of 16-235 within CCIR(16-235) or raw YCrCb(0-255).

Also IF you do correct the luma range its better to use Levels() in avisynth instead of a simple Coloryuv() incl. pc-tv scale as it gots an internal FIXED math for scaling!
A simple Levels(x,x,x,x,x) usage DOES crop the luma input first to CCIR 16-235 specs so watch out, thats why I ALWAYS use the parameter "coring=false" in levels, which uses the FULL luma input for correction via its values in the commandline, otherwise luma first will be cropped and then be corrected.

The only pro argument for just cropping the luma is that the luma histogram itself between 16 and 235 wont be affected at all, but what happens if already the source is NOT CCIR conform encoded? I got many DVDs which are in such a state. In case of a high contrast capable Beamer output its nice and supports the beamers Contrast Specs. but for TV it would be a mess or the luma could be cropped before by the SAP like my cyberhome does thats why some SAPs cant really support the THX optimizer in its Brightness/contrast Patterns in a correct way.

I have been reading a lot of posts and a special one which deals with exactly that problem of Cspace conversions and especially Tmpgenc/CCE.

The resumeé is:

Conversions between all YUV format like YV12 and YUY2 wont touch the luma range. BUT if a conversion is done between YUV and RGB in avsiynth the luma will be "stretched" to 0-255 or othewiese "cutted" to 16-235. Thats the case IF NO additional coefficient parameters are set in the ConverttoXXX() command! http://www.avisynth.org/Convert

The first risk is WHAT vfw codec is used when loading a video source into avisynth via Avisource(). Here already some codecs do stretch or expand or cut, so every codec model gots his own way of handling.

mpeg2dec3 by MarcFD doesnt touch the luma range, so a perfect 16-235 DVD in avisynth is kept by the mpeg2source command. Also a 0-255 DVD is kept at 0-255.

Lets see what happens:

A simple d2v file in a 16-235 Luma rangeout of a DVD will be imported into avisynth, no conversion etc., so YV12 is still the result out of the script:

mpeg2source("16-235.d2v")

Result:
TmpgEnc with CCIR option enabled = Cant be specified as all lays on the decoder in the system which handles the YV12 to RGB24 conversion, means mega Tricky as its individual codec specific if the luma range is kept or scaled!
TmpgEnc NO CCIR option enabled = Same as above!


mpeg2source("16-235.d2v")
ConverttoRGB24()

This causes a 16-235 to 0-255 expanding due a default YUVtoRGB Cspace conversion in avisynth when using NO coefficient parameters in the converttoXXX() command.

Result:
TmpgEnc with CCIR option enabled = Correct output! as the luma range of 0-255 will be "copressed"!! (not cropped!) to 16-235.
TmpgEnc No CCIR option enabled = 0-255 input is kept as NO luma modification is done! Whaterver comes in, it will be encoded untouched, means 16-235 stays 16-235, 0-255 stays 0-255!

Conclusion on TmpgEnc:
If you use Converttorgb24() and the CCIR option in TmpgEnc the luma will be expanded and compressed within the whole video-job. To me thats not they 100% way, better staying totally at 16-235 by converting to RGB24 using the convertto parameter to KEEP the luma 16-235! Afterwards NO CCIR option checked and you get a NON Luma touched encoding of your source.

But what happens if going with a YV12 (16-235) Cspace into TmpgEnc?? As said thats tricky AS if the decoder (YV12 to RGB24) used by the system KEEPS the 16-235 range when decoding TO rgb24 you will get a too bright encoding IF then CCIR is checked as CCIR option in TmpgEnc DOES compress the luma no matter which luma range comes in.

Now, CCE:
Very easy ALL YUV inputs will be untouched! No matter if "16-235" option in CCE settings is set or not, as that option ONLY affects RGB feeded to the CCE.

SO imho limiter() at the end makes no sense as anyway we use converttorgb24() which stretches the luma back to 0-255. Also in case if NO converttorgb24() is included as by that 2 possibilities are given:
1. The system decoder keeps the 16-255 (out of limiter()) and a CCIR based encoding in TmpgEnc will brighten too much the encoding.
2. The system decoder expands the 16-255 (out of limiter()) to 0-255 and a CCIR based encoding in TmpgEnc will result correctly.

But now there will be the question "Hmmmm If my original already IS 0-255 the converttorgb24() by this will expand my already 0-255 OUT of the range?? As the command just uses an internal FIX math??? So luma cropping via limiter() BEFORE converting to rgb for me is a MUST!"
Logically seen thats like a luma cropping where afterwards in TmpgEnc the "scaling" to CCIR also here will result like if it would have been cropped to 16-235 ;) So that would be THE SAME result as if you would use limiter() ... means also here in the script no limiter is needed.

Hmmmmm .... I did new tests, maybe TmpgEnc has changed his internal luma handling in rgb 0-255 AND rgb 16-235 input when checking/unchecking CCIR or Avisynth has changed hmmmm......

Status now .... No CCIR checked to me now seems a forced stretching to 0-255 and CCIR seems ALSO to force a stretch to 16-235 (this stretch factor is also given IF an rgb already gots correct 16-235 as input = fatal!)

Scaling isn't the option to use every time. I just did a small test on my latest capture file which reportedly has values over 235.

http://www.digitalfaq.com/archives/error.gif
http://www.digitalfaq.com/archives/error.gif
http://www.digitalfaq.com/archives/error.gif

The scaled image looks washed out when compared to the original and the limiter one. You might not notice it on this page, but save the three images and view them sequentially.

Boulder thats exactly the point:Means a not clear 0-255 or 16-235 Range ... a washed out picture results out of a "simple fixed" math of ColorYUV incl. PCtoTV params.

First you should check what range is provided by your PVR's driver, it should be 16-235 YUV (YV12), but your statement "above 235" already is serious to me.

ALSO I did find out today that the used Coefficients in YUV-RGB-YUV Conversions are a very importand thing, thats why Colormatrix does exist. Also in The ConverttoXXX() Commands you can set 601 and 709 Coefficients.

"Washed out Colours" ... does it mean to you less contrast on the whole Picture? As the TV compensates that to a TV-full-Contrast range.
On your last Picture I can clearly see more Luma Details on the Mans right Arm. Look at the reflexion of the Mirror on the Doorwindow! And even that will be more dark (in relation) on TV as 16 is Black and 235 is white.
Please do keep in mind that Im talking about TV purposed Encodings ;)

ALL captures should be set to keep the clear 16-235 Range, my 7134Cards Driver Settings got Sliders to affect the outgoing of Contrast and Brightness, on my 1734 for example if Contrast and Brightness are set to default the range is Shitty as its between 0-255 and 16-255, means no conversion afterwards would result ok if you want to keep all luma details.

So I wouldnt use ColorYUV incl. PCtoTV params., better is this:

mpeg2source(....)
Levels(0,1.0,255,0,255, false)
Histogram("Levels")

Now scroll through your capture and watch the histogram right above. Im shure you will have to tune a bit the range so it fits the correct 16-235.

So use the levels options to match the range (dont forget to set ",false" as a coring would "crop" all below 16 and above 235 before acting, ... like Limiter().

As said above limiter() isnt needed as a converttorgb24() will make all values below 16 and above 235 exceeding the rgb range, means cropped where in TmpgEncs CCIR option 16-235 again will be the encoded result.

What I mean by washed out is that the scaled image looks like there's a slight grey haze over the whole image.

I do a ColorYUV(off_y=5,gain_y=-16) on my captures, the values were determined like in the Doom9 capture guide. Basically that scales stuff to 16-235 but that doesn't turn the image into washed out look.

I just checked and the PVR outputs 0-255 values, at least according to ColorYUV(analyze=true).

Do a ......

Mpeg2source("pvrCapture.d2v")
Histogramm("Levels")

comparison with ....

Mpeg2source("pvrCapture.d2v")
ColorYUV(analyze=true)
Histogramm("Levels")

... and I saw that already Coloryuv in a simple analyze mode (not modifying mode) already does affect the source Range.

Hmm, IMHO the difference comes from the fact that ColorYUV outputs the text over the existing video which then alters the histogram.

In some cases of mine there was a significant affection of the superblack level, means a spike at 0 or between 0 and 5. Without Coloryuv in analyze mode there isnt such a spike. :wink: