Pseudo MIN bitrate with noise?
 

@vmesquita,

Just an idea.
Because setting the MIN bitrate in mencoder/ffmpeg just doesn't work correctly, and causes all kinds of side effects (quantization errors, etc.), I have a suggestion.
Could you encode a "black" source (no video) and add "noise" parameter with very low values, until you get an average bitrate of ~300Kbps :?:
This way, you have a fixed low bitrate noise stream, which guaranteed a minimum bitrate on black scenes.
I, and many others too, have problems with MIN bitrates. For example, on my JVS player, when a movie starts, right after the initial credits where it goes to total black before switching to the start of the movie, that single 1 second pause comes out to 7 or 8 seconds of "stutter", until the MIN bitrate passes a treshold, and from there on the rest of the movie is fine.
But on some rare scenes, and on some movies, where there is a total black scene, this effect could repeat.
I assume that a very low constant level of noise, will not be visible, but will maintain the video stream at a level that will help with the issue.
What do you think :?:

-kwag

@ Kwag: Better than this, there is a way to add a kind of "MA" noise?

Well, the idea is to maintain a very low noise static bitstream. When there's regular video, the "noise floor" will always be present, but won't affect the picture because the visibility of noise will be under the visible luma/chroma levels.
So this really doesn't apply to motion adaptiveness. It would really apply to "brightness" adaptiveness, and if this could be implemented, then that would be the way to go, because after the brightness drops below the treshold point (equivalent to 300Kbps), noise could be switched ON, and again turned OFF, once the brightness level trips above the treshold point.
Call it "Brightness Adaptive Linear Noise". That would be the correct term ;)

-kwag

Is for this that I put MA between quotation marks. I don't knew what expresion to use.
The idea is for to avoid adding noise (and more final size).

Yes, the idea is to only add noise *only* on very black scenes. Blockbuster does this already, so it can be used right now for this purpose.
But I'm almost sure that a constant noise floor of 300Kbps wont really add that much to the final file size, because the average of a movie is always way above that point.
As a matter of fact, if the noise floor is "weighted" to apply an average bitrate of ~300Kbps in a total black scene, this will be exactly the equivalent of having a fixed MIN bitrate of 300Kbps ;)
Anyway, it's better to use the internal noise filters in libavcodec, because that way it's platform independent, and we can use this technique in any OS supported by mencoder.
With AviSynth filters, we're sticked to Windows only.

-kwag

Can you say me how to do this whit blockbuster? I mean to obtain ever, a min bitrate of 300, even in a total black scene.

Ok, you're going to have to encode from a fake .avs script, generating black video.
Then, using *ONLY* BlockBuster in your script (after resolution and resize, of course), encode a small 1 minute or so, with very low values in BlockBuster.
Then drag your encoded .m2v into Vdub, and check the average bitrate.
Keep doing this until you get ~300Kbps average.
Remember this. The value you find above, will be the value to use :arrow: for the current resolution you used.
Other resolutions will have different values (parameters) for BlockBuster.
This is something that must be taken into consideration if implemented into MencoderME/mencoder.
Of course, if vmesquita does implement this in MencodeME, the noise floor will already be automatically set for the selected resolution (right vmesquita :?: :lol: )
But you can go ahead and do it right now with BlockBuster. Let me know :!:

Edit: read here to generate a black video: http://www.avisynth.org/index.php?page=BlankClip

-kwag

I also thought about that ;-)

ScriptClip(clip, "diff = averageLuma()"+chr(13)+"diff < X ? Blockbuster(xxxxx,diff/4,xxxx) :Nop()"
4 is here just an example for deviding the diff so a minimal blockbuster noise gain change will occur (like in tempsoften by deviding the nf)

Find out the right X as threshold and do apply an adaptive X/y within blockbuster noise gain so you can avoid nosie kick-ins.

Just adapt it to MA.

Its like an an MA but sees if a frame gets dark, so NO comaprison to the next frame BUT analyse the present frame. So a very low y would come out on very dark scenes.
THEN! We have to setup blockbuster right that a "all over" noise will be generated, because blockbuster can be set to do thresholded noise, ... not refered to present luma! but to plain surfaces and edges.

But Mencoder on dark scenes gots a very sensible quantizer threshold.
And on Credits ... it just "CUTS" high freq's, means fine noise.I tried that already by using a static plain noise when encoding a test (ANd I do mean "noise"! :lol: .

But when using an MA-Noise-on-black script, it can process a "kick in" of noise at dark parts where the quantizer engine of mencoder "could" think .... "Oh ... no black"

But maybe since then the engine changed a bit so another test would be very senseful!

:wink:

This is a very good idea. Mencoder offers basically two kinds of noise: temporal, spatial and averaged, which can by applied to luma and chroma is separated degrees.
My previous tests indicate that spatial only noise creates an effect of "the screen of the TV is dirty", while the temporal noise can create "dancing artifacts" if bitrate is too low. I used noise in MencodeME AVI filtering combo...
What kind of noise you think it would be better (temporal/spatial/averaged on luma/chroma/both)?

noise[=luma[u][t|a][h][p]:chroma[u][t|a][h][p]]
Adds noise.

<0-100> :arrow: luma noise
<0-100> :arrow: chroma noise
u :arrow: uniform noise (gaussian otherwise)
t :arrow: temporal noise (noise pattern changes be- tween frames)
a :arrow: averaged temporal noise (smoother, but a lot slower)
h :arrow: high quality (slightly better looking, slightly slower)
p :arrow: mix random noise with a (semi)regular pat- tern

The problem is ... I think theres no adaptive noise based on average luma input in mencoder like we can do in avisynth like explained above. :(

But maybe Im totally wrong :wink:

PS: Senseful would be to add a shock of chroma noise as you wont see it on already very low luma parts, but the encoder does interprete that as "dancing" details.

Here's your 350Kbps MIN bitrate ;)

BlockBuster(method="noise", detail_min=1, detail_max=1, variance=0.1, seed=5823 )

This was tested with TMPEG.
Enjoy,
-kwag

Yes, you can't vary the ammount of noise per frame. :(
Have you tested this? Because I saw the dancing details adding noise to luma only (I was trying to remove DCT blocks). Does chroma produces the same effect?

@kwag
Nice! :D 8)

Perfect Kwag!

Now try that on mencoder as it quantizes different on black frames as TmpgEnc.

So try....

ScriptClip(clip, "diff = averageLuma()"+chr(13)+"diff < X ? BlockBuster(method="noise", detail_min=1, detail_max=1, variance=0.1, seed=5823 ) :Nop()"

We have to find out the right X threshold for B.Buster kick in.
This would avoid usesless noising on average movie content as even there noise would mean less compression as seen as a whole ;-)

@ VMesquita

? Do you mean you recognised these "dancing blocks" when noising the luma only?? Maybe chroma noise is just the only we need?? We should test that :D
(or maybe that was it what you meant)

@Inc
I meant that I was noising the luma blocks (to remove DCTs) and saw the dancing blocks. But I didn't try to noise luma and chroma or chroma only to see what happens. More tests needed. :D

But looks like the avisynth idea is more suitable, specially you get the adaptative thing working, otherwise we would be wasting bits in places that doesn't need noise. Too bad it's not multiplataform. :(

Here's my final setting, to be tested with MencodeME/mencoder.

noise=3th

You can change it in MencodeME presets, for example: "DVD.fil", and it will look like this:
@vmesquita,

I changed the noise generation from "uniform" to "temporal", because uniform is a static noise, which is identical on every frame. With the temporal option, the noise is ranrom on every frame, and it's more natural.
Anyway, the value is so low, that it's barely visible, but it maintains a MIN bitrate around 290Kbps.
Give it a whirl ;)

Edit: This settings were tested on a 704x480 target, so if you use other resolutions, your MIN bitrate will be slightly different.

-kwag

And what about non black scenes, but very bright (amost white), or all of almost monochromatic scenes (full red, full, green, full other colour). IIRC the MPEG does not recognize colours, but grade of changes. This hapen very rarely but some movies are going "fade to white" not to black. Have you tested this type of scenes :?:

Very good observation Hydeus :D
In case of the mencoder noise generator, it's irrelevant, because the noise floor is statically being generated, no matter what brightness/activity there is. So the minimum bitrate will always be present, no matter what color is on the screen, even if it's a static white screen.
I thought that this would affect BlockBuster, but I just tested it with a white screen: BlankClip(length=3000, width=720, height=480, fps=23.976, color=$FFFFFF) and I got the same result. A steady MIN bitrate of around 300Kbps
I also tried a gray screen: BlankClip(length=3000, width=720, height=480, fps=23.976, color=$808080) with similar results :)

-kwag

Great 8)

This is great! In my tests I were using a higher threshold that's why I was getting dancing blocks. BTW did you try adding noise only on chroma or only on luma? The downside of temporal noise is that it's also less compressible across all frames but since you're using a small threshold this shouldn't be a problem. :D

Yes, I noticed that, and I was also seeing "smearing" on surfaces, so I suspected a too high temporal value. See the changes below. No I didn't. But the noise is so minimal, that it's barely noticeable But it does keep the minimum bitrate around the needed 300Kbps. :)These are the latest settings I'm using:

hqdn3d=3:6:2
noise=3th

I'm currently doing an encode with those parameters, and it looks pretty good :D

-kwag

Sample
 

Here's a sample, created with vmesquita's temp.conf file, that was created with MencodeME on my WIndows PC.
But I took the .conf file, and send it over to my FreeBSD machine, and encoded it there.
I rebuilt the complete mencoder/mplayer/libavcodec from CVS today.

These are the settings used for this sample:

This is my first encode on FreeBSD :D

http://www.kvcd.net/test.m2v

Min bitrate is maintained above 300Kbps :cool:

-kwag

@ kwag : What do you mean by "setting the MIN bitrate in mencoder/ffmpeg just doesn't work correctly" ?

My Pioneer DV-444 doesn't like bitrates under 800 and my mencoder scripts use vrc_minrate=800 w/o any problems. BitViewer also shows that it works.
I know it sort of screws up the KVCD compression gain but a constant 300kbps noise should do the same, right?

I do my encodings with mencoder-1.0pre4 under linux.

- Peder

It may be fixed now (we haven't tested this in a while), but setting min_rate used to make mencoder don't hit the average bitrate anymore. :wink:

no, it's not fixed (tested w/ 040508) but that harsh 'rate deviating' effect u mentioned is valid only for the low bitrate (~<1500kbps) range. however, minrate settings is (& was) always tolerated. mencoder keeps minrate much better than maxrate! (but sometimes the price is too much for that)
what kwag mentions is also true. i found that in the lb range even the smallest change in minrate affects heavily the q curve distribution. usually in the wrong direction as the q curve becomes quite erratic spanning a much wider range than w/out vrc_minrate.
anyway ... this business is much more complicated.
i've been testing mencoder for weeks & i found that min/max bitrates are heavily dependant on many options. vrc_max/minrate (of course), lmin, vratetol, qmin/max, ... i've tried to do a step-by-step investigation but as all these opts are somehow 'cross correlated' it's very hard to make a 'complete' set of experiment. what i only(?) want is finding a 'robust' & safe setting with which min&maxrate is kept (aimed 250-2500). no success so far. if u're interested i'd put some of my results.
& (just not to be so ot :-)) insted of adding noise a/o whatsoever why not push maxrate (slightly) downward ? it'd be a kinda bitrate redistribution. sure it worx but i've never tested this aspect.
the bests
y
[EDIT] hmmm ... there must be a hard coded limit of vrc_minrate as if i set it to 0 encoding goes to divx(?) instead of mpeg2(???) mencoder does it when options set outside the limits allowed (say, lmin=0 does the same) maybe, this part is really crappy.

Well yaz,
in the thread I started about vrc_minrate I commented my short experience. Yes, when setting any minrate value, mencoder keeps it OK, but it seems to overquantize, compared with the same encoding without minrate setting. But in my experience this just happens in first pass, in second pass the quantization is "controlled" more, and returns to "normal" values.
I'm following this pseudominrate thread with interest, but still didn't test it. I tested the AVI.fil with added noise (2u if I remember well) but I got low bitrates about 100.
Anyway I think it would be better to be able to use a minrate value. Has anyone else tested this on 2pass?, are min/maxrates respected?, and what about quantizers?.

Hi yaz,

Try the settings I posted above.
This morning, I woke up to a perfectly encoded MPEG-2 :!:
Look at the Mplex log:
No errors whatsoever, and target file size right on the nose (with 1-pass), and not a single under/overflow :)

-kwag

@kwag
your pleasure is mine :-) but ...
afais, u set maxrate to 2500. how's that come to thisthis instance the maxrate obtained is much higher than that set. that's what i always whine for.
i guess u set vbitrate sw to 1150 (in my experience i've found that it's pretty hard to make vbitrate deviate significantly from the setting)
& what about the minrate?
i don't want to 'wipe your smile', just asking ... :-)
the bests
y

Yes I noticed that too, and it's on some peaks. But that was a 1-pass encode. Actually, I am currently running the second pass, just because of that :) The MIN bitrate is maintaind at about 300Kbps, because of the parameter noise=3th. I verified that with Bitrateviewer, and it doesn't go below ~290Kbps.I'm still smiling ;)
But after the second pass, I'll either :arrow: :D or :arrow: :cry: if the maximum bitrate doesn't normalize to ~2,500Kbps.

-kwag

Re: Pseudo MIN bitrate with noise?
 

Umm.. I've done a few cdr's of 110 and higher movies, using 300 900 1850 as the bitrate settings. (and some 180 min season dvd's to two cdr's).

All have come out beautiful, rarely a block to be seen, and then only on vastly differing scenes (eg, a swirly flashing mutli coloured intro to Absolutley Fabulous Season 1)

What are these side effects - what do I need to look for?

<posted ini file for comparison>

of=rawvideo=1
ovc=lavc=1
nosound=1
noskip=1
sws=9
lavcopts=vcodec=mpeg1video:vhq=1:keyint=15:aspect= 4/3:vmax_b_frames=2:vrc_minrate=400:vbitrate=800:vrc _maxrate=1850:vratetol=1000:vrc_buf_size=327:vqmin =2:vqmax=24:mbqmax=20:lmin=1.5:vb_qfactor=1.2:vi_q factor=0.8:vqblur=0.3:vlelim=-4:vcelim=7:lumi_mask=0.05:dark_mask=0.01:naq=1:
intra_matrix=8,9,12,22,26,27,29,34,9,10,14,26,27,2 9,34,37,12,14,18,27,29,34,37,38,22,26,27,31,36,37, 38,40,26,27,29,36,39,38,40,48,27,29,34,37,38,40,48 ,58,29,34,37,38,40,48,58,69,34,37,38,40,48,58,69,7 9:
inter_matrix=16,18,20,22,24,26,28,30,18,20,22,24,2 6,28,30,32,20,22,24,26,28,30,32,34,22,24,26,30,32, 32,34,36,24,26,28,32,34,34,36,38,26,28,30,32,34,36 ,38,40,28,30,32,34,36,38,42,42,30,32,34,36,38,40,4 2,44
vf=scale=352:288
ofps=25

The vf=scale line is changed if the dvd is widescreen, this one is from ABFab - 4/3.
This encodes at 25fps on a p3-800. Direct from vob to mpg.

<edit> and why is my 'quote' never working!!??

Re: Pseudo MIN bitrate with noise?
 

Here's one.
From the Mplayer's group manual:

"NOTE: vratetol should not be too large during the second pass or there might be problems if vrc_(min|max)rate is used."
http://www.mplayerhq.hu/DOCS/man/en/manpage.html#GENERAL%20ENCODING%20OPTIONS%20(MENCO DER%20ONLY)

As I now don't set min (only max), I don't see problems.
I'm still on my second pass of my previous encode, to see if the bitrates are distributed, and the max bitrate stays around 2,500Kbps.

-kwag

Re: Pseudo MIN bitrate with noise?
 

Because you are "checking" Disable BBCode in this post :!:

Re: Pseudo MIN bitrate with noise?
 

:oops: :oops: :oops: :oops: :oops: :oops: :oops: :oops:

Sorry Kwag!!
<Fluffbutt blushes his furry ears in embarassment>

:oops: :oops: :oops: :oops: :oops: :oops: :oops: :oops:

I've not done a 2 pass yet, and my ratetol is nearly always set at 1000.

Actually that script I posted has been arrived at after about 50 test encodes of a 400 mb file.
They give me the best results yet, speed and quality and size -wise.
I get 105 minutes in about 740mb (inc audio at 128 or 192 kbps (depends on movie size - I'd rather drop audio then lose movie quality)

I'll have to try a 2-pass encode (I can do a 30 min dvd seasonal episode in 30 mins, so 2 pass would be 1 hour?) - Is the increase in quality worth the extra time?

Here's the log of my mux, after the second pass:

But here's the thing :!:
My 1-pass .m2v, when dragged to BitrateViewer, showed a peak of ~3,800Kbps.
My new 2-pass .m2v, shows 3,425Kbps


And here's something else. Look at the log, when processed with MPEG Validator:Specially :arrow: Peak Bitrate (1 sec) = 3550 Kbps
Obviousle, that's some kind of a spike :!:
Anyway, the file plays flawlessly on both of my SVCD capable players.

Here's a sample of the muxed 2-pass encode:
http://www.kvcd.net/payback.movie.2pass.mpg.cut.156.mpg

-kwag

@kwag
u're the lucky man :-) if u want to play further i'd suggest to lower the maxrate. it's the most effective way of controlling maxrate. some quests :
- why did u set vrc_buf_size to 1835 when u go for svcd (or did i miss sg?)
- what version of mplex do u use ? (mine does not drops that nice transcounted values (bytes/s -> bits/sec) ... ) (what a lazy ppl i'm :-))
- did u checked the 'compressibility' of ur clip ? i guess it's pretty high. yesterday i played with this limiting bitrate thingy. it seems compressibility is also an important factor concerning limiting bitrates.
yeah, here we go ... that's why i always ask how important is this problem (if at all). old(er) standalones were very rigorous about 'standard compliancy' but what about the new(er) ones ?

nice worx, anyway !
the bests
y

You didn't miss anything. I was stupid and I left the DVD setting :oops: :lol: Mplex 0.91. The GUI version. Not really, but last night I made a KDVD, and I see a pattern here :!:
I had set MAX bitrate to 7,500Kbps, and the final encoded movie had peaks of ~8,300Kbps. So it seems that there's a constant in there, where the MAX bitrate can go about ~1,000Kbps over the MAX value set. So now I just set my MAX bitrate to 7,500Kbps for KDVD, and I don't have to worry about the MAX, because it will never reach the maximum 9,800Kbps ,which is the limit for DVDs. And 7,500Kbps with mencoder, is more than plenty for action scenes ;) Well, in the two players I tested, one is new (Cyberhome 300), and my other player is much older (JVC XV-F80BK), which is a 2+ year model.

-kwag

@kwag
things are not so simple (as usual :-)) i found the same for 'high-bitrate' (1500-2500) svcd.controlling max this way is more complicated as the set & apparent max does not show linear relationship (for me)
& ... when i went below 1500 i found the peak-br lower then the max set. here i had to increase max in the setting so as to hit the limit i wanted. in the range round 1200 it meant max=6000(!) for some clips.
anyway, my results are quite uncertain cus i haven't got a reliable stream analyzer. i've tried bitrateview, vd-mpeg2, mplex, mplexgui, ... all gave different(!) results. so what i wrote is rather a tendency.
btw, for such programmers like hanging here around, would it be so hard to write a small app for a simple stream analysis ? :-) sg like statsreader for xvid. say, min/avg/max for br and q would suffice.
the bests
y

I'm going to do an analysis, simply encoding at every valid resolution with heavy gaussian noise, to force the encoder into saturation.
This way, I can see the MAX bitrate that the encoder will produce, on a worse case scenarion. Then we can adjust the MAX offset to use, once we know the "Real" peak bitrate that mencoder generates, from the value that is set.
This should solve the problem of the max bitrate going above the set limit.
Once I get the values, I'll PM vmesquita and incredible, so they can integrate the correct values into their applications ;)

-kwag

Thanx for your test kwag. It would also be nice if you post your results, for those of us that still do some of our encodings with our own command-line (although VM and Inc are making that the number of this kind of encodings is getting smaller every time... :lol: :lol: )

Sure wll :)

-kwag

Here are the test results.
I used a simple 10 second clip, using the mencoder parameter noise=99th, which causes extreme noise and saturation of the encoder.
What I did was set the MAX bitrate, and then I used an average bitrate of 200Kbps under the MAX setting, so as to keep the encoder working on a worst case scenario, by keeping average bitrate very close to MAX bitrate. As said, this is a worse case scenario, which should NEVER reach these values in reality.
Here are the results. All values in Kbps:

As you can see, all tests produced a bitrate within tolerance and below the MAX bitrate, except on 1,150Kbps test where I assume the encoder tried to maintain quality by pushing the bitrate over the max. The log for that session was full of errors, indication buffer underflows on almost every frame. Obviously, a bitrate of 1,150Kbps is simply not enough for a resolution of 704x480, which is the resolution I conducted all the tests.
The question is, why do we see peak bitrates above the MAX on our regular encodes :?:
Well, they are glitches, and I was able to confirm that even on a VOB from the movie "The Big Hit", which is a SuperBit DVD, and the bitrate peak is over 10,300 :!:
That's over the MAX DVD standard of 9,800, but that DVD plays flawlessly on all my DVD players.
So, are these peaks going to cause us problems on our DVD players, playing our KDVD encodes :?:
I don't think so :!:
So vmesquita and incredible, my recommendations are that you can disregard the issue, and leave the MAX bitrate settings as they are, without the need to use a negative correction offset.
Could this stability result be caused by the use of noise=3th on the settings :?:
Maybe, but I didn't make any more tests, because I'm very happy with the current results.
Anyway, there's really no need to set mencoder MAX bitrate above 8,000Kbps, because even at 720x480 at ~7,500Kbps, the quality results are BEYOND CCE or TMPEG at 9,800Kbps, and that's a fact :!:
mencoder (libavcodec) is way WAY ahead of them.
And that's my humble opinion, after doing many many tests.

-kwag