Buffer underflows
 

I've been getting some buffer underflows with my DVD encode tests, avg bitrate 2000-3000kbps. Is there any simple way to get rid of them or do I just have to use CCE for those encodes :lol: ?

Which build of mencoder you are using??
Are you encoding 1pass or 2pass?
As when choosing avg's of i.E. 2000-3000kbps I never had Buffer underruns again! And my Player plays it back with no Probs.
Did you check that on your DVD player?

Problem is you cant set in Mencoder vrc_minrate=xxx, as then the bitrate is broken. Thats a known mencoder issue, .. but as said above I never had Probs without vminrate setted.

Try to add some noise (first tick is the one suggested by Kwag).
But Phil is "a bit" right as he said IF you do encode a 2.35:1 source that noise will also be added to the leaved black border from 2.35:1 to 1.77:1(1.85:1) but that will also happen when using fitcd without determining the effective "movie area" before ;-)

But I did say "a bit right" above as the noise in that amount only gives the encoder information that even at black scenes there does something happen, so the min bitrate at these scenes will be rised and the "noise" will almost be quantized out! .... I did several 2.35 source tests and didnt recognise partial noise on black borders .... but Phil's thought gots its logic, and I should add a "Source 2.35:1 ?" button so other cropping values in such a case would be used so that almost no black borders even at 2.35:1 input will be left. ;-)

So try setting a bit noise where beside that the movie gots a tiny bit more natural look.
Id did came off of "plaining" sources that much when I do enjoy big media sizes like you also do have in this example IF encoding using i.E. using an avg bitrate of 2000-3000kbit.

:)

The only factor which would count for me to still use CCE is IF I want to affect DC precision to 9 or 10 ... ;-)

I used your latest CVS build (P4 optimized), 2-pass. The first pass doesn't show any underflows, they occur in the second pass.

I solved Phil's problem by cropping, resizing and filtering in the AVS script and then expanding to 704x576 in MEncoder. I've not experimented with the internal filters so I wanted to be rather safe than sorry :lol:

I think the next thing I'll try is raising the max bitrate, maybe that would help.

If you want to see "some " filters:
http://www.kvcd.net/forum/viewtopic....10637&start=16

Why do you do your whole postprocessing in avisynth but then expanding in Mencoder?? In that case do the "whole" pprocessing in avisynth and choose "no resize" or ged rid of all image manipulation parameters/commands in your mencoder line.

btw. You did use a Keyinterval of 24! And as you do a high avg bitrate encoding I assume you encode to dvd-r?
Well as I assume you know what u are doing, your SA-Player supports a gop length of 24 in DVD mode?

That's because I wanted to make sure that the min bitrate is around 300kbps so my Pioneer doesn't get hiccups. I also didn't want the noise to be put on the borders and I was unsure which Blockbuster line would be sufficient. As the "noise=3th" has been tested thoroughly, I wanted to use it.

Yes, the player has no problems with 24-frame GOPs in DVDs, at least none that I know of8)

Hi Boulder,

I stopped using longer GOPs with mencoder, because I get a "smearing" (trailing objects) when using 24.
This seems to be a peculiarity of mencoder.
Using 15 (PAL) or 18 (NTSC) doesn't show these artifacts, and compression is still very high, when using the notch matrix.
So I suggest standard GOPs with mencoder :)

-kwag

Well, in that case Mencoder's of no use to me as the main reason was the speed of CCE and ability to use 24-frame GOPs combined :cry:

I personally dislike TMPGEnc for DVD encodes since it takes ages to predict and encode. With jonny's QCCE I can encode 2-3 1h TV captures on one day which is not possible with TMPGEnc.

:?: Could you explain me then how you get a framecount of 24 within a GOP setted in CCE? :wink:
CCE allows only a max framecount of 15 (even in NTSC) within a GOP.
Or ... did I miss something recently??? 8O A Trick or a new release of CCE :wink:

According to captures ... as I did promise to DigiDoc ... the next days (if not even I get my resol. claculation engine ready meanwhile) Ill integrate resizing templates which do support the common capture source size inputs:
PAL: 768x576,704x576,352x576,352x288
NTSC: 640x480,352x480,325x240

I already did tests on YUY2 mjpeg input (PicVideo) and for shure mencoder does read them like a charme :D
captures using YV12 Xvid can also be encoded but I strongly advice to Noobs NOT to use that codec if doing capturing:
http://www.kvcd.net/forum/viewtopic.php?t=9380
(not adressed to you but to capture noobs)


My conclusion ... Mecoder (already in 1pass) does - even if a GOP of still 15/18 is used - give a better output than CCE.
And IF you still want to go beyond the std. GOP count ... in case of mencoder that would be only a special Matrix setting ... means less filtering in the non-intra 8x8 dct block.

What I meant was that the speed of encoding with Mencoder is similar to CCE and the 24-frame GOP from Mencoder (if it worked ok) would be splendid :wink:

I'll have to experiment with Mencoder a bit more before I do any final conclusions. I'll leave that special matrix to Kwag :lol:

Just to tell that I did a KDVD tonight (2pass needs 10 hours on my PC - encoding at 7/9 fps).

This morning the 2pass was near 30% and I saw 4 underflow messages on the screen. I don't know if there were some during 1st pass and I can't tell how many there will have before the jobs ends.

My question is : do you think I can securely burn the final video ?

FYI : PAL DVD, Res 704*576, Denoise to 2nd tick, Blockbuster to 1st one, no other option use, AVG=2000, max=4500, min=0.

I the past using Packshot I had only 2 encodings where "still" an underflow error occured, .... but playback was NO Problem on (my) SAP.

The quality of mencoder is still superior, even with a GOP of 18 :)

-kwag

I'm currently testing encoding with MEncoder again (with the same movie), this time with min quantizer=2 to avoid buffer underflows and lmin=1. Now it seems that the encode will end up undersized even though I raised the max bitrate from 5000 to 8000 :x

Guess I'll have to lower the lmin value to 0.5 and see what happens..

The higher you set qmin ... the more quantisized will your movie be in average!
And higher quantization on black parts means even LESS bitrate, so just try maybe vqmin=1:lmin=0.1 (just a test as you work using vey high avg bitrates ... which IMHO you dont even need in mencoder ;-) )

That's the problem..quant 1 is generally considered A Bad Thing To Do since the quality increase is pretty much negligible compared to quant 2. The filesize just gets a lot bigger.

This also worries me a bit when using vminq=1 (from the docs) : "ratecontrol will be confused resulting in lower quality".

I'll try running the encode with the values you suggested and see what comes out.

I wouldn't worry too much about what man_page says on vqmin=1: we're using this setting in lots of encodings, without a problem. I think it may be a problem in constant quantizer encoding (vqscale=1), but in our encodings, you will see in BitrateViewer tha quantizers don't keep constant in 1 but vary usually around 1.5-2, even higher.

About buffer underflows... I also don't like to see those warnings. But later I don't have a single problem when muxing/authoring/viewing.
Where I did have problems was when minrate too low, in SKVCD. But not with underflows. Anyway, it would be nice to know why it happens, and how to solve them...

I'm sorry to interfer with your discussion : just want to say that my first 2pass KDVD is finished and for what I can see it's really nice. A lot better than what I did with CCE without any script used (only Denoiser set to 2 in pack-shot and blockbuster to 1).

I'm much more enthousiast that I was with ffvfw since week ago.

Phil, no problem :wink:

By the way, Bitrate Viewer shows really funny figures..Q is sometimes negative :lol:

@kwag is the number of frames in a gop "keyint"? Because I keep getting artifacts :? So should I change that to 18?

-Koekies

I think I read Inc tellign that in a post today : yes GOP length = Keyint. And you should reduce it to 18.

Yes.Yes.

-kwag

It's a damn shame that MEncoder doesn't give the timestamp where the underflows occur. I just finished the encoding process and there's some 20-30 underflow errors 8O I'll try authoring a DVD in Maestro and see if it complains anything.

According to Bitrate Viewer the Q level is very low but I don't know if it can be trusted one bit since the Q values are sometimes below zero. Also the average bitrate isn't nearly correct.

8O
I haven't seen that on any of my encodes :!:
The Q line is always above the bitrate line.
Could it be related to resulutions, or to PAL :?:
You are encoding PAL, right :?:

I've encoded 704x480 and also 352x480, and burned as SVCD and as DVD, and I haven't seen any more underflows. At least since I started using the noise parameter.

-kwag

I'm in PAL zone, and I have seen Q lines below 0 too.

I often get Qs between 0 and 1 (below zero doesnt exist (IMHO) in BV) ... thats if a high avg bitrate is used and therefore even less quantization was needed.

IF you would rise the lmin that would mean even higher quantisation and therefore even LESS bitrate at those parts.

The Q curve in BV ONLY shows if (and now my english) many same values came out after the Inverse DCT transforming .... means good RLE compression. BUT that means nothing according to the real quality when quantizing was done by the encoder.

I do have streams from CCE TmpgEnc and mencoder at the beginning of our tests .... TmpgEnc got an Q avg of aprox. 5-6, CCE 4-5 and MEncoder 2-3 ... and ALL looked almost the same in their quality.... that was before doing tweaking the parameters of mencoder

Well, maybe we have gone off topic...
... but I'm also in PAL country, and never had quantizers below 0. It's even difficult for me to find quantizers below 1.
As Inc says, maybe too high average bitrates and too low lmin values?(not in general, but in certain films maybe it's not necessary lmin=0.3...)

I hope you all understand that well .... ;-)

Buffer underflow means bitratedrops (shure you know that)

BUT the less you quantisize (less q!), the higher the bitrate gets, logic?

So IF you suffer from too much drops of bitrate, that would mean at these scenes you need to allocate less quantization so more bitarte will come out.
Im not shure but IMHO less lmin should sove that .... well thats the logic, .. but it seems it also affects the whole bitrate curve (not Q curve only).

So dont mix Q curve with bitrate curve ... they are nearly the opposite.

And I checked in BV ... a Q lower than zero doesnt exist it goes in the Y axis from zero up to X ...

@Boulder: Have you a pic. I don't have anymore that mpg with BV negative Q curve.

@Inc.: You don't see the curve below 0, but you can see that Q curve is broken at basal line, it dissapear in one point and come back in other point little bit away from first point. I don't keep cursor over this point to check out values in Q info values.

Inc,

My second attempt with pack-shot is full of underflow !
I didn't change anythign since the previous one and the kind of source is the same (I do the sequel of the previous movie !)

What do you mean, same source, nothing changed??
That would mean you deal with an abitary behavior of mencoder?

8O

Ok, now I understand .... and "that" never happend to me

Ok guys, I'll get the pictures for you shortly.

DVDMaestro didn't complain anything so the encode should be ok. Funny how the bitrate can get too low as I encoded the movie as anamorphic and there are only small borders so it's basically a fullscreen encode. I also encoded Blade Runner as anamorphic and that movie has an AR of 2.35:1 but it didn't cause any underflows.

No, it's not the same source ! I did 'Blade', I'm doing now 'Blade 2'.

What I wanted to say is that the 2 sources are very similar in words of quality / complexity / actionness / length / ...


[/quote]

All right, here's the screenshot:

http://www.digitalfaq.com/archives/error.gif

The Q value at that point is -0.29, the green color gets all messed up with JPG compression :x

You seems to use a registered version of the tool, don't you ?

Try to install in an other directory the non registered versoin. And open again the same m2v.

It seems that bitrateviewer has a stange behaviour once registered.

Well that case in here shows me really the "issue" of BV according to Q values!

Dont trust them!!!!
(It doesnt make sense discussing Q values of BV as very big issues of BV according to Q are known ;-) )

Also there BV does say "Linear Quantizer Scale" ... but as you see its not! :lol:

So, watch the stream, ... do decide if it results good or bad ... and the only thing I would give notice if there are BITRATE (not Q) peaks or drops outside DVD compilancy, ... but even encodes including some BITRATEdrops down to 150kbit do play very well on my SAP (maybe in reality they arent that low! - or my SAP tolerates this. And if you do apply some noise as Kwag told, the issue is even less present.

BTW. Karl told me as he did noise tests also using a smash factor of 99 means full of big noise, ... the noise generating was performet only on the effective movie area, ... so if you input a 2.35:1 source where still borders even at anamorph source state do exist, these borders - it seems - that they wont be touched by the noise generating.

(EDIT: Answering to Phil)
That seems to be the case. The weird thing is that the curve CCE-encoded video gives is not like the one that MEncoder gives when using the registered version.

The MEncoder stream is also encoded with a linear quantizer scale?!

@Inc: I used AVS to crop, filter and resize and then MEncoder's noise (3th) before expanding so the noise is certainly only in the film pixel area.

Hey Boulder ;-)

I just edited my post above and added the "quantizer scale" issue of BV.


Resumeé .... BV is maybe nice, does look nice ... seems to be nice ... but I dont wanna trust him anymore ... defenitely not in case of Q and also I dont trust him "that much" in case of Bitrates. :?

Go and do an encode a mpeg1 at "linear Quant scale" using CCE 2.5 .. and a low flatline Q will output ... but the stream for shure wont look better in relation of a non linear one where Q iss more above. I tested it ;-)

Maybe there will be a new appl. released soon which gives more trustable outputs



-------------

Boulder, Im at work so I only assume:

Try modifying the BIAS as in regular the BIAS is responsable also for Bitrateallocations as we know from multipass CCE tweaking.

Do generate an AVS based input, using a sampler() or better Slicer() cutted output .... there you can see very fast how this affects the Bitratecurve. Do start cutting right from frame 0! As there the most black parts will be present :)

@boulder
i'm a bit confused.
- ur screenshot shows vbv=112. it's for svcd ... but u wrote u'r making dvd. how's it going ?
- it seems as if ur max & target bitrate were the same (~2500). if so why not a cbr=2500 ? (much safer)
- in my experience, if target bitrate is 'too close' to the max this 'underflow business' kicks in after awhile. if keeping max precisely is not a big issue, u should raise (or leave out) max.
anyway, this warning is just a warning. it does not bite. it simply means there's not enough bit in the buffer to feed a sudden bitrate peak.
- why do u bother about borders ? pioneers are pretty well with anamorphic shots. (or am i misinformed ?)

@inc
- imho, the bias values has nothing to do with mpeg2 here. they are mpeg4 related. (the manpage is quite unreliable with respect to distinguising the option types properly)
- adding noise does not necessarily means higher bitrate. the codec can circumvent this by raising the qantizer level. the other way round is changing the macroblock quantizers and keeping (more or less) the same frame quantizer. imho, this process is quite unpredictable. if we consider plain black frames it's a bit simpler but here we talk about clips/movies.
- bv is not as bad as u state. it's not as accurate as can be but what i've seen from it seemed to be quite reliable. (i used only the freeware versions so i don't know how it worx if it's paid :-))

Based on my experiences, BV shows the VBV buffer value incorrectly. It's always double in reality.

Nope, target bitrate was 3069kbps and the max 5000kbps.

And this often causes problems with standalone players.

Overscan, man, overscan :wink: