MEncoding DivXs/DVDs using 2 pass VBR
 

I decided to open a new thread to post this because the main mencoder thread is discussing mainly one pass VBR and DVD sources.

After some tweaking, I was able to come up with a internal filter combo that gave me results close to the ones obtained via DialHot v4 Script+CCE. CCE gave me 19 fps, while mencoder was working at 30 fps using my latest compile (posted in the main thread) optimized for my AtlhonXP. So if you consider that some time would be spent in prediction, let's say 20% of the encoding time, 2pass VBR in mencoder is taking near the same time 1passVBR takes in CCE with prediction. Seems very good, considering that the results are nice good, and CCE used to be the fastest encoder around.
Plus Mencoder has the advantage of using longer GOPs, while CCE only allows up to 15. If you're in NTSC you can go up to 18 without breaking the standard.

So this is avi.ini we are going to use for both passes, for DivX/XVid sources:
This is the dvd.ini for DVD sources:
Of course, scale, expand, dvd-device and must be replaced with the correct values. :wink: B-Frames are working. :D

Command-lines for AVI:
Command-lines for DVD:
Replace 1500 with the bitrate wanted, and in dvd://1, replace 1 with the VTS number (check using DVDDecrypter in which VTS the movie is)

This filtering can of course be improved, I am just putting my latest combination for AVIs. The DVD settings does not filter at all, this can be of course improved. The settings used are basically the latest from bilu in the main mencoder thread. :D

I am really thinking about implementing this in DIKO now! :D

<edit> Improved settings.
<edit2> forgot vrc_buf_size setting. :oops:

Re: MEncoding DivXs using 2 pass VBR
 

The future looks bright :lol:

-kwag

Hya,
Nice to see you're working with other mencoder possibilities, like 2-pass.
Hope Diko can get faster with Mencoder :D
I just would like them to check the rate control...and we (1-pass guys) would be back in biz.
P.S.-How did you check B-frames were ok? looking at BV?
Cheers

8O

MenDIKO :?:

:rotf: :hihi:

Then followed by "MenDikumcizer" HAHAHA :D

-kwag

:lol: :lol: :lol: :lol: :lol: :mrgreen: :mrgreen: :mrgreen: :mrgreen:

@rds_correa
B-Frames are OK, I checked in BV. But you have to use them in both passes, if you specify b-frames only in second pass, it doesn't work. :wink:

vmesquita,
just to make sure I understood your way... First pass you make it with constant quantizer (vqscale=1), but, doesn't it make useless vqmin and lmin parameters? I mean, since quantizers are fixed with vqscale, there'd be no need to set qmin and lmin.
And secong pass you make it with vqcomp=1:vqmin=1:mbqmin=1:lmin=1 (that is, constant quality) with vbitrate=1500. That's OK. But I thought we'd better use similar settings in frist and second pass, and from first to second you change your way...
Anyway, in first pass you keep quantizers at 1 (and I can imagine very high bitrate peaks) and in second pass you set a high constant quality setting, and a vbitrate value to moderate those high bitrate peaks and get a good final quality... is that a possible explanation of your way?. Tell me about your output: what bitrate and what Q value do you get with tihs approach?.

digitall.doc,

Thanks for your observations. Actually I just grabbed Bilu latest settings and did a few modifications for 2-pass. I am not really sure about how most of the parameters work.

The idea of doing first pass with vqscale=1 I took from XVid, I read somewhere that it is done that way now. But I am no t sure about that.
About your explanation: that's what I expected to get and I had good results yesterday. But I am testing a DVD source right now, and mencoder is not following the bitrate I asked... :( I am using vbitrate 1500 and it's coming 2000... :( I'll try with the default settings and vbitrate in first pass, maybe rate control is really broken.

vmesquita,
don't erase the first pass video file, and make sure if it was encoded at constant Q value of 1.
From bilu advise, I understood that you had to keep the same parameters in first and second pass, but in first pass you use vqscale=2 (1 is not advised in man_page, but also wasn't for vqmin and lmin... :roll: ) that is substituted by vbitrate in second pass. And I would also test with and without vqblur in second pass (just an idea, to see if mencoder can adjust better bitrate with some quantizer delay)

I found out the problem. Looks like vrc_max_rate is broken in current CVS, using it causes many problems. Another interesting thing: using the goal vbitrate in first pass instead of vscale=1 gives better quality. Check this:

http://www.jltoca.uaivip.com.br/images/cqvsfixed1.png
http://www.jltoca.uaivip.com.br/images/cqvsfixed3.png
http://www.jltoca.uaivip.com.br/images/cqvsfixed4.png
http://www.jltoca.uaivip.com.br/imag...ed5-smooth.png

Doing a constant quality first pass causes less smoothing in calm scenes, but on the other hand, more blockness in action scenes. Now let's see how 1pass with vbitrate and the rest of the settings the same as 2pass compare:

http://www.jltoca.uaivip.com.br/images/1passvs2pass.png
http://www.jltoca.uaivip.com.br/imag...ssvs2pass2.png
http://www.jltoca.uaivip.com.br/imag...ssvs2pass3.png

I'll edit the first post with the working settings.

sorry vmesquita, didn't understand your last post.
So you're now using the same settings in first and second pass, isn't it?. Instead of vqscale in first pass and vbitrate in second pass, you use vbitrate in both pass... did I understand you well?. Ok, and you get better quality. Nice. And you adjust quantizers through the same settings in first and second pass, with vqmin:lmin:mbqmin... Looks nice.
In your settings I just miss the rate control mode: if you don't set vqcomp=1, default is vqcomp=0.5, and that isn't constant quality. Check if in your tests you used vqcomp=1, and if not (and if you have time :roll: ) redo your test with vqcomp=1 and tell us the output.
How does it look the first pass file?, is it veeeery big?, does it give quality enough to avoid second pass? I guess not, due to impossibility to respect vbitrate in 1 pass.
In this two pass tests, did you get the desired average bitrate?. Is it good the bitrate allocation?.
I see you're not using trellis quantitation, nor mbd=2. They made wonders in my 1pass tests. But maybe in 2pass we can drop them (at least trell, that slows down encoding.... or not that much?. Did you test with and without it?).
And, finally (by now :twisted: ), tell us something about the filters you're using. Maybe it isn't necessary at all to use scplx_mask.
Sorry if I ask you a lot, but I don't have now many time, and my computer is faaaar slooow (I'm taking profit of your tests :oops: )

Yes, you got all this right.
I'll try that. Actually I was revising Bilu settings just now, and discovered I forgot about rate control (vrc_eq), I was using the default settings. Using vrc_eq=AvgTex is sbout the same thing, since the original equation is Tex^vqcomp. I am also testing with naq=1;
Using vqscale, it's very big. Using vbitrate, the average asked is more or less achived (I ask 1500 and get 1490 or 1510). Actually there's no problem with vrc_maxrate, I just tested in both passes and it's working fine, I don't know what was broken before because I tested so many settings today...
1 Pass didn't look bad. But it does what you can expect from a encoder that doesn't know what comes next: smooth less in calm scenes and block more in action scenes. I think the second pass is worth it (you saw the screenshots)

No I didn't. I just used mainly bilu settings.
The filters I posted are for AVI only. They do post-processing, then a temporal and spatial denoise in chorma, and then add some uniform noise to avoid the DCT killed to come back. It's about what DialHot V4 script for AVI does (which was my inspiration). And of course there's the resizing.
I don't know exactly what scplx_mask is for, maybe it's not needed after all.
Be my guest. :D

Edit: just tested with naq and vrc_eq=AvgTex. It looks much worse, some scenes are completelly distorted.
Edit2: naq alone made some parts better, some other parts worse. I don't think it's worth using.
Edit3: vrc_eq=avgTex also helpend in some parts and made others worse .

I advise you to use vqcomp=1 or vrc_eq=tex (it's the same) that gives, in theory, the best quality. And I think that bilu tested it and came to this same conclusion. Not vrc_eq=avgTex.
And give mbd=2 and mv0=1 a shot. I don't dare to suggest you again trell... maybe I'll test by myself before asking you such a slow (in theory) thing.
Do you think you could do a similar work with the MA script?. I've been thinking about it, as there are spatial and temporal filters. But I don't find anything similar to undot, and don't see how can we make a motion-adaptive routine... Anyway, for KDVD, with DVD as a source, we don't usually use the MA script, but something smoother (like undot, STMedianfilter with low parameters, light TemporalSoftening,...).
You're really kind, man. You make me feel at home. :D

@vmesquita

Not all my mencoder testings are on the same thread ;)

My new settings, with BV screenshots **LATEST**
http://www.kvcd.net/forum/viewtopic.php?t=10132

Mencoder ratecontrol
http://www.kvcd.net/forum/viewtopic.php?t=10101

Set of 4 Mencoder tests: mind borrowing us some CPU? ;)
http://www.kvcd.net/forum/viewtopic.php?t=9759

Need help on becoming practical ;)
http://www.kvcd.net/forum/viewtopic.php?t=9260


Bilu

naq is only needed when using *_mask settings.

If you use lmin=2:scplx_mask=0.3 or lmin=1:scplx_mask=0.4 (maximum acceptable values on video for me, not anime) ***WITHOUT*** naq, you can considerably lower the bitrate. If you use naq, avg bitrate can become bigger than when not using *_mask settings at all :!:

About vrc_eq settings, I only use vrc_eq=tex now.


Bilu

Test proposal for 2-pass encoding:

1st pass without B-frames, 2nd with B-frames.

Since B-frames are generated from info from the previous and next frames it shouldn't affect quality and should be faster.

Inspired on XVID's Turbo Mode: 1-pass disable some search techniques used by QuarterPixel (MPEG-4 only) and B-frames.


Bilu

@digitall.doc
vrc_eq=tex improved quality. Finaly! Tested many things: naq=1, mbd=1, mbd=2, trellis, mbcmp=6, mbcmp=2, encoding without b-frames. They all make quality worse or slighlly better in some parts and worse in others. I'll try mv0=1 (looks like your tips are hot) :lol: :lol: :lol:
MA is not possible with the current filters, since you can't vary their streight according to the ammount of action, which is the idea of MA... But maybe we can find a good static filtering combo to get more compressibility.

@bilu
Thanks, looks like I missed some threads. :wink: I tried naq with scplx_mask=0.3:vqblur=0:mbqmin=1:lmin=2.49 and didn't like the results. I'll try "lmin=1:scplx_mask=0.4" as you suggested.
Doing first pass without bframes doesn't work: in the second pass bframes are not created :!:

Maybe I'm completely wrong and that's not way that the Turbo mode works :roll:

Bilu

Have a look here also:
http://www.kvcd.net/forum/viewtopic.php?p=70130#70130


Bilu

Thanx vmesquita, but as bilu says these are our tips, since we all have been testing all the parameters in mencoder to take the best out of it.
And is you follow the main mencoder thread, you'll see that we dropped and then revisited lots of parameters and values.
And you're now giving it a new "empurrâo" (push) with your 2pass tests.
BTW, the results you post are always related to avi sources, or you get the same results with DVD sources?.

@bilu,
Tried mvo=1, but as the manpage suggest, make no difference with mbd=0. So I tried mbd=2:mv0=1 and got a bad result. I am now going to try "lmin=1:scplx_mask=0.4" with my best settings so far to see if it improves.

@digitall.doc
I followed the mencoder thread for a while in the begging but later I stopped reading and participating, because I was having too many issues with mencoder, looked very instable to me in the past. Not anymore. :D
I am using a DVD source to do the tests (the Lawmmover Man trailer, great movie BTW), to make sure any artifacts are not coming from the source. :wink:

Hey vmesquita,

Take a look at the source code of the encoder. It's more interesting than anything the MAN pages will ever tell ;)

-kwag

@kwag
Great idea, I'll do that later, it should be very interesting. 8O 8)

@bilu
"lmin=1:scplx_mask=0.4" created a output with slighly more artifacts. I'll try lmin=1:scplx_mask=0.249 just to be complete. I think I am very near to my optimal settings! :D 8)

<edit>I made a mistake, I'll re-do the test.

Looks like the best ssettings (at least with the trailer I am testing) is really ":scplx_mask=0.3:lmin=1:"
Now a very interesting data: I encoded the same trailer, using CCE in 2-Pass VBR. CCE encode was not bad, but much inferior to mencoder. Much more DCTs and mosquitos. With this latest settings, artifacts are near gone in mencoder encode. Also CCE has problems with fades, which mencoder doesn't, this is very visible. I'll post some screenshots later.

Also I found out what was making prediction go crazy: looks like specifying vrc_minrate causes that (not max_rate). I'll test with other stuff to see if vbitrate is working fine when vrc_minrate is not used or this was just coincidence.

Thanks bilu and digitall.doc for all the tips and previous tests :D :D :D I'll update the first post with my final settings (for now). :wink:

Not coincidence, just made that same test over the same stream I used for the screenshots. Used my settings but removed vrc_minrate, it got to an average of 2868 kbps.

I'll make my tests all over again :roll:

Bilu

But what was your target? 3000? I am currectly encoding a bigger AVI (20 minutes, my test trailer was only 2 minutes long) to see what happens.

<edit> I checked the last thread you posted later (the one with BV screenshots), and seems that you are aiming for 3000. It's not bad for 1 pass. Also you have to take on account that sometimes it's impossible to rise bitrate, because mencoder is already using the lower quantisizer possible. This AVI I am testing now I saw this happening (opened in BV and confirmed the hypotesis). So I rised the target resolution from 352x480 to 480x480 and now, with 70% of first pass completed, average is 999! :D :D :D

<edit2> Bitrate in first pass: 971.55 Bitrate in Second Pass: 1000.143! Amazing! :D :D :D
<edit3>Just for fun, I did the whole "Lawnmover man" movie from DVD to a 352x480 KSVCD. I asked for 873 kbits, got 871 in the first pass and 873 in the second pass, with an average quantisizer of 2.34. And the bitrate has been respected, max in BV was 2873 (I asked for 2500), but we all know BV is not that accurate. To achive that, there has been quantization peaks of 13.08!!!

@VMesquita

Re-did a lot of testing.

Extensive tests without vrc_minrate
http://www.kvcd.net/forum/viewtopic.php?t=10229


Bilu

Well, if you take some conclusions looking at the source code, make us know, since I'm surprised with results I get from time to time, opposite (in a way) to what I expected...

BTW, already tested your compilation (run time version for me, P4), and is FAST: about 27 fps in each pass, just double of real-time in two-passes!!

@digitall.doc
And it can get even faster, if you could recompile it to support SSE and SSE2 (but not 3dNow!). I'll make a recompiling guide (or maybe I should do multiple compilations), just give me some time. :wink:

Very helpful (mainly the last) for those of us that are... how could I say, :dunce: :?:

Hi all,
I've been testing a lot lately with mencoder 2pass to encode KDVD.
With barebones command and setting, I get about 26 fps in each pass, that means I finish 2 pass in less time it lasted with 1 pass and my last settings (I was getting 12 fps as much). And really accurate final file size.
So, it's fast and accurate... but that's not all, of course. I've been doing my tests with StarWars II (1 h 16 min 40 sec), so to fit this film twice (2 films per media) with 2 audio tracks (1 ac3 5.1 and 1 ac3 2.0) and a subtitle file, I calculated an average bitrate about 1500. So I did my tests with vbitrate=1500. I know this may be an extreme case, but wanted to see the output.
With barebones command and setting, no filter, it was fast as reported. And when seen in PC doesn't look bad, very homogeneus. But it's not good at all, I see lots of microblocks (don't know if this is the correct term, but they're not macroblocks, and affect the whole image).
I tried adding mbd=2:mv0=1 and quantizers raised a bit, and I think I saw more gibbs effects (I think they are, I don't really know how a gibb looks like). Encoding slowed from 26 to 20 fps. And kept seeing microblocks.
I tried instead with trell=1:cbp=1, and quantizers lowered a bit, bitrate raised a bit (respected average bitrate), and little slow down in speed ( from 26 to 24). But still microblocks (I cannot say if little less).
Tried then with scplx_mask and tcplx_mask, but I think even worse.
I tried with trell=1:cbp=1:mbd=2:mv0=1, that slowed down a lot the speed (to about 16 fps) and didn't improve the microblock problem.
... so I'm afraid I took the best out of mencoder 2pass for KDVD and vbitrate=1500. Maybe it's an extreme case, and maybe this won't be noticeable later when seen on TV. But I would like to improve the output: do you know how?.
I got better results with 1pass, but of course with higher average bitrate and file sizes, that wouldn't fit what I'm asking know to mencoder 2pass.
Any ideas how can we improve this?

Don't take me wrong: I'm really very satisfied with mencoder 2pass, and I think that's the way. I just did a difficult test, and want to know it is there a way to make it look better.

oK vmesquita please make some things clear for me about 2-pass encoding with mencoder.

Here are my problems. When I encode a movie with vpass=2, mencoder will immediately close, without starting to encode.
When I set it to vpass=1, mencoder will create a .m2v file, together with a growing dvx2pass.log file.
When I abort that encode, set mencoder back to vpass=2, the encoder will start, but will only encode until that amount of percentage I already encoded with vpass=1 (I figure it crashes when it has read the divx2pass.log until the end).

Now please tell me how can I automate that process? where's the trick? I don't want to do a manual first pass and a manual second pass... :roll:

There's no way to run mencoder only once and make it do 2 passes. You have to call it twice. But you can automate the process easily by using my GUI ( that you already know, there's a new release out ) or creating a batch file. :wink:

Alright, now I understand. phew... :lol:

But why is mencoder creating an m2v file in first pass? isn't it supposed to write just a log for the second pass? :roll:

I just figured out: use -o NUL in the first time. This way it won't create the 1-pass file, just the logfile. I don't know if it's faster. :?

EDIT: A quick tes: looks like -o NUL is slower :!: Well, maybe was just this time. :D

Thank you VM...
This is getting fun now. I LOVE the quality this two-pass beast spits out. Ever thought about 3-pass? :lol:

This would be fun... :D Unfortunatelly, mencoder doesn't update the logfile in second pass so no 3-Pass is possible. :roll:

3-pass info:

http://www.mplayerhq.hu/DOCS/HTML/en...eat-divx4.html


Bilu

Be aware, that Bitrateviewer and its outputted Qunatizer doesn't stand for resulted "viewable" image quality! So only do compare the encoding using your eyes!

As already said, I did see that on tests this week using CCE 2.50. CCE2.6x, TmpgEnc and mencoder.
If you choose "linear qunatizer scale" in CCE 2.50 a "wonderful" q curve of 1 constantly! comes out where the bitrates also are ok. BUT it did look even worse than the same parameters and low VBR avg bitrate used in TmpgEnc but non linear Quantizer (tmpgEnc default) which resulted in a average Q of 3.5 !!! with peaks up to 6!!
So the output from Bitrateviewers Q curve does say something different ... maybe how the quality of the stream how it will be descrambled by the decoding and therefore the usage of the matrix posted in the m2v! Means perfect descrambling BUT does NOTHING say about the quantisation DURING the encoding ..... maybe thats the reason?!

Hi all.
Sorry me for being so impatient. I posted my tests with two passes in this same thread, previous page. And posted there some results and some problems I found.
Just wanted to know if is there any way :idea: to improve the results I got.