Codecs: Best MPEG2 Encoder for metrics objectives tests
 

~ Movie Metric Benchmark Challenge ~

You think that your MPEG2 coder is the best : prove that ... !!?


A - Introduction

The purpose of this challenge is to determine which is the best codec for the metrics and only for the metrics : this test will not speak about subjective visual quality. If you want subjective visual comparison download sample and compare yourself ...



B - Rules

Rule 1 : Source

You must use this source for the test : Harry Potter II trailer, 720*576 MPEG2 MP@ML 4.5 Mbps
HPII Part1 winrar, HPII Part2 winrar, HPII Part3 winrar, HPII Part4 winrar & HPII Part5 winrar


Rule 2 : Pre-process and encoding

You must use DGindex with 32-bit SSE/MMX iDCT algorithm for .d2v project and DGDecode.dll for MPEG2 decoding in avisynth scipt. You must use MPEG2 with DVD standard MPEG2 MP@ML Pal DAR 4/3 PAR 128:117 profil (edit ... lol). You must use this avisynth script for make encoding:

Code:

---

source=Mpeg2Source("D:\Mes dossiers\B.A\Harry Potter\azerty.d2v",idct=2)
source=Crop(source,4,76,-4,-76)
source=LanczosResize(source,720,320)
source=addborders(source,0,128,0,128)
Return(source)

---

Rule 3 : Bitrate

You must use these bitrate/size for encoding:

The first round use is very low bitrate for MPEG2 : maximum bitrate is 1800 Kbps for mini-DVD profil encoding.
Mini-DVD: Maximun 1800 Kbps for Video Elementary Stream or 28125 Ko for m2v file.


Rule 4 : Metric Tests

SSIM (YV12): IMO the best metric. it uses some HVS properties but not all ...
PDF description for VSSIM

Overall PSNR (YV12): Very good metric test, particullary for Rate Control Quality test ...
MSE = Sum t = 1 -> NFrame Sum x = 1 -> width Sum y = 1 -> height (F'(x,y,t)-F(x,y,t))² / (NFrame*width*height)
Overall PSNR = 10 * Log ( 255² / MSE)

Average PSNR (YV12):The worse metric test but Frame PSNR graph is very good to compare RC strategy ...
MSE = Sum x = 1 -> width Sum y = 1 -> height (F'(x,y,t)-F(x,y,t))² / (width*height)
Average PSNR = Sum t = 1 -> NFrame 10 * Log ( 255² / MSE) / Nframe


SSIM vs PSNR

"For a given reference video and a given compressed video, it is meant to compute a quality metric, based on perceived visual distortion. Unlike the well-known PSNR measure, it's not purely mathematical, and should correlate much better with human vision."

"A higher MSE (and so lower PSNR) should mean that the compressed clip is a worse image but MSE and PSNR are flawed in this respect as numerous tests have shown. However with SSIM, according to tests carried out on the VQEG dataset, a higher Q (SSIM value) has a much better relation to the visual quality of the compressed clip. Despite this, bear in mind the SSIM metric still isn't perfect."

Here last AviSynth plugin for VSSIM in YV12


Average PSNR vs Overall PSNR

It's easy to increase artificially Average PSNR with Rate Control tweak. In certain case CBR (0% for variability) will be better than real VBR (100% for variability or constant quant). It's generaly not possible with Overall PSNR. Overall PSNR is very good to test Rate Control quality. If difference between Average PSNR and Overall PSNR is very high, "local quality difference" (difference between Pframe or Bframe for exemple) or "scene quality difference" (difference between low motion and high motion for exemple) will be too high and Rate Control for this codec not very good.

Here last AviSynth plugin for Overall PSNR in YV12


Open your avs metric script with VirtualDub. Select file > preview input for run the test. Select file > close video file for obtain final result in log metric files. You must use these AviSynth type script for metric test:

Code:

---

# --> Video Opening <--

source=Mpeg2Source("D:\Mes dossiers\B.A\Harry Potter\azerty.d2v",idct=2)
source=Crop(source,4,76,-4,-76)
source=LanczosResize(source,720,320)
source=addborders(source,0,128,0,128)

video=Mpeg2Source("D:\Mes dossiers\B.A\Harry Potter\qwaerty.d2v",idct=2)

# --> PSNR analysis <--
compareYV12(video,source,"YUV","PSNR-MPEG2.log")

---

Code:

---

# --> Video Opening <--

source=Mpeg2Source("D:\Mes dossiers\B.A\Harry Potter\azerty.d2v",idct=2)
source=Crop(source,4,76,-4,-76)
source=LanczosResize(source,720,320)
source=addborders(source,0,128,0,128)

video=Mpeg2Source("D:\Mes dossiers\B.A\Harry Potter\qwaerty.d2v",idct=2)

# --> SSIM analysis <--
SSIM(source,video,"results.csv","SSIM-MPEG2.log",lumimask=2)

---

C - Results

At this time here the best result ... but done better if you can ... :devil:

MPEG2 MP@ML : TMPGEnc encoding by Sagittaire
File: MPEG2 1800 Kbps

MPEG2 MP@ML : QEnc encoding by Sagittaire
File: MPEG2 1800 Kbps

MPEG2 MP@ML : Mencoder encoding by Sagittaire
File: MPEG2 1800 Kbps

MPEG2 MP@ML : FFDShow encoding by Sagittaire
File: Not available

MPEG2 MP@ML : HC015 encoding by hank315
File: Not available


D - Conclusion

At this time for this trailer

The best is Mencoder encoding by Sagittaire

IF YOU DON'T AGREE WITH THAT, TRY TO MAKE BETTER WITH YOUR FAVORITE ENCODER ... !!!


Annexe - Update

15.08.05 : upadate with TMPGEnc KDVD "6 hours" profil
14.08.05 : Sagittaire with QEnc, Mencoder and FFDShow
14.08.05 : hank315 with HC015

Re: [Challenge] Best MPEG2 Encoder for metrics objectives te
 

I don't agree. Video quality is not maths. Video quality is subjective.

BTW there are too much things you don't tell in the condition of your own test. Even if "MPEG2 with DVD standard MPEG2 MP@ML Pal DAR 4/3 PAR 12:11 profil" should fix things like matrix, GOP and DC prec, it does not tell anything about open GOP or not, auto I-frame insertion (scene detect), min bitrate...

But your post is nice for people that didn't know how to use metrics plugins. Thanks.

SSIM use HVS system for exemple ... but this test only speak about metric.


MPEG2 with DVD standard MPEG2 MP@ML Pal DAR 4/3 PAR 12:11 profil
- open/closed GOP are compliant with DVD MP@ML PAL standard: you can use the best for you
- custum matrix are compliant too : you use the best for you
- GOP PAL compliant : you must use GOP max 15 frames with 2 bframes no adaptatives
- DC prec is compliant too : 8, 9 or 10 ... use your best DC

From where did you get the value PAR 12:11 ?
Standard PAL DVD 720x576 playback is @ 13.5mhz using PAR 128:117

12:11 or 128:117 don't change anything for my eyes and certainely for your eyes ... lol

mencoder, ffdshow, WM9Encoder ... use 12:11 PAR for PAL DAR 4:3. But it's perhabs not exactly the good value ... I don't know and it's not important for this test. Use 128:127 if you want ...

1) Update with libavcodec core class codec (Qenc, Mencoder, FFDShow)

2) Many user think that CCE or Canopus Pro coder are the best (I don't think ... IMO Libavcodec is the best for MPEG2) prove that with this trailer ...

It's also the worst compliant encoded MPEG2 stream in the world, which causes many incompatibilities in standalone players :!:
It's full of spikes, underruns, overrruns, etc. That's why we dumped it here, until further notice.
You should take a look here: http://www.kvcd.net/forum/viewtopic.php?t=16073 for some insights about visual quality ;)
We don't need to proove anything with any trailer :cool:

-kwag

Really :?:
Any change in target aspect ratio, different from the source, will scale the image incorrectly causing jagged edges and other artifacts, gibbs, moire patterns, etc., and that WILL produce visual degraded picture on some scenes.

-kwag

As I said in my first post, I do not agree at all with this "contest" (to say the truth, in my first post I wrote "this is a loss of time" but erased it afterwards).
But as long as all encoder are treated the same way, this problem introduced by a wrong PAR is not important if you do not consider the visual quality.

And if any "contest" was to be done, then use production class encoders like CCE, TMPEG, ProCoder, etc., and not crap like all avcodec based encoders.

-kwag

really ... lol

"12:11 (=1.0909) vs 128:117 (=1.0940) can will scale the image incorrectly causing jagged edges and other artifacts, gibbs, moire patterns, etc" ... you are sure ... !!! Perhabs with DAR error but not with PAR error. I use PAR only in resizing process with lanczos4resize (1% approximation here) and not in encoding process (I choose standardised DAR value in encoder ... 4/3 PAL or 16/9 PAL ...)

Anyway it's not important in my test (dgindex decode MPEG2 without PAR or DAR but in square 1:1 formatp pixel) and you must use my avs script for encoding ...



I don't know ... but it is not a challenge for standalone players compatibility. You use perhabs adaptative quantisation, psy masking or too long motion vectors in your encoding with libavcodec ... ???



QEnc, FreeEnc, FFDShow, ffmeg ... are encoder based on Libavcodec like mencoder ... ???



In fact if metric are better for all frame of this trailer (3000 frames tested) then metric with other source (150 000 frames or more) will done similar result. When there is a great difference it is not necessary to make other test: TMPEnc will be always worse for the metrics for all the sources than you will be able to choose. Anyway this challenge is only for this HPII trailer ...



And ... ???

1) With good source and adaptative avs script I can make encoding in HDTV 720p (1280*720*60) or 1080i with high quality and low bitrate:

Batman begin trailer
1280*720*60 MPEG2 MP@HL 4000 Kbps 1,1,1,2,2 image sequence
TMPGEnc CQ 65 with Sagittaire matrix

2) screenshots don't prove anything (I, P or B bframe) ... with screenshots I can prove that shrink is the best MPEG2 coder in the world ... lol

... as with metrics :roll:

IMHO no contest can determine the best encoder. It can just determine which is the worst (generally all people agree on the bad quality of worst encoder - but there is not two identical advices about the best one).

speak about that with video codec developper and with Babayaga for exemple ...

but I repeat ...

I can read. Do you ?
Note: it seems you don't agree yourself with your own context. Just put two lines of your posts side by side :
and... kidding ?
Karl try to explain you that no libavcodec based encoder respect the MPEG2 MP@ML defined standard and that is why lot of encoders can't read their results.

:arrow: You are comparing things that respect the rules of your contest and other that don't. It's like allowing drug usage in a 100m contest.

Seems you haven't been around lately? :)
Yes, ALL are based on avcodec, and so is Mencoder, which is simply a front end to avcodec :!: Now try that with KVCD Notch matrix, and you'll be able to drop the bitrate by at least 1,000Kbps ;) If done properly, and if they are taken from high action scenes, they do prove good visual comparison.

-kwag

I thought Hank's HC Encoder was not based on the libavcodec but instead written from scratch by Hank in the programming language called Fortran?

Yes it is.

In fact I prefer my Sagittaire matrix:
1) better intra / inter repartition for my eyes (only for my eyes here ... lol)
2) better global / HF repartition for my eyes (only for my eyes here ... lol)


In fact it's impossible for video codec comparison
1) Codec A and Codec B must have the same frame type frame for same frame
2) Codec A and Codec B must have the same quant for same frame
3) Codec A and Codec B must have the same RC strategie


You must use DVD PAL profil (gop, no adaptative bframe, 2 bframe, VBV specification ...) but it's no compatibility challenge ... I test MPEG2 codec quality and not standalone players compatibility. Libavcodec work certainely with large majority of standalone players compatibility ...

Nero Burning Room is the most used (IMO the best) CD/DVD burner in the world but perhabs not the most compatible for standalone players (Libavcodec work with my old DVD player but not Nero Burning ... lol)

Lol, that is a litle vague there Phil. Is that yes it is written in Fortran or yes it is based on Libavcodec, oooorrrr. . . Both!. :D :lol:

Nope. It is very possible.
Just compare I to I, B to B, and P to P from each encoder on a same segment of footage ;) Now that will be encoder dependant, and the best encoder will give the better quantization.Again, no :!:
Every encoder will be different, so each will have different constraints applied during conversion.
In the end, the visual quality is what counts.
And again, in the I to I, B to B, and P to P, you can compare encoder X (I frame) for the same frames on encoder Y by looking at the I, B and P
specially after a scene change, where there is less compression.
That is, compare the first header of an MPEG GOP, which should be IBBP.
Then you can compare those four frames to another encoder.
Just make sure the frames you are comparing on both encoders are truly splitted on a scene detect.

-kwag

:lol: It is written from scratch in Fortran.

Anyway if you think that TMPGEnc or other encoder could be better with your setting (custom matrix, open/close GOP, DC ...) post your result ... and make visual comparison if you want ...

For exemple TMPEGEnc target bitrate is not very good and can be better I think ...

Thanks for clearing that up Phil. :D

libavec codec does not even respects the MPEG2 definition provided by the Motion Expert Group. But if you can't understand that I can't help.
I let you read that FAQ where you can learn a lot about MPEG2 encoders :
http://www.tns.lcs.mit.edu/manuals/mpeg2/FAQ
Jump to question 49 :

Sorry to drop this thread there. I use to encode to burn the result on a DVD (and read it afterwards), not just to use my PC.

For the last time it's not standalone players compatibility ... If you think that TMPGEnc is compatible with more standalone players than QEnc ... speak about this problem with libavcodec developper. Anyway I think that Libavcodec (and particulary QEnc) is compatible with the large majority of players ( I test 12 differents player with Libavcodec and no problem ... perhabs with the number 13 ... lol)

but if that can please to you ... lol
Libavcodec : 99%, 90% ... 10% (perhabs less ... i don't know)
TMPEGEnc : 99.99% (perhabs more ... i don't know)

For the last time it's not standalone players compatibility test ... :lol:

I CAN READ ! Are you sure you can ???
Let my put that in different words : libavcodec do not produced mpeg2 streams ! It is not an MPEG2 encoder
You want to compare MPEG2 codecs, don't you ? Libavec codec is not.
I don't need to speak about this with them. They wrote in plain letter in the doc that the lib should not be use to produce mpeg ! Read the doc !
Do I have to discuss that with all user of the lib that can't read a manual ?
(unfortunally mencoder docs is not online anymore, but you surely have a doc on your disc if you use it).

Something "not compatible" does not say exactly something never accepted. Even on these 12 players, you would/will suffer problems with some streams encoded with libavcodec.

You can put 100% for tmpgenc. And something below this is NOT compliant.

What do please me is that you find this "laughtable". Let me laught my turn :D.
You make me remind about these days when "compatible PC" were classified according to their "compatibility percent" against IBM PS1. Under 95% there weren't considered as compatible. Above there were.
:arrow: but that means that with a PC that was "97%" compatible only, potentially 3% of the software didn't work on it. Would you call that a "PC" if you own it nowaday ? Can you imagine buy a computer that can run only 99%, 90%... of available programms ?

For the last time : you are just loosing your time. But it seems you don't mind :D :lol: :rotf:

Let me ask you a question : you say you fix the maximum bitrate but all the encoders give you a file that is almost the same size. Are you sure you are not fixing the average bitrate ?

Edit: I'm convinced that you are mistaken average bitrate and maximum bitrate in all the thread. No problem, that occurs often. This just make me laught a little more... :P

I just want to add also something :
The problem is that the metrics give the victory to DVDDecrypter :rotf:
(metrics here are PIE/PIF)

Libavcodec seem able to encode with a large variety of codec (MPEG1, MPEG2, DivX3 or MPEG4 ASP). Libavcodec seem able to produce DVD compliant MPEG2 stream. I don't know if an "MPEG2 analyse flux" exist but my Mencoder.m2v seem compliant. QEnc 0.61 seem produce DVD compliant MPEG2 stream too ...


the challenge your best encoding with less that 1800 Kbps or 28125 Ko for m2v files. 1800 Kbps / 28 125 Ko are the maximum limte bitrate.

It's a challenge : if you can make better encoding with TMPGEnc (surelly possible) you can make test.
1) perhabs with better target bitrate (1787 for TMPEGEnc.m2v)
2) perhabs with better matrix for you

I bolded all importants words in your sentense. I won't add other things.

Note: I used to work on MPEG2 standartization when it was in the labs 15 years ago. I know a little about what is compliant and what is not. Even if I lost a lot of my knowledge now :(

Note2 : at this time I was one of the "lab rats" that did subjective tests to validate what "objective tests" found. And trust me... human eyes and maths equations are not good companion in this domain. Thousands of dollars have been put in these subjectives tests, because they were absolutly necessary.

Okay, so please use correct words because this is very confusing :
"1800 Kbps / 28 125 Ko are the maximum limte average bitrate".
"maximum bitrate" is an other thing and first read of your contest conditions gives the idea that you limit the "maximum" birate to 1800.

I don't have time for that right now (my two computers are encoding 24/7 since two weeks) but asap I will test some idea I have aimed to have the best result for the metrics whatever the quality I obtain in the end

Update with KDVD PAL "six hours" TMPGEnc Template CQ 67 highest quality and 1803 Kbps ... not very better for metric and more blur for my eyes.


Really ... lol

Questions Answers with Nic - QEnc developper

Sagittaire to Nic
"QEnc (and libavcodec) make MPEG2 DVD compliant stream ?"

Nic to Sagittaire
"Yes they do. The compatibility from libavcodec has a got a lot better over the last year or so due to some good patches."

So if you don't prove the opposites ... for me:
1) The encodings make with QEnc and libavcodec are DVD compliant. If your encoding are not compliant it's because you don't use the good setting.
2) QEnc and libavcodec are certainely very better and by far than TMPGEnc for metric (more than 1.5 dB for OPSNR and more than 2.5 for SSIM ... it's a very high difference for this quality level) in all situations for all MPEG2 DVD compliant stream (not only with HPII trailer)


http://www.cns.nyu.edu/~zwang/files/papers/vssim.pdf
Perhabs in your labs in 15 years ago ... lol. Speak about that with video codec developpers or better with SSIM developpers ... not with me. Anyway and for the last time my challenge is objective test and not subjective test ...

The purpose of this challenge is to determine which is the best codec for the metrics and only for the metrics : this test will not speak about subjective visual quality. If you want subjective visual comparison download sample and compare yourself ...
TMPGEnc 1800 Kbps
QEnc 1800 Kbps

If you want to make tests where Quality is the point then you should consider all specs like the ITU PAL par as 128/117. ok ... (as I do understand your test as a max. possible quality getting one and as YOU are the one who did set the rules)

I (was) a fan of libavcodecs mpeg2 encodings and my 20-30 latest mencoder full encodings from the last two month using special parameters did keep the min/max's.
But what about encoding #31 ? Im even waiting for an encoding incl. a peak ;) (purpose of my tests: watching if some developement on that has been made by the ffmpeg group)

But I would NEVER lay my hands on a general constant compilant mpeg2 stream output support of mencoder. My SAP supports it 100%, but its just imho no subject for a community where people do have that much individual SAP brands where about 10% do have problems playing back them. THATs the reason why we dropped mencoder big discussions in here. But to be fair - I dont know what has been changed on that core since then!

Use Bitrateviewer and you'll see drops and peaks of bitrates and Quantizers.
In Qenc Nic integrated XVID Ratecontrol for obtaining more compilant outputs - but afaik problems do still exist. Petercheat was the only one who went into the main RC core of Libavcodec and did some impressive mods, but no further developing was made as he seems to be off.

It seems that mencoder Group or ffmpeg.org did integrate the new Ratecotrol, but anyway, its not an mpeg2 encoder for general 100% confiable mpeg2 compilant encoding purposes (just imho)

Do make your tests also on movies whch contain parts of zooming to a tv screen incl. a hughe amount of tv noise like "Poltergeist" ... these parts are the best food for testing encoders spikes/peaks.

To clearify this: I didnt tested Qenc since 1-2 Month so I do not dement Nics words. But it seems you do tests using your own SAP, but youre not shure whats going on within Qenc etc. thats why you have to ask Nic. ;)

don't change anything I wrote 128:127 if you want (edit) ... I use PAR only in avisynth rezise pre-process and approximation is here only to have absolute mod16 resolution for real image (720*320 here). I use standardised DAR (4/3 PAL) only for MPEG2 encoding. Anyway don't change anything for my test because dgindex don't use DAR but square pixel 1:1 and you must use my avysynth script for encoding ...


Perhabs peak problem for 6-8 Mbps encoding but generaly not for 1,8 Mbps Average encoding. For a good security with peakbitrate use maxbitrate = 8000 or less if you want with QEnc in 2 pass mode. Anyway for my Mencoder.m2v peakbitrate is 5760 Kbps (ifoedit information stream). But it's true ... it's very difficult to use mencoder in real situation. QEnc is very better for that with very good 2 pass mode and very good quality ...

@Sagittaire,

Please stop the trolling :!:
avcodec DOES NOT create compliant MPEG2 files.
The developers of avcodec acknowledge it, and it's clearly documented in their documentation. And we can testify to that.
QuEnc is just a front end to avcodec. Nothing more :!:
No matter what QuEnc does, FreeEnc does, NuEnc does, the final product is the same. :arrow: trash :!:
They ALL rely on the same library.
:arrow: Do you understand now :?:
For me, this thread is over (and useless :!:)

-kwag

Karl, you must have read my mind, I was going to suggest locking this thread as to me it is pointless and serves no purpose to us here as we all know the issues regarding Libavcodec based encoders.

Our time is better spent helping others who need to learn KVCD not wasting our time debating with Sagittaire about whether Libav encoders can produce compliant Mpeg2 streams when we all know they cant. :)

report bug at Nic if you are specific problem ... perhaps that you have good information for him ...

Hanck315 (MPEG2 HC developper) check this files and say "they are compliant". I use me_range=128 with mencoder for make MPEG2 compliant stream:

here little demo with mencoder:
Serenity Trailer
Video:
720*576 16/9 PAL format 2.35
Real image mod8 720*432 with mod8 720*72 blackborders
MPEG2 MP@ML 3000 Kbps highest quality
Audio:
DD 2.0 16bits 48Khz
AC3 224 Kbps

if you have problem with this trailer with your hardware player send me a PM.

I've been reading this thread for 15 minutes or so, and I don't know if I wanna cry or if I wanna laugh.
It seems to me that we've been, once again, comparing apples with oranges...
@Sagittaire
I was one of the "believers" in Mencoder and libavcodec when we found out about it here at kvcd.net.
I did hundreds of tests with Mencoder in the old days.
And the same goes for QuEnc, NuEnc and every xxEnc based on libavcodec.
I had inumerous arguments with many kvcd members trying to defend libavcodec.
And I did that until one day I opened my eyes and saw that even though it's amazingly fast and has an amazing bitrate allocation, Mencoder compliancy is less than mediocre.
Bitrate spikes occur so often on noisy sources that I couldn't believe my eyes when I was looking at bitrate viewer :(.
Because I didn't want to quit on Mencoder too soon I decided to write on the mailing lists to see if the developpers could help.
And then I saw what the problem was: even the developpers didn't know that much about Mpeg1/Mpeg2.
Worst than that: they didn't care about Mpeg1/2.
They were simply looking for Mpeg4 and not much more than that :(.
That was when I rest my case with Mencoder/libavcodec because the developpers themselves told me that the problems affecting Mencoder were also present on libavcodec.
I returned to re-test libavcodec encoder some time ago, when Nic included the Xvid ratecontrol in QuEnc.
But what would you know? Even with Xvid R/C QuEnc didn't seem able to limit the bitrate spikes...
The best I ever saw around was actually NuEnc where PeterCheat rewrote many bits of the libavcodec R/C.
But unfortunately, Peter left when he was about to finally fix it 100%.
I guess the developpers didn't like the changes he made when he tried to submit his sources.
Bottom line, if you want to use Mencoder or any other libavcodec frontend be my guest.
But then, one day you might want to take one of your movies to one of your friend's place to watch it after dinner or something similar and :arrow: you might find yourself unable to play it on your friend's player even thought it's a branded product.
Just my oppinion.
I'll stick to HCEnc for the moment being.
Cheers

This is the nicer post I read since a long time.

Note : be carefull Rui, some people reported unexplained problems with HC on their SPA.

I can sign all that rds_correia say as myself did hundred of test with mencoder.
Also I did give up, tired when I saw developpers didn't fix rc with each new release.
:(

Hya Phil :),
Well I was just being honest because this is exactly what happened to me.
And once again you were right and I was wrong since you were telling us we were wrong right since the begining of the testings and we just didn't listen to you.
And now I'm worried about what you are telling me.
I can recall some weeks ago Karl also advised me that some folks can't play HCEnc's output on their SAPs.
Would you know a thread where I can find more info on that?
Cheers :)