Codecs: Best MPEG2 Encoder for Metrics Objectives Tests

Sagittaire · #1 08-13-2005, 05:46 PM

~ Movie Metric Benchmark Challenge ~

You think that your MPEG2 coder is the best : prove that ... !!?

A - Introduction

The purpose of this challenge is to determine which is the best codec for the metrics and only for the metrics : this test will not speak about subjective visual quality. If you want subjective visual comparison download sample and compare yourself ...

B - Rules

Rule 1 : Source

You must use this source for the test : Harry Potter II trailer, 720*576 MPEG2 MP@ML 4.5 Mbps
HPII Part1 winrar, HPII Part2 winrar, HPII Part3 winrar, HPII Part4 winrar & HPII Part5 winrar

Rule 2 : Pre-process and encoding

You must use DGindex with 32-bit SSE/MMX iDCT algorithm for .d2v project and DGDecode.dll for MPEG2 decoding in avisynth scipt. You must use MPEG2 with DVD standard MPEG2 MP@ML Pal DAR 4/3 PAR 128:117 profil (edit ... lol). You must use this avisynth script for make encoding:

Code:

source=Mpeg2Source("D:\Mes dossiers\B.A\Harry Potter\azerty.d2v",idct=2)
source=Crop(source,4,76,-4,-76)
source=LanczosResize(source,720,320)
source=addborders(source,0,128,0,128)
Return(source)

Rule 3 : Bitrate

You must use these bitrate/size for encoding:

The first round use is very low bitrate for MPEG2 : maximum bitrate is 1800 Kbps for mini-DVD profil encoding.
Mini-DVD: Maximun 1800 Kbps for Video Elementary Stream or 28125 Ko for m2v file.

Rule 4 : Metric Tests

SSIM (YV12): IMO the best metric. it uses some HVS properties but not all ...
PDF description for VSSIM

Overall PSNR (YV12): Very good metric test, particullary for Rate Control Quality test ...
MSE = Sum t = 1 -> NFrame Sum x = 1 -> width Sum y = 1 -> height (F'(x,y,t)-F(x,y,t))² / (NFrame*width*height)
Overall PSNR = 10 * Log ( 255² / MSE)

Average PSNR (YV12):The worse metric test but Frame PSNR graph is very good to compare RC strategy ...
MSE = Sum x = 1 -> width Sum y = 1 -> height (F'(x,y,t)-F(x,y,t))² / (width*height)
Average PSNR = Sum t = 1 -> NFrame 10 * Log ( 255² / MSE) / Nframe

SSIM vs PSNR

"For a given reference video and a given compressed video, it is meant to compute a quality metric, based on perceived visual distortion. Unlike the well-known PSNR measure, it's not purely mathematical, and should correlate much better with human vision."

"A higher MSE (and so lower PSNR) should mean that the compressed clip is a worse image but MSE and PSNR are flawed in this respect as numerous tests have shown. However with SSIM, according to tests carried out on the VQEG dataset, a higher Q (SSIM value) has a much better relation to the visual quality of the compressed clip. Despite this, bear in mind the SSIM metric still isn't perfect."

Here last AviSynth plugin for VSSIM in YV12

Average PSNR vs Overall PSNR

It's easy to increase artificially Average PSNR with Rate Control tweak. In certain case CBR (0% for variability) will be better than real VBR (100% for variability or constant quant). It's generaly not possible with Overall PSNR. Overall PSNR is very good to test Rate Control quality. If difference between Average PSNR and Overall PSNR is very high, "local quality difference" (difference between Pframe or Bframe for exemple) or "scene quality difference" (difference between low motion and high motion for exemple) will be too high and Rate Control for this codec not very good.

Here last AviSynth plugin for Overall PSNR in YV12

Open your avs metric script with VirtualDub. Select file > preview input for run the test. Select file > close video file for obtain final result in log metric files. You must use these AviSynth type script for metric test:

Code:

# --> Video Opening <--

source=Mpeg2Source("D:\Mes dossiers\B.A\Harry Potter\azerty.d2v",idct=2)
source=Crop(source,4,76,-4,-76)
source=LanczosResize(source,720,320)
source=addborders(source,0,128,0,128)

video=Mpeg2Source("D:\Mes dossiers\B.A\Harry Potter\qwaerty.d2v",idct=2)

# --> PSNR analysis <--
compareYV12(video,source,"YUV","PSNR-MPEG2.log")

Code:

# --> Video Opening <--

source=Mpeg2Source("D:\Mes dossiers\B.A\Harry Potter\azerty.d2v",idct=2)
source=Crop(source,4,76,-4,-76)
source=LanczosResize(source,720,320)
source=addborders(source,0,128,0,128)

video=Mpeg2Source("D:\Mes dossiers\B.A\Harry Potter\qwaerty.d2v",idct=2)

# --> SSIM analysis <--
SSIM(source,video,"results.csv","SSIM-MPEG2.log",lumimask=2)

C - Results

At this time here the best result ... but done better if you can ... :devil:

Code:

|-----------------|---------|---------|---------|---------|
| Codec           | Bitrate |   Size  |  OPSNR  |  SSIM 2 |
|-----------------|---------|---------|---------|---------|
| TMPGEnc         |  1803   |  28130  |  45.26  |  80.67  |
| CCE             |  1800   |  28179  |  45.83  |  78.76  |
| Canopus         |  1786   |  27464  |  44.50  |  81.20  |
| Mainconcept     |  1800   |  28200  |  45.21  |  80.54  |
| HC              |  1801   |  28252  |  46.24  |  81.53  |
| QuEnc (lavc)    |  1800   |  28210  |  46.94  |  83.00  |
| mencoder (lavc) |  1801   |  28207  |  47.19  |  83.77  |
| AQMatEnc (lavc) |  1800   |  28244  |  46.72  |  82.97  |
|-----------------|---------|---------|---------|---------|
SSIM 0: Lumimask Off
SSIM 2: Lumimask On (One2Tech Patch)

MPEG2 MP@ML : TMPGEnc encoding by Sagittaire
File: MPEG2 1800 Kbps

MPEG2 MP@ML : QEnc encoding by Sagittaire
File: MPEG2 1800 Kbps

MPEG2 MP@ML : Mencoder encoding by Sagittaire
File: MPEG2 1800 Kbps

MPEG2 MP@ML : FFDShow encoding by Sagittaire
File: Not available

MPEG2 MP@ML : HC015 encoding by hank315
File: Not available

D - Conclusion

At this time for this trailer

The best is Mencoder encoding by Sagittaire

IF YOU DON'T AGREE WITH THAT, TRY TO MAKE BETTER WITH YOUR FAVORITE ENCODER ... !!!

Annexe - Update

15.08.05 : upadate with TMPGEnc KDVD "6 hours" profil
14.08.05 : Sagittaire with QEnc, Mencoder and FFDShow
14.08.05 : hank315 with HC015

Dialhot · #2 08-13-2005, 06:08 PM

Quote:

Originally Posted by Sagittaire

this test will not speak about subjective visual quality
...
IF YOU DON'T AGREE WITH THAT, TRY YOUR YOUR ENCODER ... !!!

I don't agree. Video quality is not maths. Video quality is subjective.

BTW there are too much things you don't tell in the condition of your own test. Even if "MPEG2 with DVD standard MPEG2 MP@ML Pal DAR 4/3 PAR 12:11 profil" should fix things like matrix, GOP and DC prec, it does not tell anything about open GOP or not, auto I-frame insertion (scene detect), min bitrate...

But your post is nice for people that didn't know how to use metrics plugins. Thanks.

Sagittaire · #3 08-13-2005, 06:47 PM

Quote:

The purpose of this challenge is to determine which is the best codec for the metrics and only for the metrics : this test will not speak about subjective visual quality. If you want subjective visual comparison download sample and compare yourself ...

Quote:

Originally Posted by babayaga (Nero/ateme MPEG4 AVC developper ... the best codec in the world)
PSNR at constant quantizer gives a good idea of the raw efficiency since this is what core encoder try to optimize most of the time.
MPSNR/OPSNR add some hints about rate-control and SSIM (JND metrics or whatever) add clues about adaptive quantization.

A codec which is inferior is those 3 metrics will almost surely be seen as subjectively inferior.

Besides, subjective testing is also very difficult to do and is not always reliable.

SSIM use HVS system for exemple ... but this test only speak about metric.

Quote:

BTW there are too much things you don't tell in the condition of your own test. Even if "MPEG2 with DVD standard MPEG2 MP@ML Pal DAR 4/3 PAR 12:11 profil" should fix things like matrix, GOP and DC prec, it does not tell anything about open GOP or not, auto I-frame insertion (scene detect), min bitrate...

MPEG2 with DVD standard MPEG2 MP@ML Pal DAR 4/3 PAR 12:11 profil
- open/closed GOP are compliant with DVD MP@ML PAL standard: you can use the best for you
- custum matrix are compliant too : you use the best for you
- GOP PAL compliant : you must use GOP max 15 frames with 2 bframes no adaptatives
- DC prec is compliant too : 8, 9 or 10 ... use your best DC

incredible · #4 08-14-2005, 04:48 AM

Quote:

MPEG2 with DVD standard MPEG2 MP@ML Pal DAR 4/3 PAR 12:11 profil

From where did you get the value PAR 12:11 ?
Standard PAL DVD 720x576 playback is @ 13.5mhz using PAR 128:117

Sagittaire · #5 08-14-2005, 08:33 AM

12:11 or 128:117 don't change anything for my eyes and certainely for your eyes ... lol

mencoder, ffdshow, WM9Encoder ... use 12:11 PAR for PAL DAR 4:3. But it's perhabs not exactly the good value ... I don't know and it's not important for this test. Use 128:127 if you want ...

Sagittaire · #6 08-14-2005, 09:12 AM

1) Update with libavcodec core class codec (Qenc, Mencoder, FFDShow)

2) Many user think that CCE or Canopus Pro coder are the best (I don't think ... IMO Libavcodec is the best for MPEG2) prove that with this trailer ...

kwag · #7 08-14-2005, 10:15 AM

Quote:

Originally Posted by Sagittaire

... IMO Libavcodec is the best for MPEG2) prove that with this trailer ...

It's also the worst compliant encoded MPEG2 stream in the world, which causes many incompatibilities in standalone players

It's full of spikes, underruns, overrruns, etc. That's why we dumped it here, until further notice.
You should take a look here: http://www.kvcd.net/forum/viewtopic.php?t=16073 for some insights about visual quality

We don't need to proove anything with any trailer

-kwag

kwag · #8 08-14-2005, 10:19 AM

Quote:

Originally Posted by Sagittaire

12:11 or 128:117 don't change anything for my eyes and certainely for your eyes ... lol

Really

Any change in target aspect ratio, different from the source, will scale the image incorrectly causing jagged edges and other artifacts, gibbs, moire patterns, etc., and that WILL produce visual degraded picture on some scenes.

-kwag

Dialhot · #9 08-14-2005, 11:12 AM

Quote:

Originally Posted by kwag

Any change in target aspect ratio, different from the source, will scale the image incorrectly causing jagged edges and other artifacts, gibbs, moire patterns, etc., and that WILL produce visual degraded picture on some scenes.

As I said in my first post, I do not agree at all with this "contest" (to say the truth, in my first post I wrote "this is a loss of time" but erased it afterwards).
But as long as all encoder are treated the same way, this problem introduced by a wrong PAR is not important if you do not consider the visual quality.

kwag · #10 08-14-2005, 11:18 AM

Quote:

Originally Posted by Dialhot

But as long as all encoder are treated the same way, this problem introduced by a wrong PAR is not important if you do not consider the visual quality.

And if any "contest" was to be done, then use production class encoders like CCE, TMPEG, ProCoder, etc., and not crap like all avcodec based encoders.

-kwag

Sagittaire · #11 08-14-2005, 12:15 PM

Quote:

Originally Posted by kwag

Quote:

Originally Posted by Sagittaire

12:11 or 128:117 don't change anything for my eyes and certainely for your eyes ... lol

Really

Any change in target aspect ratio, different from the source, will scale the image incorrectly causing jagged edges and other artifacts, gibbs, moire patterns, etc., and that WILL produce visual degraded picture on some scenes.

-kwag

really ... lol

"12:11 (=1.0909) vs 128:117 (=1.0940) can will scale the image incorrectly causing jagged edges and other artifacts, gibbs, moire patterns, etc" ... you are sure ... !!! Perhabs with DAR error but not with PAR error. I use PAR only in resizing process with lanczos4resize (1% approximation here) and not in encoding process (I choose standardised DAR value in encoder ... 4/3 PAL or 16/9 PAL ...)

Anyway it's not important in my test (dgindex decode MPEG2 without PAR or DAR but in square 1:1 formatp pixel) and you must use my avs script for encoding ...

Quote:

It's also the worst compliant encoded MPEG2 stream in the world, which causes many incompatibilities in standalone players

I don't know ... but it is not a challenge for standalone players compatibility. You use perhabs adaptative quantisation, psy masking or too long motion vectors in your encoding with libavcodec ... ???

Quote:

It's full of spikes, underruns, overrruns, etc. That's why we dumped it here, until further notice.

QEnc, FreeEnc, FFDShow, ffmeg ... are encoder based on Libavcodec like mencoder ... ???

Quote:

We don't need to proove anything with any trailer

In fact if metric are better for all frame of this trailer (3000 frames tested) then metric with other source (150 000 frames or more) will done similar result. When there is a great difference it is not necessary to make other test: TMPEnc will be always worse for the metrics for all the sources than you will be able to choose. Anyway this challenge is only for this HPII trailer ...

Quote:

You should take a look here: http://www.kvcd.net/forum/viewtopic.php?t=16073 for some insights about visual quality

And ... ???

1) With good source and adaptative avs script I can make encoding in HDTV 720p (1280*720*60) or 1080i with high quality and low bitrate:

Batman begin trailer
1280*720*60 MPEG2 MP@HL 4000 Kbps 1,1,1,2,2 image sequence
TMPGEnc CQ 65 with Sagittaire matrix

2) screenshots don't prove anything (I, P or B bframe) ... with screenshots I can prove that shrink is the best MPEG2 coder in the world ... lol

Dialhot · #12 08-14-2005, 12:22 PM

Quote:

Originally Posted by Sagittaire

2) screenshots don't prove anything (I, P or B bframe) ... with screenshots I can prove that shrink is the best MPEG2 coder in the world ... lol

... as with metrics

IMHO no contest can determine the best encoder. It can just determine which is the worst (generally all people agree on the bad quality of worst encoder - but there is not two identical advices about the best one).

Sagittaire · #13 08-14-2005, 12:28 PM

Quote:

Originally Posted by Dialhot

Quote:

Originally Posted by Sagittaire

2) screenshots don't prove anything (I, P or B bframe) ... with screenshots I can prove that shrink is the best MPEG2 coder in the world ... lol

... as with metrics

speak about that with video codec developper and with Babayaga for exemple ...

Quote:

Originally Posted by babayaga (Nero/ateme MPEG4 AVC developper ... IMO the best codec in the world)
PSNR at constant quantizer gives a good idea of the raw efficiency since this is what core encoder try to optimize most of the time.
MPSNR/OPSNR add some hints about rate-control and SSIM (JND metrics or whatever) add clues about adaptive quantization.

A codec which is inferior is those 3 metrics will almost surely be seen as subjectively inferior.

Besides, subjective testing is also very difficult to do and is not always reliable.

but I repeat ...

Quote:

The purpose of this challenge is to determine which is the best codec for the metrics and only for the metrics : this test will not speak about subjective visual quality. If you want subjective visual comparison download sample and compare yourself ...

Dialhot · #14 08-14-2005, 12:37 PM

I can read. Do you ?

Quote:

Originally Posted by Dialhot

As I said in my first post, I do not agree at all with this "contest" (to say the truth, in my first post I wrote "this is a loss of time" but erased it afterwards).
But as long as all encoder are treated the same way, this problem introduced by a wrong PAR is not important if you do not consider the visual quality.

Note: it seems you don't agree yourself with your own context. Just put two lines of your posts side by side :

Quote:

You must use MPEG2 with DVD standard MPEG2 MP@ML Pal DAR 4/3 PAR 12:11 profil

and

Quote:

it is not a challenge for standalone players compatibility

... kidding ?
Karl try to explain you that no libavcodec based encoder respect the MPEG2 MP@ML defined standard and that is why lot of encoders can't read their results.

You are comparing things that respect the rules of your contest and other that don't. It's like allowing drug usage in a 100m contest.

kwag · #15 08-14-2005, 12:38 PM

Quote:

Originally Posted by Sagittaire

Quote:

It's full of spikes, underruns, overrruns, etc. That's why we dumped it here, until further notice.

QEnc, FreeEnc, FFDShow, ffmeg ... are encoder based on Libavcodec like mencoder ... ???

Seems you haven't been around lately?

Yes, ALL are based on avcodec, and so is Mencoder, which is simply a front end to avcodec

Quote:

1) With good source and adaptative avs script I can make encoding in HDTV 720p (1280*720*60) or 1080i with high quality and low bitrate:

Batman begin trailer
1280*720*60 MPEG2 MP@HL 4000 Kbps 1,1,1,2,2 image sequence
TMPGEnc CQ 65 with Sagittaire matrix

Now try that with KVCD Notch matrix, and you'll be able to drop the bitrate by at least 1,000Kbps

Quote:

2) screenshots don't prove anything (I, P or B bframe) ... with screenshots I can prove that shrink is the best MPEG2 coder in the world ... lol

If done properly, and if they are taken from high action scenes, they do prove good visual comparison.

-kwag

Zyphon · #16 08-14-2005, 12:42 PM

I thought Hank's HC Encoder was not based on the libavcodec but instead written from scratch by Hank in the programming language called Fortran?

Dialhot · #17 08-14-2005, 12:45 PM

Quote:

Originally Posted by Zyphon

I thought Hank's HC Encoder was not based on the libavcodec but instead written from scratch by Hank in the programming language called Fortran?

Yes it is.

Sagittaire · #18 08-14-2005, 12:56 PM

Quote:

Now try that with KVCD Notch matrix, and you'll be able to drop the bitrate by at least 1,000Kbps

In fact I prefer my Sagittaire matrix:
1) better intra / inter repartition for my eyes (only for my eyes here ... lol)
2) better global / HF repartition for my eyes (only for my eyes here ... lol)

Quote:

If done properly, and if they are taken from high action scenes, they do prove good visual comparison.

In fact it's impossible for video codec comparison
1) Codec A and Codec B must have the same frame type frame for same frame
2) Codec A and Codec B must have the same quant for same frame
3) Codec A and Codec B must have the same RC strategie

Quote:

You must use MPEG2 with DVD standard MPEG2 MP@ML Pal DAR 4/3 PAR 12:11 profil
it is not a challenge for standalone players compatibility

You must use DVD PAL profil (gop, no adaptative bframe, 2 bframe, VBV specification ...) but it's no compatibility challenge ... I test MPEG2 codec quality and not standalone players compatibility. Libavcodec work certainely with large majority of standalone players compatibility ...

Nero Burning Room is the most used (IMO the best) CD/DVD burner in the world but perhabs not the most compatible for standalone players (Libavcodec work with my old DVD player but not Nero Burning ... lol)

Zyphon · #19 08-14-2005, 12:59 PM

Quote:

Originally Posted by Dialhot

Yes it is.

Lol, that is a litle vague there Phil. Is that yes it is written in Fortran or yes it is based on Libavcodec, oooorrrr. . . Both!.

kwag · #20 08-14-2005, 01:12 PM

Quote:

Originally Posted by Sagittaire

In fact it's impossible for video codec comparison
1) Codec A and Codec B must have the same frame type frame for same frame

Nope. It is very possible.
Just compare I to I, B to B, and P to P from each encoder on a same segment of footage

Quote:

2) Codec A and Codec B must have the same quant for same frame

Now that will be encoder dependant, and the best encoder will give the better quantization.

Quote:

3) Codec A and Codec B must have the same RC strategie

Again, no

Every encoder will be different, so each will have different constraints applied during conversion.
In the end, the visual quality is what counts.
And again, in the I to I, B to B, and P to P, you can compare encoder X (I frame) for the same frames on encoder Y by looking at the I, B and P
specially after a scene change, where there is less compression.
That is, compare the first header of an MPEG GOP, which should be IBBP.
Then you can compare those four frames to another encoder.
Just make sure the frames you are comparing on both encoders are truly splitted on a scene detect.

-kwag