HCEnc: Slicer and HC

[rds_correia]

Hi guys,
Anybody testing Slicer function on Hank315's HC?
I've been trying since HCBatch 0.14 but with poor results.
Now with HCBatch 0.15 results are also poor.
HC only supports Constant Quantizer not Constant Quality but the encoded movies look very good.
Using an AMD AthlonXP 2400+ w/ 512MB and with a simple script like:

Mpeg2Source(...)
Letterbox(78,78,16,16) <--- to keep in 16:9 720x576 anamorphic with overscan=2
Removegrain()
Deen()

I get a framerate of ~11fps.
Not bad .
As said results are veeery nice but filesize prediction is hard.
My HC.ini looks like this:

*cpu           auto
*profile       best
*aspect        16:9
*dc_prec       10
*matrix        notch
*infile        E:\xxxxxx.avs
*outfile       E:\xxxxxx.m2v
*logfile       E:\xxxxxx.log
 *bitrate       4000
*maxbitrate    8000
 *bias          30
*cq_maxbitrate 5.556
 *noscd
 *shutdown

I know, I know, I'm not using scene change detection.
But my problem is really with *cq_maxbitrate.
Even though it's set to 5.556 when I look at the log I see:

constant Q:              5.556
...
average quant (non linear):       5.533

This completely blows up the prediction that works damn fine with Tmpgenc but poorly with HC.
If I use *cq 5.556 instead of *cq_maxbitrate 5.556, sometimes I can see spikes up to the 12000Kbit and this blows the prediction too because in the end I'll really be using *cq_maxbitrate 5.556 and NOT *cq 5.556 .
Oh, BTW on a 2h11m movie it takes ~4h30 to encode plus 8 PINGs of 1% with GOP=15 and GOPmulti=x2.
Each 1% run takes ~2m45s.
Now add 2 PONGs, the initial and the final each taking also 2m45s.
That means the whole prediction can be achieved in ~28 minutes.
It could go even lower than this if I wouldn't get things like this:
5.000-ping 18.037KB
5.000-pong 21.338KB
5.277-ping 17.706KB
5.467-ping 17.357KB
5.552-ping 17.229KB
5.567-ping 17.171KB
5.563-ping 17.171KB
5.559-ping 17.172KB
5.556-ping 17.172KB
5.556-pong 20.118KB <-- Settled with this Constant Q.
5.553-ping 17.229KB <-- Just to see if it would be much different than 5.552...

Any ideas guys?
Cheers

[Prodater64]

Hi rds: I like you are doing this test.
I also did it and developped a tool that works with ping-pong method.
First I will say you my experience.

I agree with you, cqmaxbitrate method, even it is true that permits you to obtain a streams without spikes, also, frequently, undersize. I think that it is doubt to spikes cuts.
Amnon82 tool, I think it works with cq, not cq maxbitrate, with a safe value of 3. He say that under this value, don't see spikes, but I don't now.
I don't test it but I think Danpos do it. He sure will say us something later.

Well, with this in sight, and frequents undersizes, i deal with this in the following way: I did my tool HC-Qmatic with instructions to use a safe cqmaxbitrate value. On films with a lot of undersize, it will obtain, theoretically, max a -2% filesize. If oversize, rejig will be used, always with a level over 90.
All my test until now went between almost 0 and -1.7%.

[rds_correia]

Hi Pro ,
I know that I said that it completely blows the prediction.
In a way it does but on the other hand I get similar (manual) results when compared to your's.
You say your tool undersizes between 0 and -1,7%.
Well that's exactly what I get here too: 0 and -2% (max!).
But that's enough to loose around 70-75MB per DVD.
So I call that bad .
From your experience, have you only seen undersizes or have you had oversizes too?
Because if it's only undersizes I will simply apply a correction factor of 1% and that's the end of story .
Cheers

[Prodater64]

I have seen under and oversize, bigger undersize that oversizes.
For this reason, I stick this lowering maxbitrate, enough to obtain a targeted final size. But if it is oversized, I apply rejig. Level over 90% always (I would say over 95%).
In this way you will obtain almost always a target between 0 and -1.5%.
You can download the tool and take a look at HC-Qmatic-core.bat.
Prediction is 3 decimal floating point accurate.


Edited: typos.

[rds_correia]

Quote:

Originally Posted by Prodater64

I have seen under and oversize, bigger undersize that oversizes.

I see
That's the biggest problem I'm facing here .

Quote:

Originally Posted by Prodater64

For this reason, I stick this lowering maxbitrate, enough to obtain a targeted final size. But if it is oversized, I apply rejig. Level over 90% always (I would say over 98%).
In this way you will obtain almost always a target between 0 and -1.5%.

That's my exit.
I don't want to use a transcoder.
I have seen and tried DVDShrink and Rejig and believe me, I can tell when a transcoder has been used even if it is below than 90%.
Well, since we're talking about reducing in less than 2% it won't be much noticeable but using a transcoder is definitly not in my plans.

Quote:

Originally Posted by Prodater64

You can download the tool and take a look at HC-Qmatic-core.bat.
Prediction is 3 decimal floating point accurate.

I have seen it already.
That's a really interesting batch work.
But right now, I'm looking for a manual way.
Later on I'll go for an automatic way maybe similar to CQMatic: simple, small and effective aid on video encoding only.
The rest (audio transcoding, subtitles, muxing, etc) will always be manual.
But thanks for the help Pro .
Cheers

[hank315]

Quote:

I agree with you, cqmaxbitrate method, even it is true that permits you to obtain a streams without spikes, also, frequently, undersize. I think that it is doubt to spikes cuts.
Amnon82 tool, I think it works with cq, not cq maxbitrate, with a safe value of 3. He say that under this value, don't see spikes, but I don't now.

IMHO the undersize is caused by the raising of Quant to keep the max. bitrate, the "missing" bits are not returned to the bitstream ATM.
I'm planning to keep track of these missing bits and to insert them again which means lowering the Quant for some GOPS.
Think if CQ is used instead of CQ_MAXBITRATE you can never know if you overshoot the max. bitrate, it can spike like hell... there's just not a "safe" value for it.

[Prodater64]

Quote:

Originally Posted by hank315

Quote:

IMHO the undersize is caused by the raising of Quant to keep the max. bitrate, the "missing" bits are not returned to the bitstream ATM.
I'm planning to keep track of these missing bits and to insert them again which means lowering the Quant for some GOPS.
Think if CQ is used instead of CQ_MAXBITRATE you can never know if you overshoot the max. bitrate, it can spike like hell... there's just not a "safe" value for it.

Thanks for your explanation.

[rds_correia]

Thanks Hank

[Prodater64]

I did an encode with 0.15 and 0.14 (pics in this order). Results are not good.



(Thanks to ImageShack for Free Image Hosting)

[Dialhot]

(OT : Pro, you should redue the length of your pathnames. Lot of app are known to have problems with long pathnames).

[Prodater64]

Quote:

Originally Posted by Dialhot

(OT : Pro, you should redue the length of your pathnames. Lot of app are known to have problems with long pathnames).

Thank you.

[Prodater64]

Definitively Q prediction in HC Encoder is not good, don't matter what version (0.14-0.15) do use. It is doubt to that explained by hank315 related to cqmaxbitrate.
See my previous post.

[Dialhot]

1/ Aren't you supposed to do a last pong to verify what have been found with pings ?

2/ error on the sample is near 4% in you second log and near 8% in the first one
can't you have a better precision ?

with such error on the sample, I'd never encode the whole movie, whatever the encoder.

[rds_correia]

Hi Phil ,
I'm on it it too and I can't quite get it to work reasonably...
Not that it is a big deal or anything but if there is a CQ-like mode I'd like to see it working even because the quality I've seen on my samples is wonderfull.

Quote:

Originally Posted by Dialhot

1/ Aren't you supposed to do a last pong to verify what have been found with pings ?

Yes he/we is but from what I've seen from all my test runs, I bet that if Pro had done a last Pong he would have clinched a bullseye.

Quote:

Originally Posted by Dialhot

2/ error on the sample is near 4% in you second log and near 8% in the first one
can't you have a better precision ?

He can but he'll have to switch to cq instead of cqmaxbitrate for the prediction. The problem is, for such a prediction to be truthfull he would have to encode in cq but most probably he will get big spikes.
That's why me and Pro (and others probably) are going for cqmaxbitrate...

Quote:

Originally Posted by Dialhot

with such error on the sample, I'd never encode the whole movie, whatever the encoder.

Agreed but what can we do to compensate???
Use of cq is not an option here
Though, I've refined my calculations and right now I'm consistently reaching less than 1% of undersize.
Cheers

[Dialhot]

Quote:

Originally Posted by rds_correia

Agreed but what can we do to compensate???

I'm just surprised that with a CQMAXBITRATE value that can go to 4 digits after the dot, you can't find a value that gives results closer than these 4/8%

Quote:

Though, I've refined my calculations and right now I'm consistently reaching less than 1% of undersize.

So, you see that it is possible

[rds_correia]

Quote:

Originally Posted by Dialhot

Quote:

I'm just surprised that with a CQMAXBITRATE value that can go to 4 digits after the dot, you can't find a value that gives results closer than these 4/8%

Quote:

So, you see that it is possible

It is possible but it very hard to get there...
And this cq is constant quantiser NOT constant quality.
Plus you input 5.355 and the encoder ends up doing 5.371 or something similar.
That's because it will ALWAYS compensate for the spikes...
I'll try to explain my methods, If I have some time tonight.
Cheers

[rds_correia]

@Pro64
BTW buddy, how come with the same movie you got 2 different predictions?
Are you using non-linear algorithms to predict?
Or did you just set Slicer() with different parameters on 2 encodes?

@all
As said I think I found a break through but in fact I won't be able to say anything tonight.
I had left my home PC doing 2 different encodes during the day so I could get home and see if my new calcs were going ok.
But I just called home and my son told me we had a power outage.
That means I will only be able to test this overnight.
Hopefully, tomorrow I'll have more info on it.
BTW, I wonder how Pedro sorted this on OPV in DVDREasy .
Cheers

[Prodater64]

Quote:

Originally Posted by Dialhot

1/ Aren't you supposed to do a last pong to verify what have been found with pings ?

2/ error on the sample is near 4% in you second log and near 8% in the first one
can't you have a better precision ?

with such error on the sample, I'd never encode the whole movie, whatever the encoder.

1 - After many tests, I decided don't do last pong as all times it was near the target.

2 - Error in first sample is 143/19302*100=0,74085586985804579836286395192208 %

In second one is 72/19302*100=0,0037301834006838669567920422754119 %

3 - HC Encoder is really 3 decimal floating point sensible. My tool rounds 4 to 3 decimal points. (0.0006 = 0.001)

4 - Im sure that rds is getting 1% undersize in samples, not whole movie. But I can be wrong. When he do whole encoding, he will find a surprise, I think.

5 - @rds: Please, if you find out the solution, let me know how? It would be great that I can include that maths in my tool. Thanks.

[Prodater64]

Quote:

Originally Posted by rds_correia

@Pro64
BTW buddy, how come with the same movie you got 2 different predictions?
Are you using non-linear algorithms to predict?
Or did you just set Slicer() with different parameters on 2 encodes?

First encode is with HCbatch 0.15 and second one with 0.14.
This is the reason of different prediction.

Edited: Im thinking in doing a last 5% prediction and see where is going it. With this value, to do a correction of maxbitrate. Any idea.

[Prodater64]

Quote:

Originally Posted by rds_correia

Quote:

It is possible but it very hard to get there...
And this cq is constant quantiser NOT constant quality.
Plus you input 5.355 and the encoder ends up doing 5.371 or something similar.
That's because it will ALWAYS compensate for the spikes...
I'll try to explain my methods, If I have some time tonight.
Cheers

You can see it is not so hard with my tool to obtain error less than 1% (it was hard to code it for me). But that error in sample always undersize in final encoding. This undersize as explained by Hank315 is doubt to "cuts" in bitrate that cqmaxbitrate option does to keep DVD compliant. Those bits are not bring back to the stream ATM as Hank said.
This do prediction, a very hard task, as we don't know how many bits ard discarted.
But we can suppose it. I don't know how by the moment, but if we do extras passes at (my first ones are 1%) 2 and 3% and we can discover a "model" or behaviour, we will find out a correct final cqmaxbitrate or one really near to it (I hope this).
Im working on it.
Also in few hours I will post an little update with only prediction button, then you can do all test as you want without have to abort encoding. It will permit to compare your manual results with my tool ones.