Me too
Final file size shouldn't matter with or without filters, because of the dual pass accuracy
Only CQ value will change.

-kwag

Do you think this could improve accuracy for MPEG2 prediction?

I always exceed the desired file size using CalcuMatic data into CQmatic for MPEG2 files.

It shouldn't matter what is encoded, because the accuracy is now determined by the first dual pass.
We'll see how accurate this works. I'm currently less than an hour away from finishing a 352x240 encode of "Sum of all fears", and I also have a current encode (on hold) of "Red Planet" at 704x480.
The latter is on hold, because it takes too long to encode. So I'm testing the 352x240, and then I'll continue the 704x480. My P4 is just too slow to handle two concurrent TMPEG sessions

-kwag

Information relevant to the new prediction:

[1:42 PM] krassi10: Hi Karl, can you please call me when you have some time?
[3:01 PM] krassi10: Hi Karl,
CQMatic gave me a final CQ of 63.17, before it was 35 ; -)
I'm retesting this with a bigger source range.
[3:15 PM] kvcdnet: Hey Marc, did you use 2-pass(old) or 2-pass(new)? Because all tests I did, were using "old". I got different results when I used "new" 2-pass!
[3:18 PM] krassi10: I used the new one. I'm on my 3rd run with the old one now. Forgot that Plus is having the new one as default. Doing a test with a sc of 1.40 min and 10 min. But now i see that the filesize is still bigger with a CQ of 40 and thats a nogo...
[3:20 PM] kvcdnet: Is the current test with the "old" 2-pass ?
[3:20 PM] krassi10: Yes, with padding enabled.
[3:20 PM] kvcdnet:
[3:20 PM] kvcdnet:
[3:24 PM] krassi10: Keep smiling Karl, you have done so much great things. Have you tried it with CQ_VBR?
[3:26 PM] kvcdnet: No. Not yet. I'm still focused on the 2-pass thing. Because 2-pass is predictable, I still believe I can "steal" something out of the 2-pass prediction, and apply it to CQ.
[3:27 PM] krassi10: Gee, CQ_VBR @60 is giving me a filesize of 53,744KB (should be 20,471).
[3:28 PM] kvcdnet: =:O
[3:31 PM] kvcdnet: I think the problem is clear. When we make a 2-pass VBR sample of X size, the average bitrate will ALWAYS be the one we selected. Doesn't matter if it's a 1 minute clip, or 10 minute clip. But that is the average of the total sum of the movie. Not independent chunks. So we can't target the same file size with CQ as with 2-pass on a little (small) sample. It must be much larger.
[3:34 PM] krassi10: Yeah. Maybe a dumb question: And if we would calculate the average bitrate only on the source range?
[3:37 PM] kvcdnet: Now that you mention it, we can instead of trying to match the file size that 2-pass made with CQ, we can try to encode in CQ and "match" the same average bitrate !!!!! So drop the 2-pass sample into bitrate viewer and note the average. Then encode with CQ, and start lowering CQ value until the same average is reported.
[3:37 PM] kvcdnet: DOes that sound logical?
[3:38 PM] krassi10: Still reading and trying to understand... Give me two minutes... good that you have written it down...
[3:38 PM] kvcdnet:
[3:39 PM] krassi10: I think its worth a try... I'll do one.
[3:41 PM] kvcdnet: If you get about the same average, but there is a substantial difference in file size, then that's it! But in theory, no matter how you encode, if you have two files with the same average bitrate, the file size should be about the same. It's just the way the "bits" are allocated.
[3:51 PM] kvcdnet: Finished Red Planed at 704x480. Wanted: 729,726KB. Encoded: *** 725,003KB ***
[3:51 PM] kvcdnet: I think I know what the problem is!
[3:51 PM] kvcdnet: What was your average, MIN and MAX bitrates????
[3:52 PM] krassi10: Hmm. Optimal CQ (from CQMatic would be 63, with a CQ of 60 bviewer is giving me an average of 2318 and VB had 1619.
Min: 1000 Max: 5000 (KDVD), thats what i thougt before.
[3:53 PM] kvcdnet: Ok. What was PEAK (max) bitrate you got in your 2-pass sample?
[3:53 PM] krassi10: Great Result! CQ of 40 is giving an average of 1807
[3:54 PM] krassi10: Peak is 5033 from 2-pass
[3:54 PM] kvcdnet: Is CQ of 40 about the correct value you need?
[3:55 PM] krassi10: No, it would be 63.
[3:55 PM] kvcdnet: So then you have to increase CQ to match the needed average
[3:55 PM] kvcdnet: Let me do a quick test here too....
[3:57 PM] krassi10: Sorry, i meant 63 would be the optimal CQ, but then average bitrate is way off >2300
[3:58 PM] kvcdnet: What average do you get on that sample if you encode with CQ=63?
[3:58 PM] krassi10: Currently encoding...
[3:59 PM] krassi10: Maybe we can find a constant multiplier here, but i don't expect this...
[3:59 PM] kvcdnet: Not with this crazy CQ (un)linearity
[4:01 PM] krassi10: CQ=63: Average 2357, Peak is 4772.
[4:04 PM] kvcdnet: Marc, check this out. I just checked the average of my 2-pass sample and the average of the CQ sample I created, and the averages were almost the same. This is the one I got almost perfect result (Red Planet at 704x480). Could you check the average of your 2-pass against the average of your CQ sample ( with the CQ that you got ~980MB instead of 1.4GB ) and see if the averages of both samples don't match?
[4:07 PM] krassi10: I've overwritten it but i'll redo a sample with CQ=35 (what resulted in that file). However i've changed source range but that shouldn't matter.
[4:09 PM] krassi10: Results with CQ=35: Average bitrate:1708, Peak:3609
[4:10 PM] kvcdnet: Are you using "source range", or are you encoding your 2-pass using the file "CQMatic.tpr" ??
[4:10 PM] krassi10: This time i used source range, before it was using the time slices from CQMatic.
[4:11 PM] kvcdnet: Ok. I'll do more tests here. Just trying to find a constant!
[4:13 PM] krassi10: I'm doing a prediction on my second workstation with min=1300 and max=2000 (VB and CQ of course). I think its coming from TMPGEnc's variable bitrate:
Normally its nearly sticked to the average, CQ isn't. Thats why there's such a difference in my tests.
[4:15 PM] kvcdnet: Right! There are not that many fluctuations between low and high bitrate. That's why CQ is better that 2-pass, at least with TMPEG
[4:17 PM] krassi10: BINGO. With the latest settings on my second pc theres only a file difference of less than 1000KB with a CQ of 60, before the size was nearly doubled.
Maybe you can make a test with wide-spreaded min-and-high-bitrates.
[4:18 PM] kvcdnet: I think that's the main problem!. The further apart are MIN and MAX to average, the worse the results.
[4:19 PM] kvcdnet: So maybe we should raise MIN if we raise MAX, and keep MIN bitrate at a constant percent below average bitrate (idea!)
[4:20 PM] krassi10: Yes, but personally i want to have 3 movies one on KDVD and the final quality suffers if i'm adjusting min too high, but you're right!
BTW: Nice chat!
[4:22 PM] kvcdnet:
[4:22 PM] kvcdnet: Doing a wide MIN,MAX test....
[4:24 PM] kvcdnet: SO, in theory, the higher the resolution, the wider can be the MIN and MAX range. That's why my 704x480 was exact, but my 352x240 was lower in file size!.
[4:25 PM] krassi10: Why can't players do 1600x1280
[4:25 PM] kvcdnet: So we must "scale", or find the MIN and MAX bitrate "borders" for different resolutions.
[4:26 PM] kvcdnet: Could you repeat the test you did with the .tpr, but at 704x576 and MIN=300, MAX=2,500?
[4:26 PM] kvcdnet: 1600x1280, I wish
[4:27 PM] krassi10: I'll redo with these settings
[4:28 PM] kvcdnet: If you can hit final file size target with that resolution and with those MIN/MAX values, then we know that for 704x or 720x, MIN of 300 and MAX of 2,500 will always be correct!
[4:30 PM] kvcdnet: It's going to be a circus, trying to find the optimal MIN and MAX for 352x240, 352x288, 352x480, etc, etc LOL
[4:31 PM] krassi10: Yes, and i'm the clown searching for the KDVD-settings : -)
[4:31 PM] kvcdnet: No, we both are LOL
[4:32 PM] kvcdnet: I hope this is worth it!, and not another "CQ down the drain" attempt
[4:37 PM] krassi10: Mee too. I have some time while encoding, so here something funny. Me and my friends were on vacation near Calpe (Spain). Some spanish girls talked to us. One of my friends is indonesian. They asked where we come from, we all said "from Stuttgart", except the indonesian. He said "me too". The girls understood Mi-Tu and asked, where this is located LOL
So i decided to learn spanish but wasn't very succesfull till now.
[4:37 PM] kvcdnet: LOL
[4:38 PM] krassi10: BINGO, filesize is right on target with 704x576 @300-2500. I'll try it with 720576 now.
[4:38 PM] kvcdnet: YES
[4:39 PM] kvcdnet: It should be good at 720x too. We might be able to increase to at least ~2,800 MAX for 720x (idea)
[4:40 PM] kvcdnet: 'm trying a 352x240 now, and setting MIN to 300 and MAX to 1,500. My previous encode had 1,800 as MAX, and final file size was a little low. So probably with 1,500MAX will hit target.
[4:41 PM] krassi10: But i'm still the clown concerning KDVD. I've tried with MIN=2000, Max=5000, but filesize is 13MB bigger. I'll try with 350-5000.
[4:44 PM] kvcdnet: It might be a problem with such a "wide" range! Maybe a true 2-pass full encode is the only way to accomodate such large bitrate fluctuations :\
[4:47 PM] krassi10: Yep, it's just a try. 720x576 is hitting also!!!
[4:47 PM] kvcdnet: I'll have 352x240 results in a couple of minutes.
[4:50 PM] kvcdnet: Have you noticed that Bitrate viewer gives lower MPEG-1 average bitrate than if viewed with VirtualDub?
[4:51 PM] krassi10: Yes, i've already noticed that. Thats really curious, i think it's the same with mpeg2. But the question is: which one is right
[4:52 PM] kvcdnet: VirtualDub. I manually calculated an average bitrate, and Vdub is the correct valu.
[4:54 PM] kvcdnet: I'm going to do a scale, by increasing MAX bitrate, until I find the "Roll off" point of every resolution. I think we don't have to worry too much with MIN bitrate, because the problem is mainly theMAX bitrate. So I'll figure a MAX bitrate to be used with every resolution.
[4:54 PM] krassi10: I'm a bit tired now,Kwag. I'm sorry. Do you have something important to test? If not, i'll be offline in a couple of minutes. But i think we thats means you have found a good way for alternate prediction. If you want to, you can post this chat into the thread.
[4:55 PM] kvcdnet: Hey, go to sleep Marc It's 11:PM your time!. Hopefully, I'll have a table of MAX bitrates for tomorrow, and your name will be in lights in the forum, for the findings you did
[4:56 PM] kvcdnet: Good idea! I'm Copy&Pasting this into the forum now! See you tomorrow
[4:57 PM] krassi10: Thanks Kwag. The spotlight should be on you. Good night. I think i'll login tomorrow evening.

Hi,

It explains why I was so 'out' of target: I'm using a 1800 max bitrate...
I'll try with 2500...

CU
Fabrice

After some more tests i've figured out that a long source range seems to take away the problem of high bitrates.
Currently testing my 3rd movie with a long sr. Manually calculated CQ was always nearly the same as predicted CQ-value with CQMatic.
I've used a bitrange range of 350-4000

EDIT: @fabrice: Can you try it also with a longer sample size (source range), please

With the source i used in the chat with kwag i did a test with a source range of 4:49 minutes.
The correct CQ with this sc is about 66. And thats what the optimal CQ should be.
CQMatic gave 63, i'm doing a full encode to test which one is right
(Bitrate 500-4000)

EDIT: Here the file sizes:
VB: 141.418KB
CQ@66.2: 141.965KB

Final File size with CQ=66: 1.35 GB, wanted was 1.4 GB

Hi Krassi,

Apparently, a source of 3 to 4 minutes does the trick
Actually, the longer the better. But I think ~4 minutes should be fairly accurate. It's just a matter of selecting the right footage ( mixed action segment ).
I'm trying a source range of 3 minutes right now. Then I'll do an encode to see if it matches the wanted final size. All at 352x240 ( for speed reasons )

-kwag

Hi Kwag,
my prediction results with a sr of 4 minutes:
VB: 48.558, CQ=66: 53.926. So thats not enough for a high Bitrate variance of 500-5000.
Testing with a sr of 5 min. now.
Maybe i should find a better sr, but maybe if we'd use a longer sr CQMatic would be able to set sr by itself (or you could enter the minutes manually ).
For sure a 4 minutes length should be good enough for users targeting for a KVCD

EDIT: Results of the 5 min. didn't show any improvement, think that its related to the high Max bitrate value. I'll make a test with 4000 as max.

EDIT 2 : difference was less with 4000 as max but still 3,5 MB over VB with a CQ of 66.

I've just had the following idea :

I'm testing this prediction method with CCE. My experience with CCE VBR is, that it is behaving better than the one TMPGEnc uses, it's not so "sticked" to the average bitrate.
So, in theory , this prediction method should be producing better results with CCE. Remember, CQ in CCE is the other way round, lower CQ will give higher filesize...
I've used 2-pass VBR to get filesize and then changed to CQ and matched filesize.

Current state:
I've done a prediction with a sample length (source range) of 60 seconds, 120 seconds and 5 minutes. Average bitrate is 1586, target video size is 1317910KB, min bitrate is 300 and max is 5000.

Results: CQ is different (also because CQ in CCE has a bigger CQ range):
1 min: optimal CQ is 60
2 min: optimal CQ is 55
5 min: optimal CQ is 41.
I'm doing a full encode with 41 now and let you know if it is hitting target.

Maybe i'll do a Mainconcept also, or can you setup one

Results of the CQ-encode @41:
VS: 1267375 KB
Wanted VS: 1317910 KB.

That's 4% under target.
I'll redo a test with another source range (maybe a longer one).

Hey Krassi, that's great
At least the theory of the quick "2-pass/long CQ" does make sense
I still think there must be a way to use it with TMPEG. Hopefully I'll have some time in my hands next week, and play with this more.
This week has been a little busy for me, and the little time I've had, has been playing with "MythTV". It's finally working
I hope to move the test computer to the living room ( maybe tomorrow), and then I'll take some pictures with my digital camera and post them here

-kwag

Hi Kwag,

i've redone the same prediction with CCE with a source range of 10 minutes
Final predicted CQ was 44.
Results:
Wanted VS: 1317910 KB
Final VS: 1220134 KB
This makes 7,4%
I've tested this prediction with k-pax. It's a very dark movie and i think it's hard to find a good source range. I'll repeat this test with another source.
I haven't found a possibilty to make time slices with CCE (except with avisynth for sure).

@ Krassi

What type of samples u are using??
A .AVS including Sampler(length=xx, Samples=x) ??
maybe you mentioned already.

Hi incredible,

in CCE you have the possibilty to set the source range from a start frame to an end frame (Input Files->Settings->right-click->edit).
Thats how i've selected my source range.
Maybe i'll play around with some avisynth scripts to create time slices later on.

I think you don't need that AVS Time slice way ...
Im at work so maybe lets try something "online" ...
Open the mpeg source in Bitrateviewer (this now depends if it accepts d2v's???? or Avs's including d2v imports?? otherwise forget my thinking) and whatch the peaks and lows, write down the time stamps of theese parts including some average bitrate parts. Determine these parts as timeslices im CCE and do your workaround again .... would be interesting.

Or another method:

Do the AVS time slice way ... but:
Write a sampler(xxx,xxx) based time slice a-script and another one b-script which includes a trim line which gives you an offset of about 2 or 5 mins (depending upon the used numbers of samples)... (you remember the thread in the german forum?)

Maybe this double a/b script based 2-pass/prediction will give you two values where you can see an average of both which should give you just another type of value ... it could also be more precise ????

I've already done this before with TMPGenc, the prediction was better when i've selected peaks and mins manually and "imported" them to TMPGEnc. But thats a lot of work...
We could do this automatically, but before is has to be approved.
In CCE, you can only define from-to frames, no slices...

Sure i do. It's already on my todo-list. Have you already setup a test?
Yes, in theory it could be. My previous tests were always selected with a different source range, so in theory, the average value of the 3 predictions would give the best value.

I also had the idea of making a longer VBR encode (e.g. 10 or 15 min. with slices) and then doing only a prediction with CQ divided by 2 or 4 or...
I mean you would have the size of the VBR (10 min.), e.g. 20 MB. Then you would divide it by 5. So the CQ prediction should match 4 MB when encoding 2 minutes. That's next on my list
The only problem here i can see is that you can't use the same source range, but in theory, compared to the whole video and the length of the VBR sample that should'nt play a role.
I hope you understand what i mean.

Update:
VBR encode: 30 mins. Final sample size: 348.163 KB
So, divided by 5:
6 minutes sample for CQ and a sample size of 69.632,6 KB (348.163/5).
Prediction gave a CQ of 31 (filesize is 70.026 KB).
Doing a full encode now with 31.

Tonight will be the night (I mean this evening *lol*)
Do you will stay tuned too?? If yes gimme your ICQ via PN, so we can figure out togehter

BTW: Did you recognised that Kwag changed us to moderators? I was really surprised