Quantcast Replacing DVD Rebuilder's Encode Phase with Manual Encoding? - digitalFAQ.com Forums [Archives]
  #1  
07-22-2004, 03:03 PM
SansGrip SansGrip is offline
Free Member
 
Join Date: Nov 2002
Location: Ontario, Canada
Posts: 1,135
Thanks: 0
Thanked 0 Times in 0 Posts
Okay, this has caught my interest so I figured I'd make a new thread. The original conversation started here.

The basic idea is to use DVD-RB's many awesome features but replace the encoding phase with a custom step. One would run the "Prepare" phase and then run an external tool to tweak various values and launch the encoder. Once all the cells were encoded one would then use DVD-RB's "Rebuild" phase to remux everything together.

The advantage is that we gain a significant amount of control over the encoding process, can use file size prediction, custom resolutions, overscan, GOP length, etc. We can even enable TMPGEnc support in DVD-RB.

I have other plans for it, too. I'd like to be able to run IVTC in my Avisynth scripts, but DVD-RB currently makes this impossible. Technical details follow: skip a few paragraphs if you're not interested.

The problem is that when DVD-RB generates its D2V file it sets all the frames to "TFF, no RFF" (top field first, no repeat field flag), regardless of how the flags were set in the original. The encoder is then passed the raw fields from the video stream, and the new fields are remuxed into the VOBs. When it's remuxed DVD-RB sets the field flags back to how they were originally. The purpose of this is to leave things as unmolested as possible, so that the user doesn't have to worry about forced film mode and so on. The field structure will remain exactly the same as the original.

This is great for truly progessive or truly interlaced streams, but falls down when it comes to hybrid or hard telecined material. Because Avisynth doesn't see the frames "as they're meant to be seen" it's impossible to correct problems in the original mastering, such as hard telecined material. It also makes applying filters suboptimal because with hybrid or hard telecined material one can't convert to progressive in Avisynth before running filters, which means filters can end up being run on interlaced frames. This is not good.

My plan is to replace DVD-RB's D2V file with one that actually reflects the reality of the stream's field flags so that Mpeg2Dec will produce a "what is meant to be seen" stream. One can then IVTC in Avisynth, if necessary, to make true progessive frames. These can then be filtered much more accurately, and any interlace artifacts in the original can be fixed. This still leaves the problem that, upon rebuilding, DVD-RB will set the flags back the way they were in the original. Because the tool will have made the material truly progressive (or truly interlaced, as the case may be) this approach will not be correct in many cases, and playback will be messed up. Thus the tool will need to fix up the flags. I believe that DVD-RB stores the original flags in files called "Vxxxxxxxxxxxxxx.FLG", so it should be possible to simply rewrite those files with the correct flags (for truly progressive material the correct flags would be, in DVD2AVI's parlance, 0 1 2 3 0 1 2 3 0 1 2 3... and for truly interlaced material it would be either 0 0 0 0 0... or 2 2 2 2 2... depending on whether it's top- or bottom-field first.)

End of geek talk .

So this tool wouldn't just be a custom encode step -- it would also fix what I consider to be DVD-RB's greatest weakness, which is the inability to properly fix badly mastered material because of the way it handles field flags.

I'm more than willing to make this my next coding project, so let's kick some ideas around. This kind of tool could make DVD-RB even more of a killer app than it already is, so let's think of features, perhaps figure out a friendly interface, see if we can incorporate support for TMPGEnc as well as the encoders DVD-RB already supports, etc.

We also need to thrash out the details of how the file size prediction will work. I've not looked at that stuff in a long time...
Reply With Quote
Someday, 12:01 PM
admin's Avatar
Site Staff / Ad Manager
 
Join Date: Dec 2002
Posts: 42
Thanks: ∞
Thanked 42 Times in 42 Posts
  #2  
07-22-2004, 03:30 PM
Boulder Boulder is offline
Free Member
 
Join Date: Sep 2002
Location: Lahti, Finland
Posts: 1,652
Thanks: 0
Thanked 0 Times in 0 Posts
Regarding prediction, search for incredible's posts for the Slicer idea. It has been discussed in several threads and I think that his approach will give the best results for most material.
Reply With Quote
  #3  
07-22-2004, 03:32 PM
SansGrip SansGrip is offline
Free Member
 
Join Date: Nov 2002
Location: Ontario, Canada
Posts: 1,135
Thanks: 0
Thanked 0 Times in 0 Posts
Quote:
Originally Posted by Boulder
Regarding prediction, search for incredible's posts for the Slicer idea. It has been discussed in several threads and I think that his approach will give the best results for most material.
Willdo. Thanks.
Reply With Quote
  #4  
07-22-2004, 04:45 PM
SansGrip SansGrip is offline
Free Member
 
Join Date: Nov 2002
Location: Ontario, Canada
Posts: 1,135
Thanks: 0
Thanked 0 Times in 0 Posts
I think I found the thread where this method was developed, but I'm left a little confused as to the exact method involved. Am I right in saying that it involves taking two sets of samples, one offset by 30 seconds, and averaging them?

I also read about Kwag's "random samples" idea, but I'm not sure if this will gain anything over the usual method of sampling (the only difference is that instead of being sequential the sampled segments are now out of order).

What is the current "canonical" method for file size prediction? Is there one?
Reply With Quote
  #5  
07-22-2004, 05:42 PM
Dialhot Dialhot is offline
Free Member
 
Join Date: May 2003
Posts: 10,463
Thanks: 0
Thanked 0 Times in 0 Posts
Inc should be back next week and I'm sure he will find both your posts and your ideas really freaky. He will give you any answer you need, and much more
Reply With Quote
  #6  
07-22-2004, 05:45 PM
SansGrip SansGrip is offline
Free Member
 
Join Date: Nov 2002
Location: Ontario, Canada
Posts: 1,135
Thanks: 0
Thanked 0 Times in 0 Posts
Hmm. I like the look of the Newton method. Sounds like it's way faster than a binary search.

Here's what I feel might be a good approach to this particular problem:

* Ignore first 5 minutes of movie (or a certain percentage)
* Ignore last 5 minutes of movie (ditto)
* Make each sample at least one GOP in length, probably more like 3 or 4
* Pull enough samples, evenly spaced, from what remains of the movie, to make up 1%, 2%, 3% (or whatever) of the total length
* Guess (somehow) at a Q to start, and use the Newton method to locate a good value (EDIT: Looks like r6d2's "Newton's Iteration 2" method is the way to go.)
* Maybe take another sample, with different segments of the movie, and do it again, then average?
* Create .vaf file with that Q
* Depending on accuracy, do a VBR pass using that .vaf but specify an exact average bitrate to hit the target

Sound about right?
Reply With Quote
  #7  
07-22-2004, 05:47 PM
SansGrip SansGrip is offline
Free Member
 
Join Date: Nov 2002
Location: Ontario, Canada
Posts: 1,135
Thanks: 0
Thanked 0 Times in 0 Posts
Just thought I'd point out that I don't particularly want this turning into yet another file prediction method thread -- I'm also interested in hearing feature suggestions for this (potential) tool, and comments on my idea regarding fixing field flags so we can IVTC and filter with DVD-RB.

Size prediction is just one part of the (potential) tool .
Reply With Quote
  #8  
07-22-2004, 06:10 PM
SansGrip SansGrip is offline
Free Member
 
Join Date: Nov 2002
Location: Ontario, Canada
Posts: 1,135
Thanks: 0
Thanked 0 Times in 0 Posts
Quote:
Originally Posted by SansGrip
* Create .vaf file with that Q
* Depending on accuracy, do a VBR pass using that .vaf but specify an exact average bitrate to hit the target
Mmm. These steps are complicated by the fact that we're going to be encoding each cell separately. I think what will be required, in this circumstance, is one "pass" to create all the .m2v files, then add up the file sizes to compare with the desired. We then decide if we need a second go at it using VBR on the existing .vaf files and a set average bitrate. That should work. I think...



(God I love this stuff.)
Reply With Quote
  #9  
07-23-2004, 12:18 AM
Boulder Boulder is offline
Free Member
 
Join Date: Sep 2002
Location: Lahti, Finland
Posts: 1,652
Thanks: 0
Thanked 0 Times in 0 Posts
Quote:
Originally Posted by SansGrip
Hmm. I like the look of the Newton method. Sounds like it's way faster than a binary search.

Here's what I feel might be a good approach to this particular problem:

* Ignore first 5 minutes of movie (or a certain percentage)
* Ignore last 5 minutes of movie (ditto)
* Make each sample at least one GOP in length, probably more like 3 or 4
* Pull enough samples, evenly spaced, from what remains of the movie, to make up 1%, 2%, 3% (or whatever) of the total length
* Guess (somehow) at a Q to start, and use the Newton method to locate a good value (EDIT: Looks like r6d2's "Newton's Iteration 2" method is the way to go.)
* Maybe take another sample, with different segments of the movie, and do it again, then average?
* Create .vaf file with that Q
* Depending on accuracy, do a VBR pass using that .vaf but specify an exact average bitrate to hit the target

Sound about right?
Yes, sounds good to me too. The good and bad thing about CCE is that it uses an integer value for Q so you'll get it predicted a lot faster than with TMPGEnc.
Reply With Quote
  #10  
07-23-2004, 07:37 AM
zagor zagor is offline
Free Member
 
Join Date: Feb 2004
Location: Italia
Posts: 226
Thanks: 0
Thanked 0 Times in 0 Posts
I don't know the Newton method, about inc's prediction, I have used it
time ago, but i haven't had a good results. I have used it for more divx
on a kdvd.

I think that the prediction with sample is the best.


by
Reply With Quote
  #11  
07-24-2004, 05:24 PM
incredible incredible is offline
Free Member
 
Join Date: May 2003
Location: Germany
Posts: 3,189
Thanks: 0
Thanked 0 Times in 0 Posts
Send a message via ICQ to incredible
Quote:
Originally Posted by Dialhot
Inc should be back next week and I'm sure he will find both your posts and your ideas really freaky. He will give you any answer you need, and much more
Im back and it sounds very interesting.

Some users do have problems with Ping-Pong Method as its a totally manual based job and doesnt come as an stand alone automatic aplication tool. So someone has to get "used to it"

Now ... even before my vacations I had no time to get into that DVD Rebuilder

So I only can do (till now) assume whats your intention sansgrip.

Maybe you can enlighten me a bit whats your purpose?
(Shure Ill read your posts again, but I just dropped my baggage and got in here to see what happened the last 2 weeks
Reply With Quote
  #12  
07-24-2004, 06:11 PM
Dialhot Dialhot is offline
Free Member
 
Join Date: May 2003
Posts: 10,463
Thanks: 0
Thanked 0 Times in 0 Posts
I want to add my 2 cents on this thread.

I am currently encoding a DVD (LOTR - King's return) with DVB-RB "twaeked" by RBOpt (thansk to Sansgrip to have made me discover this !).

As the encoding was awfully long, I stopped it and started to change the ecl generated to encode in OPV insteed of 2pass. Very simple to do even with a text editor. Then I did a prediction in OPV mode on the whole movie to see the Q factor to put in the ecl.

big problem !

The sample size to reach was 33 MB. At Q=20 I had a file of 45Mb. At Q=25 it was still 41 MB. And even at Q=40 it was above the target

In few words : I'm not sure OPV can replace easily the 2pass mode in any case
Reply With Quote
  #13  
07-24-2004, 07:16 PM
SansGrip SansGrip is offline
Free Member
 
Join Date: Nov 2002
Location: Ontario, Canada
Posts: 1,135
Thanks: 0
Thanked 0 Times in 0 Posts
Quote:
Originally Posted by incredible
Maybe you can enlighten me a bit whats your purpose?
As it stands right now, DVD Rebuilder is an almost perfect tool for performing DVD9-DVD5 backups. It only works on the main movie and extras, leaving the rest of the structure intact. It means that I can re-encode the main movie and extras with CCE and yet keep the menus and overall disc structure as it is. And all this with a "one-click" tool.

There are a couple of weaknesses, though. First, it encodes on a cell-by-cell basis. While in my experience this isn't much of a problem in terms of quality, it can produce some weirdness when everything is remuxed. For example, there's a known issue where certain Sony standalones will freeze up on a transition from one VOB ID to another, in certain circumstances. This may be related to a large difference between the bitrate at the end of one VOB ID and the bitrate at the start of another.

The other weakness is that one can't perform any kind of field structure correction, be it IVTC or otherwise.

My plan is to create a tool that will be run instead of (or in addition to) "phase 2" of the DVD-RB process, which is encoding. It will make it possible for DVD-RB to work with IVTC'd material, for example, and will implement a more accurate quality-based encode than the one that's built in. It could even provide support for TMPGEnc -- right now DVD-RB only supports CCE, Rejig and QuEnc.

Depending on how complex I wanted to get, it could also enable one to encode the entire video title set at once instead of on a cell-by-cell basis.
Reply With Quote
  #14  
07-24-2004, 07:23 PM
SansGrip SansGrip is offline
Free Member
 
Join Date: Nov 2002
Location: Ontario, Canada
Posts: 1,135
Thanks: 0
Thanked 0 Times in 0 Posts
Quote:
Originally Posted by Dialhot
I am currently encoding a DVD (LOTR - King's return) with DVB-RB
Wow, talk about starting off with a difficult challenge!

Quote:
I stopped it and started to change the ecl generated to encode in OPV insteed of 2pass.
This is probably a bad idea. Better to rerun the "prepare" phase in OPV mode.

Quote:
The sample size to reach was 33 MB. At Q=20 I had a file of 45Mb. At Q=25 it was still 41 MB. And even at Q=40 it was above the target
Why not run the prepare in OPV mode? It'll do the file size prediction for you. It's not always very accurate, but if an encode of mine turns out less than 10% oversized I just do a quick DVD Shrink on it to get it to the desired size.

This is part of the reason I want an external tool, to implement something much closer to RoBa, in which a final 1-pass VBR is performed if the result is oversized.
Reply With Quote
  #15  
07-24-2004, 08:00 PM
Dialhot Dialhot is offline
Free Member
 
Join Date: May 2003
Posts: 10,463
Thanks: 0
Thanked 0 Times in 0 Posts
Quote:
Originally Posted by SansGrip
Wow, talk about starting off with a difficult challenge!
???
If you find that putting 3h15 of movie on a DVD is a challenge, I prefer to stop immediatly to use DVD-RB. My regular "hand-made" KDVD contains 4h to 5h of material

Quote:
This is probably a bad idea. Better to rerun the "prepare" phase in OPV mode.
Find me the way !!!

This tool is a cell-by-cell 2pass based shit. I'm really pissed of ruining my time for that. I just saw the result of the normal 2pass process (that I relaunch after my failure with OPV) and it is already in the trash bin.

Quote:
Why not run the prepare in OPV mode?
Because that won't xhange the result !
I did my prediction for OPV using the D2V generated for the whole movie. Unless this d2v is not the same in both mode, the result I obtained is the same.

Note: I'm not sure we are talkign about the same OPV. I'm talking about a Quality based MPV and I don't see ho wDVD-RB can perform that as it needs to do file size prediction based on sample. Is it what it does ? (I won't lost again my time for testing it but I'm curious to know the answer).

Quote:
I just do a quick DVD Shrink onVB it to get it to the desired size.
You transcoded a reencoded video and you still find the result is correct of your HD tv ? I see... Definitely not the same eyes
Reply With Quote
  #16  
07-24-2004, 10:18 PM
kwag kwag is offline
Free Member
 
Join Date: Apr 2002
Location: Puerto Rico, USA
Posts: 13,537
Thanks: 0
Thanked 0 Times in 0 Posts
Quote:
Originally Posted by Dialhot
You transcoded a reencoded video and you still find the result is correct of your HD tv ? I see... Definitely not the same eyes
A ~10% transcode should not make that much difference
That is, if the first transcode result is "prime quality" on a HDTV.

-kwag
Reply With Quote
  #17  
07-25-2004, 05:37 AM
Dialhot Dialhot is offline
Free Member
 
Join Date: May 2003
Posts: 10,463
Thanks: 0
Thanked 0 Times in 0 Posts
[quote="kwag"]
Quote:
Originally Posted by Dialhot
A ~10% transcode should not make that much difference
I do not speak about a 10% transcoded DVD, I'm talking about a 10% transcoded KDVD.
I don't know if that comes from the matrix or the low bitrate, but DVDShrink comes crazy (slight ghosting and some choppiness)*.

Do the test, and tell me

*note : that can come also from long GOPs (24 picture) that I used to use. I don't remember how were my KDVD when I discovered that and I don't use long GOP anymore BTW.
Reply With Quote
  #18  
07-25-2004, 08:58 AM
SansGrip SansGrip is offline
Free Member
 
Join Date: Nov 2002
Location: Ontario, Canada
Posts: 1,135
Thanks: 0
Thanked 0 Times in 0 Posts
Quote:
Originally Posted by Dialhot
Quote:
Originally Posted by SansGrip
This is probably a bad idea. Better to rerun the "prepare" phase in OPV mode.
Find me the way !!!
In the Options -> CCE Options menu, check "One Pass VBR (w/analysis)" before doing the prepare phase.

Quote:
Note: I'm not sure we are talkign about the same OPV. I'm talking about a Quality based MPV and I don't see ho wDVD-RB can perform that as it needs to do file size prediction based on sample.
That's what it does .

Quote:
You transcoded a reencoded video and you still find the result is correct of your HD tv ? I see... Definitely not the same eyes
In my experience a transcode to drop 10% or less is barely noticible.

EDIT: For example, my re-encode of Spirited Away came out ~5% oversized, so I used DVD Shrink on it. It still looks absolutely identical to the original.
Reply With Quote
  #19  
07-25-2004, 09:09 AM
SansGrip SansGrip is offline
Free Member
 
Join Date: Nov 2002
Location: Ontario, Canada
Posts: 1,135
Thanks: 0
Thanked 0 Times in 0 Posts
Quote:
Originally Posted by Dialhot
My regular "hand-made" KDVD contains 4h to 5h of material
This is precisely the reason we need an external tool to do the encode phase. DVD-RB is limited as to the CCE options one can set (though RBOpt allows a bit more tweaking). But if you can think of an easier way to re-encode without DVD-RB and keep all the existing subs, audio streams and menus intact I'd love to hear it .

EDIT: That's assuming you use CCE for your re-encodes. If not, we need to make the tool able to fire up TMPGEnc to do it.
Reply With Quote
  #20  
07-25-2004, 10:07 AM
Dialhot Dialhot is offline
Free Member
 
Join Date: May 2003
Posts: 10,463
Thanks: 0
Thanked 0 Times in 0 Posts
Quote:
Originally Posted by SansGrip
That's what it does .
Oh! I was on the edge to start a new test with an other movie. I'm going to try OPV mode so.

Quote:
It still looks absolutely identical to the original.
To clarify the actual situation : are you using KVCD matrix or not ?

If I need a shrink after my current attempts, I'll tell you if I find the same problems I related in my previous post. As I said, that was perhaps due to a long GOP.

Edit: OH MY GOD ! That works (or seems to for the moment...) : DVD-RB had a Qbased mode using file size prediction.
I lost several nights to test the default 2pass mode and never think to look toward the proposed 1pass mode :banghead:
Why the hell this isn't the default mode ?

Note: the only drawback I see for the moment is that the Q is computed during phase 1 so we can't use the KVCD-Notch matrix (as it is introduced byt RB-opt before phase 1).

I'll tell you about the results (but the Q factor foudn for this movie is 7 : I don't see how the result can be bad )
Reply With Quote
Reply




Similar Threads
Thread Thread Starter Forum Replies Last Post
Bitrates: Problem with preparing phase Encoder Master Video Encoding and Conversion 17 01-15-2005 11:20 AM
DVD-RB / DVD-RB OPT or manual encode? break Video Encoding and Conversion 7 10-20-2004 02:44 PM
KVCD: problems with the prediction phase prior to encoding in TMPENc? Blubear Video Encoding and Conversion 5 07-06-2004 11:52 AM
HeadAC3he: Bad audio Sync, words out of phase from video CheronAph Audio Conversion 11 09-23-2003 05:30 AM
Avisynth: Interlace artifacts or phase-shifting? ozjeff99 Avisynth Scripting 0 04-29-2003 10:15 PM

Thread Tools



 
All times are GMT -5. The time now is 07:21 AM  —  vBulletin © Jelsoft Enterprises Ltd