Replacing DVD Rebuilder's encode phase with manual encoding?
 

Okay, this has caught my interest so I figured I'd make a new thread. The original conversation started here.

The basic idea is to use DVD-RB's many awesome features but replace the encoding phase with a custom step. One would run the "Prepare" phase and then run an external tool to tweak various values and launch the encoder. Once all the cells were encoded one would then use DVD-RB's "Rebuild" phase to remux everything together.

The advantage is that we gain a significant amount of control over the encoding process, can use file size prediction, custom resolutions, overscan, GOP length, etc. We can even enable TMPGEnc support in DVD-RB.

I have other plans for it, too. I'd like to be able to run IVTC in my Avisynth scripts, but DVD-RB currently makes this impossible. Technical details follow: skip a few paragraphs if you're not interested.

The problem is that when DVD-RB generates its D2V file it sets all the frames to "TFF, no RFF" (top field first, no repeat field flag), regardless of how the flags were set in the original. The encoder is then passed the raw fields from the video stream, and the new fields are remuxed into the VOBs. When it's remuxed DVD-RB sets the field flags back to how they were originally. The purpose of this is to leave things as unmolested as possible, so that the user doesn't have to worry about forced film mode and so on. The field structure will remain exactly the same as the original.

This is great for truly progessive or truly interlaced streams, but falls down when it comes to hybrid or hard telecined material. Because Avisynth doesn't see the frames "as they're meant to be seen" it's impossible to correct problems in the original mastering, such as hard telecined material. It also makes applying filters suboptimal because with hybrid or hard telecined material one can't convert to progressive in Avisynth before running filters, which means filters can end up being run on interlaced frames. This is not good.

My plan is to replace DVD-RB's D2V file with one that actually reflects the reality of the stream's field flags so that Mpeg2Dec will produce a "what is meant to be seen" stream. One can then IVTC in Avisynth, if necessary, to make true progessive frames. These can then be filtered much more accurately, and any interlace artifacts in the original can be fixed. This still leaves the problem that, upon rebuilding, DVD-RB will set the flags back the way they were in the original. Because the tool will have made the material truly progressive (or truly interlaced, as the case may be) this approach will not be correct in many cases, and playback will be messed up. Thus the tool will need to fix up the flags. I believe that DVD-RB stores the original flags in files called "Vxxxxxxxxxxxxxx.FLG", so it should be possible to simply rewrite those files with the correct flags (for truly progressive material the correct flags would be, in DVD2AVI's parlance, 0 1 2 3 0 1 2 3 0 1 2 3... and for truly interlaced material it would be either 0 0 0 0 0... or 2 2 2 2 2... depending on whether it's top- or bottom-field first.)

End of geek talk ;).

So this tool wouldn't just be a custom encode step -- it would also fix what I consider to be DVD-RB's greatest weakness, which is the inability to properly fix badly mastered material because of the way it handles field flags.

I'm more than willing to make this my next coding project, so let's kick some ideas around. This kind of tool could make DVD-RB even more of a killer app than it already is, so let's think of features, perhaps figure out a friendly interface, see if we can incorporate support for TMPGEnc as well as the encoders DVD-RB already supports, etc.

We also need to thrash out the details of how the file size prediction will work. I've not looked at that stuff in a long time...

Regarding prediction, search for incredible's posts for the Slicer idea. It has been discussed in several threads and I think that his approach will give the best results for most material.

Willdo. Thanks.

I think I found the thread where this method was developed, but I'm left a little confused as to the exact method involved. Am I right in saying that it involves taking two sets of samples, one offset by 30 seconds, and averaging them?

I also read about Kwag's "random samples" idea, but I'm not sure if this will gain anything over the usual method of sampling (the only difference is that instead of being sequential the sampled segments are now out of order).

What is the current "canonical" method for file size prediction? Is there one? ;)

Inc should be back next week and I'm sure he will find both your posts and your ideas really freaky. He will give you any answer you need, and much more :-)

Hmm. I like the look of the Newton method. Sounds like it's way faster than a binary search.

Here's what I feel might be a good approach to this particular problem:

* Ignore first 5 minutes of movie (or a certain percentage)
* Ignore last 5 minutes of movie (ditto)
* Make each sample at least one GOP in length, probably more like 3 or 4
* Pull enough samples, evenly spaced, from what remains of the movie, to make up 1%, 2%, 3% (or whatever) of the total length
* Guess (somehow) at a Q to start, and use the Newton method to locate a good value (EDIT: Looks like r6d2's "Newton's Iteration 2" method is the way to go.)
* Maybe take another sample, with different segments of the movie, and do it again, then average?
* Create .vaf file with that Q
* Depending on accuracy, do a VBR pass using that .vaf but specify an exact average bitrate to hit the target

Sound about right?

Just thought I'd point out that I don't particularly want this turning into yet another file prediction method thread -- I'm also interested in hearing feature suggestions for this (potential) tool, and comments on my idea regarding fixing field flags so we can IVTC and filter with DVD-RB.

Size prediction is just one part of the (potential) tool :).

Mmm. These steps are complicated by the fact that we're going to be encoding each cell separately. I think what will be required, in this circumstance, is one "pass" to create all the .m2v files, then add up the file sizes to compare with the desired. We then decide if we need a second go at it using VBR on the existing .vaf files and a set average bitrate. That should work. I think...

;)

(God I love this stuff.)

Yes, sounds good to me too. The good and bad thing about CCE is that it uses an integer value for Q so you'll get it predicted a lot faster than with TMPGEnc.

reply
 

I don't know the Newton method, about inc's prediction, I have used it
time ago, but i haven't had a good results. I have used it for more divx
on a kdvd.

I think that the prediction with sample is the best.


by

Im back and it sounds very interesting.

Some users do have problems with Ping-Pong Method as its a totally manual based job and doesnt come as an stand alone automatic aplication tool. So someone has to get "used to it" ;-)

Now ... even before my vacations I had no time to get into that DVD Rebuilder 8O

So I only can do (till now) assume whats your intention sansgrip.

Maybe you can enlighten me a bit whats your purpose?
(Shure Ill read your posts again, but I just dropped my baggage and got in here to see what happened the last 2 weeks :lol:

I want to add my 2 cents on this thread.

I am currently encoding a DVD (LOTR - King's return) with DVB-RB "twaeked" by RBOpt (thansk to Sansgrip to have made me discover this !).

As the encoding was awfully long, I stopped it and started to change the ecl generated to encode in OPV insteed of 2pass. Very simple to do even with a text editor. Then I did a prediction in OPV mode on the whole movie to see the Q factor to put in the ecl.

:arrow: big problem !

The sample size to reach was 33 MB. At Q=20 I had a file of 45Mb. At Q=25 it was still 41 MB. And even at Q=40 it was above the target :-(

In few words : I'm not sure OPV can replace easily the 2pass mode in any case :x

As it stands right now, DVD Rebuilder is an almost perfect tool for performing DVD9-DVD5 backups. It only works on the main movie and extras, leaving the rest of the structure intact. It means that I can re-encode the main movie and extras with CCE and yet keep the menus and overall disc structure as it is. And all this with a "one-click" tool.

There are a couple of weaknesses, though. First, it encodes on a cell-by-cell basis. While in my experience this isn't much of a problem in terms of quality, it can produce some weirdness when everything is remuxed. For example, there's a known issue where certain Sony standalones will freeze up on a transition from one VOB ID to another, in certain circumstances. This may be related to a large difference between the bitrate at the end of one VOB ID and the bitrate at the start of another.

The other weakness is that one can't perform any kind of field structure correction, be it IVTC or otherwise.

My plan is to create a tool that will be run instead of (or in addition to) "phase 2" of the DVD-RB process, which is encoding. It will make it possible for DVD-RB to work with IVTC'd material, for example, and will implement a more accurate quality-based encode than the one that's built in. It could even provide support for TMPGEnc -- right now DVD-RB only supports CCE, Rejig and QuEnc.

Depending on how complex I wanted to get, it could also enable one to encode the entire video title set at once instead of on a cell-by-cell basis.

Wow, talk about starting off with a difficult challenge! ;)

This is probably a bad idea. Better to rerun the "prepare" phase in OPV mode.

Why not run the prepare in OPV mode? It'll do the file size prediction for you. It's not always very accurate, but if an encode of mine turns out less than 10% oversized I just do a quick DVD Shrink on it to get it to the desired size.

This is part of the reason I want an external tool, to implement something much closer to RoBa, in which a final 1-pass VBR is performed if the result is oversized.

???
If you find that putting 3h15 of movie on a DVD is a challenge, I prefer to stop immediatly to use DVD-RB. My regular "hand-made" KDVD contains 4h to 5h of material ;-)

Find me the way !!!

This tool is a cell-by-cell 2pass based shit. I'm really pissed of ruining my time for that. I just saw the result of the normal 2pass process (that I relaunch after my failure with OPV) and it is already in the trash bin.

Because that won't xhange the result !
I did my prediction for OPV using the D2V generated for the whole movie. Unless this d2v is not the same in both mode, the result I obtained is the same.

Note: I'm not sure we are talkign about the same OPV. I'm talking about a Quality based MPV and I don't see ho wDVD-RB can perform that as it needs to do file size prediction based on sample. Is it what it does ? (I won't lost again my time for testing it but I'm curious to know the answer).

You transcoded a reencoded video and you still find the result is correct of your HD tv ? I see... Definitely not the same eyes ;-)

A ~10% transcode should not make that much difference :!:
That is, if the first transcode result is "prime quality" on a HDTV.

-kwag

[quote="kwag"]I do not speak about a 10% transcoded DVD, I'm talking about a 10% transcoded KDVD.
I don't know if that comes from the matrix or the low bitrate, but DVDShrink comes crazy (slight ghosting and some choppiness)*.

Do the test, and tell me :-)

*note : that can come also from long GOPs (24 picture) that I used to use. I don't remember how were my KDVD when I discovered that and I don't use long GOP anymore BTW.

In the Options -> CCE Options menu, check "One Pass VBR (w/analysis)" before doing the prepare phase.

That's what it does :).

In my experience a transcode to drop 10% or less is barely noticible.

EDIT: For example, my re-encode of Spirited Away came out ~5% oversized, so I used DVD Shrink on it. It still looks absolutely identical to the original.

This is precisely the reason we need an external tool to do the encode phase. DVD-RB is limited as to the CCE options one can set (though RBOpt allows a bit more tweaking). But if you can think of an easier way to re-encode without DVD-RB and keep all the existing subs, audio streams and menus intact I'd love to hear it :).

EDIT: That's assuming you use CCE for your re-encodes. If not, we need to make the tool able to fire up TMPGEnc to do it.

Oh! I was on the edge to start a new test with an other movie. I'm going to try OPV mode so.

To clarify the actual situation : are you using KVCD matrix or not ?

If I need a shrink after my current attempts, I'll tell you if I find the same problems I related in my previous post. As I said, that was perhaps due to a long GOP.

Edit: OH MY GOD ! That works (or seems to for the moment...) : DVD-RB had a Qbased mode using file size prediction.
I lost several nights to test the default 2pass mode and never think to look toward the proposed 1pass mode :banghead:
Why the hell this isn't the default mode ? :-)

Note: the only drawback I see for the moment is that the Q is computed during phase 1 so we can't use the KVCD-Notch matrix (as it is introduced byt RB-opt before phase 1).

I'll tell you about the results (but the Q factor foudn for this movie is 7 : I don't see how the result can be bad ;-))

Nope. As you say, OPV mode is somewhat inflexible because the prediction pass is done before one is able to change parameters. Another reason we need an external tool to do it :).

Yep. Cool, huh?

I think because it's still not always very accurate. But we can fix that.

It won't be bad. 7 is very low. It should be excellent quality :D.

Hum.... it is not. REALLY not. Dancing block on all the walls and otehr flat areas.
I think I definetly can't suffer anymore the default mpeg matrix ! :puke:

I do a last try with a patched CCE with KVCd matrix and if it does not work, DVD-RB will return to the trash bin.

EDIT: I forgot that there is no way to patch a matrix in 2.67 as it handles custum matrixes. So DVD-RB always switch back to "standard" matrix and that gives crap :-(

EDIT2: Ok, matrix patched manually into the exe. I restart the process. Once again...

That's very strange. What does your Avisynth script look like?

I think read somewhre that to make it use a custom matrix you have to delete the standard one. Or maybe not...

Very basical :
Basical but that is the script I use for all my KDVD and that gives perfect results.

You can't. There are 4 (why/which ? I do'nt know) matrix hard coded in the exe. I changed them all. The others available matrix are templates saved in the registry database. They includes the standard matrix but if you erased it AND not put any matrix in the ecl, then the hard coded standard one is used.

info
 

with dvd-rb i'ts possible "rebuild" two dvd (with menu+1audio+1subtitle) on 1 dvd.

byby

As I know all CCE matrices are stored in the registry like when we do choose Notch whn doing our KDVDs using CCE.

Now could it be that if DVD-RB uses standard CCE Matrixes, that the FIRST matrix known as standard will be used out of the registry?? If yes, you can try to change the standard Matrix Values to Notch ones.
Just a thougt.

Inc, as I said : I patched the exe. I can swear to you that there are hard coded matrix in the exe. Open ccspt.exe with an exe editor and search for "08 10 13" (begining of standard matrix). You will find four of them.

I'm not sure you read my last post before to write that but I answer to all of that in it. And the answer is NO.

I try to use DVD-RB with Notch Matrix since 10 days, you can imagine that this was the first trick I tried )

I've never seen dancing blocks on any CCE encode I've done. Perhaps Undot and Deen are removing too much noise?

But if you're not happy with the CCE encodes you're doing, then that's not a problem with DVD-RB, it's a problem with CCE. DVD-RB, obviously, doesn't do the encoding itself.

I've not been around long enough to know if you usually use CCE or TMPGEnc, but whatever encoder you use, it should be possible to precisely duplicate your usual encode settings if we develop an external tool to do the encode phase.

Don't blame DVD-RB for flaws you notice in the output. Instead think how great it would be if we had an external tool to do the encode phase in line with the usual KDVD techniques and settings, combined with DVD-RB's ability (unique amongst freeware) to remux that new video back into the original VOBs so that we can effortlessly retain menus and subs in our backups...

That being so, it would be very helpful if you could summarise the "canonical" approach to making a KDVD so that I can begin designing this tool and, if necessary, research ways of automating TMPGEnc, etc.

I never had that with my manual KDVD that I do also with CCE ! The only diff ? The matrix !
The encode (with KVCD-Notch matrix patched) was finished yesterday but I encoded on my laptop and I want to transfer it on my main PC because the laptop screen is not the best. I'll tell you tonight.

I should have mentionned that I already did 50 KDVD with CCE ;-)
Even if the first ones had some problems we named "movie paint" aspect due to the script that removed too much.

Basivally I encode using the CCE 2.67 template VMesquita developped for his guide , here :
http://www.kvcd.net/forum/viewtopic.php?t=5142
And the script I already gave to you.

As long as DVD-RB does not allows to use KVCD-Notch matrix in OPV mode, can I ? ;-)
Note: did you notice that RB-OPT is delivered with a bunch of matrixes but NOT the KVCD one ? Doom9... tsss...
(even if CCE is delivered with old (obsoletes) KVCD matrixes!)

By canonical you mean "by hand" ?

This is my method :
- DVDShrink (no compression mode) to isolate the movie and cut the credits in the same operation
- DVD2AVI + CCE for the video
- AC3Delay or AC3Machine for adding the delay or reencode the audio
- subrip for turning subtitles in SRT then srt2sup to use them into IFOEdit
- Ifoedit to do a 2 audio + 1 subtitle VOB(s) (+ chapters if I need them here)
:arrow: supremap to set correct colors in subtitles :!:
- DVDShrink (no compression) again to merge these vob in one single file
- DVDLab Pro to compile the DVD and menu if I need only a 2 movie disc or I want to set different chapters than original ones.

Actually blue part is the one that takes me the more time (encoding time excluded of course). It's boring but you can avoid it if you do only 1 audio movies OR use an other authoring programm taht DVDLab Pro.
Red part can also be avoided if you don't want to make menus (IFOedit produces things ready to burn)

@Dialhot
A bit OT here, but:
Why don't you use spumux to mux the subs and dvdauthor to author without menu? Both are free and take nearly no time (except for processing itself). You can import the VOBs in DVD-Lab later.

1/ spumux : I read quickly the manual but it seems that to use a graphical type of sub (easy to obtain from the vobs with subrip) you need to generate a xml file to use with spumux. I'm i wrong ? I never took the time to do a perl script to generate such xml.

More: spumux must be used ON THE VOBS already generated, don't they ? How can you be sure that all is in synch ?I have sometimes 5 second of delay in my subtitles. With a srt file I can adjust them before to convert them.

And even with srt file, the gain won't be big as this is the OCR step of subrip that is boring and time consuming.

2/dvdauthor : Again I read quickly the manual and never see if it was abble to do 23 movie DVD with 2 audio track + subtitle. Is that possible ?

Just to tell : I compared the two encoding I did of the same movie on the same PC and screen (so all filters and hardware were the same) : the KVCD Notch is CLEARLY better ! All is smooth and pleasant as it always be. MPEG standard matrix really sucks :-)

And an other cool thing :
Standard matrix : Q factor found by DVD-RB for movie : 7, for extras : 32
KVCD-Notch : Q =6 and 29.

Okay. So let's get started on this tool.

The basic process would be, I think:

1) Run DVD-RB "prepare" phase
2) Run our tool, which does file size prediction and then updates the rebuilder.ecl file
3) Run DVD-RB "encode" phase
4) Run DVD-RB "rebuild" phase

Shouldn't be too difficult.

No it isn't and it was my idea. One of this idea I never have the time to develop and test :-(. That's the problem when sleeping 4 hours per night : there are 4 lost hours :-P

Note: a was a little too much enthousiastic abotu result with KVCD. Always blocks in the very dark areas (but lot less). I wonder if they aren't on the source ? Hum... No time to check, I have ot burn this disc and go for the next one :-d

OK quit talking about it and get to work :D ,just kidding.

sounds like a great tool and i can't wait

ARe we going to be able to use the notch matrix.

Are we on KVCD.net or not ? :wink:

In older versions, you could use graphical .sub/bmp files, the same input that winsubmux takes. And you can convert from vobsub format using vsconv. But I am not sure it's still possible (I hope it is, otherwise the implementation in DIKO will be harder)
You must mux the audio and video using MPLEX, and then mux subs with spumux. I am not sure about how to adjust, maybe subtitle workshop can help?
Yes, I know, that's why I am suggesting all this.
DVDAuthor can do nearly everything the DVD specs allow, so you can expect to be able to do 99 VTS, 32 subtitles and 8 audio tracks. I am just not sure about seamless branching and multiangle, but the rest, get your XML ready and you're all set.

yes we are :D but it doesn't seem like its that easy to do

The problem is that you are also saying that what you are suggesting does not work anymore :-)

So can you clear this up ? What usage can I have of spumux starting from a VOB source ?

In fact I was talking about the GUI : I knwo that the exe can do all that, but as you say, you need the XML. And that is the GUI that helps in this task that not seems to be manipulate all the stuff I need.

Is there someone that use it and can answer to me ?

Actually I am saying that I don't know if it still works. It worked before XML support was added. I never tried this again after XML support was added (I used this to mux subs rendered in MastroSBT in SVCD mode, so after they added XML it became much simpler to use SRT straight)

Unfortunatelly, I haven't used for a long while... :( But I'll be doing some tests this week because I want to get this working in DIKO, so I'll get back to you after that.
Somebody posted a GUI in DVD Authoring forum, I think. But I have not tried.