Simple approach to re-encoding? (NOT transcoding)
 

I've been playing around with DVD Shrink and, while I like the program, am not really satisfied with the quality. Needless to say, CCE and TMPGEnc do a far better job.

If I wanted to make a disc with just the movie on it, and no menus or extras, it would be simple to encode the video, remux with the audio and stick it on the disc. But menus and extras introduce a certain level of complexity, and I'd rather not reconstruct the entire DVD (minus subs and non-English language audio tracks) in an authoring program.

I've spent the last few days first trying to find information on IFO and VOB files, and then trying to understand what I did find. I'm thinking what is required is this:

1. Locate the VTS or PGC you want to compress
2. Demux the video from the relevent VOB(s) into an .m2v
3. Re-encode the video to your new desired size
4. Mux the video back into the VOB, maintaining the same (wrt frame count) cell/program/PGC/PTTS structure as the original
5. Update the VOB and/or IFO to reflect the changes

I've been downloading tool after tool and found none that can do the above, not even a combination of them, unless I'm missing something obvious. I'm considering writing a tool to do it -- kind of like DVD Shrink's "full disc" mode except that compression is increased through manual re-encoding instead of automatic transcoding.

Does anyone here either know of tool that does this already, or know enough about the structure of VOB and IFO files to give me some pointers towards what would need to be updated in order to remux the video in this manner?

UPDATE (warning, things get a bit technical from now on):

I just discovered VOB files are MPEG-2 program streams, and I also happened across the full ISO 13818-1 program/transport stream specifications document. It runs to 160 pages, which I figure should give me a decent start on parsing a VOB ;).

UPDATE:

Looks like demuxing will be the easy part, though the structure of DVDs raises some interesting questions. What I envisage is the software presenting the user with a summary of the DVD's structure (at first the VTSs, the PGCs, the PTTSs, the PGs and the cells; will worry about the VMG later) and the user being able to extract (demux) any of those things.

For example, on a disc containing several episodes of a TV show, each show in its own PGC within a VTS, one could extract the PGCs separately, thus pulling out the video and audio from each episode and without having to know the IDs of each cell within the PGC, as you do right now with IfoEdit. It could also dump that PGC's PTTSs (chapters) into a text file for later use, maybe.

But here's one of those interesting questions I mentioned earlier: how does one deal with, for example, PGs with multiple angles? Which angle should be demuxed? Presumably the user would be allowed to choose. But what about cell commands? Should they be obeyed? In order to demux the PG "as it really is" one would have to take into account the fact that cell commands can cause a completely different path to be taken than simply moving to the next cell in the program. For example, I recently encountered a disc that used a cell command in the second-to-last cell to unconditionally skip playback of the last cell in the program, which was about 10 seconds of black. If the program blindly demuxed the entire program without regard to the cell commands, that excess blackness would be erroneously demuxed at the end of the video. And if one is demuxing an entire program chain, should one also follow the pre- and post-commands?

I think this kind of "smart" demuxer would be a valuable tool, but there's lots of issues that need to be sorted out: following commands would involve parsing the IFO file as well as the VOBs, and writing a command interpreter. I hope those with at least some familiarity with the concepts I'm talking about will jump in and contribute. This is starting to sound like it might be a fairly complex project and I'm not sure I want to figure it all out on my own ;).

Re: Simple approach to re-encoding? (NOT transcoding)
 

You missed dvdrebuilder on doom9. But it works SO badly that is not even necessary to sepak about it (let imagine that it does a 2pass job CELL BY CELL and you can figure how bad is the result).

:ole:

If you are on doom9 you can still ask to the designer of dvdrebuild. The guy that does it did ifoedit some time ago.

Whow... you are starting from zero :-). This info I was abble to tell you :-)

There is no real added value if you do not handle the VGM. I mean that in such a case (TV show), you will have to process ep by ep, then reauthor them onto a single DVD. This is already done "by hand" with dvddecrypter (to isolate the ep) + dvd2avi + cce/tmpgenc.

Yes they are. Cell and angles are sometimes used as a tricky copy protection. See on doom9 there is a guide about how to backup such disc (Nemo for instance). And that a tought story !

I had a trickier case for you : I have a DVD Boxset with a carton TV show. Each DVD fits 10 episodes but the start and end credits are duplicated only ONE time, in cell 1 nd 2. Ep one is in cell 3, ep two is in cell 4... The playing order is then 1-3-2, 1-4-2, 1-5-2...

I'm not really aware about taht. Can't answer...

You see now why you didn't find any tool already done on the market :-)

Hi SansGrip,

This sounds like a very complex project and I wish you luck in your task.

Im sure there are many ppl here who would be more than happy to help out as there are many ppl here who do know a lot about audio/video structures.

Like they say nothing is impossible if you put your mind to it, I look forward to seeing the development of this project. :)

Best of luck to you buddy. :D

Re: Simple approach to re-encoding? (NOT transcoding)
 

Actually I was considering a cell-by-cell approach, if only because it avoids the problem of figuring out cell play-order. But that's not to say one would have to encode each cell separately (which would severely limit the encoder).

What if the tool worked on a cell-by-cell basis, but when it demuxed them it combined them into one elementary stream? That one file would then be re-encoded, and split (with exact frame accuracy, of course) back into the separate cells.

One would need to ensure that the cell boundaries began with an I frame, but if the GOP size is known in advance that's possible.

That's not a bad idea. IfoEdit is a pretty cool program. I just wish it were a little easier to use...

Yep. When I was doing KVCDs VOB files were just "black boxes" to me. I let DVD2AVI and Mpeg2Source() deal with that stuff...

I've only inspected a few discs in detail so far and I've not seen any that contain anything important in the VMG. That said, it should be possible to demux from the VMG too just in case...

Interesting, thanks. I'll do a search for that thread.

That makes sense -- why encode cells 1 and 2 more than once if they're identical? But how was this play order achieved? I'm assuming the episodes were all in one PGC, and were given one PG each, with the program specifying "cell 1, cell 3, cell 2" or "cell 1, cell 4, cell 2" and so on. In that case I'm pretty sure cell commands aren't involved, though it could conceivably be done with general registers and cell commands...

I think a good "smart" demuxer would allow cells to be demuxed individually, but also allow an entire PG, PTTS, PGC or VTS to be demuxed too. In the case of demuxing more than one cell, there are two ways of doing it:

1) The "dumb but safe" way, which is to demux all the cells in the PG, PTTS, PGC or VTS

2) The "smart but dangerous" way, which is to attempt to follow cell commands as part of the demuxing process. This is more difficult than it sounds, though. Imagine the following scenario, designed by a Bastard DVD Author From Hell to mess up this program:

- The FP PGC runs the usual FBI warnings and trailers, but in one of the cell commands deep in the VMG it sets register 8 to 0

- The main menu is displayed, and the "Play Movie" button chosen. It links to VTS 3, PGC 2

- VTS 3, PGC 2 contains nothing except a black clip and a post-command which sets register 8 to 1, then jumps to VTS 3, PGC 1

- Cell 1 of PGC 1, which contains the movie and is the one we're trying to demux, contains cell commands which mess up the playback order if register 8 is 0, perhaps by running in a loop

This kind of "copy protection" would be hard to circumvent, because it would require the program to begin processing at the FP PGC and run all the way through to the PGC or even PG that we want to demux. In the case above, this would even include prompting for user input upon reaching a menu.

As you can see, there are difficulties with blindly following cell commands. I'm going to have a think on it, but I'd appreciate input from anyone who has any thoughts about this.

Yes, but I'm still surprised. On the one hand we have the very, very powerful and very, very hard/tedious to use IfoEdit, and on the other "one-click" solutions that offer very little opportunity for customisation. There is definitely a need for a tool "in the middle" of these two, that is powerful and configurable yet doesn't require an in-depth knowledge of each IFO structure...

It's starting to sound complex, yes ;).

Sansgrip,

you may want to drop Nic at doom9 a PM as I believe his next version of rejig was along the lines of what you are proposing. Not sure if its dead in the water but he may be able to give a few pointers on it,

cheers,

Re: Simple approach to re-encoding? (NOT transcoding)
 

That is exacly why DVD-RB works also cell-by-cell. And I perfectly understand that choice. But the bad choice is to encode in 2pass AND cell-by-cell.

If you want to encode a movie in small part, that can be achieve only in a quality based encoding mode else the bitrate repartition is completly screwed up. Do you understand or may I developp ?

Yep. When I was doing KVCDs VOB files were just "black boxes" to me. I let DVD2AVI and Mpeg2Source() deal with that stuff...

Isn't the VGM supposed to replace the VTS Root menu on multi-vts disc ?

I do not exactly understand the diff between the two option in the end of your sentence, but for the begining, yes: the disc was authored as you say.

I don't remember exactly but it seems taht this IS the tricky copy-protection I told you above and for which you can find a guide on doom9 (I will look for it if you do not find it).

Not really but just for you to have this in mind : let think also about seamless branching.

It would seem to me that instead of attempting to follow cell commands, one should allow the user to select the exact order in which cells are extracted from a program. This way if the cells are out of order for whatever reason the demuxed ES will still be correct.

Joining cells into one ES would be fairly straightforward, but the problem comes when splitting the re-encoded stream back into its component cells. Let's say we have three cells, the first 56 frames long, the second 28 frames long and the third 102 frames long. When muxed together this would produce a 186-frame ES, which could then be re-encoded. But it's essential to ensure that frames 57 and 85 are I-frames, and not P- or B-frames, so that the file can be split between frames 56/57 and 84/85.

CCE apparently has a facility to do this, by setting chapter points, but I'm not sure how well it works. The other option is to attempt to pad each cell with extra frames, if necessary, to match the GOP size used by the encoder. This way we know the I-frame will fall on the frame we require, and can (presumably) remove the extra frames we added upon splitting the re-encoded ES back into cells...

Is this viable? Is there a simpler way?

Thanks for the tip -- I'll check out rejig and get in touch with him :).

Tmpgenc too ! In Manual VBR mode.

And use "always use closed GOP" can't help ?

Re: Simple approach to re-encoding? (NOT transcoding)
 

I understand. Multipass encoding is basically useless when you're dealing with a bunch of separate files (each one possibly no bigger than a hundred megs or so). As you say, some sort of "constant quality" setting would work better, but I'd still rather be able to do a 2- or 3-pass VBR over a much larger clip, which means somehow joining and then resplitting the cells.

I've examined a couple of multi-VTS discs and both of them had the main menu in the same VTS as the movie. I believe the VMG menu is provided so that the author can implement a "title select" menu upon disc insertion, but in practice seems only to contain copyright warnings and studio animations, as well as stills for region protection and parental control stuff. I don't remember ever playing a disc that displayed a menu before the main one, but then again I've not really been looking...

What I meant was something along these lines:

A menu is displayed to choose the episode you want to watch. The button for that episode sets a register (1 == 1st episode, 2 == 2nd, etc.), then jumps to the opening credits cell. This cell contains a command which checks the register and jumps to the correct cell to play the episode itself, which then (via a cell command) jumps to the end credits cell. I know that a cell can only contain one command, and those commands are limited to a subsection of the entire command set, so I don't know if the above is actually possible. I'd need to look at the command set in more detail to say for sure.

Yes, that's also an issue. I believe that seamless branching has something to do with navigation packs, but I know little else about it. Do you know any details?

Yep. But I don't really like the sound of either method -- I'd rather it was done "automatically" by the tool. Having to set chapter points, or mess with GOP structures, manually for every movie would be a pain in the ass.

I'm not sure. As I said, I know little about the details of MPEG video. I need to read ISO 13818-2. But I'd prefer for this kind of manipulation to be done by the tool, instead of relying on the user to set the correct option in their encoder...

Re: Simple approach to re-encoding? (NOT transcoding)
 

I like to talk with people that can understand a problem in 2 seconds :-)

How ! They have a lot !
Any Disney DVD (at least the ones sold in Europe) have a "pre-menu" where you select the language you want to read the disc. Then the next menu (root menu of the DVD) is displayed in the chosen language, and when you select "play" the movie is also palyed with that language.

BTW, I suggest you to look into Disney production because they are often the trickiest DVD on the market :-)

Okay. I didn't know all that about commands either.

No :-(

Re: Simple approach to re-encoding? (NOT transcoding)
 

I've never watched a Disney DVD, let alone one in Europe ;). I believe we have a couple of Disney discs somewhere, so I'll check them out. The R1 version might contain a language selection (maybe English and Spanish for the US, and a French option too for Canada). I figured that the "pre-menu" would most often be used for this purpose.

Thanks for the tip. I don't remember which Disney movies we own (well, apart from Kill Bill...), but I can certainly rent one to check out the authoring. Any particular title? (Is Finding Nemo a Disney movie?)

Re: Simple approach to re-encoding? (NOT transcoding)
 

Yes, all pixar movies are include in what I call "Disney production". But be carrefull : a lot of rental edition are different (less complicated) than commercial releases.
I have "The lady and the tramp" in R1 and it has a language selection menu for using between en/fr/sp :-)


(I didn't notice you are form Ontario. I'm all day long in contact with Ottawa as I work for Nortel Networks now ans my team is there :-))

I also think that the pre-menu is a R2 invention. I think I've got several DVDs that have a menu which asks for the menu language before it is shown, and most of these are Scandinavian releases (=basically Finland, Sweden, Norway and Denmark).

Good luck with the tool, just beware that it won't suck you dry too soon. We've seen too many tools abandoned because there just has been too much to fix and add :wink:

Re: Simple approach to re-encoding? (NOT transcoding)
 

Okay. I'll ask around the neighbours and find a retail version ;).

Ah yes, Nortel. I'm glad you didn't lose your job when they were downsizing recently... ;)

I have some plans for this tool which I think might make it unique enough to warrant a commercial release (perhaps a cut-down free version and a more complete "pro" version), so I'm planning on developing it for a while.

One of the features I'd like to be able to add is the ability to "deauthor" a disc -- that is, demux every single bitmap, transparency, clip, sub, audio track, etc. etc., and record all the information necessary to recreate the DVD from the component parts. This would allow one to edit, for example, a single menu graphic, or a misspelled sub, or replace video with a re-encoded version, or whatever, and then automatically reauthor the entire thing. It would place no restrictions on what gets put back into the final package -- you could remove entire titlesets, strip audio tracks, even change the running time of video clips, and it would simply adjust the IFOs upon rebuild.

It would be complicated to implement (pre-, post- and cell commands in particular need to be handled carefully) but I don't know of any other tool, free or commercial, which does the same thing...

Your tool does sound like one I have sometimes needed. I've had some poorly mastered DVDs and sure would have liked to do my own version, the best example is probably the Scandinavian release of Citizen Kane. I've written some thoughts about it in the forum, and have to say that my version looks a lot better than the original :D

I might have done something with IfoEdit but it's way too complex for me to start learning it without any extra spare time. That is, a year or two :lol:

Well, you probably know where you'll find all your beta testers.

Cool -- I'll do a search for your posts.

I've not written much about reauthoring, just the basic usage of Restore24(). I still don't understand how they could create such crap from a film NTSC source->PAL transfer..

I was thinking about how one could split the re-encoded elementary stream into its constituent cells without confusing the MPEG decoder, and came up with a simple solution. It's simple to describe, but probably very difficult to implement ;).

Imagine we have two cells, A and B, which we joined in order to re-encode. The end of cell A looks like:

...-B-I-B-P-B

and the beginning of cell B looks like:

I-B-P-B-...

Joined, that would become:

...-B-I-B-P-B- <joined here> I-B-P-B-...

But let's say after re-encoding, the same frames look like:

...-P-B-I-B-P- <split here> B-I-B-P-...

We can't split there because the new cell B must start with an I frame, and also because the end of the new cell A could potentially be corrupted due to the missing end of the GOP (in this instance it wouldn't, because it's missing only a B frame, but a missing I or P frame could cause problems).

The solution is to convert, before splitting, the last GOP of cell A and the first GOP of cell B to I frames. In our example:

...-B-I-I-I-I- <split here> I-I-I-I-...

With this achieved the split could be made at precisely the right point without losing or corrupting any frames, since I frames are self-contained and do not reference other frames or GOPs.

This would probably require the join code to implement a mini MPEG decoder and encoder, but should be possible.

Thoughts?

Why don't you just decide to split "to the next I frames found after the original cut point" ? Okay that can change a little the cutting point...

But that can be "the nearest" I-frame, and the error will by of 9 frames max (18 frame max in a NTSC GOP).

It can change the cutting point, and that's the problem. My goal is to keep the exact same structure as the original VOB.

Here's a paper describing one technique for splicing motion-compensated data in the middle of a GOP.

Hi SansGrip,

Nice to see you around with such an amazing tool idea. :D
There's a MPEG splitter that can split in any frame, converting B/P frames in I frames by reencoding. I read about it in doom9 a while ago, but unfortunatelly I can't remember the name. I'll do a search and get back to you later, maybe it's opensource so you can check how it works. :wink: Sorry for the bad memory. :oops:

Hi SansGrip,

Split at header points. Not at I frames.
That way, every splitted chunk is a valid mpeg file.
That's the way I split files with "DIKumciser".
I also add a valid ending signature.
You don't need that last signature if you plan on joining them again, but you do need it if you want to import the pieces into an authoring program.

-kwag

What are header points? The GOP header?

Valid, yes. Split at precisely the same point as the original program (or PGC, or title, or whatever), no.

gr8 idea, but got a few questions...

wil you be able to put more then one dvd, on a dvd-r, keeping the menus etc.. ?

bcuz, i bought a 4 in 1 dvd from pakistan, you could tell it was not legit, but it was quite good..

it had a first menu, which takes u to all four movies, then wen u choose one, it actually shows the proper menus, like on the retail dvds. with all extras, etc. the quality was ofcourse near dvd, with ac3 sound, but u could tell it had been trascoded.

it was a DVD-9, with 4 movies, so we should get 2 movies, with original menus, extras, etc on it with no problem..

but the question is how...?

i leave it to you sansgrip.

Bazzy

That is technically possible by creating a VMG menu above all imported menus, tht have to be VTS menu then. It seems that is what your pakistanish DVD use.

yes, but how do i do that ?

thats the question - not is it possible :? :wink:

dvd lab
dvd lab pro
tmpgenc
meastro
spruce
scenerist ($30000 8O )

??

Bazzy

You are posting your question in a thread that is on a "under study" software ! Your question use the future ! And I am supposed to understand that you ask how to do that NOW ??? :banghead:

"Yes it WILL (perhaps) be possible" was the answer to the question you had posted before. "NO it is not possible" is the one to the question you ask now.

thank you very much, for you kindest of replies.

but i was only asking.

Bazzy

There is no problem in the question (or even questions). The issue was only in the sarcastics smilies you used after my answer while the fault was in the asking itself.

ok sorry

It might turn out nice if I can ever get my head around MPEG2 program streams, MPEG2 video, VOBs, and IFOs ;).

That would be awesome if you could. Splitting the re-encoded elementary stream looks to be the most difficult part. Demuxing looks (relatively) simple. Remuxing is more complicated, but I don't think as complicated as the splitting.

I have a whole bunch of ideas in my head and I want to try to nail them down, so I'm going to use this post to try to sort out some thoughts. You don't have to read it, it's more for my benefit than anyone else's ;).

I've been trying to back up The Shield (season 1, volume 1). It's one dual-layer DVD with some VMG nonsense (copyright, studio animation, etc.) and four titlesets. The four episodes are in VTS 3, each in its own PGC. There are about 22 cells per PGC, divided into 10 or 11 PTTs. The root menu is also in VTS 3.

The first steps were easy: copy to HD with DVD Decrypter and remove unwanted junk with DVDReMake. I then used IfoEdit to pull out the cells for episode 1 into their own VOB (actually, two VOB files, the second only small), and DVD2AVI to demux the audio and prepare the d2v file. I wrote an Avisynth script to IVTC and smooth, and used CCE to re-encode the 45-minute episode.

Now the fun begins. I cannot remux the video into the existing VOBs because neither VobEdit nor IfoEdit can remux CCE-encoded video (the end result jumps and jerks because of some timing bug). I cannot simply replace the VTS VOBs with new ones because the re-encoded video is in one big chunk, not split into cells, and this would cause the scene selection menu to stop working.

What I need to be able to do is mux the m2v with the audio (again, easier said than done, except by authoring a new DVD in IfoEdit and then deleting the IFO and BUP files it creates) and then somehow split that up into sections corresponding to the cells on the original. I then need to be to join those VOBs to form a whole one again, and replace the original VTS 3 VOBs with the new ones. In theory they should be identical except for the stripped streams and the file size, so I could then rewrite the IFO file(s) to reflect the new data. Once this is achieved all the menus should still work, including the scene selection screens, and I can happily burn away...

I've tried every single tool I can find to do this, but none of them seem to be able to do it. Either there's a bug (IfoEdit video remuxing, for example) or the tools can't handle the muxed VOB (because it's two files, and they only work with one). Right now I can't see any way of doing it while maintaining the functionality of the scene selection menus.

What I'd like my tool to do, then, is list the contents of the DVD and allow me to "check out" certain elements (cells, programs, PGCs, etc.). It will demux these elements from the VOBs and present them in the destination directory.

I can then manipulate these elements in whichever way I see fit, and "check them in" to the DVD again. The tool would take care of fixing up the VOBs and IFOs to accomodate the new material.

I really don't think that's too outrageous an idea. It seems a pretty obvious way of doing it, to me.

I guess I need to start with code to enumerate the contents of the VOBs, identifying every element within them, from buttons to video clips. This code will also need to parse the IFO files in order to present the information in an appropriate way. (I might do what DVDReMake does and have both a "simple" and "advanced" view of the structure.)

I'm done rambling now. Feel free to comment, if you managed to make it down this far ;).

Whether or not this proposed tool will be able to do it, I don't know. I don't want to make it an authoring package -- there are already plenty of those. I'd prefer to concentrate on manipulating existing VOBs.

As for your question, you should be able to create a VMG menu in any authoring package. Scenarist will do it for sure, and I'd be very surprised if DVDMaestro doesn't.

8O puzzled me, but as i was reading, it made very good sense...

but like they say, in one ear, out the other.. similar effect :lol:


but seriously - this seems like a very gr8 idea!

revolutionize the kdvd scene...

Bazzy

but the thing is, i want to re-encode, not transcode u see... thats why im so interested in your tool you are creating

The matter is not to do a VMG but to do that above imported menus. All authoring tools I used imports vobs but not complete dvd structures. Except for tmpgenc author but it does not handle multi audio trak :-(

Now, for sure I used Scenarist only once and that is too few to tell I know it well :-)

u payed for scenarist??? 8O 8O 8O 8O 8O

or did u pay :wink: for scenarist

joke :lol:

Bazzy