DVD-RB itself is pretty simple, once you get the options set up right.
First, go to Options -> Setup. Find the EclCCE executable and select it in "Path to CCE SP (New)" (or "Path to CCE SP 2.50" ifyou're using 2.50, or "Path to CCE Basic" if you're using CCE Basic). You shouldn't need to choose paths to MPEG2DEC3DG or DECOMB.DLL if you have them in your plugins directory. I also check "Run encodes minimized." Finally, under "Audio to remove," do a "check all" and then uncheck the languages you want to keep by default, and do the same with subpictures (though I tend to keep all subs, they rarely take up more than a few megs).
Leave Options -> AVS Options alone, the defaults are fine.
In Options -> CCE Options, choose your CCE version. Do
not choose "One Pass VBR (w/analysis)," since this will use DVD-RB's built-in prediction and you won't be able to use the Notch matrix or any other custom CCE parameters.
In Mode, ensure "CCE Mode" is checked. Also check "Verbose Status Window" and "Dynamically Assign Cell Bitrates". Uncheck everything else in that menu.
That's it!
To do a movie, choose the source path (either the original VIDEO_TS from DVD Decrypter, or the directory created by DvdReMake or whatever) and the working path (this should be an empty directory, it'll be used for temporary files and the resulting output). Hit Prepare.
Now run RB-Opt. Figure out which VOB IDs correspond to the movie and join them together. Do the same for extras. Then, for each collection of VOB IDs, go to the CCE settings dialog and set 8-bit DC precision, 15 frames per GOP, and choose a "Custom" matrix. Select your Notch matrix, which is a text file in the CCE matrix format (each line starts with a tab, 8 fields per line, etc.). Now hit the "OPV Prediction" button and choose your parameters. (Here's where the testing comes in...)
When the prediction is done, close RB-Opt and hit the Encode button in DVD-RB. Wait hours. Hit the Rebuild button. Check the size of the final encode. Celebrate or punch a wall depending on the result
.