You'll be misled by a lot of dogma on amateur sites, and by users on the amateur sites that have forums, when it concerns re-encoding and loss. However, most of that -- the idea that "all re-encoding incurs quality loss" -- isn't necessarily true. It's done daily with everything you watch. This, of course, is good news for you!
Loss is mostly determined by the input and output, as well as filtering you do after input and before output. When it comes to most DVD recorders,
major loss happened simply by using SP mode. A portion of your filter work may actually be spent undoing that damage.
VirtualDub is a fairly open-ended editor/encoder, so you can output to uncompressed video. This means you won't incur any added re-compression loss at that stage. TMPGEnc is a family of products (from Pegasys Inc aka Tsunami), although I'll assume you're referring to the
TMPGEnc Plus 2.5 MPEG encoder. While MPEG format can have loss, it's not necessarily guaranteed to happen or even noticeable when it does happen. Much of that would depend on your end settings.
GSpot should be accurate for most streams -- almost 100% reliable for DVD recorder-made discs -- so I would say it's safe to assume 720x480 is accurate. That would correspond to SP mode on virtually all DVD recorders.
There's really no need to configure
HuffYUV. By default, it should be set to the best setting. In fact, it literally says "best setting" by the two best settings! Because I almost never change these, I would have to open Vdub, and look at them to double-check what those are. So here I go...
1. You can honestly run with either the "(best)" or the "(fastest)" settings. There's no affect on image quality. Judging from my two main encoding systems, I have one set to one, and the other set to the other. I can't tell any difference. If I had to pick one, I'd say go for "best". This really affects decoding speed and CPU use more than anything else. In real-world use, in an age of multi-core CPUs and multi-TB hard drives, there's no detectable advantage of one over the other.
2. Use what you need. If your source is sharpened VHS video, then 720x480 may work well for you. If it's standard VHS quality, then 352x480 Half D1 lets you both (1) put more on a disc and (2) do it with a higher bitrate allocation. Look at the
MPEG bitrate charts (bottom of that page in
the link).
3. Again, see the
video bitrate charts. I try to get video as close to 8000kbps (8.0Mbps) for 720x480 as I can, or 4000k when using 352x480. If the content is noisy, sometimes I want superbit rates -- 9000k for 720, or 5000k for 352. There's not correct answer without seeing the content and running a few brief tests with short clips. More compression generally means using 2-pass VBR for retaining quality. Expanding to DVD+R DL media is another option.
4. Nope.
VirtualDub can "direct stream copy" the audio (no encoding/change), and TMPGEnc will use re-encode only if given audio input. The only foreseeable need for demux may come for authoring. Much of that would depend on what you're using. BatchDemux and/or TMPGEnc Plus demux great, as does the Womble software. A number of tools can demux, but those come to mind first, as I use them near-daily.
That it?