So, just an update ... I went ahead and bought MainConcept Reference for Mac (MPEG-2 module), and thankfully got $100 off due to a coupon code. Total was $415. I played with demos from Episode and Sorenson (which uses MainConcept's MPEG-2 encoder, though I am not sure which version), tweaked ffmpeg and mpeg2enc every which way I could, and nothing comes close to the quality of MC Reference when played on the SD TV I have. It looks better than the original VHS when playing, maybe due to a little color and gamma correction we did in FCP. Thanks so much for pointing out MC Reference, I think we're going to appreciate it's capability.
In summary, my son and I are doing VHS capture and transfer to DVD, and the current flow is:
DVHS/TBC/ProcAmp -->[PC] ATI AIW 9000 (
HuffYUV/AVI) --> [Mac] Apple Compressor to ProRes422 HQ w/ PCM audio --> Final Cut Pro --> Save as ProRes .mov (no additional compression) --> MainConcept MPEG-2 --> DVDSP4 (dual layer DVD)
We may be tweaking things for awhile, but overall the last set of test runs we did with a 10-year old family wedding video turned out great. The VHS quality is very good on this tape, but has some noisy low-light shots in it that served as good tests, and MC Reference was the best at keeping that noise in check in the final MPEG2. Others were amplifying it or showing artifacts.