I actually rely on
MainConcept Reference for this, but x264 can do well if tweaked and setup just right. I'm currently testing some x264 GUI frontends, to do some re-comparison tests in the next month or two.
As far as H.264 from VHS, for Blu-ray archiving -- we're thinking alike. That's something I want to look into further in early summer, after acquiring necessary hardware. Already have the software.
It's value will mostly come in on noisy VHS tapes, especially that shaky hand-held stuff, or any kind of SLP mode TV recordings. For anything else, DVD and H.264 would likely be transparent to one another, assuming each was clean and encoded with good settings in good software (or hardware).
I've been gathering info for months now, following software and hardware developments/advancments/updates, pricing, and other viability factors.
I don't know that noise is a good test. Something realistic like fog, rain, smoke, or fire -- those tend to work well for encoding tests. Fast multi-camera sporting event audiences shots are good too -- wrestling is nice. Go one further and test with a old SLP VHS recording of a wrestling event! I already wore out an old tape of this, used it many times for tests in past years.