DVD-Video is MPEG-2 GOP-medium encoding, stored in non-square pixels. VHS was an interlaced video format with an overscan. What this means is your original VHS video is full of lines, with noise all around the edges, and the DVD version is also compressed with a too-large wrong-shape video for using online It has to be cleaned up and re-compressed for online viewing.
You'll want three basic tools for this:
1. Avisynth, with the appropriate import/deinterlace plugins. It's freeware.
2. VirtualDub, for further cleanup, cropping, resize to 4:3 or 16:9 square pixel aspect ratio, and lossless output encoding (i.e.,
HuffYUV lossless codec) as an intermediary for re-encode to high-compression format. Also a freeware program (no cost, donations encouraged).
3. An encoder capable of good H.264 -- this is what sticks most people, because professional encoders cost $K-range (example:
MainConcept Reference), and freeware leaves you wanting for quality. The one exception may be x264, but it's a command-line tool. The available third-party x264 GUIs tend to be unstable, making it hard to use. If you happen to have a pro NLE like
Adobe Premiere Pro CS3/CS4/CS5, or current versions of
Vegas Pro, then you can import the AVI, drop to timeline, then export with their MainConcept SDK encoders. That's a good second-choice option. The best encodes come out of Reference, and skilled encoders know how to fully tweak all the various settings.
However, if you're relying on a third-party site like Youtube, then you can encode out to an alternative codec. Youtube, of course, must re-encode that content, using their justly-criticized choice of software and settings, which results in reduced quality. Alternate codecs include Windows Media (WMV), Divx and Xvid.
I'll look at adding a guide on this sometime in late December or early January, which goes over the long process of conversion.