For me, it comes down to this:
Is it worth figuring out?
Since both ATI MMC and
VirtualDub can capture AVI files just fine, probably not.
I would just use
VirtualDub if it's working correctly.
Just use ATI MMC for MPEG-2 capturing, whether it's I-frame high bitrate captures, or direct-to-DVD formatted MPEG-2 files.
The hardware, drivers and software is rather complex, so from time to time, odd errors do pop up on systems, and it's hard to resolve. In this situation, I think you're safe to just avoid the problem, rather than spending time solving it. Spend time on the video work -- forget this technical curiosity.