I have had several similar issues with a player I bought for my girlfriend. One issue Kwag helped me realize is that I was encoding at 480x480. It played PERFECTLY on my Sony player. It was jerky on this new player. Also, I have had bad luck with using too low of a minimum bit rate.
Additionally I have had really bad luck with Divx files I re-encode as KVCD. Then I play the Divx back and realize that there was plenty of error in the source that was just enhanced in the KVCD output.
I use ReClock on PC playback which will smooth out video that is not smooth in the source. (Sometimes from missing video frames or strange framerates).
If this was a direct DVD rip yourself... and it was a PAL source... That is a strange problem. Some players don't like VBR. Those few players can choke on KVCD?
Just a couple of ideas. Hope one helps.
Thanks for giving Kwag the idea for the latest Optimal Script change. I am really liking the new results. (15% faster on episodes of The Simpsons... and a bit nicer looking too!).
Grantman