The video rendering setting changes how the preview image is displayed. I think overlay means it uses some video functionality on the graphics card but I'm not entirely sure.
VirtualDub has similar options. Also had some issues with overlay being broken one of the win10 computers here (which has led me to using amarectv on that one), I suspect it's something with the nvidia driver but I'm not certain. The setting should not matter for the captured video though.
I don't remember the exact settings, though I don't think I've changed anything from default. I think the priority was ticked on and set to realtime at least, I'll have a look at it later.