QTGMC reduces jagged shapes created by deinterlacing procedures. This is one of its major benefits, a side effect is the one you experimented on a "real" jagged shape.
QTGMC can rarely be used with its defaults parameters, whatever preset you choose (BTW "fast" in only recommended for speed reason, not for optimal results).
Try to use QTGMC's lossless and sourcematch features:
Code:
QTGMC(Lossless=2, SourceMatch=3, ...)
For a "natural" look I sometimes use the following set, but you have to experiment with your own material:
Code:
QTGMC(preset="slow", matchpreset="slow", matchpreset2="slow", sourcematch=3, tr1=2, tr2=1, NoiseTR=2, sharpness=0.1)