Some codecs like OGM and Matroska require audio/video post-processors like Vosub and Directshow subtitle filter. Is it a good idea to leave these filters installed? I'm afraid they might affect the performance of the decoders since they are processed like this
AVI Splitter --> Decoder (Divx/Xvid) --> Post-processor (Subtitle Mixer)

These are Directshowfilters and won't be used by avisynth if the script load the source with "Avisource(...)".

That is why you should never use "Directshowsource(...)" in script or chek the DirectShow support in Tmpgenc. Else you are right, this kind of filters can affect the encoding.