It's an known issue that for doing avi, CQMAtic is less accurate. But your problem here is not that.
You are using a res that is way too low ! 1h20 can be achieved in 544*576 ! And with Maximum to 2200, not 1800
For these two reason, CQMatic encode always at 90 (that is the maximum CQ it uses because above that, the quality is the same, only the file size is different) : you obtained a file that can't be better (in quality) and do not need more space for that !
The ideal CQ is between 70 and 85. Use 544*576 for that movie and raise the max to 2200. Your CQ will be in this range and the result much closer than what you want.