The special federal paymaster is trying to persuade the American International Group, the insurance giant, to reduce a coming bonus payment of $198 million, but is running into legal hurdles despite rules curtailing bonuses at companies receiving assistance from the Treasury.