Does this modem support 11 bit operation

Yes, make sure you have firmware version 1.17C or newer. If you need firmware, then contact Tech Support for resolution 308.
You must enable 11 bit on both ends and you may need to set the parity also.
The command for 10 or 11 bit operaton is:
10 bits - $EB0 (default)
11 bits - $EB1
The command for parity is:
#P0 = no parity
#P1 = odd parity
#P2 = even parity