You need to make sure you have the 355AF RJ45/RS232 adapter as provided through Avaya Direct or a reseller to connect to the SMDR port. The cost of this is over $50 I believe so you might want to make your own to save a few bucks. Another post further back on this forum lists the pinouts if you want to make your own connector.
I believe the standard Merlin com settings (8 bit - No parity - 1 stop bit) should be fine for what you are connecting it to. There's no place I'm aware of in Merlin to configure the actual com settings. Just places to specify call lengths, in/out calls, UDP calls, etc.
Hope this helps!