I haven't tied together a Definity G3 and a Merlin Legend so I cannot speak from experience. But looking through my Legend 6.1 Network Reference book I see the Definity ECS and Definity ProLogix Solutions mentioned as being compatible with the Merlin Legend. The Legend should be at least r6.1 to offer the best features. Personally if you DO have to upgrade the Legend I would up it to r7.0 v14.2 since that's the latest and greatest.
According to a disclaimer in the manual, a Definity and a Merlin CANNOT share a Centralized Voice Messaging system. You CAN have UDP so that all extensions appear to be pseudo-internal, ARS call routing patterns between locations, ISDN dial plans, UDP transfers, and all of the other slick features of tandem PRI. If you have a choice I would always go with PRI rather than E&M tie lines for networking phone systems.
For the Switch Type, the Merlin must be programmed to match what is defined in the Definity's Interface field. By default I believe the Definity Interface is setup as User. Then the Merlin would have to be setup as Legend-PBX. If the Definity Interface is setup as Network, then the Merlin would have to be setup as Legend-Ntwk.
For the Switch ID, the Merlin must be programmed with a unique ID number in the 41-50 range if the other end is more than 200 miles away. If the other end is less than 200 miles away, then the Merlin Switch ID must be a unique number in the 51-60 range.
The PRI signaling is internal, so it's handled differently from ordering a certain configuration from telco for PSTN dial tone. The Merlin will show the Network Service as being Electronic Tandem Network (Legend UDP). It WON'T show up as being 5ESS, DMS-100, etc. like a traditional circuit would. As for the Definity I'd check their docs on how this is defined for that end in terms of defining the Network Service type as being internal.
Check Avaya's website for the manuals if you need more details on the Merlin. The Network Reference should be out there. Hope this helps!