The two dip switches are for setting the processor in either "permanent" key mode or "PBX/Hybrid" mode. Both switches have to be either "closed" or "open". If one switch is "open", but the other "closed" - the processor card will have problems or not work at all.
Both switches "closed" = permanent key mode.
Both switches "open" = PBX/Hybrid mode.
If the processor is in "permanent key mode", you would have the problem you first experienced - i.e., being unable to change modes through system programming.
In "PBX/Hybrid" mode - you can change modes through system programming.
On a Legend/Magix processor, I believe "open" = both switches down or towards the board edge.