Jake,
Assuming you did not create a new workgroup file and you are using the system default workgroup, you need to ensure you remove the same permissions from the Admins group as you do for Admin.
The Admins group is specific to the workgroup file but all default workgroups are the same so anyone using the default workgroup could have the perimission of the Admins group.
Group memebership is maintained in the workgroup file, and admin is a member of Admins by default. So even if you remove admin from admins in your workgroup, everyone else's default workgroup is still a member and I have permission.
Going beyond the scope of the question, the PID's specified in the creation of the workgroup control permission not the workgroup files themselves. The workgroup files are only keepers of PID's. I could have several Workgroups with the same PID's. The PID of the workgroup is the PID of the Admins group.
If I copy a workgroup file, then you change your password and then you tell me what your old password is, I can use my copy of the workgroup to log in with your old password. This makes workgroup security relatively weak.