how does the web server know who's accessing it if there is no requirement to authenticate?
If anon access is enabled it won't prompt for login credentials and it won't know who is logged on. The IUSR_servername account is the one that is being used for authentication when its anonymous so that is the account that would need to be blocked... but of course that would block all anonymous access (and is not how you would do it, but just to explain the reason why what you are doing won't work)
In Short, Deny would superceed all other rights but only if the server knows who is logged on in the first place. Either people have to sign in, or they log on anonymously, not both.