the N value i have is 10 but it is keeping more than that."
That doesn't make sense, unless you have some other issue.
Is the box checked next to the "Delete excess instances" on the LIMITS tab on the Object HISTORY in the CMC...?
Are there any other LIMITS settings for users/groups on that page that are in conflict with the 10 instances you have set at the top?
Have you applied any CE or CR 9 patches to that server, that you did not do a physical reboot after...?
"how safe is it to start whacking folders in the output folder?"
Not very safe - your even if you remove them from the folder, the CMS database will still have a record of them and show them in the Instance History. You will then get "Object Not Found" errors if those Instance links are clicked in CE-9.
If the Instance LIMIT function doesn't work you safest method is to delete the Instances from CMC or ePortfolio. You can do this manually, or use a third-party tool like APOS InstanceManager 9.