As for me, I'm absolutely against this.
Think about people with disabilities.
Right now, when you are young and your sight is good, it doesn't matter to you. But what will be later? Go to any site with tiny letters that are very hard to see - the same thing is for those with sight problems who come to your site, even if it seems good enough for you.
Not allowing people to change the text size by their wish is very bad.
There are two options: allow users to set the text properties by their needs (ever if will affect badly on your design), or lose visitors.
P.S. By the way, Opera browser has marvelous zoom tool (this is one of it's unique features that I like) - it's zooms the intire page, like scaling images in graphic editors.