I wouldn't be in a hurry to install sp4 just yet. I did what you are suggesting and followed by a clean install. Ever since I have had trouble with IE where I have to visit Task Manager a lot to kill it, then re-boot.
When I get time I will return to sp3.
Cheers,
Brodie