You should feel the way you feel
It depends on at least a few things that come to my mind:
How long you have been with the company you are leaving, and the terms you are leaving it on, the role you will be playing in the company you are going to. You will be saying good-bye to the day to day relationships of friends, and mentors, there will be some sadness and the occasional jerk whom you will be relieved to see no more.
Joy to saying good-bye to the lousy IT processes and management weaklings, but this is usually a tradeoff as the company you are going to will have its IT process and management problems too.
I have felt relief, sadness, joy, fear, and anxiety when I have left companies. (sounds like a first date

)
I think the most important thing is to keep in touch with the good people and relationships that you have worked with at your old company. This will increase your network of alliences and resources which you can use and enjoy for the rest of your career.
Good luck on your new job. In not now, when?
If not here, where?
If not us, who?
Just do it!!