If there are any readers who really do like that sidebar, then switch it off and download this (free) http://www.desktopsidebar.com/ It has several improved functions and more "gadgets"
You must have a special sidebar. Mine takes about 500 MB with four standard gadgets. I have 4 GB now, so it's all good, but it's still a resource whore.
I have the default Vista sidebar with CPU usage, Network meter, and uptime, and it's using 21MB of RAM. There must be something wrong with your install if it's using 500MB...