I will assume you do not want to use 3D on your system. So you do not have to install them. I believe there are sound drives as well as Video, but it you play games, the sound might be good. Do you know which drivers you might not install?
They started putting the 3D drivers on the 260 when they upgraded the drivers for the newer cards. I do not know if the drivers degrade system performance but you might at least disable them in the control panel if you can. Some older threads talk system performance, but they are older and the problems may have been worked out by now.