Some information about your setup would help. What devices are you using, how is you network set up, any anti-viruses or Firewalls?
Is this only with internet traffic or internal LAN traffic? Is this you total network connectivity for one download, or could something else be taking up part of it?
Have you spoken to your ISP about the problem.
You can leave Task Manager open and watch the transfer rate. Maybe when it drops you will notice something else going on..
Resource Monitor might also be useful to show what connections are open and if something might be running in the background.