I have been fighting this problem on and off on some of my XP machines over the past decade and, more recently, on Windows 7 (both Ultimate and Enterprise, 64-bit and 32-bit). Often it seems to be some application that's somehow interfering with rdp services. Interestingly, the problem arises more often (in my case) after I *kill* (instead of "diconnecting") an RDP client session that was created on some *particular* computers (so some interference between client software and the server might be present also).
I was unable to improve this rare situation (this happens to me just once per approx. 200 reconnections to the same server) using any of the prescriptions found here or elsewhere, however, recently I identified a piece of software that needs to be terminated on the server to make it possible for me to reconnect to my remote session in Windows 7 64 bit Enterprise running on the server. This piece of software, in my case, with my all other soft running in background on the server also is dwm.exe - Microsoft's own Desktop Window Manager. I found that, to be able to reconnect to my server after rdp protocol error arises, all I need to do is to terminate dwm.exe on the server (it restarts automatically after that)! In my opinion, this is, again, pointing toward some interference among the programs I run on the server, so, as much as many people might be surprised, I would not blame MS for that at all, as they do not control the environment as tightly as Apple does, so I have to live with this DIY.
Anyway, killing dwm.exe works! It would be very interesting to collect some statistics if this solution would work for *you*.
Here is how *I* take care of killing dwm.exe:
1) I have cygwin installed on my server,
2) sshd deamon (from cygwin) is running on my server allowing interactive shell login (and all other services that are based on secure tunnell btw, it might be cygwin that gives me trouble in the first place, as I always run cygwin on all my Windows machines, but I don't care - I just needed a working solution and now I have it),
3) when I experience rdp protocol error, I log in to my server using ssh and
4) figure out process id for dwm.exe using (for those unfamiliar with this, it's a UNIX) command(s)
ps -W | grep "dwm"
here is an example of what I see on the screen in my terminal window:
$ ps -W | grep "dwm"
5220 0 0 5220 ? 0 09:26:28 C:\Windows\System32\dwm.exe
5) then I use Windows's own taskkill.exe to terminate the task:
a) first need to navigate to where taskkill.exe is located (e.g. c:\Windows\system32\taskkill.exe in a 32-bit OS, but mine is 64 bit, so I do the following from my ssh window):
cd c:\Windows
cd SysWOW64
(obviously, my system disk is "c", SysWOW64 indicates I am running a 64-bit Windows,
it has nothing to do with World of Warcraft)
and then
b) ./taskkill.exe /PID PROCID /F
where PROCID is process id for dwm.exe I figured out using ps -W | grep "dwm"
(i.e. 5220 in the example above)
this has been working just great for me over the past few months, while I experienced rdp protocol error only a few times since than, taking care of (i.e. terminating) dwm.exe worked like a charm every time (but, of course, my statistics are limited). It happened to me again this morning and, this time, I thought I should share my solution with people on this forum, as I have been here many times to learn how to solve various problems.