Windows 8 WIndows logs me out, if video memory is full?

It could be.

It could also be hardware related rather than anything Microsoft can help.

The general rule of thumb is never to use all of your hard drive space, one of the reasons being to prevent your computer not working the way it should.

When you use up all of your RAM, the computer will then turn to your hard drive for virtual memory. There are some settings you can tweak to change how Windows handles your virtual memory to help you get more performance if you need it, but it likely won't be enough to truly save you if this is being caused by not enough RAM for what you're doing.

First things first, you might want to scan your computer for malware (using a program like Malwarebytes) and then see if it's still happening.

How much RAM have you got and what are you doing with the PC?

As for just the signing out part, are you trying to use the mouse while it's unresponsive? Could it just be your power settings kicking in and putting the computer to sleep? You by default will have to sign back in to Windows after waking it from sleep, though this can still be altered I believe from Group Policy settings if you have Windows 10.