Some files, such a .cab and compressed may take longer than normal. Some files may need special treatment.
I don't know if the time is related to the number of files scanned, but you can watch the scan and see if any particular files or particular locations take longer than you would think.
I will usually exclude certain locations on my system the may keep the Windows cab files, or Windows Update downloads and backups, etc. Maybe you could reduce the scan time using that option.
I wouldn't go through and pick out file by file. I certainly not exempt folders that contained .exe files. But what you exclude is up to you. I just think archived files in know Windows storage areas are not dangerous. If you watch the scan some, you may see what I mean.