It would be an issue with a program or task that is running not just something that is installed. When you start to see the slow down open the task manger and watch for the process that is sucking up memory or cpu cycles.
Just saw your edit, running prime for an hour or two without issues would mean it would be less likely to be a hardware issue yeah.