Shyguy that doesn't sound right, The video cards ram should not be viewed or added to the system ram, Its video dedicated, 32-bit uses a max of 3.5gb of Ram, I may have misunderstood you but from what i got from your post is that the Ram from your Video, cache, X-Fi cards, and even NIC cards is added to your system Rams total meaning that is you have 2gigs of Ram + 1gig Video card + lets say Killer NIC with 256mb + X-FI with 256 that all that equals your total 3.5gb that your 32-bit comp can only read.
I may have misunderstood you cuz thats the idea i got, 32-bit does only see 3.5gb of ram it shouldn't add your total memory in your system, 3.5gb of RAM, and should only see 3.5 gb of Video ram IE: CrossfireX or Quad SLI, those will pose a problem.