DVI-D cable is what you need. What you observed using it is that the DVI output from the NVidia chips has poor signal quality. This becomes obvious at 1280x1024 or higher, and is the reason why the more expensive cards have a separate transmitter chip on, ignoring the poor one integrated into the main NVidia chip.
In other words, use a different graphics card. Get ATI if you want clean DVI at high resolutions.
hardware
prices 
