NO, you need to connect BOTH. There is a reason there are two 4 pin molex connectors going into a 6pin power connector for the video card. It isn't there for looks. Not only do you need to connect both of them, but you should do so from two seperate branches coming from the PSU.
If your PSU already has a 6pin SLI power connector, you don't need to use that adapter at all, simply plug that in directly to your video card.