If your average ping to Yahoo is 65ms then there doesn't seem to be an obvious problem on your end. Without monitoring your outgoing traffic, it is hard to determine if something occasionally starts sending data and consumes your bandwidth.
The path to the CS server is different than that to Yahoo so there could be problems along that route. In that case, all you can do is ask your ISP to investigate. They may be able to route traffic to that destination differently.