I'd run 5/8" wire core cable. I'll have to measure to see how many feet I have on mine winch. It's a bunch. What you DON'T want to do is fill the spool full of cable. Do that and you have no place for the cable to go it it stacks up on one side. I've seen a 10,000# Garwood have the bottom shaft blown out of the case due to the cable stacking up on one side.
The solution to the problem described in post 2 is to arrange the work so the cable enters the capstan in a straight line. If the winch does not have a level winder use a snatch block to straighten the pull. If the the bottom shaft was blown out of the winch the user likely had a steel bolt substituted for the authorized aluminum shear pin.