Galactic winds are observed in many spiral galaxies with sizes from dwarfs up to the Milky Way, and they sometimes carry a mass in excess of that of newly formed stars by up to a factor of ten. Multiple driving processes of such winds have been proposed, including thermal pressure due to supernova-heating, UV radiation pressure on dust grains, or cosmic ray (CR) pressure. We here study wind formation due to CR physics using a numerical model that accounts for CR acceleration by supernovae, CR thermalization, and advective CR transport. In addition, we introduce a novel implementation of CR streaming relative to the rest frame of the gas. We find that CR streaming drives powerful and sustained winds in galaxies with virial masses M_200 10^{11} Msun), CR streaming is able to drive fountain flows that excite turbulence. For halo masses M_200 > 10^{10} Msun, we predict an observable level of H-alpha and X-ray emission from the heated halo gas. We conclude that CR-driven winds should be crucial in suppressing and regulating the first epoch of galaxy formation, expelling a large fraction of baryons, and – by extension – aid in shaping the faint end of the galaxy luminosity function. They should then also be responsible for much of the metal enrichment of the intergalactic medium.