High-energy (1–100 keV) electrons can coherently couple to plasmonic and dielectric nanostructures, creating cathodoluminescence (CL) of which the spectral features reveal details of the material’s resonant modes at a deep-subwavelength spatial resolution. While CL provides fundamental insight in optical modes, detecting its phase has remained elusive. Here, we use Fourier-transform CL holography to determine the far-field phase distribution of fields scattered from plasmonic nanoholes, nanocubes, and helical nanoapertures and reconstruct the angle-resolved phase distributions. From the derived fields, we derive the relative strength and phase of induced scattering dipoles. Fourier-transform CL holography opens up a new world of coherent light scattering and surface wave studies with nanoscale spatial resolution.