Gauge fields in condensed matter physics give rise to nonreciprocal and topological transport phenomena and exotic electronic states1. Nanomechanical systems are applied as sensors and in signal processing, and feature strong nonlinearities. Gauge potentials acting on such systems could induce quantum Hall physics for phonons at the nanoscale. Here, we demonstrate a magnetic gauge field for nanomechanical vibrations in a scalable, on-chip optomechanical system. We induce the gauge field through multi-mode optomechanical interactions, which have been proposed as a resource for the necessary breaking of time-reversal symmetry2,3,4. In a dynamically modulated nanophotonic system, we observe how radiation pressure forces mediate phonon transport between resonators of different frequencies. The resulting controllable interaction, which is characterized by a high rate and nonreciprocal phase, mimics the Aharonov–Bohm effect5. We show that the introduced scheme does not require high-quality cavities, such that it allows exploring topological acoustic phases in many-mode systems resilient to realistic disorder.