The response of driven frustrated media stems from interacting hysteretic elements. We derive explicit mappings from networks of hysteretic springs to their abstract representation as interacting hysterons. These mappings reveal how the physical network controls the signs, magnitudes, symmetries, and pairwise nature of the hysteron interactions. In addition, strong geometric nonlinearities can produce pathways that require excess hysterons or even break hysteron models. Our results pave the way for metamaterials with geometrically controlled interactions, pathways, and functionalities, and highlight fundamental limitations of abstract hysterons in modeling disordered systems.