Cell polarization underlies many cellular processes, such as differentiation, migration, and budding. Many living cells, such as budding yeast and fission yeast, use cytoskeletal structures to actively transport proteins to one location on the membrane and create a high-density spot of membrane-bound proteins. Yet, the thermodynamic constraints on filament-based cell polarization remain unknown. We show by mathematical modeling that cell polarization requires detailed balance to be broken, and we quantify the free-energy cost of maintaining a polarized state of the cell. Our study reveals that detailed balance cannot only be broken via the active transport of proteins along filaments but also via a chemical modification cycle, allowing detailed balance to be broken by the shuttling of proteins between the filament, membrane, and cytosol. Our model thus shows that cell polarization can be established via two distinct driving mechanisms, one based on active transport and one based on nonequilibrium binding. Furthermore, the model predicts that the driven binding process dissipates orders of magnitude less free energy than the transport-based process to create the same membrane spot. Active transport along filaments may be sufficient to create a polarized distribution of membrane-bound proteins, but an additional chemical modification cycle of the proteins themselves is more efficient and less sensitive to the physical exclusion of proteins on the transporting filaments, providing insight in the design principles of the Pom1/Tea1/Tea4 system in fission yeast and the Cdc42 system in budding yeast.