We combine experiments with simulations to investigate the fluid-structure interaction of a flexible helical rod rotating in a viscous fluid, under low Reynolds number conditions. Our analysis takes into account the coupling between the geometrically nonlinear behavior of the elastic rod with a nonlocal hydrodynamic model for the fluid loading. We quantify the resulting propulsive force, as well as the buckling instability of the originally helical filament that occurs above a critical rotation velocity. A scaling analysis is performed to rationalize the onset of this instability. A universal phase diagram is constructed to map out the region of successful propulsion and the corresponding boundary of stability is established. Comparing our ...