Stream stars removed by tides from their progenitor satellite galaxy or globular cluster act as a group of test particles on neighbouring orbits, probing the gravitational field of the Milky Way. While constraints from individual streams have been shown to be susceptible to biases, combining several streams from orbits with various distances reduces these biases. We fit a common gravitational potential to multiple stellar streams simultaneously by maximizing the clustering of the stream stars in action space. We apply this technique to members of the GD-1, Palomar 5 (Pal 5), Orphan, and Helmi streams, exploiting both the individual and combined data sets. We describe the Galactic potential with a Stäckel model, and vary up to five parameter...