Physical model-trained neural network generalizes to external datasets in novel behavioral contexts.

(a) Example sequences of raw images from datasets collected with other imaging systems, and superimposed 2-D projected poses from our neural network: ten 2-D backbone coordinates (green) and two 2-D centroids of eyes (red). Sampled frames are displayed for: (i) Multiple 2-D free swimming swim bouts from Ref. [38,40] showing diverse larval poses. (ii) A 2-D prey capture swim bout from Ref. [37], with the larva approaching a paramecium at 0 ms and successfully capture its prey over the next 128 ms. (iii) Multiple swim bouts from Ref. [39] showing a successful capture of a paramecium over the last three frames. (iv) 3-D prey capture swim bouts captured using a 2-camera setup from Ref. [37] showing a larva hunting a paramecium. (b) Pose prediction scores for all frames of the swim bouts are to the right of each example. The shift between pose prediction scores for different datasets reflects differences between imaging conditions and our physical model rendering.