Detecting arbitrary keypoints on limbs and skis with sparse partly correct segmentation masks

Analyses based on the body posture are crucial for top- class athletes in many sports disciplines. If at all, coaches label only the most important keypoints, since manual anno- tations are very costly. This paper proposes a method to de- tect arbitrary keypoints on the limbs and skis of professional ski jumpers that requires a few, only partly correct segmen- tation masks during training. Our model is based on the Vision Transformer architecture with a special design for the input tokens to query for the desired keypoints. Since we use segmentation masks only to generate ground truth labels for the freely selectable keypoints, partly correct seg- mentation masks are sufficient for our training procedure. Hence, there is no need for costly hand-annotated segmen- tation masks. We analyze different training techniques for freely selected and standard keypoints, including pseudo la- bels, and show in our experiments that only a few partly cor- rect segmentation masks are sufficient for learning to detect arbitrary keypoints on limbs and skis