In this paper we continue the study of the superconformal index of four-dimensional N =2 theories of class S in the presence of surface defects. Our main result is the construction of an algebra of difference operators, whose elements are labeled by irreducible representations of A N −1. For the fully antisymmetric tensor representations these difference operators are the Hamiltonians of the elliptic Ruijsenaars-Schneider system. The structure constants of the algebra are elliptic generalizations of the Littlewood-Richardson coefficients. In the Macdonald limit, we identify the difference operators with local operators in the two-dimensional TQFT interpretation of the superconformal index. We also study the dimensional reduction to di...