Dynamically composing languages in a modular way: supporting C extensions for dynamic languages

Many dynamic languages such as Ruby, Python and Perl offer some kind of functionality for writing parts of applications in a lower-level language such as C. These C extension modules are usually written against the API of an interpreter, which provides access to the higher-level language’s internal data structures. Alternative implementations of the high-level languages often do not support such C extensions because implementing the same API as in the original implementations is complicated and limits performance. In this paper we describe a novel approach for modular com-position of languages that allows dynamic languages to support C extensions through interpretation. We propose a flexible and reusable cross-language mechanism that allows composing mul-tiple language interpreters, which run on the same VM and share syntax trees. This mechanism allows us to efficiently exchange run-time data across different interpreters and also enables the dynamic compiler of the host VM to inline and optimize programs across multiple language boundaries. We evaluate our approach by composing a Ruby interpreter with a C interpreter. We run existing Ruby C extensions and show how our system executes combined Ruby and C modules on average over 3 × faster than the conventional implementation of Ruby with native C extensions, and on average over 20 × faster than an ex-isting alternate Ruby implementation on the JVM (JRuby) call-ing compiled C extensions through a bridge interface. We demon-strate that cross-language inlining, which is not possible with native code, is performance-critical by showing how speedup is reduced by around 50 % when it is disabled