The Polyglot's Dilemma: Conformance Testing a Dozen Specs in as Many Languages

A Unified YAML Test Format for MongoDB Drivers

A. Jesse Jiryu Davis; Jeremy Mikola; Jeff Yemin· 2026· DOI 10.48550/arXiv.2608.18039

The core problem

MongoDB maintains client libraries in a dozen programming languages, used by tens of thousands of organizations and millions of developers. Most are implemented natively rather than as wrappers around a shared core. Ensuring consistent behavior across these libraries, comprising millions of lines of code, is hard but essential. Over eleven years, the authors developed a specification-based testing approach: tests are written once in YAML and executed by language-specific interpreters for each library. This digest describes the evolution from many ad-hoc formats to a Unified Test Format, which allowed the team to delete over 22,000 lines of test code. The rate of nonconformance bugs fell up to 86% in drivers that adopted YAML tests, though results varied. The paper reports lessons learned about declarative test design, test architecture, schema evolution, and the limits of unification.

The adoption of YAML-based tests led to a significant reduction in nonconformance bugs. The paper reports that the rate of nonconformance bugs fell up to 86% in drivers that adopted YAML tests, though results varied across drivers. Additionally, the Unified Test Format allowed the team to delete over 22,000 lines of test code, reducing maintenance burden and duplication. The number of drivers adopting the format grew over the eleven-year period, with most drivers eventually using the unified format. The reduction in bugs is attributed to the consistency enforced by shared tests and the ability to quickly propagate fixes across all drivers. However, the authors note that results varied, likely due to differences in driver maturity, interpreter quality, and adoption timing.

MongoDB maintains client libraries in a dozen programming languages, used by tens of thousands of organizations and millions of developers. Most are implemented natively rather than as wrappers around a shared core. Ensuring consistent behavior across these libraries, comprising millions of lines of code, is hard but essential. Over eleven years, the authors developed a specification-based testing approach: tests are written once in YAML and executed by language-specific interpreters for each library. This digest describes the evolution from many ad-hoc formats to a Unified Test Format, which allowed the team to delete over 22,000 lines of test code. The rate of nonconformance bugs fell up to 86% in drivers that adopted YAML tests, though results varied. The paper reports lessons learned about declarative test design, test architecture, schema evolution, and the limits of unification.

The core methodology is specification-based testing: a single test specification is written in YAML and then executed by language-specific interpreters for each MongoDB driver. This approach ensures that all drivers are tested against the same behavioral expectations without duplicating test logic in each language. The evolution from many ad-hoc formats to a Unified Test Format (UTF) was central. The UTF defines a declarative schema for test cases, including setup, execution, and assertion phases. Each driver implements a small interpreter that reads the YAML and translates it into native operations. This architecture is illustrated below:

The authors discuss several lessons learned. First, declarative test design is powerful but requires careful schema evolution to avoid breaking existing tests. Second, test architecture must balance flexibility and simplicity; interpreters should be as thin as possible. Third, schema evolution should be managed with versioning and migration tools. Fourth, unification has limits: some language-specific behaviors cannot be expressed declaratively, requiring escape hatches. The paper also notes that while the Unified Test Format reduced duplication, it introduced a new dependency: all drivers must keep their interpreters up to date with schema changes. The authors conclude that specification-based testing is effective for polyglot environments but requires ongoing investment in tooling and communication. The approach is generalizable to other multi-language projects, as long as the team is committed to maintaining the shared test format.

CS practitioners and researchers

Opening member content…