On October 19, 2026, OpenAI published new results on programming in languages underrepresented in training data. OpenAI's COBOL research includes building a SQLite-compatible engine and migrating existing programs. Scores improved on the company's own benchmarks, though the central challenge for anyone maintaining legacy systems remains the same: preserving software behavior, including undocumented rules that a seemingly cleaner rewrite could silently break.
What MultiLangBench and the COBOL Tests Actually Measure
OpenAI separates two distinct capabilities. MultiLangBench tracks how much working software a model can build across twenty programming languages. The second evaluation covers translating COBOL programs while preserving their outputs. Both are vendor-published benchmarks, not a record of completed bank migrations. The announcement updates the picture of OpenAI's agentic coding work: it makes specific tasks and success criteria observable, providing a concrete basis against which to verify the progress claimed.
On MultiLangBench, OpenAI reports an average test-pass rate of 87.4% for GPT-6 Astra at maximum reasoning, compared to 26.8% for GPT-5.6 Sol, according to OpenAI's October 2026 benchmark release. In COBOL migration, GPT-6.1 Sol reaches 86.6% at low reasoning versus 5.4% for its predecessor; at higher reasoning, that figure climbs to 97.9%. GPT-6 Luna scores 40.3% on MultiLangBench. These evaluations use different criteria, so those percentages don't form a single unified ranking.
↵Our latest model advances bring stronger capabilities for building and migrating software in low-resource programming languages such as COBOL, Fortran, and Visual Basic.
— OpenAI (@OpenAI) October 19, 2026
These languages power software that banks, governments, and businesses depend on. Progress here means… pic.twitter.com/Mulo5P3iCO
For anyone tracking enterprise AI adoption, the key question is what you're actually asking the system to do. Writing a new function and replacing a program that's been running for decades may require overlapping skills. They carry different risks. In the second case, a simplification that looks reasonable can quietly change a rule nobody documented anywhere else. Proper evaluation must therefore account for what the new code is required to preserve, not only what it adds.
How a Rebuilt SQLite Engine Gets Checked
The public MultiLangBench repository explains the protocol. The model receives SQLite 3.40.1 documentation and must produce an executable in the requested language. Each test case starts from an initial database, runs a query against the candidate program, and verifies the result using the reference implementation. This approach evaluates implementations written in different languages through a common interface, rather than simply judging whether the code looks plausible to a human reviewer.
The README also distinguishes between available material and the suggested sample. Around 24,000 cases were converted from the original suite; for cost reasons it recommends a subset of 272 tests, while permitting broader runs as long as they're declared. Conflating the total number of cases in the repository with the number actually run in any given evaluation would be a mistake. That detail matters because a percentage only becomes comparable when the sample, environment, and execution rules are all explicit.
The instructions require an internet-free environment and the removal of any SQLite copies the model might reuse. These are conditions designed to isolate the capability being observed. In benchmark comparisons, the scope of available tools is part of the result itself. Here the question is whether the model can build a working implementation from scratch: allowing it to call the original program would change the task entirely and make the score far less informative.
Why Preserving Outputs Is Harder Than Rewriting
The more interesting criterion, in our reading, is behavioral continuity. A program can compile, run fast, and produce a valid output file while handling an edge case differently from its predecessor. Consider a records archive where an empty field means “pending review.” A rewrite that converts it to zero might pass most standard tests while silently changing the meaning of every incomplete record still in the system.
Comparing new output against old output helps identify differences like that one. It doesn't decide on its own whether the old rule was correct. Two separate tasks are involved: replicating existing behavior and deliberately choosing which behaviors to change. Conflating them makes it difficult to tell whether a discrepancy is a migration error or an intentional update. Human oversight remains necessary precisely when the decision depends on a business rule, not on language syntax.
OpenAI describes a concrete example: COBOL's fixed-point arithmetic truncates excess decimal digits when rounding isn't explicitly requested. Rewriting that section with rounding produces different numbers without any runtime error. That's why the migration test requires byte-for-byte correspondence of observable outputs and counts a migration as successful only if it passes every test case. The definition of “success” here is more demanding than compilation alone.
