On October 19, OpenAI published a post about "low-resource programming languages" with two evals. One asks a model to write a SQLite-compatible database in each of 20 languages. The other migrates COBOL programs to other languages and requires byte-for-byte identical output.

The numbers are big. On COBOL migration, GPT-5.6 Sol scores 5.4%, GPT-6.1 Sol scores 86.6% at low reasoning effort and 97.9% at higher effort. All of these are OpenAI's own figures. What I think is more useful is the eval design: it doesn't ask whether the code looks right, only whether behavior under hidden tests is exactly the same. If you hand migration or refactoring jobs to an agent, you can build your acceptance checks the same way.

"Low-resource" here means languages with little training code and documentation, such as COBOL, Fortran, Visual Basic and OCaml. The post notes they underpin critical systems like banking and weather forecasting.

Eval 1: MultiLangBench (build software from scratch). The task is to implement a SQLite-compatible database engine in each of 20 languages. Each implementation gets an initial database and a SQL query, and its output database is checked against a reference SQLite. The tests are adapted from SQLite's own test suite. Metrics are pass rate and API cost.

Eval 2: COBOL migration (preserve behavior). Migrate COBOL programs to C++, Java, Kotlin, Python or Rust. Hidden tests require byte-for-byte identical observable output, and a migration counts only if every test passes. The post calls out details such as COBOL's fixed-point rounding as a risk.

The key idea in a migration eval is to treat the old program as the answer key. The same hidden tests run on the old program first and the output is recorded. Then they run on the program the agent produced, and outputs are compared byte for byte. One differing test fails the whole migration.

That is far more reliable than asking another model whether the code looks right, because it doesn't depend on anyone's judgment, only on output. The cost is that you need tests that cover the business rules first. OpenAI also says in the post that migrating real production systems still needs expert review and tests against the system's business rules and dependencies.

Numbers from the post (all OpenAI's own results; I found no independent reproduction):

• MultiLangBench at maximum reasoning effort: GPT-6 Astra averages 87.4% versus 26.8% for GPT-5.6 Sol. It scores above 83% in 19 of 20 languages; COBOL, the hardest, is 72%.
• GPT-6 Luna (the cheaper model): 40.3% average at about $0.77 per sample, versus about $3.48 for GPT-5.6 Sol, roughly 78% cheaper.
• COBOL migration: GPT-6.1 Sol hits 86.6% at low reasoning effort at about 61% of GPT-5.6 Sol's API cost, and 97.9% at higher effort, close to saturating the benchmark. GPT-5.6 Sol scores 5.4%.

My take: going from 5.4% to 86.6% is a big jump, but "close to saturating" also means the eval may soon stop telling models apart. That is my speculation; the post doesn't say it.

The repo is openai/MultiLangBench (Apache-2.0). Key points from the README:

• The task is to implement SQLite from the SQLite 3.40.1 docs. Of roughly 51k TCL tests, about 24k were converted to the eval format.
• Each test has three parts: initial_database, user_query and validation_query. The validation query runs on real SQLite 3.40.1, and the output must match the expected string exactly.
• The model's program takes a database path as a command-line argument and reads SQL from stdin.
• Environment setup: put the docs in, remove every trace of SQLite (the repo has an example scripts/remove_sqlite.sh, and the README tells you to verify it yourself), and block internet access.
• The standard eval uses 272 tests (names starting with 00, 01 or 02); running the full set is allowed but must be noted. Crashes, mismatches and timeouts over 60 seconds count as failures; writing in the wrong language sets the pass rate to 0.

The README gives no run commands and doesn't name the 20 languages, so reproducing it means writing your own runner. I can't fill that gap for it.

• Fix the answer key first. Before migrating, record a set of inputs and outputs from the old system (real traffic, business samples, edge values), then let the agent start.
• Accept on output only. Don't let the agent decide "I got it right." Compare byte by byte with a script and attach the input to every diff so a human can read it.
• Hide the tests from the agent. The eval uses hidden tests so the agent can't hard-code answers. Keep your own acceptance set separate from any examples the agent can see.
• Watch the usual traps. OpenAI names fixed-point rounding. Money amounts, dates, string truncation and zero-padding rules in old systems deserve their own tests.
• Cut the network, clean the environment. MultiLangBench blocks internet access and removes SQLite so the model can't call an existing implementation. Think about this for any from-scratch rewrite task.

These are my takeaways from the eval design, not official OpenAI advice.

The post doesn't release the COBOL migration eval itself (only MultiLangBench is on GitHub), so the 86.6% and 97.9% figures can't be reproduced by you. I also didn't see how large or complex the COBOL programs in the test set are.

One more thing: the eval measures matching output, not maintainable code. Whether a team can take over the migrated Java is a separate question this post doesn't cover.

Low-resource language

A programming language with little training data and documentation, such as COBOL or Fortran.

Hidden tests

Tests the agent can't see, so it can't hard-code answers to pass them.

Byte-for-byte

Exactly identical output, down to spaces and decimal places.

Fixed-point rounding

Decimal math with a fixed number of places. Its rounding rules differ from floating point, so a migration can end up a cent off.

Reasoning effort

A setting for how long the model thinks. Higher is usually more accurate and more expensive.

• OpenAI: Advancing coding in low-resource programming languages (2026-10-19)
• GitHub: openai/MultiLangBench (README)

All numbers come from these two pages and are OpenAI's own results. I found no independent reproduction. Defer to the originals.