Methodology

MOMENT Eval measures model performance on structural engineering tasks contributed by engineers at Thornton Tomasetti. Thornton Tomasetti is a global structural engineering firm with deep experience in building design, bridge engineering, forensic investigation, performance-based design, and advanced analysis, so the benchmark is grounded in the kind of judgment structural engineers use in practice.

The current benchmark contains 91 questions spanning bridge loads, building loads, environmental loads, foundations, retaining walls, engineering mechanics, structural analysis, connection design, protective design, and member design in concrete, masonry, steel, and timber. The problems are intended to probe code reasoning, load path understanding, numerical calculation, geometric interpretation, and engineering decision-making.

All tasks are text-only. When a source problem uses a drawing, the benchmark prompt replaces it with a verbal geometry description that is sufficient to solve the problem. The model does not see images.

Each problem asks for a specific engineering result, such as a force, moment, pressure, stress, member size, or code classification. The model is asked to return JSON that follows a strict task-specific schema. Chains of thought and reasoning traces are useful for the model, but the verifier evaluates the structured answer.

The benchmark is designed to test engineering skill and judgment, not memorization. If a practicing engineer would normally need a table value, coefficient, section property, or similar reference excerpt to solve the problem, the prompt provides that information.

The prompt does not provide a worked solution or tell the model which equations to use unless that equation or table excerpt is itself the necessary reference material. The model still has to identify the relevant load path, choose the appropriate checks, perform the calculation, and return the requested result.

Each model receives the same problem prompt and has access to two tools: a calculator and a sandboxed Python interpreter. The Python environment includes numpy for array and numerical work, scipy for scientific computation, and sympy for symbolic algebra. These tools are provided because engineers routinely use calculators and scripts for quantitative work.

Models cannot search the internet, install additional libraries, or access files outside the sandbox. Each task is run with a 64,000 output-token budget and a 20-call tool budget. These limits keep runs comparable and prevent a model from consuming unbounded compute on a single problem.

Each task has a deterministic verifier. Numeric answers are checked against a task-specific tolerance; exact-answer tasks are checked against the expected option or classification. A run's score is the average score across the task set.

A task fails if the model returns an answer outside the acceptable tolerance, returns the wrong exact answer, or uses the entirety of its token budget without returning an answer. If the model returns invalid JSON, the harness asks it to repair the response and retries while token budget remains.

Parsing errors and provider infrastructure failures are treated differently from wrong engineering answers. If a response is not valid JSON, or if the provider has an outage, throttling event, transport error, or similar transient failure, the harness retries where possible.

Retries stop once the model returns a valid answer or exhausts its token budget. A wrong value is not retried as a parsing failure; it is scored as the model's answer.

In addition to score, MOMENT Eval records total tokens, estimated provider cost, inference time, and average tool calls per task. Cost is calculated from the provider pricing configured for the run and the token usage reported by the provider API.

Timing reflects the observed benchmark run from our runner environment, including provider latency and tool-use overhead. It is useful for comparing broad behavior, but it is not a controlled latency benchmark.

Runs are reviewed before publication. We check for harness errors, provider outages, obvious schema issues, and other artifacts that should not be interpreted as engineering performance.

Only published runs appear on the public leaderboard. Deprecated targets remain visible by default for historical comparison and can be hidden with the model selectors.

MOMENT Eval is not a substitute for professional engineering review, and a high score does not qualify a model to practice engineering. The benchmark measures performance on a curated set of constrained tasks with known answers.

The goal is to make model strengths and failure modes visible: whether a model can read a structural problem carefully, use provided references appropriately, compute reliably, and return a verifiable engineering result.