A long-task agent: strong execution, incomplete delivery

https://touch.long-arena.com/en/ai-evaluations/max-long-task-2026-09/

Evaluation date: 2026-09-11

An independent review of the Max daily text workspace and sales-expert capability plans. Substantial implementation did not yet add up to reliable acceptance.

Implementation difficulty: 8.5/10

Delivery performance: 60/100

Acceptance verdict: blocked (reviewed-historical-snapshot)

Reviewer: agent + model: Codex + GPT-6 Astra

Subject: agent + model: dsh + DeepSeek V4.1 Flash

Agent and model identities were supplied by the commissioner on September 11, 2026. Execution sessions, request IDs and pinned model versions were not independently verified.

Codex was the reviewer, not a competing implementation. This is not a same-task Codex-versus-DeepSeek experiment, a base-model ranking or a vendor endorsement.

The task

The subject was the coding-agent and model combination delivering an existing application, not the model serving Max business conversations. The scope crossed Vue and TypeScript UI, backend services, organization permissions, state persistence, migrations and acceptance tooling.

Stage A: daily text workspace
Document state, revision history, localization, narrow-container layout and mobile navigation. Local UI fixes were real; scope isolation and complete editing journeys were still missed.
7/10; 55/100

Stage B: sales-expert capability layer
Process facts, B2B/B2C flows, authorization, provenance, migration preparation, governed tools and evaluations. The integration and safety constraints made this the harder stage.
9/10; 62/100

Stage scores are supporting judgments, not acceptance ratios. The overall score is calculated from the seven dimensions below, not by averaging the two stages.

Evaluation method

Blind review; identity disclosed after scoring

This is a single-case review with identity disclosed after scoring, under the commissioner’s stated blind-review method. There was no retained preregistration, random assignment or independently verified masking protocol. Repository tool markers could reveal clues, so complete blinding and double blinding are not established.

Compare the plan with delivery
Distinguish an implementation claim from a reachable business entry point, persisted result and completed acceptance item.

Rerun bounded engineering checks
Use type checks and existing targeted tests. Keep local checks separate from earlier real-login or live-model records.

Challenge critical invariants
Probe trusted identity, scope transitions, repeated normalization, persistence round trips and the failure behavior of release checks.

Score before adding identity metadata
The recorded review and numerical assessment preceded the identity information supplied for this publication. Identity disclosure did not change the scores.

Score breakdown

Overall score = sum of each dimension’s score × its weight.

Task decomposition and sustained execution: 85/100; 15%; 12.75
The agent sustained work across modules and adjusted the learning scope when available feedback was insufficient.

Implementation and integration: 75/100; 25%; 18.75
There was working code and meaningful compatibility evidence, but some implemented components were not wired into a complete user journey.

Security boundaries and data consistency: 35/100; 20%; 7.00
Trusted identity, document scope and provenance round trips exposed major unresolved invariants, including inherited behavior missed by this acceptance.

Test design and counterexamples: 65/100; 15%; 9.75
Real-runtime evaluation and can-fail checks were improvements. Adjacent states, round trips and failures of the checks themselves remained under-tested.

Business closure and UI acceptance: 40/100; 10%; 4.00
A visible component or loading screen is not a completed task. Write/readback, recovery and rollback evidence remained incomplete.

Evidence accuracy and conclusion calibration: 35/100; 10%; 3.50
Statements such as “the only blocker” and “structurally impossible” exceeded the scope of the retained evidence.

Operational discipline and traceability: 85/100; 5%; 4.25
Authorization and branch boundaries were respected, and changes were recorded. Refusing an unauthorized registration was correct, not a penalty.

On this rubric, 60–69 means substantial output that still needs strong independent review and rework. It does not mean acceptance passed.

Evidence layers

Passing checks and failing counterexamples answer different questions. They are not combined into a synthetic success rate.

Existing targeted tests passed: 840 (681 + 116 + 43)

Targeted boundary checks failed: 7; Critical problem groups: 3

Rerun locally
Type checks, targeted frontend/backend/industry tests, and synthetic boundary probes. No real business writes were introduced by this independent review.

Inspected, not rerun live
Earlier real-login screenshots and live-model evaluation records were reviewed. A function precheck was not counted as a browser journey; a loading screenshot was not accepted as a completed task.

Still not established
Complete business write/readback journeys, exception recovery, rollback and the other original acceptance gaps. Registration authorization alone cannot turn the result into Ready.

Technical findings

01 / Identity must remain a trusted fact
An authorization decision could be affected by untrusted presentation metadata rather than only the trusted actor and current organization’s permissions.
authorize(trustedIdentity, currentScope)
// Display labels are not authorization facts.
Hold trusted identity constant while varying labels, untrusted metadata and neighboring organization membership. None may upgrade the granted authority.

02 / A revision number is not a document identity
Workspace state was insufficiently isolated across documents and organization changes. A matching revision number did not prove that the content belonged to the active document.
revisionKey = (organization, user, workspace, document, revision)
restore(scopeB) must not retain scopeA.content
Exercise two documents with the same revision and two scopes with different drafts. Assert cleanup at scope transitions and ownership immediately before using content.

03 / Normalization must not invent evidence
A default provenance value could become “stated” after another normalization or persistence round trip. Reported coverage rose without new source evidence.
coverage(normalize(normalize(x))) == coverage(normalize(x))
coverage(decode(encode(x))) == coverage(x)
// No new evidence means no higher verified coverage.
Preserve the distinction between unknown and verified data through repeated transformations and normal reads. Test the actual read chain, not only the first normalization.

04 / A check that cannot read must not pass
A change-inspection error was treated as an empty result. The process returned success without successfully establishing whether a relevant change existed.
readFailure => checkFailure
checkFailure => nonzeroExit
// An unreadable change set is not an empty change set.
Inject read errors and oversized inputs. A gate must fail closed and propagate a nonzero exit status; test the checker as well as the business code.

What worked

Implemented meaningful cross-module changes and repaired real localization and responsive-layout issues.

Adjusted automatic learning to collection-first when real feedback was too sparse, with a switch-off test independent of the threshold.

Respected authorization limits instead of creating a real organization merely to obtain a passing result.

What limited reliability

Local success was sometimes promoted to a full-chain completion claim.

Single-case fixes did not consistently cover adjacent inputs, scope transitions and persistence round trips.

Acceptance evidence did not always demonstrate the actual business end state.

No points were deducted for refusing unauthorized real-world registration or writes. The deductions concern unwired functionality, missed checks, overstated passes and omitted blockers.

Use the agent to implement; separate the acceptance decision.

Keep independent review for trusted identity, data isolation, evidence semantics and real user journeys. A long-task agent can contribute substantial engineering work without yet being a reliable sole signatory for its own delivery.

Scope and sources

One delivery snapshot, assessed retrospectively with an expert rubric. No repeat-run distribution, control group or statistical model comparison.

Precise tool versions, model snapshots, prompts, inference settings, token usage, cost and complete elapsed-time measurements were not verified for this publication. No productivity or cost claim is made.

The repository and harness context can affect the outcome. Findings cannot be attributed solely to DeepSeek V4.1 Flash or generalized to every dsh run.

This is a historical implementation review, not a claim that the current production system has been attacked or that subsequent fixes were already verified.

Evidence provenance

Implementation plans dated September 8 and 9, 2026, together with the delivered code, tests and acceptance materials.

Independent local review on September 10–11, 2026, followed by the seven-dimension scoring assessment.

Commissioner-supplied identity and blind-review metadata on September 11, 2026.

Private code, raw conversations, credentials, customer identifiers and actionable vulnerability details are not distributed. The PDF is a score summary, not the full private audit evidence pack.

Download score summary · Chinese PDF: https://touch.long-arena.com/reports/2026-09-11-agent-long-task-assessment.pdf

3 pages · Chinese · historical score summary