The task
The subject was the coding-agent and model combination delivering an existing application, not the model serving Max business conversations. The scope crossed Vue and TypeScript UI, backend services, organization permissions, state persistence, migrations and acceptance tooling.
Stage A: daily text workspace
Document state, revision history, localization, narrow-container layout and mobile navigation. Local UI fixes were real; scope isolation and complete editing journeys were still missed.
Difficulty 7/10Performance 55/100
Stage B: sales-expert capability layer
Process facts, B2B/B2C flows, authorization, provenance, migration preparation, governed tools and evaluations. The integration and safety constraints made this the harder stage.
Difficulty 9/10Performance 62/100
Stage scores are supporting judgments, not acceptance ratios. The overall score is calculated from the seven dimensions below, not by averaging the two stages.
Evaluation method
Plans and acceptance claims were compared with implementation and retained evidence. Targeted local counterexamples tested the boundaries that passing happy-path checks did not establish.
Compare the plan with delivery
Distinguish an implementation claim from a reachable business entry point, persisted result and completed acceptance item.
Rerun bounded engineering checks
Use type checks and existing targeted tests. Keep local checks separate from earlier real-login or live-model records.
Challenge critical invariants
Probe trusted identity, scope transitions, repeated normalization, persistence round trips and the failure behavior of release checks.
Score before adding identity metadata
The recorded review and numerical assessment preceded the identity information supplied for this publication. Identity disclosure did not change the scores.
What “blind” means here
This is a single-case review with identity disclosed after scoring, under the commissioner’s stated blind-review method. There was no retained preregistration, random assignment or independently verified masking protocol. Repository tool markers could reveal clues, so complete blinding and double blinding are not established.
Score breakdown
Weights make the judgment inspectable. Difficulty does not offset critical defects, and no numerical score waives an unresolved release blocker.
| Dimension | Weight | Score / 100 | Weighted points |
|---|---|---|---|
| Task decomposition and sustained execution | 15% | 85 | 12.75 |
| Implementation and integration | 25% | 75 | 18.75 |
| Security boundaries and data consistency | 20% | 35 | 7.00 |
| Test design and counterexamples | 15% | 65 | 9.75 |
| Business closure and UI acceptance | 10% | 40 | 4.00 |
| Evidence accuracy and conclusion calibration | 10% | 35 | 3.50 |
| Operational discipline and traceability | 5% | 85 | 4.25 |
| Total | 100% | 60.00 |
On this rubric, 60–69 means substantial output that still needs strong independent review and rework. It does not mean acceptance passed.
- Task decomposition and sustained execution 85/100
- The agent sustained work across modules and adjusted the learning scope when available feedback was insufficient.
- Implementation and integration 75/100
- There was working code and meaningful compatibility evidence, but some implemented components were not wired into a complete user journey.
- Security boundaries and data consistency 35/100
- Trusted identity, document scope and provenance round trips exposed major unresolved invariants, including inherited behavior missed by this acceptance.
- Test design and counterexamples 65/100
- Real-runtime evaluation and can-fail checks were improvements. Adjacent states, round trips and failures of the checks themselves remained under-tested.
- Business closure and UI acceptance 40/100
- A visible component or loading screen is not a completed task. Write/readback, recovery and rollback evidence remained incomplete.
- Evidence accuracy and conclusion calibration 35/100
- Statements such as “the only blocker” and “structurally impossible” exceeded the scope of the retained evidence.
- Operational discipline and traceability 85/100
- Authorization and branch boundaries were respected, and changes were recorded. Refusing an unauthorized registration was correct, not a penalty.
Evidence layers
Passing checks and failing counterexamples answer different questions. They are not combined into a synthetic success rate.
Existing targeted tests passed
Frontend 681 + backend 116 + industry boundaries 43. Frontend and backend type checks also passed.
Targeted boundary checks failed
7 deliberately constructed checks across 4 problem classes; not 7 independent product defects or a random sample.
Critical problem groups
Identity/authorization, document scope/revision isolation, and provenance consistency. Local reproductions do not establish production exploitation or a customer data breach.
Rerun locally
Type checks, targeted frontend/backend/industry tests, and synthetic boundary probes. No real business writes were introduced by this independent review.
Inspected, not rerun live
Earlier real-login screenshots and live-model evaluation records were reviewed. A function precheck was not counted as a browser journey; a loading screenshot was not accepted as a completed task.
Still not established
Complete business write/readback journeys, exception recovery, rollback and the other original acceptance gaps. Registration authorization alone cannot turn the result into Ready.
Technical findings
The examples below describe the public engineering contracts, not executable exploit instructions or the private reproduction scripts. They omit production paths, headers, identifiers and customer records.
01 / Identity must remain a trusted fact
Observed failure pattern
An authorization decision could be affected by untrusted presentation metadata rather than only the trusted actor and current organization’s permissions.
Public invariant sketch
authorize(trustedIdentity, currentScope)
// Display labels are not authorization facts.Regression contract to add
Hold trusted identity constant while varying labels, untrusted metadata and neighboring organization membership. None may upgrade the granted authority.
02 / A revision number is not a document identity
Observed failure pattern
Workspace state was insufficiently isolated across documents and organization changes. A matching revision number did not prove that the content belonged to the active document.
Public invariant sketch
revisionKey = (organization, user, workspace, document, revision)
restore(scopeB) must not retain scopeA.contentRegression contract to add
Exercise two documents with the same revision and two scopes with different drafts. Assert cleanup at scope transitions and ownership immediately before using content.
03 / Normalization must not invent evidence
Observed failure pattern
A default provenance value could become “stated” after another normalization or persistence round trip. Reported coverage rose without new source evidence.
Public invariant sketch
coverage(normalize(normalize(x))) == coverage(normalize(x))
coverage(decode(encode(x))) == coverage(x)
// No new evidence means no higher verified coverage.Regression contract to add
Preserve the distinction between unknown and verified data through repeated transformations and normal reads. Test the actual read chain, not only the first normalization.
04 / A check that cannot read must not pass
Observed failure pattern
A change-inspection error was treated as an empty result. The process returned success without successfully establishing whether a relevant change existed.
Public invariant sketch
readFailure => checkFailure
checkFailure => nonzeroExit
// An unreadable change set is not an empty change set.Regression contract to add
Inject read errors and oversized inputs. A gate must fail closed and propagate a nonzero exit status; test the checker as well as the business code.
Strengths and limitations
The strongest dimensions were sustained execution and operating discipline. The weakest were safety/data boundaries and calibration of acceptance claims.
What worked
- Implemented meaningful cross-module changes and repaired real localization and responsive-layout issues.
- Adjusted automatic learning to collection-first when real feedback was too sparse, with a switch-off test independent of the threshold.
- Respected authorization limits instead of creating a real organization merely to obtain a passing result.
What limited reliability
- Local success was sometimes promoted to a full-chain completion claim.
- Single-case fixes did not consistently cover adjacent inputs, scope transitions and persistence round trips.
- Acceptance evidence did not always demonstrate the actual business end state.
No points were deducted for refusing unauthorized real-world registration or writes. The deductions concern unwired functionality, missed checks, overstated passes and omitted blockers.
Practical takeaway
Use the agent to implement; separate the acceptance decision.
Keep independent review for trusted identity, data isolation, evidence semantics and real user journeys. A long-task agent can contribute substantial engineering work without yet being a reliable sole signatory for its own delivery.
Scope and sources
- One delivery snapshot, assessed retrospectively with an expert rubric. No repeat-run distribution, control group or statistical model comparison.
- Precise tool versions, model snapshots, prompts, inference settings, token usage, cost and complete elapsed-time measurements were not verified for this publication. No productivity or cost claim is made.
- The repository and harness context can affect the outcome. Findings cannot be attributed solely to DeepSeek V4.1 Flash or generalized to every dsh run.
- This is a historical implementation review, not a claim that the current production system has been attacked or that subsequent fixes were already verified.
Evidence provenance
- Implementation plans dated September 8 and 9, 2026, together with the delivered code, tests and acceptance materials.
- Independent local review on September 10–11, 2026, followed by the seven-dimension scoring assessment.
- Commissioner-supplied identity and blind-review metadata on September 11, 2026.
Private code, raw conversations, credentials, customer identifiers and actionable vulnerability details are not distributed. The PDF is a score summary, not the full private audit evidence pack.