AS/400 reverse-engineering eval, 15 October 2026

Executive summary

The offline skills engine and the offline custom engine produce the same scores on these four targets. Both use the deterministic offline extractor. Mean rule recall is 0.0813. Mean documented-rule precision is 0.1528. Mean objective is 0.5267.

Mutation-detection rate is 0 on every offline run. Several patches do not match the pinned source text. The patches that do apply do not change the extracted rule blob, so the detector records a miss.

Anthropic extract on claude-sonnet-5 is in from the pull_request llm job. Skills mean recall is 0.4313. Custom mean recall is 0.4813. Skills mean documented-rule precision is 0.1472. Custom mean documented-rule precision is 0.124. Skills mean objective is 0.3409. Custom mean objective is 0.3503.

Skills mean mutation-detection rate is 0.7084. Custom mean mutation-detection rate is 0.7084. Combined Anthropic extract cost across the eight rows is $9.223. Execution stays not applicable on every corpora run.

Labels other than cognidev are LLM-drafted and pending human review. Cognidev uses the CogniDev understand spec as primary truth.

Methodology

You score a target by cloning its public upstream at the SHA in source.yaml, running extract, and matching kept rules to corpora rules.json. Matching is a case-insensitive needle search over title, description, condition, and effect. Needles come from the rule statement, the rule id, and citation file names.

The four targets are cognidev-order-mgmt, flight400, ibmi-inventory, and ibm-company-system. The corpora repo is ecociel/as400-eval-corpora at 4b3f612d478ef7437592ae032caf9d143d93a855. Fetch uses evals/corpora/scripts/fetch.py. Source stays in a gitignored cache. Cleanup deletes it after scoring.

Each engine runs once on the untouched clone, then once per mutation case after a semantic patch on a copy. A mutation counts as detected when the patch applies and the extracted rule blob changes, or when the replacement text appears only after the patch.

This Cloud Agent token cannot clone the private corpora repo. CI tries that clone first. If GitHub returns 404, CI copies evals/corpora/derived/ (JSON, YAML, and Markdown only) and still fetches the four public upstreams anonymously.

CI links: pull request 7, pull_request evals run, push evals run. The llm job uses the ANTHROPIC_API_KEY secret and model claude-sonnet-5.

Findings

Offline baseline (skills and custom)

LLM extract, claude-sonnet-5 (skills and custom)

Recall by target, offline skills

Objective by target, offline skills

Recall by target, Anthropic skills

Recall by target, Anthropic custom

On Anthropic, skills and custom diverge. The engines no longer share one deterministic extractor, so the two CLI paths are a real comparison.

Anthropic skills mean recall is 0.4313 against offline mean recall 0.0813.

Anthropic keeps far more rules than the offline extractor. Documented-rule precision does not rise with recall, because the extra rows do not match the corpora list. Mean objective falls because citation precision and noise get worse, and execution stays not applicable.

Flight400 has the highest offline objective (0.6779). IBM Company System has the lowest. Offline extract keeps file-write and call-graph rows. It rarely restates a CHECK constraint, a message file, or a buried RPG III compare, so needle match stays low.

Skills and custom match exactly on every offline cell. That is not two methods agreeing. It is one offline extractor reached through two CLI engines.

cognidev-order-mgmt

Cognidev rules use the CogniDev understand spec as primary truth. Source origin is corpora-cache.

Scores

Matched ground-truth ids on offline skills: ord-line-updates-article-stock.

Matched ground-truth ids on Anthropic skills: ord-deliver-qty-le-ordered, ord-qty-ge-already-delivered, ord-unknown-customer, ord-insert-sets-customer-last-order-date, art-vat-code-required.

Mutation cases, offline skills

Mutation cases, Anthropic skills

flight400

Labels other than cognidev are LLM-drafted and pending human review. Source origin is corpora-cache.

Scores

Matched ground-truth ids on offline skills: frs-main-menu-is-reservation-system.

Matched ground-truth ids on Anthropic skills: frs-customer-unique-number, frs-main-menu-is-reservation-system, frs-order-display-uses-order-number.

Mutation cases, offline skills

Mutation cases, Anthropic skills

ibmi-inventory

Labels other than cognidev are LLM-drafted and pending human review. Source origin is corpora-cache.

Scores

Matched ground-truth ids on offline skills: inv-stock-nonnegative.

Matched ground-truth ids on Anthropic skills: inv-working-days-skip-holidays.

Mutation cases, offline skills

Mutation cases, Anthropic skills

ibm-company-system

Labels other than cognidev are LLM-drafted and pending human review. Source origin is corpora-cache.

Scores

Matched ground-truth ids on offline skills: none.

Matched ground-truth ids on Anthropic skills: cmp-dept-pk, cmp-employee-pk, cmp-employee-belongs-to-dept, cmp-dept-admin-cascade, cmp-phone-numeric-range, cmp-dept-cost-is-sum-of-pay.

Mutation cases, offline skills

Mutation cases, Anthropic skills

Miss analysis

The counts below use the offline skills run.

cognidev-order-mgmt (offline skills)

Miss classes: dds 5, other 1, screen 1, sql 1, sqlrpgle 1.

• ord-deliver-qty-le-ordered. Delivered quantity on a line must be less than or equal to ordered quantity. Classified as sqlrpgle.
• ord-qty-ge-already-delivered. Ordered quantity cannot be lowered below the quantity already delivered. Classified as dds.
• ord-unknown-customer. An order requires a known customer; unknown customer codes are rejected with F4-select guidance. Classified as dds.
• ord-unknown-article. An order line requires a known article. Classified as dds.
• ord-insert-sets-customer-last-order-date. After an ORDER insert, CUSTOMER.CULASTORD is set to the order date. Classified as sql.
• cus-credit-limit-tracked. Customer master stores credit limit (CULIMCRE) and current credit (CUCREDIT). Classified as screen.
• art-vat-code-required. Each article carries a VAT code (ARVATCD) resolved through VATDEF. Classified as dds.
• ref-unknown-country-family-provider. Country, family, and provider codes must exist; unknown values prompt F4 select. Classified as other.
• soft-delete-flags. Article, customer, provider, family, and VAT rows carry a one-character delete flag rather than physical delete-only lifecycle. Classified as dds.

flight400 (offline skills)

Miss classes: dds 3, other 1, screen 3.

• frs-signon-required. The reservation system is entered through FRS000 sign-on before the main menu. Classified as screen.
• frs-order-needs-customer-flight-tickets. An order stores customer number, flight number, departure date, tickets ordered, and class. Classified as dds.
• frs-flight-has-seats-available. Each flight tracks seats available and ticket price. Classified as screen.
• frs-zero-seats-is-special-path. Flight inquiry treats zero seats as a distinct path (SSEATS equal zeros). Classified as screen.
• frs-customer-unique-number. Customers are uniquely identified by customer number. Classified as dds.
• frs-order-number-unique. Orders are uniquely identified by order number and may reference an agent. Classified as dds.
• frs-order-display-uses-order-number. Order display programs FRS409/FRS410 are parameterized by order number. Classified as other.

ibmi-inventory (offline skills)

Miss classes: other 2, sql 7.

• inv-movement-types. Movement type must be I (in), O (out), or A (adjust). Classified as sql.
• inv-active-item-warehouse. Posting requires an existing active item and active warehouse. Classified as sql.
• inv-commit-stock-and-movement. Stock update and movement insert commit or roll back together under commitment control. Classified as sql.
• inv-lock-and-recheck. The stock row is re-read with a lock and quantity is checked again before posting. Classified as sql.
• inv-optimistic-item-lock. Item change uses optimistic locking: display without lock, re-read with lock before update. Classified as other.
• inv-item-price-and-min-nonnegative. Item unit price and minimum quantity cannot be negative. Classified as sql.
• inv-reorder-when-below-min. STOCK_STATUS marks reorder when on-hand is below the item minimum. Classified as sql.
• inv-api-same-posting-rules. REST POST /api/movements uses POST_MOVEMENT; insufficient stock is HTTP 409 / SQLSTATE 75006. Classified as sql.
• inv-working-days-skip-holidays. DATEUTIL working-day math skips weekends and HOLIDAY_CALENDAR rows. Classified as other.

ibm-company-system (offline skills)

Miss classes: sql 5, sqlrpgle 3.

• cmp-dept-pk. Department is uniquely identified by DEPTNO. Classified as sql.
• cmp-employee-pk. Employee is uniquely identified by EMPNO. Classified as sql.
• cmp-employee-belongs-to-dept. EMPLOYEE.WORKDEPT references DEPARTMENT. Deleting a department sets WORKDEPT to null. Classified as sql.
• cmp-dept-admin-cascade. DEPARTMENT.ADMRDEPT references DEPARTMENT. Deleting an admin department cascades. Classified as sql.
• cmp-phone-numeric-range. Employee phone number must be a 4-character value between 0000 and 9998. Classified as sql.
• cmp-new-employee-needs-dept-and-numeric-salary. NEWEMP cannot continue without a department and rejects blank or non-numeric salary. Classified as sqlrpgle.
• cmp-dept-list-opens-employees. Department maintenance can open the employee program for a selected department. Classified as sqlrpgle.
• cmp-dept-cost-is-sum-of-pay. Department detail net income is the sum of employee salary, bonus, and commission. Classified as sqlrpgle.

The counts below use the Anthropic skills run.

cognidev-order-mgmt (Anthropic skills)

Miss classes: dds 2, other 2, screen 1.

• ord-unknown-article. An order line requires a known article. Classified as dds.
• ord-line-updates-article-stock. Insert, update, and delete of DETORD lines adjust article quantities through ORD700. Classified as other.
• cus-credit-limit-tracked. Customer master stores credit limit (CULIMCRE) and current credit (CUCREDIT). Classified as screen.
• ref-unknown-country-family-provider. Country, family, and provider codes must exist; unknown values prompt F4 select. Classified as other.
• soft-delete-flags. Article, customer, provider, family, and VAT rows carry a one-character delete flag rather than physical delete-only lifecycle. Classified as dds.

flight400 (Anthropic skills)

Miss classes: dds 2, screen 3.

• frs-signon-required. The reservation system is entered through FRS000 sign-on before the main menu. Classified as screen.
• frs-order-needs-customer-flight-tickets. An order stores customer number, flight number, departure date, tickets ordered, and class. Classified as dds.
• frs-flight-has-seats-available. Each flight tracks seats available and ticket price. Classified as screen.
• frs-zero-seats-is-special-path. Flight inquiry treats zero seats as a distinct path (SSEATS equal zeros). Classified as screen.
• frs-order-number-unique. Orders are uniquely identified by order number and may reference an agent. Classified as dds.

ibmi-inventory (Anthropic skills)

Miss classes: other 1, sql 8.

• inv-stock-nonnegative. Stock on hand cannot go below zero. Database CHECK is the last line of defence. Classified as sql.
• inv-movement-types. Movement type must be I (in), O (out), or A (adjust). Classified as sql.
• inv-active-item-warehouse. Posting requires an existing active item and active warehouse. Classified as sql.
• inv-commit-stock-and-movement. Stock update and movement insert commit or roll back together under commitment control. Classified as sql.
• inv-lock-and-recheck. The stock row is re-read with a lock and quantity is checked again before posting. Classified as sql.
• inv-optimistic-item-lock. Item change uses optimistic locking: display without lock, re-read with lock before update. Classified as other.
• inv-item-price-and-min-nonnegative. Item unit price and minimum quantity cannot be negative. Classified as sql.
• inv-reorder-when-below-min. STOCK_STATUS marks reorder when on-hand is below the item minimum. Classified as sql.
• inv-api-same-posting-rules. REST POST /api/movements uses POST_MOVEMENT; insufficient stock is HTTP 409 / SQLSTATE 75006. Classified as sql.

ibm-company-system (Anthropic skills)

Miss classes: sqlrpgle 2.

• cmp-new-employee-needs-dept-and-numeric-salary. NEWEMP cannot continue without a department and rejects blank or non-numeric salary. Classified as sqlrpgle.
• cmp-dept-list-opens-employees. Department maintenance can open the employee program for a selected department. Classified as sqlrpgle.

Limitations

• Flight400, IBM i inventory, and IBM Company System labels are LLM-drafted and pending human review. Treat recall as a draft-label score, not a human gold score.
• The GitHub App token for this agent sees repository_selection: selected with only as400-re-harness. It cannot clone ecociel/as400-eval-corpora or ecociel/rpg-demo-stack. Derived labels in this PR are the 4b3f612 snapshot, not a live checkout.
• Corpora scenarios are not executable in this Linux job. Execution is recorded as not applicable, not as 0. The rpg-demo-stack rpg_interp --trace interpreter is used when that snapshot is present. These four trees are not that order-to-cash fixture.
• Mutation find_pattern values are exact substrings. Line-wrapped message text and *ZEROS versus ZEROS make some patches fail to apply. Applied patches that only change a CHECK or a message still leave the offline rule blob unchanged, so offline detection stays 0. Anthropic extract often changes the blob, so mutation-detection rises on those same patches.
• Never changing AS/400 or SAP objects means this run does not execute green-screen programs on IBM i.

Next steps

1. Have a human review the non-cognidev rule lists before using them as a holdout gate.
2. Mint a GitHub App token that includes the corpora repo, or add that repo to the Cloud Agent environment repo list, so CI stops using the derived fallback.
3. Teach extract to emit CHECK constraints, message-file text, and RPG III indicator compares so needle match can rise without leaking ground truth into extract.
4. Align mutation find_pattern strings with the pinned source, still without committing that source.