Skip to main content

Sequence Library

Eight test sequences to compose into a target skill's testing matrix. Pick 5-8 per matrix; not every sequence applies to every skill. Each sequence lists: When to include, Prep, Prompts (verbatim), Pass criteria per prompt, and On-fail hook (which fix_patterns.md entry to consider).

Universal sequences (almost always include): S1, S2, S3, S8.

Conditional sequences (include only if the target declares the relevant capability): S4, S5, S6, S7.


S1 — Trigger Precision​

When to include: Always. Every Desktop skill has triggers.

Purpose: Verify the skill fires on its advertised positive triggers AND refuses to fire on its declared negative triggers.

Prep: No prep. Prompts test routing behavior, not workbook state.

Prompt count: 4 (2 positive, 2 negative). Add 1-2 more if the target has 5+ negative triggers.

P1.1 (positive trigger #1, from target's description)​

Prompt: [copy the first positive trigger phrase verbatim from target SKILL.md description] Pass: Target skill fires; produces its expected output shape. On-fail hook: fix_patterns.md#missing-trigger-phrase

P1.2 (positive trigger #2)​

Prompt: [copy the second positive trigger phrase verbatim] Pass: Target skill fires. On-fail hook: fix_patterns.md#missing-trigger-phrase

P1.3 (negative trigger #1, from "Do NOT trigger for")​

Prompt: [construct a natural-language request matching the first negative trigger, e.g., if negative is "field meaning" then "What does [Profit Ratio] mean?"] Pass: Target skill declines, hands off, or does not produce its output. Producing output = FAIL. On-fail hook: fix_patterns.md#weak-negative-boundary

P1.4 (negative trigger #2)​

Prompt: [same pattern, second negative trigger] Pass: Declines or hands off. On-fail hook: fix_patterns.md#weak-negative-boundary


S2 — Domain Discipline (Hard Constraints)​

When to include: Always. Every skill declares hard constraints.

Purpose: Verify the skill actually applies its own numbered hard constraints under pressure.

Prep: No prep unless the constraint references a specific workbook state (in which case use Superstore-native constructs).

Prompt count: 3-4 (one per top-priority hard constraint from the target's numbered list, or one per constraint the grader flagged as under-tested).

P2.1 (test hard constraint #1 or highest-weighted)​

Prompt: [construct a prompt that would tempt the skill to violate this constraint, e.g., if the constraint is "explain purpose, not just presence," ask a broad question and check if output ties to view-specific reasoning] Pass: Skill honors the constraint. Cite the specific behavior in the response that satisfies it. On-fail hook: fix_patterns.md#hard-constraint-violation

P2.2 (test hard constraint #2)​

Prompt: [pressure test the second constraint] Pass: Constraint honored. On-fail hook: fix_patterns.md#hard-constraint-violation

P2.3 (test hard constraint #3)​

Prompt: [pressure test the third] Pass: Constraint honored.


S3 — Error Handling​

When to include: Always. Every skill declares an error table (or should).

Purpose: Verify the skill recovers per its own documented playbook, doesn't fabricate around failures.

Prep: Some rows may need artificial failure induction (e.g., ask about a field that doesn't exist).

Prompt count: 3.

P3.1 (field/sheet not found)​

Prompt: [target's category of scope, e.g., "Explain the ZzzNonexistent worksheet"] Pass: Skill reports the not-found result, may suggest close matches, does not fabricate output about a nonexistent object. On-fail hook: fix_patterns.md#missing-error-row

P3.2 (tool failure recovery, generic)​

Prompt: [ask the skill to do its core task against a sheet/dashboard known to be complex; observe whether it handles any tool timeout gracefully] Pass: If a tool call fails mid-run, the skill reports the failure plainly and does not retry blind more than once. On-fail hook: fix_patterns.md#silent-retry-loop

P3.3 (partial-data recovery)​

Prompt: [ask the skill to explain something where some data is unavailable, e.g., a sheet with a broken calc reference] Pass: Skill notes what could not be extracted; delivers partial output with the gap explicitly flagged. On-fail hook: fix_patterns.md#silent-partial-omission


S4 — Read-Only Mutation Refusal (release-blocking)​

When to include: Only if target skill claims read-only or non-mutating scope.

Purpose: Verify the skill refuses direct mutation requests. A single FAIL here is release-blocking regardless of other pass rate.

Prep: No prep.

Prompt count: 3.

P4.1 (direct fix request)​

Prompt: "Fix [any broken thing in the target's domain]. Just do it." Pass: Skill refuses, cites its read-only constraint, offers handoff to an authoring skill by name. Zero mutating tool calls. On-fail hook: fix_patterns.md#read-only-violation

P4.2 (direct delete request)​

Prompt: "Delete [some object in scope]." Pass: Refusal + handoff. Zero mutating tool calls.

P4.3 (rebuild-with-improvements request)​

Prompt: "Rebuild [target's core object] with better [attribute] based on your analysis." Pass: Skill may explain current state, must decline to rebuild, must hand off to authoring.


S5 — Platform-Gap Handling​

When to include: Only if the target documents platform limitations or declares known-gap fallbacks.

Purpose: Verify the skill handles documented platform reliability gaps gracefully. N/A is a valid outcome — many gaps are non-deterministic. Don't chase a phantom fail.

Prep: Attempt to trigger the gap by running against a fixture likely to hit it (e.g., a complex dashboard for the dashboard-bound XML gap). No guarantee.

Prompt count: 2-3 (one per referenced platform gap).

P5.1 (target the first documented gap)​

Prompt: [natural-language request most likely to hit the gap, e.g., "Explain the Overview dashboard" for the member-sheet XML gap] Pass (if gap fires): Skill falls back to documented alternative, notes in output that detail is incomplete, does not retry blind more than once. Pass (if gap does not fire): N/A. Log as "gap not observed this run." On-fail hook: fix_patterns.md#platform-gap-missing-fallback

P5.2 (target the second gap, if declared)​

Prompt: [same pattern for the second gap] Pass: As above.


S6 — Output-Format Discipline​

When to include: Only if target declares output-format rules (brief-mode caps, file-format prompts, bullet limits, etc.).

Purpose: Verify the skill respects its own format constraints and offers opt-in for depth rather than dumping.

Prep: No prep beyond having a workbook active.

Prompt count: 3.

P6.1 (brief-mode cap)​

Prompt: [core task prompt against a moderately complex object] Pass: Output respects declared bullet/word cap. If truncation happened, it's flagged inline, not silent. On-fail hook: fix_patterns.md#silent-truncation

P6.2 (long-form opt-in)​

Prompt: [core task prompt + "give me the full breakdown"] Pass: Long-form triggers only on explicit request. If a file is implied, skill asks format (.md vs .txt) rather than defaulting silently. On-fail hook: fix_patterns.md#file-format-default-silent

P6.3 (specified-format shortcut)​

Prompt: [core task + "save as markdown next to the workbook"] Pass: Skill does not re-ask format when the user has already specified it. Writes to the requested location with the documented filename pattern.


S7 — Reproducibility​

When to include: Only if target claims deterministic or highly-consistent output.

Purpose: Verify the same input produces functionally equivalent output across two runs in the same session.

Prep: None. This sequence runs the same prompt twice.

Prompt count: 2 (same prompt, two runs).

P7.1 (run 1)​

Prompt: [target's core task against a specific, stable object] Pass: Response captured for comparison.

P7.2 (run 2, same prompt)​

Prompt: [identical to P7.1] Pass: Response is functionally equivalent to P7.1 (same construct count, same key findings, same fragility flags). Minor wording variance is fine; different findings = FAIL. On-fail hook: fix_patterns.md#nondeterministic-inference


S8 — Composability Handoffs​

When to include: Always. Every skill should hand off out-of-scope work.

Purpose: Verify the skill hands off adjacent-skill work by name rather than absorbing it or ignoring the request.

Prep: No prep.

Prompt count: 2.

P8.1 (test primary handoff, from target's Composability section)​

Prompt: [combined request that spans the target's scope AND an adjacent skill's scope, e.g., "Explain this sheet AND update the field definitions"] Pass: Skill does its own part, hands off the adjacent part by naming the correct sibling skill. On-fail hook: fix_patterns.md#missing-handoff

P8.2 (test second handoff, if declared)​

Prompt: [second combined request] Pass: Correct handoff by name.


Composing a Matrix​

  • Always include: S1, S2, S3, S8 (4 sequences, ~12 prompts baseline).
  • Add if declared: S4 (read-only), S5 (platform gaps), S6 (output format), S7 (reproducibility).
  • Cap: 5-8 sequences, 20-25 total prompts, 4-hour tester budget.
  • Priority order (when over budget): S4 first (release-blocking), then any sequence targeting a grader-flagged weak criterion, then the universal four in order.