name: skill-studio
description: "Comprehensive skill evaluation and debugging framework for testing agent skills. Use when users need to (1) evaluate a skill's recall rate (how often it triggers correctly), (2) test a skill's accuracy against expected outputs, (3) analyze skill performance with various prompts, or (4) generate improvement recommendations for existing skills. Requires test cases with standard answers."

Skill Studio

A systematic framework for evaluating and debugging agent skills through automated testing of recall rates and output accuracy.

Overview

Skill Studio provides tools and workflows to:
- Test if a skill triggers correctly (recall rate)
- Verify skill outputs match expected results (accuracy rate)
- Evaluate code outputs with specialized metrics (syntax, structure, functionality, quality)
- Generate actionable improvement recommendations
- Track performance across iterations

Core Workflow

Step 1: Collect Test Requirements

Gather from the user:
- Target skill name: The skill being evaluated (e.g., "pptx", "docx", "custom-skill")
- Test cases file: Path to test prompts (or help create one)
- Standard answers file: Path to expected outputs (or help create one)
- Test parameters: Number of prompt variations, testing approach

Step 2: Prepare Test Cases

If test cases don't exist, help create test_cases.json using the format in references/test_format.md.

Key elements:
- Diverse prompts that should trigger the skill
- Negative cases (prompts that should NOT trigger)
- Category labels for analysis
- Expected behaviors

Step 3: Run Recall Testing

Execute recall rate evaluation:

python3 scripts/test_recall.py \
  --skill-name <skill-name> \
  --test-cases <path-to-test-cases.json> \
  --variations 5 \
  --output outputs/recall_results.json

What it does:
- Generates prompt variations for each test case
- Tests if the skill triggers with each variation
- Calculates recall rates overall and by category
- Identifies false positives and false negatives

Key metrics:
- Overall recall rate: % of prompts that correctly triggered
- Per-category recall: Performance by test case type
- False positive rate: Incorrect triggering

Step 4: Run Accuracy Testing

Execute output accuracy evaluation:

python3 scripts/test_accuracy.py \
  --skill-name <skill-name> \
  --test-cases <path-to-test-cases.json> \
  --standard-answers <path-to-standard-answers.json> \
  --workspace <path-to-workspace-directory> \
  --artifacts-dir test_artifacts \
  --output outputs/accuracy_results.json

What it does:
- Runs the skill with test prompts
- Captures file changes (for coding agent skills)
- Compares outputs to standard answers
- Evaluates both exact matches and semantic similarity
- Saves before/after states and diffs
- Categorizes errors by type

Key parameters:
- --workspace: Optional. Path to workspace directory for capturing code changes
- --artifacts-dir: Optional. Directory to save test artifacts (default: test_artifacts)

Key metrics:
- Exact match accuracy: % of perfect matches
- Semantic similarity: Average similarity score (0-1)
- Error categories: Common failure patterns

Artifacts saved (when --workspace is provided):

For each test case, the following artifacts are saved in test_artifacts/<test_id>/:
- metadata.json: Test case information
- before_state.json: File states before skill execution
- after_state.json: File states after skill execution
- changes.json: Summary of added/modified/deleted files
- diffs/<file>.diff: Unified diffs for each modified file
- full_output.json: Complete skill response

This allows you to:
- Review exact code changes made by the skill
- Compare actual changes to expected changes
- Debug why tests failed
- Track how the skill evolves over iterations

For code outputs:
- Syntax validity: Does code parse correctly?
- Structure match: Has expected functions/classes/imports?
- Functionality: Passes test cases?
- Code quality: Comments, docstrings, error handling, type hints

Step 5: Analyze and Generate Recommendations

Run the analysis to get improvement suggestions:

python3 scripts/analyze_results.py \
  --recall-results outputs/recall_results.json \
  --accuracy-results outputs/accuracy_results.json \
  --skill-path <path-to-skill-folder> \
  --output outputs/recommendations.md

Output includes:
- Performance summary with visual indicators
- Specific issues identified (low-performing categories)
- Concrete recommendations for SKILL.md improvements
- Suggested trigger phrases for description
- Priority-ranked action items

Step 6: Implement Improvements

Based on recommendations:
1. Review the generated recommendations document
2. Update the skill's SKILL.md frontmatter description
3. Enhance instructions in SKILL.md body
4. Add clarifying examples or decision trees
5. Re-run tests to validate improvements

Iterate until target performance is achieved.

Interpreting Results

Recall Rate Thresholds

• >90%: Excellent - skill triggers reliably
• 70-90%: Good - minor description improvements helpful
• 50-70%: Fair - description needs refinement
• <50%: Poor - major triggering issues, likely unclear description

Accuracy Rate Thresholds

• >85%: Excellent - outputs are high quality
• 70-85%: Good - minor instruction improvements helpful
• 55-70%: Fair - instructions need clarification
• <55%: Poor - major workflow or instruction issues

Common Issues and Solutions

Low Recall Rate:
- Description is too vague or generic
- Missing key trigger keywords
- Overlaps with other skills' descriptions
- Solution: Add specific file types, actions, or scenarios to description

High False Positive Rate:
- Description is too broad
- Includes common generic terms
- Solution: Be more specific about exact use cases

Low Accuracy Rate:
- Instructions are unclear or incomplete
- Missing critical steps in workflow
- Examples don't match user expectations
- Solution: Add decision trees, more examples, clearer step-by-step guidance

Variable Performance by Category:
- Skill handles some use cases well but not others
- Solution: Add specific guidance for underperforming categories

Best Practices

Test Case Design

• Include 20-30 diverse test cases minimum
• Cover all major use cases the skill should handle
• Add negative cases (should NOT trigger) at ~20% ratio
• Use realistic, varied phrasing
• Label test cases by category for granular analysis

Standard Answers

• Define clear quality criteria
• Use semantic similarity for flexible matching when appropriate
• Include both structure and content expectations
• Specify file formats and key elements

Iterative Testing

• Test after each modification to track progress
• Focus on lowest-performing categories first
• Keep a log of changes and their impact
• Aim for incremental improvements (5-10% gains per iteration)

Skill Description Optimization

• Include specific file extensions (e.g., ".docx", ".pptx")
• List concrete actions (e.g., "creating", "editing", "analyzing")
• Mention key scenarios (e.g., "when user uploads", "when user requests")
• Use numbered lists for multiple trigger conditions
• Avoid generic terms that apply to many skills

Resources

scripts/

• test_recall.py: Measures skill triggering consistency
• test_accuracy.py: Evaluates output quality
• analyze_results.py: Generates improvement recommendations
• generate_variations.py: Creates prompt variations for testing

references/

• test_format.md: Detailed test case format specification
• answer_format.md: Standard answer format specification
• code_evaluation.md: Code output evaluation format and best practices
• metrics_explained.md: Deep dive into evaluation metrics