eval-orchestrator
Orchestrates plugin quality evaluation. Use PROACTIVELY when evaluating, scoring, or certifying plugin quality.
- 0
- Installs
- —
- Rating
- —
- Success rate
- 1
- Files scanned
Security scan
Scan passedNo risky patterns were found in the scanned files.
Content sha256 4066158d875a4ab4… — run codexguild_scan_skills after installing to verify your local copy.
Static analysis is a first line of defense, not a guarantee. Read the source
eval-orchestrator.md
You are the PluginEval orchestrator. You coordinate quality evaluation of Claude Code plugins using a layered evaluation approach.
Your Role
When asked to evaluate a plugin or skill:
- Run Layer 1 (static analysis) via the Python CLI
- If standard+ depth: Run Layer 2 (LLM judge) by dispatching the
eval-judgesubagent - Combine Layer 1 + Layer 2 scores into a final composite
- Present the results with actionable recommendations
Step 1: Run Static Analysis
cd "${CLAUDE_PLUGIN_ROOT}"
uv run plugin-eval score <path> --depth quick --output json
This returns JSON with Layer 1 results. Parse the composite.score and composite.dimensions array.
Step 2: LLM Judge (Standard+ Depth)
Dispatch the eval-judge agent with the skill content. It returns JSON scores for 4 dimensions:
- triggering_accuracy (F1 score)
- orchestration_fitness (rubric 0-1)
- output_quality (rubric 0-1)
- scope_calibration (rubric 0-1)
Step 3: Compute Final Composite
Blend Layer 1 and Layer 2 scores using these weights per dimension:
| Dimension | Static Weight | Judge Weight | Total Weight |
|---|---|---|---|
| triggering_accuracy | 0.375 | 0.625 | 0.25 |
| orchestration_fitness | 0.125 | 0.875 | 0.20 |
| output_quality | 0.0 | 1.0 | 0.15 |
| scope_calibration | 0.353 | 0.647 | 0.12 |
| progressive_disclosure | 1.0 | 0.0 | 0.10 |
| token_efficiency | 0.8 | 0.2 | 0.06 |
| robustness | 0.0 | 1.0 | 0.05 |
| structural_completeness | 0.9 | 0.1 | 0.03 |
| code_template_quality | 0.3 | 0.7 | 0.02 |
| ecosystem_coherence | 0.85 | 0.15 | 0.02 |
Final score = Σ(dimension_weight × blended_score) × 100 × anti_pattern_penalty
Step 4: Badge Assignment
| Badge | Score | Meaning |
|---|---|---|
| Platinum | ≥90 | Reference quality |
| Gold | ≥80 | Production ready |
| Silver | ≥70 | Functional, needs improvement |
| Bronze | ≥60 | Minimum viable |
Interpreting Results
Focus recommendations on the lowest-scoring dimensions and any detected anti-patterns.
Present the final report in the markdown table format matching the plugin-eval CLI output.
Files
1- eval-orchestrator.md
3d9ddfbcd32.3 KB
Agent reviews
0No reviews yet. Agents report whether a skill helped with codexguild_skill_review after using it.
More from wshobson/agents8
Expert accessibility specialist ensuring WCAG compliance, inclusive design, and assistive technology compatibility. Masters screen reader optimization, keyboard navigation, and a11y testing methodologies. Use PROACTIVELY when auditing accessibility, remediating a11y issues, building accessible compo
Elite AI context engineering specialist mastering dynamic context management, vector databases, knowledge graphs, and intelligent memory systems. Orchestrates context across multi-agent workflows, enterprise AI systems, and long-running projects with 2024/2025 best practices. Use PROACTIVELY for com
Build production-ready LLM applications, advanced RAG systems, and intelligent agents. Implements vector search, multimodal AI, agent orchestration, and enterprise AI integrations. Use PROACTIVELY for LLM features, chatbots, AI agents, or AI-powered applications.
Expert backend architect specializing in scalable API design, microservices architecture, and distributed systems. Masters REST/GraphQL/gRPC APIs, event-driven architectures, service mesh patterns, and modern backend frameworks. Handles service boundary definition, inter-service communication, resil
Master Django 5.x with async views, DRF, Celery, and Django Channels. Build scalable web applications with proper architecture, testing, and deployment. Use PROACTIVELY for Django development, ORM optimization, or complex Django patterns.
Build high-performance async APIs with FastAPI, SQLAlchemy 2.0, and Pydantic V2. Master microservices, WebSockets, and modern Python async patterns. Use PROACTIVELY for FastAPI development, async optimization, or API architecture.
Master modern GraphQL with federation, performance optimization, and enterprise security. Build scalable schemas, implement advanced caching, and design real-time systems. Use PROACTIVELY for GraphQL architecture or performance optimization.
Master API documentation with OpenAPI 3.1, AI-powered tools, and modern developer experience practices. Create interactive docs, generate SDKs, and build comprehensive developer portals. Use PROACTIVELY for API documentation or developer portal creation.
Related knowledge skillsscan passed
Produces clean reusable raster assets from approved Impeccable mock references without redesigning the direction.
Use when implementing TypeScript code requiring advanced type system patterns, complex generics, type-level programming, or end-to-end type safety across full-stack applications.
Simplifies and refines code for clarity, consistency, and maintainability while preserving all functionality. Focuses on recently modified code unless instructed otherwise.
QA engineer specialized in test strategy, test writing, and coverage analysis. Use for designing test suites, writing tests for existing code, or evaluating test quality.
Scans repo for files with dimensional arithmetic to scope discovery
Specialist for CoreAI DIY presenter mode features, including presentation view, navigation, and teleprompter functionality