AI-Powered Sherpa Feedback Processing Loop
1. Overview
When your Sherpa provides feedback on your planning specifications, the AI-powered feedback loop processes those suggestions in seconds - evaluating quality, surgically applying improvements, and showing before/after scores. You see exactly what changed, why it changed, and the measurable impact on spec quality before you approve. This eliminates hours of back-and-forth while maintaining your control over the final spec.2. Step-by-Step Guide
- Submit your feedback - On the Sherpa Planning Review page, enter your feedback in the text field (minimum 10 characters). Click Save. You’ll see a processing status showing estimated completion time of 15-20 seconds.
- Wait for AI processing - The system runs three phases automatically: evaluating your feedback for quality impact, surgically applying validated improvements, and re-scoring the enhanced spec. Real-time status updates appear on screen.
-
Review the results - When processing completes, the Feedback Iteration View displays your iteration number, processing time, cost, and token usage. Scroll through four key sections:
- AI Evaluation shows valid suggestions (green badges) with reasoning and estimated impact, plus invalid suggestions (orange badges) with reasoning and risks
- Quality Impact displays before/after grades (e.g., C 72% to B 85%), overall score delta, and breakdowns across five dimensions: story completeness, requirement testability, success criteria measurability, entity coverage, and scope clarity
- Changes Applied lists each modification with before/after text and AI reasoning
- Spec Comparison shows side-by-side markdown with color-coded diffs - green for additions, red for deletions
-
Make your decision - Three action buttons appear at the bottom:
- Approve Changes locks in the enhanced spec and updates your planning artifact
- Add More Feedback opens a new input field to submit additional suggestions, creating iteration 2
- Revert Changes restores the original spec if quality degraded or you disagree with changes
- Track iteration history - Access past iterations via the review page. Each iteration shows its number, status, score changes, and your decision. Approved iterations cannot be reverted. Reverted iterations cannot be approved.
3. Common Questions
Q: What happens if I submit feedback that isn’t really about quality improvement?A: The AI evaluation phase assesses every suggestion critically. If you submit something like “Nice work” with no actionable improvement, the system may return zero valid suggestions, apply no changes, and show an unchanged score. You’ll see this clearly in the AI Evaluation section. Q: Can I process multiple rounds of feedback at once?
A: No. The system prevents concurrent processing. If an iteration is already running (status PENDING or PROCESSING), you’ll get an error “Processing already in progress” when attempting to submit new feedback. Wait for the current iteration to complete, then submit your next round. Q: What if the enhanced spec has a worse score than the original?
A: The system detects quality degradation when the score drops by more than 5 points. You’ll see a red negative delta in the Quality Impact section and receive a warning recommending you revert the changes. The iteration still completes, but the warning helps you make an informed decision. Q: How much does each feedback iteration cost?
A: Processing cost appears in the iteration summary, calculated from Anthropic token usage (input tokens at 0.015 per 1,000). Typical iterations cost between 0.30 depending on spec length and feedback complexity. Q: What happens if the AI makes a mistake or misunderstands my feedback?
A: Review the Changes Applied section carefully. If you disagree with any modifications, click Revert Changes to restore the original spec. You can then submit revised feedback with clearer instructions to guide the next iteration.
4. Troubleshooting
Issue: System shows “Processing timeout” error after 60 secondsThe processing worker hit a timeout limit (60 seconds per phase). This usually happens with extremely large specs. The iteration is marked FAILED with details about which phase timed out. Contact support with your iteration ID if this happens repeatedly. Issue: Enhanced spec is missing section IDs or cross-references
This indicates a bug in the surgical enhancement phase. The system is designed to preserve all section IDs and cross-references. If you notice missing structural elements, revert the changes immediately and report the iteration ID to support. Do not approve broken specs. Issue: Cannot submit new feedback - button is disabled
Check if another iteration is currently processing (status shows PENDING or PROCESSING). Only one iteration runs at a time per review. Wait for completion, then submit new feedback. If no processing is active and the button remains disabled, refresh the page. Issue: Approved iteration still shows old spec content
After approving an iteration, the planning artifact should update with the enhanced spec immediately. If you still see old content, check the artifact’s updatedAt timestamp. If it didn’t change, the approval may have failed silently. Try refreshing and re-approving, or contact support if the issue persists.