Tech Stack
AI Evaluation Pipeline
1Written Response Evaluation
Candidate answers are evaluated against employer-defined criteria using Claude. Each criterion receives a 1-5 score with reasoning. The system is calibrated to use the full scoring range and avoid artificial strictness.
2URL Verification
Demo URLs are fetched and verified for accessibility. Page content is extracted and analyzed for relevance to the challenge requirements.
3Video Demo Analysis
Videos are processed on Railway (FFmpeg). Frames are extracted at regular intervals and audio is transcribed using Whisper. Claude Vision analyzes the visual demonstration alongside the transcript to evaluate:
- Does it work? — Functioning solution vs. mockups
- Do they understand it? — Technical explanation quality
- Can they communicate it? — Presentation clarity
4Triage & Drafts
Candidates are automatically triaged into Priority, Maybe, or Skip based on combined scores. The system generates interview invitation drafts for top candidates and rejection drafts for others.
System Architecture
┌─────────────────────────────────────────────────────────────────────┐ │ CANDIDATE │ │ │ │ │ Submit Application │ │ ▼ │ ├─────────────────────────────────────────────────────────────────────┤ │ │ │ ┌──────────────┐ ┌──────────────┐ ┌──────────────────────┐ │ │ │ Vercel │ │ Supabase │ │ UploadThing │ │ │ │ (Next.js) │◄──►│ (PostgreSQL) │ │ (Video Storage) │ │ │ └──────┬───────┘ └──────────────┘ └──────────┬───────────┘ │ │ │ │ │ │ │ Trigger Evaluation │ │ │ ▼ │ │ │ ┌──────────────┐ │ │ │ │ Claude │ ◄── Written answers │ │ │ │ (Anthropic) │ + URL content │ │ │ └──────────────┘ │ │ │ │ │ │ │ Trigger Video Eval │ │ │ ▼ ▼ │ │ ┌──────────────────────────────────────────────────────────────┐ │ │ │ Railway │ │ │ │ ┌─────────┐ ┌─────────┐ ┌─────────┐ ┌─────────┐ │ │ │ │ │ FFmpeg │───►│ Whisper │───►│ Claude │───►│Supabase │ │ │ │ │ │ (Frames)│ │ (Audio) │ │ (Vision)│ │ (Save) │ │ │ │ │ └─────────┘ └─────────┘ └─────────┘ └─────────┘ │ │ │ └──────────────────────────────────────────────────────────────┘ │ │ │ ├─────────────────────────────────────────────────────────────────────┤ │ EMPLOYER │ │ │ │ │ View Dashboard │ │ ▼ │ │ Triaged Candidates │ │ (Priority / Maybe / Skip) │ └─────────────────────────────────────────────────────────────────────┘
Key Design Decisions
Split Processing Architecture
Text evaluation runs on Vercel (fast, serverless), while video processing runs on Railway (requires FFmpeg, longer runtime). This optimizes for both speed and capability.
Frame-Based Video Analysis
Rather than processing full video, we extract frames at regular intervals and combine with audio transcription. This enables Claude Vision to analyze demos efficiently while keeping costs manageable.
Calibrated Scoring
The evaluation prompts are carefully calibrated to use the full 1-5 scoring range and avoid artificial strictness. A 5 means "thoughtful and specific" — not perfection.
Built by Zack Dorward