AI Model Comparison Framework

Act as an AI model evaluation specialist responsible for benchmarking large language models (GPT-4, Claude 3.5, Gemini 1.5, Llama 3) across accuracy, reasoning, creativity, safety, and cost dimensions for enterprise AI selection. Generate a complete model comparison framework for a specific use case (coding, writing, analysis, translation, summarization, classification, generation) including test prompts, evaluation criteria, scoring methodology, and cost-benefit analysis. Begin with use case specification including primary task type, required output format, latency requirements (real-time, batch, overnight), volume estimates (requests per day/month), accuracy requirements (acceptable error rate), and budget constraints (per-request, monthly). Develop test prompt suite including 10-20 standardized prompts covering edge cases (long context, formatting requirements, multi-turn conversations, ambiguous instructions, contradictory information, adversarial inputs, out-of-domain requests, multilingual tasks, numerical reasoning, common sense reasoning), golden answers for each prompt (ideally curated from human experts), and difficulty classification (basic, intermediate, advanced). Create evaluation criteria including accuracy subdimensions (factual correctness 0-5, instruction following 0-5, precision of claims 0-5, completeness 0-5), reasoning (logical coherence 0-5, step-by-step explanation quality 0-5, counterfactual handling 0-5), creativity (novel solutions 0-5, appropriate tone 0-5, engagement 0-5), safety (harmful content avoidance 0-5, bias detection 0-5, privacy protection 0-5), and format (structure quality 0-5, readability 0-5, specified format compliance 0-5). Implement scoring methodology including blinded evaluation (evaluators don't know which model produced output), rubric-based scoring with detailed anchors, inter-rater reliability calculation (minimum 0.7 Kappa), statistical significance testing (paired t-test, Wilcoxon signed-rank), confidence intervals for each score, and composite score weighting by dimension importance. Add cost measurement including API pricing per 1K input/output tokens, monthly projected spend at volume, latency measurements (time-to-first-token, total generation time), token efficiency (prompt length optimization, output conciseness), and feature-based pricing differences (fine-tuning, batch API, streaming). Provide decision matrix including minimum acceptable scores per dimension, trade-off analysis (higher cost for higher accuracy), implementation complexity by model, data privacy considerations (data retention, training opt-out), and vendor lock-in risks. Include recommendation framework with primary model selection, fallback model for edge cases, caching strategy for repeated prompts, and fine-tuning viability assessment for domain-specific improvements.