Skip to main content

Best Practices

Best Practices​

1. Configuration Management​

✅ DO:

  • Use descriptive names: config/gpt4_vs_claude_alignment.yaml
  • Include version in benchmark config for tracking
  • Add comments in YAML explaining parameter choices
  • Keep separate configs for experiments vs. production

❌ DON'T:

  • Overwrite configs without versioning
  • Use generic names like config1.yaml
  • Hard-code parameters in source code

2. Reproducibility​

✅ DO:

  • Set temperature to 0.0 for deterministic results
  • Use at least 3-5 runs for reliable statistics
  • Save all configurations in version control
  • Specify exact model names (e.g., gpt-4-0613, not just gpt-4)
  • Record timestamps and config hashes in results

❌ DON'T:

  • Use temperature > 0 without documenting why
  • Run benchmarks only once
  • Delete old result files
  • Use "latest" model versions without recording specifics

3. Cost Management​

✅ DO:

  • Start with 1-2 questions for testing
  • Use cheaper models (gpt-3.5-turbo) for development
  • Monitor API usage dashboards
  • Cache results to avoid re-running
  • Set max_tokens appropriately for each metric

❌ DON'T:

  • Run full benchmarks during development
  • Use GPT-4 for all testing
  • Ignore rate limits
  • Re-run unnecessarily

4. Data Quality​

✅ DO:

  • Validate JSON schemas before running
  • Ensure source_material paths exist
  • Review questions for well-formedness
  • Test with small dataset first
  • Include learning objectives when available

❌ DON'T:

  • Skip validation steps
  • Assume JSON is well-formed
  • Run benchmarks on untested data

5. Metric Selection​

✅ DO:

  • Choose metrics relevant to your research question
  • Start with core metrics (alignment, clarity, cognitive level)
  • Create domain-specific metrics for specialized content
  • Test metric prompts manually before automation
  • Document metric rationale in config comments

❌ DON'T:

  • Enable all metrics without purpose
  • Use metrics inappropriate for question type
  • Deploy untested custom metrics

6. Result Interpretation​

✅ DO:

  • Look at trends across multiple quizzes
  • Check for consistency (low std dev = reliable)
  • Cross-validate with multiple evaluators
  • Review outliers and raw LLM responses
  • Consider context (domain, audience, objectives)

❌ DON'T:

  • Over-interpret single data points
  • Ignore high variance warnings
  • Trust one evaluator blindly
  • Compare scores across different metrics