Best Practices

Best Practices

1. Configuration Management

✅ DO:

• Use descriptive names: config/gpt4_vs_claude_alignment.yaml
• Include version in benchmark config for tracking
• Add comments in YAML explaining parameter choices
• Keep separate configs for experiments vs. production

❌ DON'T:

• Overwrite configs without versioning
• Use generic names like config1.yaml
• Hard-code parameters in source code

2. Reproducibility

✅ DO:

• Set temperature to 0.0 for deterministic results
• Use at least 3-5 runs for reliable statistics
• Save all configurations in version control
• Specify exact model names (e.g., gpt-4-0613, not just gpt-4)
• Record timestamps and config hashes in results

❌ DON'T:

• Use temperature > 0 without documenting why
• Run benchmarks only once
• Delete old result files
• Use "latest" model versions without recording specifics

3. Cost Management

✅ DO:

• Start with 1-2 questions for testing
• Use cheaper models (gpt-3.5-turbo) for development
• Monitor API usage dashboards
• Cache results to avoid re-running
• Set max_tokens appropriately for each metric

❌ DON'T:

• Run full benchmarks during development
• Use GPT-4 for all testing
• Ignore rate limits
• Re-run unnecessarily

4. Data Quality

✅ DO:

• Validate JSON schemas before running
• Ensure source_material paths exist
• Review questions for well-formedness
• Test with small dataset first
• Include learning objectives when available

❌ DON'T:

• Skip validation steps
• Assume JSON is well-formed
• Run benchmarks on untested data

5. Metric Selection

✅ DO:

• Choose metrics relevant to your research question
• Start with core metrics (alignment, clarity, cognitive level)
• Create domain-specific metrics for specialized content
• Test metric prompts manually before automation
• Document metric rationale in config comments

❌ DON'T:

• Enable all metrics without purpose
• Use metrics inappropriate for question type
• Deploy untested custom metrics

6. Source Material Management

For single-file source materials:

✅ DO:

• Use clear Markdown formatting with proper headers
• Include all content referenced in questions
• Keep PDFs as text-based (not scanned images)
• Verify file encoding is UTF-8

For multi-file source materials (folders):

✅ DO:

• Group related content in a single folder
• Name files with numeric prefixes for consistent ordering: 01_, 02_, etc.
• Include a mix of formats where appropriate (PDFs for textbooks, Markdown for notes)
• Test with 2-3 files before adding many
• Document topic coverage in folder name (e.g., module_1_basics/)

❌ DON'T:

• Mix hundreds of files in a single folder
• Use unclear file names without descriptive labels
• Include unrelated materials in the source folder
• Rely on alphabetical ordering without explicit naming
• Use scanned/image-based PDFs (system can't extract text)

Performance Tip: Multiple small files (5-20 files) generally perform better than one massive file, as LLMs handle structured, chunked material more effectively.

7. Result Interpretation

✅ DO:

• Look at trends across multiple quizzes
• Check for consistency (low std dev = reliable)
• Cross-validate with multiple evaluators
• Review outliers and raw LLM responses
• Consider context (domain, audience, objectives)

❌ DON'T:

• Over-interpret single data points
• Ignore high variance warnings
• Trust one evaluator blindly
• Compare scores across different metrics

---