RaDialog Copilot
Project Overview
RaDialog Copilot explores how confidence-aware NLP can support radiologists during report drafting. The system integrates calibrated confidence methods into a feedback loop that aligns AI suggestions with clinician input, making the drafting process both faster and more verifiable. Radiology report drafting needs assistive AI that is transparent and verifiable in real time.
Methods & Architecture
Confidence Estimation Methods
We compared logit-based entropy, sentence-embedding calibration (Duan), and entailment-based NLI approaches, balancing latency against calibration quality.
| Method | Family | Latency | Notes | Verdict |
|---|---|---|---|---|
| Simple Entropy | Logit‑based baseline | ~2s | Baseline calibrationLow semantic signal | Baseline only |
| Duan (med. ST) | Sentence embeddings (medical) | ~30s | Best ascending calibrationDomain‑trained | Most promising |
| Kuhn (NLI) | Entailment (DeBERTa‑MNLI) | ~9.5s | Bi‑directional entailmentNon‑linear < 0.6 | Inconsistent below 0.6 |
System Architecture
- React + Tailwind
- Rich editor with acceptance keybinds
- Opacity = confidence visual feedback
- Findings checklist, PDF export
- Flask + RaDialog integration
- Image preprocessing pipeline
- Confidence modules (entropy, embeddings, NLI)
- Caching + streaming for responsiveness
- Greedy base with controlled variance
- Calibration analysis (binning, metrics)
- Semantic rule‑based stopping
- Mode switch: fast vs. SOTA
Demo & Key Findings
Complete Workflow Demo
Unlike one-shot text assistants, RaDialog Copilot responds as the radiologist types, surfacing confidence-weighted completions that can be accepted with a keystroke or ignored without disruption.
Complete RaDialog Copilot Interface
- • Duan method yields best monotonic calibration curves
- • Temperature ↑ improves calibration alignment
- • Domain‑trained encoders outperform general models
- • 0.7 confidence cleanly separates relevant from noise
- • Evaluation framework for confidence in clinical NLP
- • Practical confidence‑aware copilot UI for reporting
- • Evidence for domain‑trained encoders in calibration
Analysis & Impact
- Generic text assistant.
- Long completions requiring pruning.
- No confidence cues; manual edits heavy.
- Confidence‑weighted inline suggestions.
- Early‑stop at clause boundary.
- Type‑constrained snippets; faster acceptance.
These gains translate into smoother workflows: fewer edits, faster acceptance, and reduced frustration with over-confident AI text.
Key Design Decisions
| Decision | Rationale | Trade‑off |
|---|---|---|
| Fast vs SOTA modes | 2s for UX; 20s when high certainty needed | Speed vs calibration quality |
| 0.7 threshold | Balances acceptance and precision | May hide rare low‑confidence finds |
| Hybrid early‑stop | Avoids jitter and run‑on sentences | Occasional clipped clause edge cases |