P Luthfi's Playground

Experiment · 2026-11-02

Few-shot vs zero-shot for review summarization

Tiga contoh few-shot menaikkan accuracy 6.2 poin dan menurunkan latency 14%, tanpa menambah biaya berarti.

Done prompting eval claude-sonnet-5-5 LA
TL;DR

  1. 3-shot prompt: 91.4% accuracy vs 85.2% baseline (+6.2 pt).

  2. Output lebih pendek dan konsisten, jadi p50 latency turun dari 950 ms ke 820 ms.

  3. Chain-of-thought gagal (timeout) di input > 4k token. Tidak direkomendasikan untuk batch.

Question

Apakah contoh few-shot cukup untuk menggantikan fine-tuning di task ringkasan review?

H1

Few-shot improves accuracy by ≥ 5 pt.

Supported +6.2 pt
H2

Chain-of-thought beats few-shot.

Rejected timeout
H3

Few-shot matches fine-tuned model.

Inconclusive 1.7 pt gap, 1 seed

Setup

Semua variant memakai dataset, model, dan seed yang sama.

Model
claude-sonnet-5-5
Temperature
0.2
Samples
1,200
Seed
42
Dataset
data/reviews-v3.jsonl
Metric
normalized EM

How to reproduce

  1. Install
    pip install -r requirements.txt
  2. Run all variants
    python run.py --all --n 1200
  3. Results

    Written to results.csv and run.log.

Prompt change

prompts/system.txt
Summarize the review. Be concise. Answer in one sentence. Examples: {{3 examples from examples.json}}

Results

n = 1,200 per variant.

Best accuracy 91.4% 6.2 pt
p50 latency 820 ms 14% faster
Cost / 1k runs $3.10 0%
Variant Accuracy p50 latency Cost / 1k
Zero-shot (baseline)85.2%950 ms$3.10
3-shot91.4%820 ms$3.10
Fine-tuned93.1%700 ms$4.80
Chain-of-thought—timeout—
 Tabel 1. Fine-tuned lebih akurat tapi 55% lebih mahal. 3-shot dipilih.

Accuracy by variant

  • Zero-shot
    85.2%
  • 3-shot
    91.4%
  • Fine-tuned
    93.1%

3-shot output breakdown

  • Correct 91%
  • Partial 6%
  • Wrong 3%

Sample output

Zero-shot
The reviewer says the app is calming, but they have also noticed that the timer sometimes resets on its own.
out 24 tok980 msToo long
3-shot
Calming app, but the timer resets randomly.
out 9 tok610 msCorrect
Run log excerpt (run.log)
INFOLoaded 1200 samples
WARNRate limited, retrying in 2s
ERRORCoT sample 77: timeout after 120s
INFODone. 3-shot accuracy=0.914

Conclusion

Few-shot adalah default terbaik untuk task ini: hampir menyamai fine-tuned model (−1.7 pt) dengan biaya 35% lebih rendah dan tanpa pipeline training.

Hasil H3 belum bisa disimpulkan karena hanya satu seed. Perlu diulang dengan 3 seed sebelum keputusan final.

Limitations

Satu dataset, bahasa Inggris saja, satu seed.

Next steps

  • Pilih 3-shot sebagai default
  • Ulangi dengan 3 seed
  • Uji dataset bahasa Indonesia

Files

File mentah di folder ini. Jangan diedit setelah run.

results.csv

4 variants × 1,200 rows · 310 KB

run.log

Full run log · 2.1 MB

run.py

Eval script

examples.json

3 few-shot examples