第三課：先寫判準再送出Write Your Criteria Before You Submit

為什麼這件事會出錯Why this goes wrong

askLLM 的 LIMITATIONS.zh-TW.md 用一個示意範例示警：有人拿 iris 資料集問問題， 卻沒有勾選「Attach data summary to prompt」——結果 AI 完全是在憑空回答， 憑空生出「8 個樣本，分成 Saguaro／Palo Verde／Ironwood 三個物種」這種與 iris 資料八竿子打不 著的內容（iris 其實有 150 筆、三個物種是 setosa/versicolor/virginica）。如果 使用者送出前沒有先想好「什麼樣的回覆我才會接受」，看到一段語氣篤定、格式工整 的答案，很容易就直接信了。判準要在收到回覆之前寫好，不然人會被回覆本身 的自信說服，失去自己原本的判斷基準。

askLLM’s LIMITATIONS.zh-TW.md illustrates the risk with a made-up example: someone asked a question about the iris dataset without ticking “Attach data summary to prompt.” The AI then answered from pure guesswork, inventing “8 samples across Saguaro / Palo Verde / Ironwood.” The real iris data holds 150 rows across three species, setosa, versicolor, and virginica. Decide in advance what answer you would actually accept. Without that decision, a confident, well-formatted reply is easy to believe at face value. Write your criteria before the reply arrives, or the reply’s own confidence will overwrite your judgment.

判準長什麼樣子What a criterion looks like

一條好判準要具體到「看得出來有沒有命中」，不能只是「答得好」這種空話。 verify/checklist-guider.qmd 給的範本是：

回覆必須指名一個 T-Tests 底下的分析，並說明為何不是 ANOVA。

這句話同時做了兩件事：界定「哪個分析家族」（T-Tests，而非隨便一個統計名詞）， 以及要求「說明理由」（不是只給名稱）。用 iris 的例子練習，判準可以寫成：

回覆必須提到 iris 的物種變項有三個水準，若建議做組間比較，必須指名一個 ANOVA 而非兩組 t 檢定；若引用樣本數或物種名稱，必須與我勾選變項後看到的 摘要一致。

這條判準把「LIMITATIONS 記錄過的失效模式」（物種數量、樣本數被編造）直接寫 進了驗收條件裡，回覆一送到就能對著這幾點逐條檢查。

A good criterion is specific enough for you to tell, on sight, whether it was met. “A good answer” is too vague to qualify. The template in verify/checklist-guider.qmd reads:

The reply must name a specific analysis under T-Tests and explain why it is not an ANOVA.

That sentence does two things. It pins down which analysis family is expected, namely T-Tests rather than any statistical term at all. It also demands a reason, not just a label. Practicing with the iris example, a criterion could read:

The reply must recognize that iris’s species variable has three levels; if it recommends a group comparison, it must name an ANOVA rather than a two-group t-test; and any sample size or species name it cites must match the summary I see after selecting my variables.

That criterion writes the exact failure mode documented in LIMITATIONS, including invented species counts and sample sizes, straight into the acceptance condition. The moment a reply arrives, you have specific points to check it against.

判準怎麼接上查核流程How criteria feed the verification workflow

先寫判準這件事，在 verify/workflow.qmd 的流程裡 叫 P0（送出前寫判準），對應的是 README 提到的「設定問題獲得解決或需要 再探討的明確指標」。收到回覆後，拿判準逐條核對的步驟叫 C1（結案時對照 判準），結果只有兩種：solved（全部命中，依建議執行）或 reopen（任一項沒過， 要換個問法或換個 persona 再試）。P0 與 C1 是一組：沒有事先寫好的判準，C1 就沒有東西可以對照，只能憑感覺覺得「這回覆聽起來不錯」——而 iris 的例子已 經證明，聽起來不錯不等於是真的。

Writing criteria first is step P0 in the verify/workflow.qmd process, meaning you write your criteria before you submit. P0 answers the README’s call for “a clear indicator of whether the problem is solved or needs further exploration.” Checking the reply against those criteria once it arrives is step C1, the comparison you run when closing out. C1 has exactly two outcomes. Everything checks out, so the case is solved and you proceed with the recommendation. Or something failed, so you reopen it and either rephrase the question or try a different persona. P0 and C1 are a pair. Without criteria written in advance, C1 has nothing to check against, and you are left judging by how convincing the reply sounds. The iris example already shows that “sounds convincing” is not the same as “is correct.”

動手做（5 分鐘）Try it (5 minutes)

  1. 在送出問題之前，先在紙上或筆記軟體寫一句判準，格式仿照：「回覆必須 指名 __ 底下的分析，並說明為何不是 __」。
  2. 把你打算問的問題，連同勾選的變項，送進 askLLM 的 jamovi Module Guider。 這次刻意取消勾選「Attach data summary to prompt」一次，觀察回覆是 否出現你意料之外、與你資料完全無關的數字或名稱（對照 LIMITATIONS 的 iris 示意範例）。
  3. 重新勾選「Attach data summary to prompt」，再送出同一個問題。
  4. 拿出你在步驟 1 寫的判準，逐條核對這次的回覆：命中幾條？哪一條沒過？
  5. 依 C1 的兩種結果做結論：全部命中 → solved，依建議執行；任一項沒過 → reopen，記下要換的問法或 persona，留到下一課的協作日誌裡。
  1. Before you submit anything, write one criterion on paper or in a note, following the pattern: “The reply must name an analysis under and explain why it is not .”
  2. Submit your question, with your selected variables, to askLLM’s jamovi Module Guider. This time, deliberately untick “Attach data summary to prompt” once, and watch for numbers or names with no connection to your actual data. Compare against the iris illustration in LIMITATIONS.
  3. Re-tick “Attach data summary to prompt” and submit the same question again.
  4. Take the criterion you wrote in step 1 and check this reply against it, point by point: how many did it meet? Which one failed?
  5. Conclude using the two C1 outcomes: everything met → solved, proceed with the recommendation; anything failed → reopen, and note what question or persona to try next. You will use this in the collaboration log in a later lesson.

帶走這句話One thing to remember

判準要在回覆出現之前寫好，因為回覆一旦出現，它的自信語氣會讓你忘記自己原本 想核對什麼。

Write your criteria before the reply exists. Its confident tone will make you forget what you meant to check.