第一課：LLM 看得到什麼、看不到什麼What the LLM Sees, and What It Doesn’t

為什麼這件事會出錯Why this goes wrong

很多使用者會用askLLM直接問AI：「這筆資料有沒有離群值？」，然後得到語氣篤定的回覆：「我檢查過資料，第47 筆的數值明顯偏高，建議排除。」這樣聽起來很專業，但是這句話從頭到尾是AI編造的。askLLM從來沒有把第 47筆、或任何一筆原始資料，傳給 AI。預設狀況下，AI收到的只是你勾選變項的摘要統計，它連資料有幾欄都不一定看得完整，更不可能指認「第幾筆」。這種錯誤之所以危險，是因為肯定的語氣猶如宣稱答案正確。

Many users ask the AI directly through askLLM: “Does this dataset have outliers?” The reply comes back in a confident tone: “I checked the data. Row 47 is clearly high, so I would exclude it.” That sounds expert, but the AI fabricated the whole answer. askLLM never sends row 47, or any other raw row, to the AI. By default the AI receives only summary statistics for the variables you ticked. It may not even see how many columns your data has, and it certainly cannot name a specific row. This kind of error is dangerous because the confident tone itself claims the answer is correct.

這個對話是示意範例，不是實際記錄。母專案的 askLLM/docs/LIMITATIONS.zh-TW.md 用同一類示意來示警：未勾選 Attach data summary 時，AI 可能憑空生出「8 個樣本， 分成 Saguaro／Palo Verde／Ironwood 三個物種」這種與手上資料毫無關係的內容。AI 沒有資料時不會說「我沒有資料」，它會 編一份出來。

That exchange is illustrative, not a transcript. The parent project warns about the same thing in the same way. askLLM/docs/LIMITATIONS.zh-TW.md gives its own illustration: with the data summary switched off, an AI can invent “8 samples across Saguaro / Palo Verde / Ironwood” for a dataset containing nothing of the kind. Given no data, the AI does not say it has no data. It makes some up.

askLLM 實際送出的內容What askLLM actually sends

askLLM 的資料摘要函式 summarize_data() 為每個變項產生固定格式的區塊。連續 變項（數值型）長這樣：

summarize_data(), askLLM’s data-summarizer, produces a fixed-format block per variable. A continuous (numeric) variable looks like this:

age [numeric]:
  n: 118, missing: 2
  mean: 34.6, sd: 9.82, median: 33, min: 19, max: 61

類別變項（factor）長這樣：

A categorical (factor) variable looks like this:

group [factor]:
  n: 120, missing: 0, 2 levels
  levels by count: control(61), treatment(59)

就這樣。唯一的例外是有序類別變項（ordered factor）會多一行 level order: 低 < 中 < 高，說明水準的順序。除此之外沒有別的了——AI 收到的資訊，跟你在這裡讀到的一模一樣，不多也不少。 勾選「Attach data summary to prompt」時才會送出這段摘要；若取消勾選，AI 完全 是在憑空猜測。

That is all. The one exception is an ordered factor, which adds one more line, level order: low < medium < high, to show how the levels are ordered. Beyond that there is nothing — the AI receives exactly what you just read, nothing more. This block is attached only when “Attach data summary to prompt” is ticked; untick it and the AI is guessing with no data at all.

摘要看不到什麼What the summary cannot show

摘要只有連續變項如n、missing data、mean、sd、median、min、max，與類別變項像是各水準次數。整欄原始資料不會在摘要裡面，所以以下這些事，AI 原則上不可能真的檢查過，即使它宣稱檢查過：

  • 哪一筆資料是離群值（只知道 min 和 max，不知道哪一列）
  • 資料是否常態分布（min/max/median 不足以判斷分布形狀）
  • 遺漏值集中在哪些組別（只知道遺漏「總數」，不知道遺漏的樣式）
  • 變項背後的實際測量方式或研究設計

看到AI回覆聲稱做了以上任何一件事，那就是越界（overreaching），使用者要謹慎判斷。

The summary reports n, missing, mean, sd, median, min, and max for a continuous variable, and level counts for a categorical one. The raw column values never reach the summary. In principle, therefore, the AI cannot have checked any of the following, whatever it claims:

  • which specific row is an outlier. Min and max give you the range, not the row.
  • whether the data are normally distributed. Min, max, and median cannot reveal the shape of a distribution.
  • whether missingness clusters in particular groups. The summary gives you the total missing count, not the pattern.
  • what the variable actually measures, or how the study was designed.

When an AI reply claims to have checked any of these, the reply is overreaching. Judge it carefully.

動手做（5 分鐘）Try it (5 minutes)

  1. 在 jamovi 開啟你的資料，任選三個變項（至少一個連續、一個類別）。
  2. 開啟 askLLM 的 jamovi Module Guider，把這三個變項勾進 Variables to describe。
  3. 確認「Attach data summary to prompt」是勾選的。
  4. 問一句簡單的問題，例如「請描述這三個變項的分布」，勾選 Submit 送出。
  5. 收到回覆後，把回覆裡提到的每一個數字——n、mean、sd、median、min、max、 各水準次數——逐一對回 Exploration ▸ Descriptives 的輸出，看有沒有對不上。
  6. 檢查回覆有沒有提到摘要裡不存在的東西（例如「第幾筆資料」「常態性」「離群值」 這類需要看原始列或做額外檢定才能判斷的內容）。有的話，標記為越界。
  1. Open your dataset in jamovi and pick three variables (at least one continuous, one categorical).
  2. Open askLLM’s jamovi Module Guider and tick those three variables into Variables to describe.
  3. Make sure “Attach data summary to prompt” is ticked.
  4. Ask a simple question, e.g. “Describe the distribution of these three variables,” and tick Submit.
  5. When the reply arrives, match every number it cites — n, mean, sd, median, min, max, level counts — against the Exploration ▸ Descriptives output. Note any mismatch.
  6. Check whether the reply mentions anything the summary cannot support (a specific row, normality, outliers). If it does, flag it as overreaching.

帶走這句話One thing to remember

AI 只讀得到你勾選變項的摘要數字，你在 jamovi 裡看到的原始資料，它一個字都 看不到。

The AI only ever sees the summary numbers for the variables you ticked — never a single row of the raw data you see in jamovi.