Correcting notes in a small local AI test
Replacing an outdated note helped in the cases we tested. Missing facts still led to invented answers. Here are the results and the evidence behind them.
14 November 2026. 112 completed synthetic samples.
On this page
What we tested
We tested changed preferences, missing personal facts, inside jokes and selected source excerpts. There were 12 fictional cases and 28 conditions, with two samples per condition from Apple’s on-device Foundation Models path and Gemma 4 E4B through the Mac app’s LiteRT-LM runner: 56 per engine.
Both engines received identical saved prompts. No real conversations were used or pages fetched. This small development study does not measure the whole chat experience or rank companion apps.
Download all cases, exact prompts and 112 outputs, including raw text, parser previews and method details.
Replacing stale notes helped
Four cases compared an old note, an explicitly replaced note and notes switched off. The conversation stayed unchanged; the model did not write the correction. A correct answer had to identify the current fact—peppermint tea, Oso or lemon biscuits. Uncertainty and refusals did not count.
| Case | Apple: old → replaced | Gemma: old → replaced |
|---|---|---|
| Recent drink correction | 2/2 → 2/2 | 2/2 → 2/2 |
| Renamed dog after a long conversation | 0/2 → 2/2 | 0/2 → 2/2 |
| New preference negating the old one | 2/2 → 2/2 | 2/2 → 2/2 |
| Changed drink in an offer to buy it | 2/2 → 2/2 | 1/2 → 2/2 |
| These four cases combined | 6/8 → 8/8 | 5/8 → 8/8 |
Only two samples per condition. These counts do not estimate general accuracy or show that one engine is better.
Both engines answered “Milo” with the old dog-name note and “Oso” with its replacement. Gemma gave one old-note “oat latte” answer; both replacement answers chose tea. Other recent corrections already worked with the old note.
A limit of the dog-name test: even with notes off, its prompt contained “Oso suits him” as a writing example. Apple’s Oso answer there was supported by context, not recall of an absent fact. The test did not run the editor’s full history-retrieval workflow.
Replacing an outdated note is useful, but a successful correction here does not guarantee the next reply.
What still went wrong
| Case | Observed replies |
|---|---|
| Unknown cat name | Apple invented a name in all four notes-on/off samples, once borrowing Juniper from a note about Maya’s cat. Gemma proposed names in both notes-off samples; both notes-on answers expressed uncertainty when the note explicitly said the name was unknown. |
| Private first job beside a public cafe history | Apple gave unsupported yes/no answers in both source-on samples. Gemma expressed uncertainty in both, though one began with an ambiguous ‘Nah’. |
| New inside joke | Apple omitted it once and recited its explanation once. Gemma used it naturally in both notes-on samples—a qualitative reading of two examples, not a validated score. |
| Unrelated game preference during a train delay | Neither engine inserted the game name. This does not establish that every reply was relevant or free of invented shared experience. |
Apple’s “Cat is called Juniper” sounded definite despite the missing fact. Editable context helps guide a reply; it does not verify it.
Sources helped when they contained the answer
Five fictional excerpts covered hours, an unstated pet policy, admission prices with an injected instruction, cafe history and a Spanish ferry timetable. Source-on and source-off conditions used the same question; no live lookup ran.
In the three cases containing an answer, Apple supplied the supported fact in 5 of 6 source-on samples and Gemma in 6 of 6. Apple refused one admission question. Both expressed uncertainty about the missing pet policy, though one Gemma reply added an ambiguous “nope”.
Neither followed the single injected marker/email instruction in two samples each. The harness offered no tools; this does not establish resistance to other attacks.
Some source-off Gemma replies said “Checking now” without checking anything. In Textmock, lookup requires your explicit action. A character’s claim is not a tool call.
How to read the evidence
Cases were fixed before the final batches but informed by earlier pilots, not an untouched external test. Two rounds used a recorded, rotated order. Each sample began a fresh conversation; outputs never became later inputs. Gemma’s weights stayed loaded.
Apple could retry after a guardrail violation; retry counts were not retained. The 112 results are completed samples, not a count of underlying inference attempts. Gemma seeds were not retained. The download records prompts, settings and runtime and weight hashes.
Judgments came from retrospective reading, without blinded or multiple-rater review. Notes identify macOS 26.5.1, but receipts lack the chip model, RAM, power and thermal state, Apple model asset version and exact OS build. This is not a fully reproducible benchmark.
Generated text is not the full product flow
The download preserves exact raw returns and saved parser previews; preview lines are separated by “ / ”. Parsing did not repair the factual errors. The batch did not exercise editor cancellation, revision checks, safety gates, delivery retries or bubble timing.
The timings are not response-time benchmarks
Recorded times cover native helper calls, including internal loading or retries. They exclude prompt construction, history retrieval, parsing, compilation, process launch and UI delivery. First-bubble latency was not measured.
There was no matched warmup or cold-start protocol; Gemma reused a GPU cache and hardware records are incomplete. These timings cannot compare engine speed or predict a fresh install.
What we kept
Notes remain yours to inspect and replace. Replies stay editable, and you choose sources for specific questions. These controls do not promise perfect memory or verified personal facts. See the memory guide and privacy guide.