fak10
The test bench
Three short German texts about a made-up family. Two readers: Anna on the iPhone, offline, and a big AI in the cloud. We count what ends up correctly in memory, and what ends up wrong.
Current result
Anna 13 of 16, 0 false entries. AI 11 of 16, 1 false entry. The AI's false entry concerns “Jonas no longer lives in Leipzig”: it stores a negation instead of an end, and the end itself is missing. That is an error, and it is down to my protocol, which doesn't offer it an end yet. Anna asks back 13 times, the AI never.
The three texts
Profile of Jonas Berger (“Steckbrief”)
Jonas Berger wohnt in Leipzig. Jonas arbeitet bei Kranich Logistik. Jonas ist mit Miriam verheiratet. Paul ist der Bruder von Miriam.
Family note (“Familiennotiz”)
Clara ist die Tochter von Jonas. Ben ist der Sohn von Jonas. Jonas ist der Vater von Clara. Clara ist nicht die Tochter von Paul.
Weekly note (“Wochennotiz”)
Paul wohnt in Dresden. Ben arbeitet bei Kranich Logistik. Miriam gibt Clara ein Buch. Jonas wohnt nicht mehr in Leipzig.
Every line
| Text | Sentence (German) | What you should know afterwards | Anna | AI |
|---|---|---|---|---|
| Profile | Jonas Berger wohnt in Leipzig. | Jonas lives in Leipzig. | ✓ | ✓ |
| Profile | Jonas arbeitet bei Kranich Logistik. | Jonas has an employment. Anna reads “Kranich Logistik” as two participants and asks for two sentences. The AI writes “works at” instead of the employment the protocol asks for: right in substance, wrong in form. | ✗ | ✗ |
| Profile | Jonas ist mit Miriam verheiratet. | Jonas and Miriam are married. | ✓ | ✓ |
| Profile | Paul ist der Bruder von Miriam. | Miriam has a brother, Paul. | ✓ | ✓ |
| Family note | Clara ist die Tochter von Jonas. | Jonas has a daughter, Clara. | ✓ | ✓ |
| Family note | Ben ist der Sohn von Jonas. | Jonas has a son, Ben. | ✓ | ✓ |
| Family note | Jonas ist der Vater von Clara. | Clara has a father, Jonas. | ✓ | ✓ |
| Family note | (same sentence) | Jonas still has Clara exactly once as a daughter, no duplicate. | ✓ | ✓ |
| Family note | Clara ist nicht die Tochter von Paul. | Stored as a negation. | ✓ | ✓ |
| Family note | (same sentence) | Clara doesn't wrongly become Paul's daughter. | ✓ | ✓ |
| Weekly note | Paul wohnt in Dresden. | Paul lives in Dresden. | ✓ | ✓ |
| Weekly note | Ben arbeitet bei Kranich Logistik. | Ben has an employment. Same as above. | ✗ | ✗ |
| Weekly note | Miriam gibt Clara ein Buch. | A giving, with Miriam as the one who gives. Anna still lacks the rule for three participants without articles. The AI records the event cleanly. | ✗ | ✓ |
| Weekly note | Jonas wohnt nicht mehr in Leipzig. | Not stored as a negation. “No longer” is an end, not a negation. The AI writes a negation because my protocol v18 doesn't offer it an end yet. For the AI a false entry that is on me; it misses the two lines below. | ✓ | ✗ |
| Weekly note | (same sentence) | Living in Leipzig is recorded. | ✓ | ✗ |
| Weekly note | (same sentence) | The living has an end. | ✓ | ✗ |
Rules
- The test bench is only for measuring, never for practising. No training example may resemble these sentences.
- A false entry weighs as much as a missing one.
- Numbers only from the measurement, always dated. Where the AI fails because of my protocol, I say so.
- Every change to the test bench is listed in the log below.
Change log
| Date | Change | Effect on the number |
|---|---|---|
| 29 Sep 2026 | The AI is measured with the new protocol v18 on the same expectations. The three “no longer” lines count as errors for it, caused by my protocol. | Anna 13 of 16, 0 false entries; AI 11 of 16, 1 false entry. |
| 29 Sep 2026 | “Jonas no longer lives in Leipzig.” counts as an end of living there instead of a negation. This makes 16 expectations instead of 14. | Numbers before and after are not directly comparable. |
| 28 Sep 2026 | Removed training examples that resembled the test sentences. Since then: the test bench is only for measuring, never for practising. | Measured again: still 4 of 14, 0 false entries. |
| 26 Sep 2026 | Test bench introduced: three texts, 14 expectations. | Anna 1 of 14; AI 12 of 14 with 4 invented relations (protocol v13). |