Claims and limits

C7

tmct solves a taught Hanoi puzzle in under a second up to this many disks.

7disks, largest solved under 1s

The river row is the classic wolf-goat-cabbage crossing, played here with a fox (mudiii.html's fourth scenario), scaled past the three passengers the classic puzzle names: it still solves the classic optimum at 3 passengers under a second, and the search first runs past 10 seconds at 182 passengers — not because the search slows down looking, but because past that size no first move can leave every guarded pair covered, so it reports the honest miss straight away and the slowdown is the miss itself getting more expensive to state, never a shortened plan standing in for a real one.

What this does not mean: The search does not scale without bound. Past the "largest under 1s" size, each domain eventually runs over 10 seconds and is killed by its own timeout rather than left to search indefinitely. See the "first over 10s" column above.

Meets the standard: the scaling wall sets the denominator here, and you can recheck it yourself with the device button below.

Source: results/claims/planner.json, regenerated by npm run claim:planner. view JSON · try it in plan.html

L1

Unedited Simple Wikipedia prose grounds into a stored fact this often, sentence by sentence.

18.65% of 520 sentences

What this does not mean: This is raw, unedited prose pulled in unfiltered, not the corpus tmct ships with. 423 of 520 sentences were skipped, most for an ungrounded content word such as “dessert” or “each”.

Meets the standard: external input, unedited Simple Wikipedia prose tmct’s author did not write or curate.

Source: results/claims/prose-band.json, regenerated by npm run claim:prose-band. view JSON · try it in chat.html

L2

Asked a five-way commonsense multiple-choice question, tmct picks an option only when one grounds against the graph, and picks the right one this often.

6of 100 questions

22 answered, 22 refused, 56 abstained.

Of the 100 questions, 30 had the answer stated in the graph at all; 2 of those got picked. The lane read the question's own topic in 31 of them.

1 of the ties were split because one option answered something else the question said, and 1 of those were right. Every other tie is reported as a tie.

Of the 56 abstentions, 25 matched no relation cue in the stem at all — a judgment call or a small scene, not a stated relation the graph could hold in the first place. Three real examples: “A story about World War II would be set when?”, “What do professors primarily do?”, “In what country are the most fast food restaurants?”.

What this does not mean: This is not a reasoning score. CommonsenseQA's distractors were pulled from ConceptNet on the same relation as the answer, so an option having an edge to the source concept is close to no evidence at all. A refusal here means several options grounded equally well and tmct reported the tie rather than breaking it. An abstention means nothing in the graph connected the question to any option. Neither is a wrong answer, and neither is dressed up as one. Three of the stems it missed: “Eating is part of living, but your body doesn't use it all and the next day you will be doing what?”, “How might a automobile get off a freeway?”, “What do animals do when an enemy is approaching?”.

Meets the standard: external input: the CommonsenseQA dev split, authored outside this repository from ConceptNet, the same source the shipped corpus slice is cut from, selected by a committed deterministic rule and attributed in test-benchmarks/claims/commonsenseqa-sample.NOTICE.

Source: results/claims/commonsenseqa.json, regenerated by npm run claim:commonsenseqa. view JSON · try it in chat.html