The Markovian Ceiling: Coherence ≠ Relevance — A Clean Empirical Proof

Posted by: Chetan Sharma Date: 15 May 2026 Location: Kolkata, India You can teach a language model to speak perfectly. It can write grammatically flawless sentences. It can mimic your style, your vocabulary, your cadence. It can be coherent. But can it mean what it says? This morning, at approximately 5:30 AM IST, I ran a simulation that answered this question definitively. The Setup Three models. All trained on the same 20,000 sentences. UniLSTM — a standard Markovian language model. It predicts the next word based on the previous words. It never sees a goal. Bidirectional Seq2Seq LSTM — reads the goal with a bidirectional encoder, then generates the sentence. Non-Markovian. Multidirectional Transformer — full self-attention across the goal, then generates. Non-Markovian. The task: given a three-word goal like [fox, under, sofa], generate a sentence that contains those elements in the correct order. The test goals are novel combinations — never seen during training. The Result Model ...