Markov vs Transformer: Text Generation Experiment
This project compares classic statistical models (Markov chains & n-grams) with modern transformers (GPT-2) on the same text corpus (Pushkin’s poetry). It highlights the jump from local memory to long-context reasoning in text generation.
Motivation
In 1906, Andrey Markov showed how letter dependencies could model text. Fast forward a century — transformers now handle hundreds of tokens of context with attention. This repo recreates that evolution, side by side.
Models Compared
Markov Chains (n=1,3,5)
Generate text from local character windows.
Capture letter frequencies, small fragments of fluency.
Fail on long-term dependencies → loops & gibberish.
GPT-2 (medium)
Pretrained transformer with 345M parameters.
Extends prompts into fluent, poetic lines.
Still prone to degeneration without sampling controls.
Results
Prompt: “I loved you” (Pushkin corpus)
Markov vs GPT-2 on Pushkin (Prompt: “I loved you”) Model Sample Output (excerpt) Markov (n=3) “I longer trouble you so tenderly, sorrow. I loved in my so sincerely extinguished in my shyness, no loved now by jealousy. I loved you may be loved you may be loved you so tenderly…” Markov (n=5) “I loved you silently, without hope, Tormented now by jealousy. I loved you: and perhaps this flame Has not entirely extinguished in my soul; But let it no longer trouble you; I do not entirely extinguished in my soul; But let it…” GPT-2 medium “I love you with all my heart, without reserve. I am in love with you now, and have never been. I am in love with you now, and will never be… I love you with all my heart, without reserve…”
Key Observations
Markov chains: good for local coherence, but collapse quickly.
Transformers: sustain global structure, more creative continuations.
Both models show failure modes — repetition loops highlight why sampling strategies matter.
Demonstrates the leap from statistical modeling → neural networks → generative AI.
How to Run
Markov chains
mc = NGramMarkov(n=5) mc.train(corpus) print(mc.generate("", 200))
GPT-2 (via 🤗 Transformers)
from transformers import pipeline
generator = pipeline("text-generation", model="gpt2-medium")
print(generator("", max_length=80, do_sample=True))
✨ Author
Developed by [Naga Adithya Kaushik (GenAIDevTOProd)], assisted with AI(Debug, text corpus generation only)
For research, debug, and teaching purposes.