Data Contamination

Talkie is a 1930 language model with a modern contamination problem

A 13B model trained on pre-1931 text is less a nostalgia demo than a practical test bed for clean data, synthetic tuning, and what language models really learn from the web.