"Differences in climate made butter and milk a mark of otherness—unlike cheese which kept better in warmer climes. Since the presence of large, powerful cities and cultures in the far north is a relatively modern development, this linked any non-cheese dairy with inferior (or at least less powerful) groups.

Drinking milk wasn’t unknown in places such as Ancient Rome. But for similar reasons, it was linked with the country and lower classes. Until the age of refrigeration, very little fresh drinking milk was consumed in the Middle East. In Rome, due to the inevitability of spoilage, and because fresh milk was available only on farms, it was consumed mostly by the farmers’ children and by peasants who lived nearby, often with salted or sweetened bread. This led to fresh milk’s being widely regarded as a food of low status. Drinking milk was something that only crude, uneducated rural people did and was rare among adults of all social classes."

https://www.atlasobscura.com/articles/history-of-milk