There are a few main infertlity journals I read: Human Reproduction, Fertility and Sterility (F&S) and the Journal of Clinical Endocrinology & Metabolism. It is in these journals the latest fertility research is published, useful topics are reviewed and in F&S they have a section called "Inkings." This is the section I enjoy the most, honestly. Inklings are short musings, generally by well respected fertility doctors and scientists. They write about their "gut feelings" or their pet theories about disease or treatments. I like them because I think we can see the future of research and of medical treatment in many of them. I've read some that theorize about the role of coenzyme Q 10 in egg quality, the meaning of vitamin D deficiency, the role of progesterone in miscarriage. The Inkling I read tonight was on dairy consumption and male factor infertility.
Milk consumption in the developed world has increased at a staggering rate over the past 50-60 years. This demand has prompted farmers to push cows to produce milk for consumption, even milking them throughout their pregnancies. Indeed 70% of milk in the world comes from cows who are pregnant. Pregnant cows have high levels of steriods like estrogen and progesterone, which appear in their milk. These hormones concentrate in milk fat predominantly so fattier dairy products like butter and cheese have even higher hormone levels than plain milk.
If girls and women consume extra estrogen and progesterone from dairy products it may not matter as amount of these hormones in dairy products are relatively tiny compared to the amount our bodies are making. Extra estrogen and progesterone in boys and men might matter, though. Men who consume dairy products might then have higher estrogen and progesterone (female) hormone levels. In men, estrogen and progesterone can affect the pituitary gland which controls sperm production in the testicles.
Studies in the US and Japan have indeed suggested that men who consume high fat dairy products like cheese and butter do have lower sperm counts and motility than those who do not. Studies out of the Netherlands did not find such a correlation so more studies are needed to sort out if there is an effect.
Given there is a trend to lower sperm count and quality over the past few decades, the role of dairy consumption should be considered as a possible explanation. At least it's one group's inkling.
Reference: Haimov-Kochman R, Shore LS, Laufer N. The milk we drink, food for thought. Fertil Steril. 2016 Nov;106(6):1310-1311.