ASP.net used to do user agent sniffing extensively. It used it for feature detection, but also to serve different HTML and JavaScript to different browsers. I think it still does, actually, but I've not looked at it in a while.
Microsoft obviously thought that user agent sniffing was a great idea until pretty recently, because they use it a lot.
Use of user agent sniffing is built right into the framework, and also into Webforms (controls, server-side events, and the like). Often, completely different HTML and JavaScript is generated for different browsers (whether you want it to or not).
Worse still - things built on top of ASP.net used user agent sniffing even more. For example, ClickOnce required that the browser report which versions of the .net framework are installed, and the server does different things depending on that information. IE's user agent strings can get very long, reporting the browser version, OS version, CPU architecture, various service pack levels or revisions for different things, half a dozen versions of .Net, Infopath, Windows Media Center, Zune software, whether it's a TabletPC or not, RTC API, and a whole bunch of other fragments that I don't even recognize (SLCC?), and even occasionally advertising that the browser has some malware installed.
It's nearly impossible to switch this stuff off in ASP.net, and the browser database is often out of date. So new browsers or browser versions are often served broken pages (because the "unknown browser" fallback doesn't actually work). This happened when IE 10 was released, for example. Existing browsers are often detected incorrectly, and served a degraded version of the page (happened a lot to Firefox and Opera, back in the IE 6 / IE 7 era).
At least in ASP.net MVC you can ignore the browser detection most of the time. You can't really disable it in ASP.net itself.
Modern ones. Checking for "Mozilla" is one (admittedly not very good) way to check if you're dealing with a browser or with some other kind of user agent, such as a search engine bot or something like that.
Unfortunately, yes. Every mom-and-pop PHP-based website that was done by a teenager in the last 15 years probably still does this at some level. There are also plenty of old web servers out there that look for those headers to decide how to respond (to account for browser bugs).
