No, I don't think I ever agreed on that. It's buried very deeply in the settings (Settings -> Privacy -> Location Services -> System Services -> Significant Locations). You however can opt out.
It's not enabled on any of my iOS devices, so that's one data point. They were upgraded to iOS 13, if that makes a difference (maybe it's opt-out on a fresh install?). I tried searching online and I found an article saying it's opt-in[1], but also found anecdotes similar to yours about it being enabled without their knowledge.
isn't that opt in?