Measuring as spend ROI is standard practice on any large scale ad campaign. The trick about having a holdout population is a neat twist.
It boggles my mind when HN commenters start claiming that measuring ad campaigns is impossible or that ads are universally ineffective. Anyone who has spent time working with large ad campaigns should know the tools and methods used to measure these things.
I suspect the HN sentiment comes from a common feeling among techies that they are somehow immune to influence from advertising, combined with a high adoption rate of ad blockers. HN commenters tend to assume that other consumers are just like themselves, which is far from the truth. In the real world, advertising (when done right) is not only very effective but not that difficult to measure using modern technology.
Every campaign has TWO holdouts. First, the universal holdout who gets nothing from any campaign, and a second additional holdout who doesn't get a specific campaign but gets other campaigns. We have an entire group of people designing the holdout matrix to make sure they are sound and don't cross over. That's the elementry school version of things.
The graduate level stuff we run is using decioning engines with adaptive models down to EACH email, switching up not just offers but also ad copy, images, time of delivery etc. that drive VERY complex holdout matrix's which can limit the number of things we can test at once.
One of the executive decisions is when or if the drop customers out of holdout populations. Usually keep the holdout going for X period of time as a standard practice, but if a campaign is showing very strong response we might drop half the holdout if we can still make a read on campaign. We have a formal notice of decision that is published when this happens. Holdouts have alot of opportunity cost, but finance doesn't see those numbers.
Some of the digital exhaust from our primrary product can be used to solve marketing attribution issues for major brands in an area that has until now lacked attribution. Our product beta is just hitting the market.
> I suspect the HN sentiment comes from a common feeling among techies that they are somehow immune to influence from advertising, combined with a high adoption rate of ad blockers.
I don't believe this at all. The reason I am skeptical of the effectiveness of advertising is despite engaging with dozens of people over the years, I have never had a single person provide me with convincing evidence that it works. Instead, they simply repeat "but of course it works!" similar to your comment.
I manage advertising spend for a small technology company. We spend in the 10's of thousands per month on Google and Facebook.
We can measure customer acquisition cost and we do using normal tools like Google Analytics.
We've had days where we pause our campaigns, and we get fewer new customers that day. We've had platforms pause for a period of time because of some issue, and we get fewer new customers over that period of time.
It's one data point. I'm aware of plenty of other companies with similar results to what we see.
It boggles my mind when HN commenters start claiming that measuring ad campaigns is impossible or that ads are universally ineffective. Anyone who has spent time working with large ad campaigns should know the tools and methods used to measure these things.
I suspect the HN sentiment comes from a common feeling among techies that they are somehow immune to influence from advertising, combined with a high adoption rate of ad blockers. HN commenters tend to assume that other consumers are just like themselves, which is far from the truth. In the real world, advertising (when done right) is not only very effective but not that difficult to measure using modern technology.