> without any attempt to validate their investments.
I've also worked in adtech, for quite a long time on all sides, and my experience is that far more companies do research to justify their ad spend not validate it.
I have managed plenty of A/B tests in my day, each one claiming to show some improvement, 10%, 15% etc (some, as you said, showing no improvement). Even though these tests are often run "correctly", essentially nobody goes back and asks "wait, we had a 10% increase multiple times in the past year, but is our <metric> really showing the cumulative improvement we expected?"
The greatest trick in the industry is that because VC are pouring money into everything, everywhere, every metric appears to grow. At every startup, everywhere, pre-pandemic, numbers were going up because people were pouring money into the system.
I've worked at companies who I know for a fact their adtech product cannot and does not work, yet their business continues to explode because in recent months nearly all ad spend has been on digital ads.
I've talked to companies whose entire function is bidding optimization who literally do not understand how to optimize bidding given the information you have.
Absolutely people are running "experiments" but the function of the "experiments" is to justify that the ad team is worth having, and then that the VP of marketing is doing their job and then that the CEO has hired some super smart people, and then that the VCs might have really found a unicorn. Everyone sees what they want and no one really wants to ask the question "wait, does this really work? are these tests really able to capture the complexity of the environment?" And if you are one of those ornery people that insists on probing into the details and seeing if any of this is working, you will eventually get fired.
Ad tech is largely a scam, but a huge number of rich and smart people benefit from the illusion that it is not, so we continue to see experiments showing that everything works as expected.
I agree that there a lot of (smaller) players in the ad tech space who are scamming companies out of money. And especially if you've been in this industry for a while, this practice was much more prevalent a decade ago.
I don't think your view is representative of how most F500 companies, or many of the new DTC brands, invest on major ad tech platforms though (FB/G/Snap/etc.). Experienced marketers try to measure metrics as close to core business KPIs as possible, and comparing ads based off of simple A/B tests is increasingly becoming a technique of the past. Measuring the incrementality of ad campaigns through long-term holdout groups gives companies a much more accurate read of their investment, while bringing higher statistical rigor as well. So rather than looking for cumulative increases on <metric> after advertising for a year, you could instead point to group A (who saw no ads) and see that group B (who saw ads) drove a 1.5x higher <metric>.
I'm not sure how you can say advertising is a scam when there are clear examples of popular brands that most likely wouldn't exist today without digital ads? Take Allbirds as an example. There are dozens of consumer shoe brands that are already in physical stores and have higher brand recognition; how can you argue that advertising didn't help them cut through the noise and grow their bottom line?
I've also worked in adtech, for quite a long time on all sides, and my experience is that far more companies do research to justify their ad spend not validate it.
I have managed plenty of A/B tests in my day, each one claiming to show some improvement, 10%, 15% etc (some, as you said, showing no improvement). Even though these tests are often run "correctly", essentially nobody goes back and asks "wait, we had a 10% increase multiple times in the past year, but is our <metric> really showing the cumulative improvement we expected?"
The greatest trick in the industry is that because VC are pouring money into everything, everywhere, every metric appears to grow. At every startup, everywhere, pre-pandemic, numbers were going up because people were pouring money into the system.
I've worked at companies who I know for a fact their adtech product cannot and does not work, yet their business continues to explode because in recent months nearly all ad spend has been on digital ads.
I've talked to companies whose entire function is bidding optimization who literally do not understand how to optimize bidding given the information you have.
Absolutely people are running "experiments" but the function of the "experiments" is to justify that the ad team is worth having, and then that the VP of marketing is doing their job and then that the CEO has hired some super smart people, and then that the VCs might have really found a unicorn. Everyone sees what they want and no one really wants to ask the question "wait, does this really work? are these tests really able to capture the complexity of the environment?" And if you are one of those ornery people that insists on probing into the details and seeing if any of this is working, you will eventually get fired.
Ad tech is largely a scam, but a huge number of rich and smart people benefit from the illusion that it is not, so we continue to see experiments showing that everything works as expected.