"For more than a decade now, the fact that people have a hard time gaining actionable insights from their data has been blamed on its size."
The real issue is that business people usually ignore what the data says. Wading through data takes a huge amount of thought, which is in short supply. Data Scientists are commonly disregarded by VPs in large corporations, despite the claims about being "data driven". Most corporate decision making is highly political, the needs of/whats best for the business is just one parameter in a complex equation.
I used to joke that Data Scientists exist not to uncover insights or provide analysis, but merely to provide factoids that confirm senior management's prior beliefs.
I did several experiments, and noticed that whenever I produced analysis that was in line with what management expected - my analysis was praised and widely disseminated. Nobody would even question data completeness, quality, whatever. They would pick some flashy metric like a percentage and run around with it.
Whenever my analysis contradicted - there was so much scrutiny in numbers, data quality, etc, and even after answering all questions and concerns - analysis would be tossed away as non-actionable/useless/etc.
if you want to succeed as a Data Scientist and be praised by management - you got to provide data analysis that supports managements ideas (however wrong or ineffective they might be).
Data Scientist's job is to launder management's intuition using quantitative methods :)
> if you want to succeed as a Data Scientist and be praised by management - you got to provide data analysis that supports managements ideas (however wrong or ineffective they might be).
> Data Scientist's job is to launder management's intuition using quantitative methods :)
It’s no different than the days when grey bearded wisemen would read the stars and weave a tale about the great glory that awaits the king if he proceeds with whatever he already wants to do.
The beards might be a bit shorter or nonexistent, but the story hasn’t changed.
Absolutely. If you don't like what K-Means is telling you, change a variable and re-run. (that's one great thing about business data: there's no shortage of variables! True, there's usually a shortage of independent variables, but fixing that is difficult and underfunded).
And you'd better hope the bones actually say something useful.
I was the infra lead on a data lake project and got take part all the way to breaking down the data and turning into PowerBI reports. The result was "sell more" and to clients who marketing already identified, years ago, as whales.
There were some interesting other insights, esp. w/r/t to niche products that sold around weird dates (Easter, Memorial Day, 4th July -- but not obvious gift days like Valentines or X-Mas), but it led to a lot of "you're doing it wrong!" recriminations and follow up projects.
> Data Scientist's job is to launder management's intuition using quantitative methods
Ouch. This is savage, but sadly correct in many cases.
HOWEVER, to play devil's advocate here, I've also seen corporate data scientists overstate the conclusions / generalizability of their analysis. I've also seen data scientists fall prey to believing that their analysis proves would should be done, rather than what is likely to happen.
The role of an executive or decision maker is to apply a normative lens to problems. The role of the data scientist / economist / whatever is to reduce the uncertainty that an action will have the desired effect.
Good point. Data is one aspect of making a decision. The other aspect is understanding the industry and environment. Often data scientists give just one variable needed to make a decision. In health care for example you need to factor in a whole host of legislation. You also need to factor aspects of the industry not reflected in the data. As an example doctors not wanting to use iPads is something you can't measure and can't force as company. Even though data analysis might suggest this is the way to go.
> The role of an executive or decision maker is to apply a normative lens to problems. The role of the data scientist / economist / whatever is to reduce the uncertainty that an action will have the desired effect.
Where do business analysts fit into this dichotomy? Their whole job is to poke around in Tableau in order to surface high-ROI strategies for the business to pursue. (Where, in choosing which proposals to surface to management, they're effectively making 90% of the strategic decisions.)
Or how about corporate buyers in trading and retail companies?
People who poke around in Tableau might not get a lot of respect in the hierarchy of DataFolk, but descriptive statistics and thoughtfully chosen visualizations can be immensely useful. Exploratory data analysis sometimes reveals patterns that are so obvious that to apply statistical inference is just vanity.
If understanding the data generating processes is the goal, I'd rather see some useful plots than wade through a technical description of some model whose assumptions were flagrantly violated.
Positive claims are about what is true. Normative claims are about what should be true, or rather what decisions we should make. Put another way, positive claims deal only with facts while normative claims deal also with values.
GP is saying that it's the data-scientist's job to give the executive the facts and it's the executive's job to decide what to do about the facts.
> I used to joke that Data Scientists exist not to uncover insights or provide analysis, but merely to provide factoids that confirm senior management's prior beliefs.
Here's my take on this not listening to the "expert":
A few years ago there was a problem with storm-water infiltration into my (elderly at the time) mother's property from her neighbor. I, being a dutiful son and a civil engineer, investigated it and came up with the probable cause, the likely effects of non-action and the most cost-effective solution. I presented it to my mother in the most layman-like terms that I could. She said she'd think about it – meaning she'd refer i.e. defer – to her daughters. In the meantime I had a very layman-like chat with my mother's carer and told her the situation in layman's terms. The carer listened and said that what I said I made total sense to her. Later on, one of my sisters accosted me and stated that it was completely obvious what the problem and the solution was – "even the carer could see it". Human foreheads don't have the real estate for where my eyebrows wanted to ascend.
My advice is to consider whether the message should be separated from the messenger somehow.
My Mother-in-Law was called by a tech support scammer. Her bank was unwilling to accept their charges, and the scammer wanted her to call the bank to tell them to accept them anyway. My Brother-in-Law was telling her "no, this is a scam, do not do this", but she was unwilling to listen. Eventually, he told her to call me, thinking if she wouldn't listen to her son, maybe she'd listen to her son-in-law. Which she did.
Parents can be listening to their kids 99% of the time it will be transparent and uneventful. When there’s confrontation/divergence in opinion, by definition it didn’t work out through the usual channel, and of course a third party weighting in the balance will have visible effects.
As a consultant with roots in backend dev, I fully understand the scrutiny that we receive because unfortunately, it is often very warranted... It feels a bit refreshing to read your comment and see someone articulate what I am trying to convey to my clients. I am a tool, and yes, this pun is intended.
Sure, it is actually not very complicated in my case. I did backend development for a short while during and after university and then moved into IT consulting fairly quickly.
It was a LinkedIn recruiter message which I usually ignore. However, my SO did not (she is in IT as well) and convinced me to join a hiring event. I ended up liking it a lot and went through the hiring process. Soon, I started out on the most junior level and joined my first project with 3 very senior colleagues after a few weeks.
The learning curve was very steep both on the technical level and also regarding the consulting aspect - at first there was nothing I could 'consult' on due to lack of experience. This changed with growing experience, with the guidance of senior colleagues and my private efforts to gain skills and expertise.
This almost reads like my trajectory so far, but I'm at the point where I can't really consult due to the lack of experience, but I did make a good impression so far. Can I ask you, into what efforts should I put my private time? More technical knowledge? Into very fine details, or brief insights into different areas? Any good resources?
In the news business, if your story or opinion backs up the preconceived notions of the investigative reporter then you are a 'source' otherwise you are a 'conspiracy theorist'.
This could be a reason why Data Scientist as a job title exploded in last years, every middle manager could afford one/two/few headcounts of data scientists to produce analysis that advances that middle manager's corporate agenda (more growth, empire building, expansion to certain de-novo areas, etc).
Recent tech layoffs is the other side of that growth, when cheap money is gone and company is forced to stick to core competencies and shutdown growth plans
This would match what psychologists say about humans in general: we feel first, then we use our brain to justify that feeling. We’re not rational beings.
I think the answer is simpler: people care about their careers and their family first. Think, "If the data says something that gets in the way of my career well I don't care about the data."
Had the same problem when I was an economics researcher -- publication bias for what stakeholders want to hear (often the government) is rampant because that's where funding for the economics department mostly comes from.
It's only rational. The company certainly doesn't care about that individual first, as evidenced by e.g. its decision to lay them off when it doesn't think the individual is serving them, so why should the individual put the company first?
This is also known as The Iron Law of Institutions.
We totally are, it's just that rationality is a tool, not a guide. If you want to work out the truth, rationality will help you do that, but if instead you want to justify a decision you already made, well, it'll help you do that too.
Hypothesis don't come from rationality either, they result from well informed intuition. All of the formality of science is about tricking ourselves into discovering our intuition is wrong using a rational series of steps even when everything in our nature is to use that ability to reason to do the opposite.
Thats because psychologists dont understand or choose to ignore how chemistry influences our personalities and emotions. An extremely simple example from the same medical/health profession is the use of SSRI's to make people feel happy. The legal system recognises how chemicals influence our feelings because of the laws that exists on illegal drugs or drink driving.
The definition of rational is being informed enough to know what said chemicals will do in the short term and long term in order to make an informed decision, but then I'm reminded we dont get taught any of the above unless we specialise at a Uni, so most people cant make any sort of informed decision.
Yes.
I worked in the data org of a moderately sized financial firms tech org.
The tech org claimed to be hugely data driven. Was in the org mottos and all of that.
Nonetheless, the CTO went on a multi-year, 10s of millions of dollars, huge data tech stack & staffing reorg shake up... with really zero data points explaining the driver, or what we would measure to determine it was successful.
So it became a self referential decision that we are successful by doing what he decided, and we are doing it because he decided it.
Huh. I've not thought of it as laundering, but I think you've basically summarised consulting in healthcare. Pay to legitimize and push through a pre-existing idea (eg let's close down a few ERs) or a delusion (e.g. lean, we don't need a waiting room) and say it was recommended by consultants to stakeholders and the public.
all consulting is like that, Partners/MDs at consulting companies meet with Board/CEOs to get rough idea of what they want/need, and quickly negotiates a consulting engagement contract to create PowerPoint with all the evidence and analysis gathered that supports CEO's initial idea.
This is the only reason why a 60+ PowerPoint slide deck can cost several millions dollars
Also a big reason McKinsey and BCG exists - provide cover for business plans intended by management to protect them from shareholder lawsuits. My friend did a sojourn at McKinsey and 6 months of his life was producing PowerPoints and memos backing up an expansion to AIPAC region. Was already happening but he was providing all manner of business justification for board meetings and whatnot.
This phenomenon is true to varying degrees in academic medicine (maybe all of academia) as well - personally have seen excellent data and methods disregarded when they don't confirm existing agendas. The choice for the researcher can become one of burning out trying to do good work and getting nowhere, or acquiesce and only present data that is uncontroversial. Huge existential threat to knowledge advancement.
This isn't just "Data Scientist" but scientist as well. The more a finding is in contradiction, either with existing scientific consensus or even with just popular culture, the more the science is criticized. I've seen unequal criticism based on how much people wanted the results to be true/false and even after responding to the criticism I've seen people just ignore science they don't like.
The skepticism isn't a problem, the unequal application of it, the potential to harm careers, and the chilling effect as people wisen to how best meet their own personal goals is.
> I used to joke that Data Scientists exist not to uncover insights or provide analysis, but merely to provide factoids that confirm senior management's prior beliefs.
The SNAFU principle: communication is only possible between equals. When an hierarchical divide exists the subordinate will tell the superior what he wants to hear.
>The SNAFU principle: communication is only possible between equals. When an hierarchical divide exists the subordinate will tell the superior what he wants to hear.
Sadly true. As humorously depicted here[0]:
In the beginning was the DEMO Project. And the Project was
without form. And darkness was upon the staff members
thereof. So they spake unto their Division Head, saying, "It
is a crock of shit, and it stinks."
And the Division Head spake unto his Department Head,
saying, "It is a crock of excrement and none may abide the
odor thereof." Now, the Department Head spake unto his
Directorate Head, saying, "It is a container of excrement,
and is very strong, such that none may abide before it." And
it came to pass that the Directorate Head spake unto
the Assistant Technical Director, saying, "It is a vessel of
fertilizer and none may abide by its strength."
And the assistant Technical Director spake thus unto the
Technical Director, saying, "It containeth that which aids
growth and it is very strong." And, Lo, the Technical
Director spake then unto the Captain, saying, "The powerful
new Project will help promote the growth of the
laboratories."
And the Captain looked down upon the Project, and He saw
that it was Good!
Economics is the go-to for conservative, status-quo maintaining arguments because there's a wealth of statistics and information available for "how things have always been", and precious little-to-none for "how things could be if..."
It's easier to poke holes in predictions of the future than in interpretations of the past, especially when the people making those decisions have likely reached their decision-making status through "how things have always been".
I think this depends a lot on the org. In a place I used to work we collected and analysed a lot of data which convinced management to significantly change the spec of the product and spend a lot more time and effort on testing, because the product was being used in unexpected ways.
I would say it was a very engineering driven org however, so if you could present compelling data it could go a long way.
Authoritarian types consider any information derived by science which is contrary to their position as invalid or irrelevant because facts challenge their authority and ability to exercise control.
yes. I used to think the Church had a honest disagreement with Galileo about heliocentricity. When I grew up I realized the Church never cared about orbits at all, what they care about is maintenance of status quo.
And then when I got old, I realized, there is even a reason that some people want status quo... because they have usually been around long enough to see society fall apart into anarchy and mass murder, so in their mind, they are doing the right thing.
"The Church" wasn't then and isn't now a monolith of opinion.
A modern characterisation of "The Galileo Affair" would be that he was SWAT'ed by someone he was really really mean online to.
Thus the whole "Galileo affair" starts as a conflict initiated by a secular Aristotelian philosopher, who, unable to silence Galileo by philosophical arguments, uses religion to achieve his aim. [1]
and
While delle Colombe was almost alone in arguing publicly against Galileo, there was a group of scholars and churchmen who supported his Aristotelian views. After Galileo referred disparagingly to delle Colombe as 'pippione' ('pigeon'), his close friend the painter Lodovico Cigoli coined the nickname 'Lega del Pippione' ('The Pigeon League') for delle Colombe's group.[2]
Galileo literally refered to delle Colombe (and friends) as Simplicio (simple minded) and worse in his highly popular Dialogue Concerning the Two Chief World Systems [3] and within a year or so the Pigeon League got their revenge, using their influence to have religuous charges bought against Galileo.
The affair was complex since very early on Pope Urban VIII had been a patron to Galileo and had given him permission to publish on the Copernican theory .. this was very much a case of personal vendettas and internal politics rather than a straight up case of "The Church Versus Galileo".
The church was not upset about heliocentrism. They were upset that Galileo was attempting to reinterpret the words of Bible in order to bolster his astronomical authority.
"[Therefore,] when God willed that at Joshua’s command the whole system of the world should rest and should remain for many hours in the same state, it sufficed to make the sun stand still. In this manner, by the stopping of the sun, the day could be lengthened on earth—which agrees exquisitely with the literal sense of the sacred text.”
>>I did several experiments, and noticed that whenever I produced analysis that was in line with what management expected - my analysis was praised and widely disseminated. Nobody would even question data completeness, quality, whatever. They would pick some flashy metric like a percentage and run around with it.
>> Whenever my analysis contradicted - there was so much scrutiny in numbers, data quality, etc, and even after answering all questions and concerns - analysis would be tossed away as non-actionable/useless/etc.
It's a good sign at the company that I run, anytime our analysts/data scientists come up with metrics that say we're killing it, or that our ideas should bear a ton of fruit, the kneejerk reaction is to be extremely skeptical of the results. Usually they're still right.
When the data scientists say we're fucking something up, we tend to pay a lot more attention.
I’ve been there, we wanted to release a feature, it kept coming back with issues that made it perform much worse than control, after 5 or so iterations with bug fixes it came back positive.
It took a lot of analysis and time to clarify to higher ups that we weren’t just P-hacking , but at least they were concerned about that.
I wonder what a data scientist could really find out about executive (over?) compensation. employee compensation. working from home. office cubicle size and layout. tool expenditure for employees vs productivity.
> I used to joke that Data Scientists exist not to uncover insights or provide analysis, but merely to provide factoids that confirm senior management's prior beliefs.
As a data scientist at a large corporate I find this is often the push… but I resist every time and tell people what they don’t want to hear. Maybe I’m playing this whole corporate ladder thing wrong :/
how is this really different from any other aspect of life? Very few people really like to be told counter information, and it is always easier when providing data that aligns with the current group think. Doesn't matter if it is business, politics, or really anything. Being the outlier trying to change the direction of things is a struggle.
I found it incredibly stressful to discover and provide analyses (even experimental results) that wasn't expected, or contradicted prior beliefs. The findings were always very harshly scrutinized, and typically lead to tons of pointless extra work to 'understand what is going on'.
That's like a portrait artist that finds success by painting people more beautiful than they really are vs a starving one that paints them true to life due to sense of artistic integrity.
Reminds me how Garth Brooks started doing metal after becoming a country music star.
My friend runs a successful market research agency and she says she gets called in when management have decided they need to make a change but need evidence to sell it to the shareholders and staff.
I mean, that makes sense does it not? If you're confirming something people already had a hunch about, why would they challenge it? And if it does go against their belief, they are going to want to make sure the data is correct before they change the course of the ship.
Agree with some of what you've said, but disagree with a lot:
> Most corporate decision making is highly political, the needs of/whats best for the business is just one parameter in a complex equation.
100% Individual humans are emotional creatures with their own wants and needs, and it's important to understand how organizational incentives drive decision making.
> Data Scientists are commonly disregarded by VPs in large corporations, despite the claims about being "data driven".
This has not been my experience, though. The more common thing I've seen is that, sometimes data is boring and doesn't really show much actionable insight, but as everyone wants to justify their job, I've seen data scientists come up with really questionable conclusions that fell apart on further inspection (call it "p-hacking the enterprise").
Plus, a lot of this data in these data wearhouses is messy. Often times data scientists are siloed at the end of the process, but then you get "garbage in/garbage out" results, where there is some bug in data tracking that isn't understood until it's too late. Much better in my opinion to have data engineers and data scientists work much more closely with product engineering teams up front so they can help ensure the data they collect is accurate.
What “data driven” normally looks like: business people ask questions that are not answerable, but the DS team often goes off and tries to answer them anyway. I’ve been in so many meetings, in multiple organizations, where a senior leader asks a question like, “why did this number go up? We need an owner to deep dive this.” Then, someone disappears for a few hours or days and comes back with some nice narrative that more or less might make some sense, and is directionally consistent. Then, best case, they go “hmm, ok” and worst case, they say it’s an interesting view and we need to add it to the tracking. Then, move on to a new question + action item, rinse and repeat. And then, 6 months or a year later, some brave soul goes, “geeze, there are way too many things to look at here, can we streamline this reporting to focus on the handful of most important metrics, and then assign owners to review the others and surface anything interesting?” And then all those things people asked for over time get stuffed into an appendix or deleted, and the whole thing starts all over again.
I find a lot of organizations don't have the discipline to harness whatever power their data may have. Sure collect everything, but god forbid you have any sort of data governance, or spend a single resource minute of time manually tagging or organizing or validating it. Then they try to make shitty ML models or products out of it, but don't care if the models actually work or not, just that they have AI now. Then a year later when the model has provided no value they are like, oh well big data is worthless I guess.
Oh, and a bunch of data scientists with zero domain knowledge for whatever data they are analyzing, preferably with PhDs in maths, but some ML background will do. And agile, because of course all those Palantir dashboards can only be developed using agile.
Once all is said and done, zero insight was created but a whole lot of consultants, contractors and project managers have been paid handsomely, while some higher ups can now put "implemented agile and big data at X" on their LinkedIn profiles.
I'm one of those PhDs with zero domain knowledge analyzing data and I share the sentiment.
Most of my analyses provide very little value because they are sort of common sense to people with domain knowledge. When I ask people what could be more useful, one of two things usually happen: 1) it's impossible due to data and/or infrastructure limitations, 2) what they ask turns out to be nonsensical in further analysis (like asking for average of something that follows a very fat tailed distribution with a few observations dominating the phenomenon. Of course it's usually impossible to explain this to people).
The more I think about this, the more I think that in truly data powered companies, both the decision making and data analysis have to be carried out by more or less the same people. The organizational hierarchies have to be much flatter. Essentially the employees will have to be some kind of "secret agents" who have both the skills and the mandate to steer the company in the direction they see fit. I sort of see this already happening in the FAANG companies where, or so I hear, it's very difficult to get hired, the staff count is quite small compared to traditional companies and the senior engineers have a lot of power in the company.
Using math PhDs or Palantir or whatever as a sort of modularized black box for "insights" while giving them no real skin-in-the-game does not work.
I can confirm this way of worling with data from my time at Amazon operations. We used data all the time, everywhere and for everything. But we did not have data scientist in our time, we did it ourselves. Quite peculiar, but so damn effecient and effective. I kind of miss that. It also showed that most of the data analytics stuff I know, Six Sigma, is just plain overkill for a lot of practical applications.
My favorite example is the WW2 bomber diagram shown to illustrate survivorship bias. Sure, working from data and first principle one could identify the vulnerable spots of the bomber. Or one could asl the designers or have an engineer, heck even a contemporary field mechanic, take a look at the actual drawings of the plane. And reach the same conclusion, faster, with a ton of additional insight and improvement ideas that can actually be implemented...
Are you going to take the time / money to set up a warehouse, get all the data into with an ETL product, set up dbt or some other transformation layer, set up a BI tool and build the reports and dashboards, etc.
Regardless the size of your data, you still need to get it in one place and model it in a way it's actually usable.
Exactly. It isn't just time to set up all the data in a way that makes the right query possible. It is also having queries fast enough to be able to run a vast number of them in order to find what you are looking for (or even things you were not looking for).
It’s moving the data around that is slow… and expensive. Getting the data into the data warehouse, then getting it to the processors then moving it around to filter and transform.
Getting your data to the cloud is expensive, but then you can’t do anything with it because distributing it to process in multiple stages is too expensive and you’re already paying so much to keep all that useless data.
Agree on the size not being the issue. I transitioned from a data engineering manager to a data product manager at a new company. You know how much data a typical customer generates? Less than 1TB a year.
I told my VP that the engineering foul ups in the current product are easily fixable. Standard tooling and patterns exist to re-architect and solve the bottlenecks. What is much harder is a data architect to make sense of the complex data and make sure there is good value for our customers.
Guess what position I don't have on the team, and won't have due to budget issues.
"customer obsession" was always at the mercy of the real obsession: "making money hand over fist". The former will ALWAYS lose out to the latter given enough cycles.
Yeah, "customer obsession" really just means "market share / growth obsession" which is a means to (eventually) making monopoly profits. Which Amazon seems to have achieved.
All the amazon corporate values are derived from making profit. Two-pizza teams? More like "three slices isn't frugal", one or two should be enough for you.
You can also “coast upwards” for quite awhile - as an example we now pay Microsoft less than we were for periodic upgrades to Office for the entire Microsoft 365 suite (including email hosting, etc) but all the machines are now Macs. They make more from us in one way, but less in total dollars.
The other thing to consider is that data simply has nothing of value. Part of the marketing of big data is the almost fairy tale belief in "insights" existing in any data set if you just look hard enough.
Correct, much data has no value. The cost for storing the data, maintaining the data [in the day-and-age of privacy requirements especially], and combing through the data is often much greater than the value obtained from the data itself.
The expertise we need in the industry is people who understand applications in-and-out and make great decisions on what data is worth keeping for present and future applications. And what data is needed to be kept, but only in aggregates (or anonymized, which reduces costs of maintenance)
"data" is usually just twisted to make leadership look good or justify what decision they wanted to make anyway. Analytics data is sliced at arbitrary time periods to make growth in whatever metric look good, certain subsets are just removed, etc.
doesn't help that most of this data goes through multiple layers of BS where each person is putting it through filters to make themselves look better. And a good chunk of people don't have enough understanding of stats to understand when they are being tricked
I think the recent industry layoffs reflect that in part.
Though I think there’s a better way— that is executive data science, what I do at Zapier. The key is that I’ve built up a huge amount of econometric and economic / business skills that I apply to affect good change in collaboration with company leaders. It allows me to work with Execs using sensible analysis. I improve growth and output by helping us catch errors of assumption before they go into production and cost growth / bad surprises. I also help the executives gain alignment around good information. This multiplies their departments’ output by allowing them to work better together, more in concert. That helps avoid issues with data being bent to decisions.
They typically carry a lot of hard-won valuable domain knowledge (that I combine with my economic-statistic knowledge and skills for rigor).
It’s my job to ensure Execs start with good sensible information regarding objectives. They usually ask fantastic questions and share a lot of great analysis of their own.
There are times I learn about what I might call controversial implications. This is typical of innovation using technology. It’s in these moments that I feel I create the most value by highlighting the trade offs I believe we face / potential regret.
I get to apply my causal inference and measurement skills to a lot of compelling situations in collaboration with some really intelligent and skilled people. I truly feel like I get to perform technically excellent work, and I feel very privileged in that way.
My day to day is mostly economic / econometric analysis, conceptualization and formulation of technology, and writing, but also occasional meetings to gather information, feedback, answer questions, teach and collaborate with others.
I try to effectively “sample” the org by interacting with people up and down the org, so I can assist Execs in incorporating directors, managers and ICs knowledge / pool good information / systematize planning, etc.
> Data Scientists are commonly disregarded by VPs in large corporations, despite the claims about being "data driven".
I worked in a "data-driven" company of 5,000+ employees, with 4 data scientists who were spilt into two separate non-collaborating teams. In effect, they were so under resourced they got nothing done.
I've found this to be true in the (albeit small) # of samples I've run up against.
There's an agenda by some level of management, and they use "data" to forward their agenda, or disregard it due to "unexplainability" (a legitimate concern for BD/ML) if it disagrees with the agenda.
I would argue that insights from statistical models “[are] just one parameter in a complex equation.” No doubt PR from political issues can help the bottom line of a company. Can also hurt it. Just to be clear I’m not arguing political choices are inherently good or bad.
Replace “data scientists” with “software engineers” and you have another accurate insight. They don’t want to listen to us about how to write software or derive value from data.
The real issue is that business people usually ignore what the data says. Wading through data takes a huge amount of thought, which is in short supply. Data Scientists are commonly disregarded by VPs in large corporations, despite the claims about being "data driven". Most corporate decision making is highly political, the needs of/whats best for the business is just one parameter in a complex equation.