> Don’t ask questions that can be answered with a web search.
These days, that’s equivalent to saying “don’t ask questions about what the day to day job requires one to know”.
Sites like stackoverflow contain answers to problems that people encounter in their day to day jobs, and if you eliminate those as potential questions, then what should you ask? And what is the value in asking some obscure question, other than making the interviewer feel smart and special?
Here's a question you can't properly answer with a web search for a data engineering role
"I'm emitting 10TB of data a day with this schema {X} - it's produced in Avro. Currently we upload it into S3. But, instead, I want to stream into Apache Druid, but also want to make it available for ad-hoc analysis by BAs.
Your question turns into a question of "have you worked with Avro, S3, Apache Druid..." which will filter out competent people that don't have experience with the specific list of tools.
The keywords there were "streaming" and "also ad-hoc analysis".
Answering those questions requires no knowledge of Avro or Druid, but rather, how to build streaming pipelines, and also make the data available for a query engine.
But that said, if you're in data engineering and are unaware of what Avro is, you're probably not at the level of knowledge I'd expect, and I'm happy to filter you out.
Likewise, if you're unfamiliar with S3, why are you even applying for a data engineering role?
But how can I be familiar with S3 without any data engineering work? Or maybe I should put it like this, what exactly does "familiar with S3" mean? Does it mean that I used it to store something (I mean even a non-programmer can do that), or that I should be able to manipulate (i/o) the store using Python with Boto (Again this can be done with a tutorial), or something else?
I believe that "something else" usually indicates some previous data engineering work though, so it's an egg and chicken problem.
I think it's a bit too specific. But for sure someone can relate to similar experience. Something like "For sure I haven't dealt with Druid specifically but in my job I tackled with a similar problem for a different set of tools...".
BTW from my experience we are streaming a lot of data (similar to 10TB but definitely more) produced in JSON then converted to Avro using Spark and finally loaded into Vertica and the analysis pipeline. I have never used Druid but from Googling it is a column based data store which I guess is not too different from Vertica. Is there any specific difficulty you think would be encountered with Druid? Because the wording seems to indicate that Druid is not particularly good with ad-hoc analysis.
> Because the wording seems to indicate that Druid is not particularly good with ad-hoc analysis
Bingo ;)
I was trying to phrase it like I've heard people do so in interviews.
And tbh, a candidate saying "I'm not familiar with Druid, but it sounds like you can't use it do ad-hoc queries with it? Can you tell me more?" would be a very good sign.
The best ones are the one that take the CV and for each project listed they ask you to say how it was and then some tehnical challenges you encountered. Depending on how you lay out the problems, a competent interviewer will know at what level you are.
These days, that’s equivalent to saying “don’t ask questions about what the day to day job requires one to know”.
Sites like stackoverflow contain answers to problems that people encounter in their day to day jobs, and if you eliminate those as potential questions, then what should you ask? And what is the value in asking some obscure question, other than making the interviewer feel smart and special?