To me, it's not about magic, it's about the abstraction. Most implementations of this style of database demand that you consider internals of the engine before storing anything. If you don't design your keys the right way, then you're screwed down the road, and there's no way to fix it without just copying the data to a new table (and losing metadata like TTL). When I use something like Redis, or any SQL databse, I don't need to consider the inner workings of it unless I'm trying to absolutely max out my performance (one slight exception being atomicity of Redis commands).
But with LSM style databases, you are going to have a really bad time if you don't have a team member who has dedicated serious time to understanding the internal workings of LSM itself, and the details of your chosen implementation. That's a real mark against it, IMO.
tldr; LSM databases are like a colander in the world of leaky abstractions.
There is MyRocks, you work like usual mysql. When you want distrubted-db, yeah you need to design. But comparing RocksDB vs LMDB, the second is easier but not by much ?
Think citusdb without consideration you're gonna expect good performance in sharded environment?
> When you want distrubted-db, yeah you need to design
But one shouldn't be designing for implementation details. Usually, we start technologies out with leaky abstractions, and gradually get better at it. A good example is game development, where it used to be that you always used the drawing method of the display and the clock speed of the cpu to your advantage. Nowadays, we've moved past that, because it was working on horrible abstractions, and because the technology underneath improved.
I'm not saying I have a solution, and I agree that this problem rears it's head the most when you start bringing in distributed storage. But my point stands: these databases run on a highly leaky abstraction, and that's a big problem going forward.
