large unstructured blobs and large files are among the things not well suited to foundationdb and couchdb 4 actually reduced supported blob size in the transition to foundationdb. it looks like object/blob storage systems are at the moment rather seperating more from key/value and document storage than growing together. but this is a good thing because the tradeoffs are very different and it allows each system to focus on what it does best. blob stores will hopefully move even more to content addressing and merkle dag similar to git and ipfs.

Can you elaborate on what requirements these blob systems should have?

My understanding is object stores are typically “flat” by design to scale well (in contrast to a tree structure found in filenames).

For content addressing, are people using the keys in sophisticated ways or are the values being indexed? Any reason to push this complexity into the storage layer as opposed to composing the functionality?

well there is often no hierarchy as in folders on an old school filesystem but if the system uses chunking, the organization of blob chunks is very important for the performance and scaling characteristics. The chunking algorithm needs to be performant but also lead to sensible chunk size and count and in addition can also do data based boundaries so chunks can be reused even if blob data changes at the start of the data. This can be different depending on your specific application (eg. the read/write ratio and average file sizes) and requirements for optimal use of the underlying filesystem, that's one reason why no de facto standard chunker has been established so far. There are many tradeoffs for key organization too. Do you need more sophisticated range queries or only single keys? How balanced is growing and shrinking of your data structure vs performance? What is the clustering story? How do you handle rebalancing/cleanup/pruning? Is your primary key organization content hashes like in ipfs or more arbitrary strings as in s3/minio? Is your metadata/ secondary keys system completely integrated or more independent?

Thats exactly what your last questions points to. If you are lets say dropbox and have probably a super sophisticated key value store setup i can imagine you would want your content addressable layer to be as simple and narrowly optimized as possible and develop and optimize the indexing, metadata and key queries system nearly completely separately. If you are working on some system that also should scale down to run on individual machines like ipfs, git annex or minio before their focus on kubernetes you want a system that can run as a single daemon but also where users can reason about the whole system as an integrated concept.

I see. Some of these, like ipfs, are more general purpose systems (basically communication protocols) than I was thinking of.

I’m curious about this as well. Is anyone working on building text search on top of FDB? It’s kind of astounding to me that last time I checked Elasticsearch was still essentially the only game in town.

its pretty hard to catch up with lucene, there is just so much work, features and brainpower in there at this point. as many features of foundationdb such as the transaction guarantees and reliability are not super important for fulltext search i cannot imagine any company even apple or ibm being able to justify that gigantic investment, instead im sure nearly any soluion willcontinue to use lucene under the hood for the forseeable future.

Tantivy is giving it a good go: https://github.com/tantivy-search/tantivy

The good thing about Lucerne existing is you’re allowed to also use their good ideas.

Lucene is Java, right? There should be space for a native implementation, like ScyllaDB is doing to Cassandra (and DynamoDB, though the gap is not of the same shape in the last case). Or am I missing something?

I used ElasticSearch and run one cluster in production in the past and found it horrible. Maybe I'm missing something, since they're so successful even as a public company...

There’s a lot to Lucene, and there’s also a lot to ElasticSearch. And I think they’re fairly tightly coupled.

But I do think that a well-funded and skilled startup team could take a run at ES. They’re a monopoly in their niche. There has to be money in disrupting them.

Tantivy (written in Rust) uses MIT. I can't speak to the quality of the code, but their aim is to rewrite Lucene-type functionality as a Rust crate.

https://github.com/tantivy-search/tantivy

yes its java and also yes there is definitely a space for a native implementation, its not that i don't want that to happen, on the contrary. but the reality is that its really hard to do and i dont see only getting rid of java to currently motivate a relevant player or large enough dev community.

Tantivy? Written in Rust. More like Lucene than Elastic, though.

https://github.com/tantivy-search/tantivy

just out of interest, what was horrible about es?

It was brittle beyond belief. The process would run out of memory and crash really easily. You are supposed to dimension resources well, but of course you'd like to handle things more gracefully when a huge inflow of data or requests comes in, good engineering is "graceful failure" such as just rejecting requests, instead of total mayhem and destruction. It was "interesting" to discover that the monitoring plugins included by default would bring down the entire cluster. 5 beefy AWS machines brought down when pointing their own monitor at them! Yes there was significant input traffic, but I would expect more solid behavior from production-ready software.

Shard migration and duplication was a beauty to watch when it worked well, but very often it wouldn't. I tried to establish a procedure to bring instances back up safely and repeatably, but sometimes I would just have to try and try again, no procedure guaranteed restoring operation in all cases. I do remember spending more than one weekend babysitting that cluster. There's a special place in my heart for that memory, and it's not a beautiful one.

I was very surprised to see them IPO successfully - this test was before they were public. I assumed the open source version was just missing the kind of tools and know how to be able to work solidly without paying them or their consultants. These days, I think it's poor engineering resulting in brittleness, we shouldn't have put so much traffic to it, and probably everyone out there experiences the same behavior unless they ensure much safer margins, which probably makes it expensive but I guess who's complaining.

I ended up designing and writing my own (much simpler) distributed system for what I needed (much simpler than everything ElasticSearch does, of course), and it's been in reliable operation since summer 2015. Good engineering goes a long way. Now in the process of turning that approach into a startup. If I get 1/10th as successful as ES I will be a happy camper :)

peruse the fdb forum. they produce document and record layers now. there are community layers of varying quality for a network block device, a filesystem, and a few other things.