cbfs - a couchbase large object store
Thirteen days ago, Marty Schoch and I created an empty
directory called cbfs and started typing some go code into
it. The idea was to create an answer to the frequent question, “how
do I store large items in Couchbase?”
I think we’re both pretty pleased with the results and would like to share what we’ve made a bit more broadly.
cbfs is essentially a read/write HTTP server with a minimal RESTful API for getting a bit more meta information out of it and serving apps on couchbase. We had a few goals and were able to demonstrate almost all of them within a week of birth. Things like:
- No single point of failure.
- No single point of contention.
- No limits to the size or volume of data that can be served.
- No invalid states to reconcile.
- Adding and removing nodes and other management tasks should be as easy as possible.
Also, we wanted to be able to bring back something along the spirit of couchapps (but much easier!).
We haven’t written up enough on it yet, but I did do a demonstration to our team today in a google hangout. You can see that quick intro in the first half of the following video.
The slides are available as well. They are meant to be served from a cbfs instance doing the demo, but I made the demos also work when offline.
Some neat stuff has been built around this since we started.
Trond Norbye wrote a FUSE interface so we could mount it locally. I’m synchronizing our Dropbox stuff into cbfs and watched Trond browse around in a terminal and interact with the files as if they were local.
Marty’s written a pretty awesome admin console. I’ve done some demos of other stuff built on the API as well (including the built-in cluster monitoring console and the commandline tool).
The wiki page was mostly written in the first four days, but describes what the idea was pretty well. It’s a lot more complete now, though the issues list shows where we want to add more polish.
I’m running a cluster at home and in the office and serving content and apps out of it, so it’s definitely self-hosting and stuff.
Come join us, help finish tasks, think of new ones, find ways you think it could be better…
But, as a reminder, this isn’t two weeks old yet, so if you want something that’s actually been in production with PBs of data, look at mogilefs. We wrote this because we wanted it to exist and wanted to have some answers for specific questions we’d been asked.
