Dustin Sallings2023-10-17T07:32:50+00:00http://dustin.github.com/Dustin Sallingsdustin@spy.netGoProFS - All My Video2023-10-16T00:00:00+00:00http://dustin.github.com/2023/10/16/goprofs<h1 id="goprofs">GoProFS</h1>
<p>So, I end up shooting a lot of video. As I mentioned way back in my
<a href="/2020/04/29/gopro-plus.html">GoPro Plus</a> post, I mostly shoot with a GoPro and use the
GoPro+ service.</p>
<p>I’d been backing up all of my cloud stuff to a USB hard drive on a
Raspberry pi. That allows me to keep all of my Davinci Resolve
projects around and reference media without having to do anything
weird.</p>
<p>However, I ran into a problem.</p>
<div>
<img src="/images/goprostor.png" alt="storage dashboard" title="GoPro Storage Dashboard" width="100%" />
</div>
<p>Looks like I just crossed 18.5TB raw media. The 20TB hard drive I’ve
got seems to provide a bit over 17TB of raw storage.</p>
<h2 id="obvious-solutions">Obvious Solution(s)</h2>
<p>Now, there are a few obvious things one might do in this situation.
I’ll briefly describe why I didn’t do any of these things.</p>
<h3 id="create-a-raid0-thing">Create a Raid0 Thing</h3>
<p>I <em>could</em>, but if any disk fails, I lose all the things. Also, my
Raspberry Pi doesn’t have enough USB (or memory for that matter).</p>
<h3 id="create-a-proper-raidz-pool">Create a Proper RAIDZ Pool</h3>
<p>In addition to the USB/Memory issues, this is now getting rather
expensive. I’m already paying GoPro to store my media, creating an
expensive local solution isn’t desirable.</p>
<p>I do <em>have</em> a TrueNAS box with raidz2 (and am using it for some
overflow), but that’s expensive storage. This is <em>mostly</em> a backup.</p>
<h3 id="use-davinci-resolve-media-management">Use Davinci Resolve Media Management</h3>
<p>I could ask Davinci Resolve to pull out only the parts of media that I
actually use and give me a much smaller set of media to work with.
That works for some projects, but I sometimes go back and find things
that I missed when I was first reviewing footage.</p>
<p>But in general, I want everything easily available in source form.</p>
<h2 id="ok-so-what-is-goprofs">OK, So What is GoProFS?</h2>
<p>My initial thought was to build a FUSE-based filesystem that provides
access to data stored in the GoPro cloud as a single magic mechanism.
I built that and it’s pretty great. I wrote the bulk of it in <code class="language-plaintext highlighter-rouge">go</code>
because I didn’t have a Haskell FUSE library that worked on OS X and
wanted to get a prototype up and running instead of solving this
library deficiency, while still being able to use the bulk of my
Haskell code via the web interface.</p>
<p>It does more than just provide cloud access, but let’s just start with
that.</p>
<h3 id="cloud-file-access">Cloud File Access</h3>
<p><code class="language-plaintext highlighter-rouge">goprofs</code> will create a local directory that will give you a tree like
this:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>ls -l $GOPROFS/2014/07
total 0
dr-xr-xr-x 1 root wheel 0 Jul 1 2014 m0ao4NWMQBWvn
dr-xr-xr-x 1 root wheel 0 Jul 1 2014 pJ0WlypDKXJX1
</code></pre></div></div>
<p>where each directory contains the source media you uploaded to the
service.</p>
<p>The contents are fetched dynamically in chunks. For example, if you
open an mp4 file with QuickTime Player, you’ll find that it reads a
bit of the beginning of the file and then a bit from the end of the
file, then a few other parts near the end, then the beginning again,
then more near the end for a while, etc…</p>
<p>If you’re opening a 10GB file, it’d be rather unfortunate to have to
wait to pull the entire content down just to see the first frame, so
the underlying mechanism slices the file into 8MB (arbitrary number)
chunks and maps each byte range requested to a series of chunks,
blocking reads until all of those chunks are satisfied. The chunks
themselves are stored in a sparse file for the duration of the
reader’s session, pulling any new blocks the user may request.</p>
<p>When the application using the file is done with it, we check to see
if we’ve found all the blocks. If we have, then we rename the file to
a permanent “this is all the parts” name. Otherwise, we write down
which blocks we had and reload them for the next file read.</p>
<p>This means you only need the parts of the video you are actually
using.</p>
<h3 id="but-i-already-downloaded-everything">But I Already Downloaded Everything</h3>
<p>Since I’ve been using my <code class="language-plaintext highlighter-rouge">backuplocal</code> command, I already had a bunch
of the footage locall (until I ran out of disk), so if I’ve already
got a file, I should be able to just use that instead of bothering the
GoPro Plus service for a signed URL to download it again. <code class="language-plaintext highlighter-rouge">goprofs</code>
has a <code class="language-plaintext highlighter-rouge">-source</code> flag allowing it to check for a local file (in the
same file layout as the backups) before going to the internet. Better
yet, you can supply <code class="language-plaintext highlighter-rouge">source</code> multiple times to have it look in
multiple places.</p>
<h3 id="why-is-my-data-in-multiple-places">Why Is My Data in Multiple Places?</h3>
<p>It turns out, it was quite easy to get the <code class="language-plaintext highlighter-rouge">backuplocal</code> command to,
itself, look in multiple places to see what it has, and then write to
one of those. So I can just add another storage location and do
something kind of like striping where the backup itself can be over
multiple distinct filesystems.</p>
<p>This is similar to the raid0 solution discussed above, but without any
direct filesystem support or even needing the sources to be on the
same host. In my case, I’ve got my old big USB disk on my Raspberry
Pi and a bit of overflow on my raidz2 volume until I can get another
cheap disk.</p>
<h3 id="tldr-where-does-it-look">tl;dr: Where Does it Look?</h3>
<p>For a given artifact, it checks:</p>
<ol>
<li>Each <code class="language-plaintext highlighter-rouge">-source</code> location</li>
<li>Its download cache location</li>
<li>Uses the cloud and grabs individual chunks as they’re read.</li>
</ol>
<h3 id="sounds-like-it-does-a-lot">Sounds Like it Does a Lot!</h3>
<p>It does! But there’s one more feature: Proxies</p>
<p>Davinci Resolve will automatically link a proxy for <code class="language-plaintext highlighter-rouge">F.MP4</code> if it
finds a file named <code class="language-plaintext highlighter-rouge">Proxy/F.MP4</code> (or <code class="language-plaintext highlighter-rouge">Proxy/F.mov</code> or whatever).
While most of the filesystem is read-only (since it’s showing you your
own prerecorded content with no ability to break stuff), we do want to
be able to create these <code class="language-plaintext highlighter-rouge">Proxy</code> directories. So we do that with a
read/write loopback for the <code class="language-plaintext highlighter-rouge">Proxy</code> directory of any medium overlaying
a different disk location. This is pretty important when you are
trying to pull the sizes of stuff we’re talking about.</p>
<h2 id="sounds-awesome--hows-it-work">Sounds Awesome. How’s it Work?</h2>
<p>So far, it works as I’d expect, <em>but</em> Blackmagic Proxy Generator seems
to hang sometimes while generating proxies. As far as I can tell,
it’s not my bug, but I can’t tell what it’s doing because OS X is
pretty hostile to system introspection these days. I may just try
different software for this.</p>
<p>It’s in the goprofs tree of my <a href="https://github.com/dustin/gopro">gopro project</a> on github.
Note that it does require my Haskell GoPro web services to be running
as an endpoint. I could make it talk directly to the GoPro Plus
service if I wanted to, but that’s a lot of work to reproduce what
I’ve already been running for years and doing so in a language that’s
harder to work in.</p>
Monads are Tedious in Go2021-06-23T00:00:00+00:00http://dustin.github.com/2021/06/23/monads-are-tedious-in-go<h1 id="monads-are-tedious-in-go">Monads are Tedious in Go</h1>
<div>
<img src="/images/gonad.png" alt="mqtt" title="Monads in Go" class="floatleft" width="200" height="272" />
</div>
<p>Most of my personal projects are written in Haskell these days.
I’ve heard people say “Haskell is hard” or whatever for a long time,
but the reason I write most of my projects in Haskell isn’t because
I’m smart and want to do the most impressive smart person thing
possible, but because I’m dumb and want better tools to help me
understand things more easily and avoid the kind of bugs that dumb
people like me write a lot.</p>
<p>On any given work day, I review at least one piece of
<a href="https://golang.org">go</a> code. Go kind of has a similar thing in
theory about not having “clever” features that might confuse people.
Some of this is really nice, but some of it is tedious. I’m going to
get into the latter a bit here.</p>
<p>Much of the code I end up reviewing contains many reimplementations of
<code class="language-plaintext highlighter-rouge"><$></code> or <code class="language-plaintext highlighter-rouge">>>=</code>, sometimes buggy. These are scary looking things to
someone who doesn’t write any Haskell, but they’re so fundamental to
what most code is doing that you just absorb them quickly.</p>
<p>This post isn’t meant to be a tutorial on Haskell operators, but <code class="language-plaintext highlighter-rouge"><$></code>
is also spelled <code class="language-plaintext highlighter-rouge">fmap</code> and basically means “apply this function inside
that thing.” e.g., you might apply a function to each element of a
list, or to the value inside an optional (think nullable pointer).
<code class="language-plaintext highlighter-rouge">>>=</code> is the monadic “bind” operator and basically is used to combine
monadic actions.</p>
<p>At this point, you’re either confused by jargon or angry for how
inaccurate my descriptions are. I intend to make things clearer as I
write. I’m actually intending to write more about go, so let’s get
going.</p>
<h2 id="error-handling-in-go">Error Handling in Go</h2>
<p>I actually rather like how error handling works in go. Mostly. I’ve
worked in lots of languages with exceptions and I’ve disliked most
exception handling I’ve encountered (including Haskell). You either
end up with a completely opaque error path that you are unable to
reason about (e.g., any line of code may fail with any exception) or
you end up with an incomplete list of exceptions you might have to
care about, but generally can’t do anything about at a particular site
(e.g., java checked exceptions which is always an incomplete list).</p>
<p>In go, a function that might fail will return an error. This is
really nice. You can see what might fail and decide what to do about
it. In most cases, you just pass it up, but you don’t have the
situation where you’ve forgotten to handle a particular exception type
and your program crashes instead of just failing gently.</p>
<p>There are two downsides, though:</p>
<ol>
<li>You have to add the dreaded <code class="language-plaintext highlighter-rouge">if err != nil { return err }</code> code
everywhere.</li>
<li>You have to check errors and ignore values on error by
<em>convention</em>.</li>
</ol>
<p>The first point is mildly annoying. It seems unnecessary and there’s
been <a href="https://github.com/golang/go/issues/32437">exhaustive discussions</a> around how to
improve the particular case. Looking at it with my Haskell glasses
on, it seems really weird to even consider writing a special case
built-in just to cover what is a super generic concern. This all for
what is just a special use case of <code class="language-plaintext highlighter-rouge">>>=</code>.</p>
<p>The second point comes up a lot in code review. You’re not supposed to
use values if you also got an error. You’re not supposed to <em>return</em>
a useful value if you wish to return an error. This is a bit beyond
the scope of what I wanted to discuss here, but it’s something you
have to consider every single time you return a <code class="language-plaintext highlighter-rouge">(T,error)</code> and every
time you receive one. The easy way to demonstrate what this code
<em>could</em> be accidentally doesn’t have this problem either, so I wanted
to bring it up.</p>
<h2 id="idiomatic-go-is-the-either-monad-sort-of">Idiomatic Go is the Either Monad (sort of)</h2>
<p>In go, if you want to return a value of type <code class="language-plaintext highlighter-rouge">T</code> or an error, you
generally return <code class="language-plaintext highlighter-rouge">(T, error)</code> and expect the user to only use one of
those values. I’m going to contrast this to Haskell’s <code class="language-plaintext highlighter-rouge">Either</code> type
which is <em>almost</em> the same. <code class="language-plaintext highlighter-rouge">Either a b</code> can give you either <code class="language-plaintext highlighter-rouge">Left a</code>
or <code class="language-plaintext highlighter-rouge">Right b</code> (<code class="language-plaintext highlighter-rouge">a</code> and <code class="language-plaintext highlighter-rouge">b</code> are types here). The primary difference is
the second point above… you <em>can’t</em> get both values. You either
get an error or value.</p>
<p>As a Monad, <code class="language-plaintext highlighter-rouge">Either a</code> will effectively short-circuit any failure and
continue forward with any value.</p>
<p>Let’s look at an example where we have a function that takes two
numbers as strings, adds them together, and returns the value as a
string:</p>
<div class="language-go highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">func</span> <span class="n">readInt</span><span class="p">(</span><span class="n">s</span> <span class="kt">string</span><span class="p">)</span> <span class="p">(</span><span class="kt">int</span><span class="p">,</span> <span class="kt">error</span><span class="p">)</span> <span class="p">{</span>
<span class="k">return</span> <span class="n">strconv</span><span class="o">.</span><span class="n">Atoi</span><span class="p">(</span><span class="n">s</span><span class="p">)</span>
<span class="p">}</span>
<span class="k">func</span> <span class="n">add</span><span class="p">(</span><span class="n">a</span><span class="p">,</span> <span class="n">b</span> <span class="kt">string</span><span class="p">)</span> <span class="p">(</span><span class="kt">string</span><span class="p">,</span> <span class="kt">error</span><span class="p">)</span> <span class="p">{</span>
<span class="n">ai</span><span class="p">,</span> <span class="n">err</span> <span class="o">:=</span> <span class="n">readInt</span><span class="p">(</span><span class="n">a</span><span class="p">)</span>
<span class="k">if</span> <span class="n">err</span> <span class="o">!=</span> <span class="no">nil</span> <span class="p">{</span>
<span class="k">return</span> <span class="s">""</span><span class="p">,</span> <span class="n">err</span>
<span class="p">}</span>
<span class="n">bi</span><span class="p">,</span> <span class="n">err</span> <span class="o">:=</span> <span class="n">readInt</span><span class="p">(</span><span class="n">b</span><span class="p">)</span>
<span class="k">if</span> <span class="n">err</span> <span class="o">!=</span> <span class="no">nil</span> <span class="p">{</span>
<span class="k">return</span> <span class="s">""</span><span class="p">,</span> <span class="n">err</span>
<span class="p">}</span>
<span class="k">return</span> <span class="n">strconv</span><span class="o">.</span><span class="n">Itoa</span><span class="p">(</span><span class="n">ai</span> <span class="o">+</span> <span class="n">bi</span><span class="p">),</span> <span class="no">nil</span>
<span class="p">}</span>
</code></pre></div></div>
<p>(I included <code class="language-plaintext highlighter-rouge">readInt</code> just so the types are visible)</p>
<p>This is pretty straightforward, idiomatic go. The rough equivalent in
Haskell would look something like this:</p>
<div class="language-haskell highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">readInt</span> <span class="o">::</span> <span class="kt">String</span> <span class="o">-></span> <span class="kt">Either</span> <span class="kt">String</span> <span class="kt">Int</span>
<span class="n">readInt</span> <span class="o">=</span> <span class="n">readEither</span>
<span class="n">add</span> <span class="o">::</span> <span class="kt">String</span> <span class="o">-></span> <span class="kt">String</span> <span class="o">-></span> <span class="kt">Either</span> <span class="kt">String</span> <span class="kt">String</span>
<span class="n">add</span> <span class="n">a</span> <span class="n">b</span> <span class="o">=</span> <span class="kr">case</span> <span class="n">readInt</span> <span class="n">a</span> <span class="kr">of</span>
<span class="kt">Left</span> <span class="n">x</span> <span class="o">-></span> <span class="kt">Left</span> <span class="n">x</span>
<span class="kt">Right</span> <span class="n">ai</span> <span class="o">-></span> <span class="kr">case</span> <span class="n">readInt</span> <span class="n">b</span> <span class="kr">of</span>
<span class="kt">Left</span> <span class="n">x</span> <span class="o">-></span> <span class="kt">Left</span> <span class="n">x</span>
<span class="kt">Right</span> <span class="n">bi</span> <span class="o">-></span> <span class="kt">Right</span> <span class="p">(</span><span class="n">show</span> <span class="p">(</span><span class="n">ai</span> <span class="o">+</span> <span class="n">bi</span><span class="p">))</span>
</code></pre></div></div>
<p>That’s kind of worse in that it seems to march off to the right. What
if we wanted to add three numbers!?</p>
<p>But <code class="language-plaintext highlighter-rouge">Either a</code> is a monad, so we can use <code class="language-plaintext highlighter-rouge">>>=</code> to get things done.
This is the equivalent function without <code class="language-plaintext highlighter-rouge">case</code>:</p>
<div class="language-haskell highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">addM</span> <span class="o">::</span> <span class="kt">String</span> <span class="o">-></span> <span class="kt">String</span> <span class="o">-></span> <span class="kt">Either</span> <span class="kt">String</span> <span class="kt">String</span>
<span class="n">addM</span> <span class="n">a</span> <span class="n">b</span> <span class="o">=</span>
<span class="n">readInt</span> <span class="n">a</span> <span class="o">>>=</span> <span class="nf">\</span><span class="n">ai</span> <span class="o">-></span>
<span class="n">readInt</span> <span class="n">b</span> <span class="o">>>=</span> <span class="nf">\</span><span class="n">bi</span> <span class="o">-></span>
<span class="n">pure</span> <span class="o">.</span> <span class="n">show</span> <span class="o">$</span> <span class="n">ai</span> <span class="o">+</span> <span class="n">bi</span>
</code></pre></div></div>
<p>This does the same thing – we’ll either get <code class="language-plaintext highlighter-rouge">ai</code> as an int, or it’ll
short-circuit the rest of the function and return the error we got
from trying to parse the value. i.e., it does all the <code class="language-plaintext highlighter-rouge">if err != nil
{ return err }</code> bits for you.</p>
<p>Much like in the initial version, <code class="language-plaintext highlighter-rouge">ai</code> is the <code class="language-plaintext highlighter-rouge">Int</code> form of of the
<code class="language-plaintext highlighter-rouge">String</code> <code class="language-plaintext highlighter-rouge">a</code> as <code class="language-plaintext highlighter-rouge">bi</code> is for <code class="language-plaintext highlighter-rouge">b</code>. <code class="language-plaintext highlighter-rouge">ai</code> and <code class="language-plaintext highlighter-rouge">bi</code> are arguments to
lambda functions that do stuff with those <code class="language-plaintext highlighter-rouge">Int</code> values. It should be
clear here that there’s no possible way to get <code class="language-plaintext highlighter-rouge">ai</code> if <code class="language-plaintext highlighter-rouge">readInt</code>
returned a <code class="language-plaintext highlighter-rouge">Left</code> (error), so the only thing the code <em>can</em> do is
return that error and not push the value into the next lambda.
There’s not a way to get this wrong.</p>
<p>In the wild, you’d probably be more likely to see this written in the
<code class="language-plaintext highlighter-rouge">do</code> syntax, which is just syntactic sugar for the above:</p>
<div class="language-haskell highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">addM'</span> <span class="o">::</span> <span class="kt">String</span> <span class="o">-></span> <span class="kt">String</span> <span class="o">-></span> <span class="kt">Either</span> <span class="kt">String</span> <span class="kt">String</span>
<span class="n">addM'</span> <span class="n">a</span> <span class="n">b</span> <span class="o">=</span> <span class="kr">do</span>
<span class="n">ai</span> <span class="o"><-</span> <span class="n">readInt</span> <span class="n">a</span>
<span class="n">bi</span> <span class="o"><-</span> <span class="n">readInt</span> <span class="n">b</span>
<span class="n">pure</span> <span class="o">.</span> <span class="n">show</span> <span class="o">$</span> <span class="n">ai</span> <span class="o">+</span> <span class="n">bi</span>
</code></pre></div></div>
<p>Note that this isn’t a Haskell language feature that knows how to do
fancy stuff with <code class="language-plaintext highlighter-rouge">Either a</code> – that’s just how the library is
defined. You can make your own monad that works differently (as long
as it’s <a href="https://wiki.haskell.org/Monad_laws">lawful</a>). The
definition for Either is just this:</p>
<div class="language-haskell highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="kr">instance</span> <span class="kt">Monad</span> <span class="p">(</span><span class="kt">Either</span> <span class="n">e</span><span class="p">)</span> <span class="kr">where</span>
<span class="kt">Left</span> <span class="n">l</span> <span class="o">>>=</span> <span class="kr">_</span> <span class="o">=</span> <span class="kt">Left</span> <span class="n">l</span>
<span class="kt">Right</span> <span class="n">r</span> <span class="o">>>=</span> <span class="n">k</span> <span class="o">=</span> <span class="n">k</span> <span class="n">r</span>
</code></pre></div></div>
<p>i.e., if we get a <code class="language-plaintext highlighter-rouge">Left l</code>, we ignore our second param (the function)
and return the <code class="language-plaintext highlighter-rouge">Left l</code>. If we get a <code class="language-plaintext highlighter-rouge">Right r</code>, we pass <code class="language-plaintext highlighter-rouge">r</code> to that
function (named <code class="language-plaintext highlighter-rouge">k</code> here).</p>
<p>Of course, I wouldn’t write it that way either, since monads are also
applicative functors. My brain automatically rewrites that using
<a href="https://hackage.haskell.org/package/base-4.15.0.0/docs/Control-Applicative.html#v:liftA2">liftA2</a>:</p>
<div class="language-haskell highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">addA</span> <span class="o">::</span> <span class="kt">String</span> <span class="o">-></span> <span class="kt">String</span> <span class="o">-></span> <span class="kt">Either</span> <span class="kt">String</span> <span class="kt">String</span>
<span class="n">addA</span> <span class="n">a</span> <span class="n">b</span> <span class="o">=</span> <span class="n">show</span> <span class="o"><$></span> <span class="n">liftA2</span> <span class="p">(</span><span class="o">+</span><span class="p">)</span> <span class="p">(</span><span class="n">readInt</span> <span class="n">a</span><span class="p">)</span> <span class="p">(</span><span class="n">readInt</span> <span class="n">b</span><span class="p">)</span>
</code></pre></div></div>
<p>Again, same error handling, etc…</p>
<h2 id="monadic-go">Monadic Go</h2>
<p>Now, imagine we had a similar monadic functionality in go. We’d
write something like:</p>
<div class="language-go highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">func</span> <span class="n">add</span><span class="p">(</span><span class="n">a</span><span class="p">,</span> <span class="n">b</span> <span class="kt">string</span><span class="p">)</span> <span class="p">(</span><span class="n">ErrorOr</span><span class="p">[</span><span class="n">String</span><span class="p">])</span> <span class="p">{</span>
<span class="n">ai</span> <span class="err">⩴</span> <span class="n">readInt</span><span class="p">(</span><span class="n">a</span><span class="p">)</span>
<span class="n">bi</span> <span class="err">⩴</span> <span class="n">readInt</span><span class="p">(</span><span class="n">b</span><span class="p">)</span>
<span class="k">return</span> <span class="n">strconv</span><span class="o">.</span><span class="n">Itoa</span><span class="p">(</span><span class="n">ai</span> <span class="o">+</span> <span class="n">bi</span><span class="p">)</span>
<span class="p">}</span>
</code></pre></div></div>
<p>This is similar to the <a href="https://github.com/golang/go/issues/32437">try</a> specification mentioned earlier, but
with the dream of having an arbitrary binding mechanism that lets
specific types (in this case, <code class="language-plaintext highlighter-rouge">ErrorOr[T]</code>) decide what it means to
either move values forward or fail.</p>
<h2 id="tedium">Tedium</h2>
<p>I wasn’t thinking about this because I want to sell people on Haskell.
It’s more about how I see the same stuff written in go (and other
languages) every day and have to be super vigilant to make sure nobody
is introducing bugs in their error handling. I catch bugs in code
like this regularly. These types of bugs can’t be expressed in code
that will compile if they just embraced the monads they were
reinventing and got to work at a higher level.</p>
<p>The terminology is probably confusing to folks from strange lands, but
much like the olde Gang of Four patterns, you see the same stuff a lot
and name the patterns. Except folks also make libraries that just
<em>do</em> the patterns, so we don’t all have to reinvent things so
frequently.</p>
SubTree2021-04-20T00:00:00+00:00http://dustin.github.com/2021/04/20/subtree<div>
<img src="/images/mqtt.png" alt="mqtt" title="MQTT" class="floatright" width="200" height="172" />
</div>
<h1 id="subtree---an-mqtt-subscription-tree">SubTree - An MQTT Subscription Tree</h1>
<p>I do a lot of iot junk and use <a href="https://mqtt.org">mqtt</a> quite a bit. Somehow I
ended up writing my own <a href="https://github.com/dustin/mqttd">mqtt broker</a> in Haskell along the way.
There was one core part of the server that I thought would be really
hard, but ended up being one of my favorite pieces of code, though
it’s only about 50 lines long. I wanted to describe why.</p>
<p>First, a bit of background.</p>
<h2 id="mqtt">MQTT</h2>
<p>MQTT is basically a pubsub service for IoT like things. Little
devices connect to an MQTT broker so other devices connected to the
MQTT broker can pick up those messages via subscriptions. It’s a bit
complicated and there are a lot of details, but I’ll try to simplify
to the relevant points.</p>
<h3 id="publishing">Publishing</h3>
<p>Messages are published to topics that are slash-separated strings,
e.g. <code class="language-plaintext highlighter-rouge">a/b/c</code>. The broker’s job is to decide who wants to receive
messages published to <code class="language-plaintext highlighter-rouge">a/b/c</code> and delivers a copy of the message to
that subscriber.</p>
<h3 id="subscribing">Subscribing</h3>
<p>Multiple clients can subscribe to topics either specifically, e.g.,
<code class="language-plaintext highlighter-rouge">a/b/c</code> or by a couple different wildcards. <code class="language-plaintext highlighter-rouge">+</code> can replace any
single path element, e.g., <code class="language-plaintext highlighter-rouge">+/b/c</code> or <code class="language-plaintext highlighter-rouge">a/+/c</code> or <code class="language-plaintext highlighter-rouge">+/+/+</code> or whatever.
<code class="language-plaintext highlighter-rouge">#</code> can appear as the last element in a path and matches anything
below the current path. Also: topics whose first character is <code class="language-plaintext highlighter-rouge">$</code> are
not automatically matched by a toplevel <code class="language-plaintext highlighter-rouge">#</code>. i.e., <code class="language-plaintext highlighter-rouge">$x/blah</code> will not
be matched by <code class="language-plaintext highlighter-rouge">#</code> but it will be matched by <code class="language-plaintext highlighter-rouge">$x/#</code>.</p>
<p>The specific rules aren’t too important, but it’s complicated enough
that you can’t just use a simple Map to locate subscribers and there
may be multiple subscribers for every published topic.</p>
<h3 id="other-fancy-things">Other Fancy Things</h3>
<p>There’s also a concept of “shared subscriptions” that allows multiple
subscribers to round robin messages and we need to be able to deal
with unsubscribing or timing out clients and forgetting subscriptions,
so we need to handle a couple other cases specially. These are all
doable without excessive consideration of the data structure.</p>
<h2 id="interesting-type-classes">Interesting Type Classes</h2>
<p>Setting aside the specific goals for a bit, let’s look into a few
type classes that we’d like our <code class="language-plaintext highlighter-rouge">SubTree</code> type to satisfy to make
things easier to think about.</p>
<h3 id="semigroup">Semigroup</h3>
<p>A <a href="https://typeclasses.com/semigroup">Semigroup</a> at a high level just means you have an
associative binary operation that is used to “combine” two things.
This is a bit hand wavy, but if you think about it as set union or
list concatenation, you’re on the right track for most of the purposes
here.</p>
<p>I wanted my <code class="language-plaintext highlighter-rouge">SubTree</code> to be a <code class="language-plaintext highlighter-rouge">semigroup</code> such that <code class="language-plaintext highlighter-rouge">a <> b</code> does
whatever you might think of as a natural combination of two
<code class="language-plaintext highlighter-rouge">SubTree</code>s. <code class="language-plaintext highlighter-rouge">SubTree</code> itself is <code class="language-plaintext highlighter-rouge">* -> *</code>, so it’s parameterized on a
type. A <code class="language-plaintext highlighter-rouge">Semigroup</code> of <code class="language-plaintext highlighter-rouge">SubTree a</code> is only meaningful when <code class="language-plaintext highlighter-rouge">a</code>
itself is a <code class="language-plaintext highlighter-rouge">Semigroup</code>. i.e., if you have a <code class="language-plaintext highlighter-rouge">SubTree [Int]</code> then
combining two <code class="language-plaintext highlighter-rouge">SubTree</code> values would produce a new <code class="language-plaintext highlighter-rouge">SubTree</code> with all
of the same subscribers for all of the topics from both values.</p>
<h3 id="monoid">Monoid</h3>
<p>A <a href="https://typeclasses.com/monoid">Monoid</a> is basically just a <code class="language-plaintext highlighter-rouge">Semigroup</code> with an “identity”
value. The “identity” value can be combined to either end via the
semigroup <code class="language-plaintext highlighter-rouge"><></code> operator and you’ll end up with the same value. e.g.,
<code class="language-plaintext highlighter-rouge">"a" <> "" == "a"</code> and <code class="language-plaintext highlighter-rouge">"" <> "a" == "a"</code>.</p>
<p>Similarly to <code class="language-plaintext highlighter-rouge">Semigroup</code>, when I have a <code class="language-plaintext highlighter-rouge">SubTree a</code> and <code class="language-plaintext highlighter-rouge">a</code> is a
<code class="language-plaintext highlighter-rouge">Monoid</code>, I wanted <code class="language-plaintext highlighter-rouge">SubTree a</code> to also be a <code class="language-plaintext highlighter-rouge">Monoid</code>, making <code class="language-plaintext highlighter-rouge">mempty</code>
do the right thing for constructing an empty <code class="language-plaintext highlighter-rouge">SubTree</code>.</p>
<p>This doesn’t look like much so far, but it gets more powerful as we
go.</p>
<h3 id="functor">Functor</h3>
<p>Since my <code class="language-plaintext highlighter-rouge">SubTree</code> is parameterized and conceptually a container, it
makes a lot of sense to have it be a <a href="https://typeclasses.com/functortown">Functor</a>.</p>
<p>A <code class="language-plaintext highlighter-rouge">Functor</code> gives you a way to take a <code class="language-plaintext highlighter-rouge">SubTree a</code> and convert it to a
<code class="language-plaintext highlighter-rouge">SubTree b</code> if you have a function <code class="language-plaintext highlighter-rouge">(a -> b)</code>. One way to think of
this is to imagine your function <code class="language-plaintext highlighter-rouge">f :: a -> b</code> being placed in front
of every <code class="language-plaintext highlighter-rouge">a</code> in the <code class="language-plaintext highlighter-rouge">SubTree a</code> which naturally makes every <code class="language-plaintext highlighter-rouge">a</code> into a
<code class="language-plaintext highlighter-rouge">b</code>.</p>
<p>e.g., if I have a <code class="language-plaintext highlighter-rouge">SubTree [String]</code> where I map subscription filters
to a list of strings and I want to have a <code class="language-plaintext highlighter-rouge">SubTree Int</code> where I map
filters to the number of subscribers, then that’s just <code class="language-plaintext highlighter-rouge">fmap length</code>
and we’re done.</p>
<p>The laws require the “shape” of the structure not change while
performing such a transformation. Mapping a list over a function
gives you a new list with the same number of elements. Simiarly,
mapping a <code class="language-plaintext highlighter-rouge">SubTree</code> over a function gives you the same subscription
structure, but changes only the values.</p>
<h3 id="foldable">Foldable</h3>
<p><a href="https://wiki.haskell.org/Foldable_and_Traversable">Foldable</a> is an abstraction for doing stuff to all of the
elements of a container. It gives you such great bits as <code class="language-plaintext highlighter-rouge">foldr</code> and
<code class="language-plaintext highlighter-rouge">fold</code> and <code class="language-plaintext highlighter-rouge">toList</code>. Where <code class="language-plaintext highlighter-rouge">Functor</code> gives you a shape-preserving
mechanism to operate across a value, <code class="language-plaintext highlighter-rouge">Foldable</code> provides catamorphisms
allowing you to reduce a value to a structure of a different shape.</p>
<p>For example, if you want to know how many subscribers are found within
a <code class="language-plaintext highlighter-rouge">SubTree [String]</code> named <code class="language-plaintext highlighter-rouge">t</code> you can write something like <code class="language-plaintext highlighter-rouge">foldMap (Sum
. length) t</code> and your <code class="language-plaintext highlighter-rouge">Foldable</code> implementation and the <code class="language-plaintext highlighter-rouge">Sum</code> <code class="language-plaintext highlighter-rouge">Monoid</code>
does the rest of the work for you.</p>
<h3 id="traversable">Traversable</h3>
<p><a href="https://wiki.haskell.org/Foldable_and_Traversable">Traversable</a> is a bit of a fancier <code class="language-plaintext highlighter-rouge">Functor</code> that allows
for effects and a few other things that aren’t necessarily interesting
for this discussion, but when you’re building a container, it’s good
to have around.</p>
<div>
<img src="/images/subtree-ex.png" alt="mqtt" title="MQTT" class="floatright" width="150" height="258" />
</div>
<h2 id="finally-the-subtree-type">Finally, the SubTree Type</h2>
<p>So now that we’ve covered all of the desirable properties, the
long-dreaded <code class="language-plaintext highlighter-rouge">SubTree</code> ended up being mostly just this:</p>
<div class="language-haskell highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="kr">data</span> <span class="kt">SubTree</span> <span class="n">a</span> <span class="o">=</span> <span class="kt">SubTree</span> <span class="p">{</span>
<span class="n">subs</span> <span class="o">::</span> <span class="kt">Maybe</span> <span class="n">a</span>
<span class="p">,</span> <span class="n">children</span> <span class="o">::</span> <span class="kt">Map</span> <span class="kt">Filter</span> <span class="p">(</span><span class="kt">SubTree</span> <span class="n">a</span><span class="p">)</span>
<span class="p">}</span> <span class="kr">deriving</span> <span class="p">(</span><span class="kt">Show</span><span class="p">,</span> <span class="kt">Eq</span><span class="p">,</span> <span class="kt">Functor</span><span class="p">,</span> <span class="kt">Foldable</span><span class="p">,</span> <span class="kt">Traversable</span><span class="p">)</span>
</code></pre></div></div>
<p>It’s a tree that may have subscribers at any given level, and it may
have children below it. I think I hand-wrote <code class="language-plaintext highlighter-rouge">Functor</code> et. al. before
just trusting the compiler to do the right thing (there should be only
one valid implementation, anyway). So at this point, we’re almost
done with most normal storage bits.</p>
<p><code class="language-plaintext highlighter-rouge">Semigroup</code> and <code class="language-plaintext highlighter-rouge">Monoid</code> require a bit more more work, so let’s
implement those really quickly before we move on:</p>
<div class="language-haskell highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="kr">instance</span> <span class="kt">Semigroup</span> <span class="n">a</span> <span class="o">=></span> <span class="kt">Semigroup</span> <span class="p">(</span><span class="kt">SubTree</span> <span class="n">a</span><span class="p">)</span> <span class="kr">where</span>
<span class="n">a</span> <span class="o"><></span> <span class="n">b</span> <span class="o">=</span> <span class="kt">SubTree</span> <span class="p">(</span><span class="n">subs</span> <span class="n">a</span> <span class="o"><></span> <span class="n">subs</span> <span class="n">b</span><span class="p">)</span> <span class="p">(</span><span class="kt">Map</span><span class="o">.</span><span class="n">unionWith</span> <span class="p">(</span><span class="o"><></span><span class="p">)</span> <span class="p">(</span><span class="n">children</span> <span class="n">a</span><span class="p">)</span> <span class="p">(</span><span class="n">children</span> <span class="n">b</span><span class="p">))</span>
<span class="n">empty</span> <span class="o">::</span> <span class="kt">SubTree</span> <span class="n">a</span>
<span class="n">empty</span> <span class="o">=</span> <span class="kt">SubTree</span> <span class="kt">Nothing</span> <span class="n">mempty</span>
<span class="kr">instance</span> <span class="kt">Monoid</span> <span class="n">a</span> <span class="o">=></span> <span class="kt">Monoid</span> <span class="p">(</span><span class="kt">SubTree</span> <span class="n">a</span><span class="p">)</span> <span class="kr">where</span>
<span class="n">mempty</span> <span class="o">=</span> <span class="n">empty</span>
</code></pre></div></div>
<p>(note that <code class="language-plaintext highlighter-rouge">empty</code> exists so we can have non-monoidal empty
<code class="language-plaintext highlighter-rouge">SubTree</code>s)</p>
<h3 id="modification">Modification</h3>
<p>In order to add, change, or remove subscriptions in a <code class="language-plaintext highlighter-rouge">SubTree</code>, we
introduce the <code class="language-plaintext highlighter-rouge">modify</code> function. It’s the most general mechanism for
performing any modifications, so it gets a pretty generic name. It
looks like this:</p>
<div class="language-haskell highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">modify</span> <span class="o">::</span> <span class="kt">Filter</span> <span class="o">-></span> <span class="p">(</span><span class="kt">Maybe</span> <span class="n">a</span> <span class="o">-></span> <span class="kt">Maybe</span> <span class="n">a</span><span class="p">)</span> <span class="o">-></span> <span class="kt">SubTree</span> <span class="n">a</span> <span class="o">-></span> <span class="kt">SubTree</span> <span class="n">a</span>
</code></pre></div></div>
<p>i.e., for a given <code class="language-plaintext highlighter-rouge">Filter</code> and a function that takes a maybe-existing
value and returns a new maybe-existing value, we can do our thing.</p>
<p>The actual implementation leverages <code class="language-plaintext highlighter-rouge">Data.Map</code>’s <code class="language-plaintext highlighter-rouge">alter</code> function
which does most of the work here, but the actual implementation is
just a couple of lines:</p>
<div class="language-haskell highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">modify</span> <span class="o">::</span> <span class="kt">Filter</span> <span class="o">-></span> <span class="p">(</span><span class="kt">Maybe</span> <span class="n">a</span> <span class="o">-></span> <span class="kt">Maybe</span> <span class="n">a</span><span class="p">)</span> <span class="o">-></span> <span class="kt">SubTree</span> <span class="n">a</span> <span class="o">-></span> <span class="kt">SubTree</span> <span class="n">a</span>
<span class="n">modify</span> <span class="n">top</span> <span class="n">f</span> <span class="o">=</span> <span class="n">go</span> <span class="p">(</span><span class="n">splitOn</span> <span class="s">"/"</span> <span class="n">top</span><span class="p">)</span>
<span class="kr">where</span>
<span class="n">go</span> <span class="kt">[]</span> <span class="n">n</span><span class="o">@</span><span class="kt">SubTree</span><span class="p">{</span><span class="o">..</span><span class="p">}</span> <span class="o">=</span> <span class="n">n</span><span class="p">{</span><span class="n">subs</span><span class="o">=</span><span class="n">f</span> <span class="n">subs</span><span class="p">}</span>
<span class="n">go</span> <span class="p">(</span><span class="n">x</span><span class="o">:</span><span class="n">xs</span><span class="p">)</span> <span class="n">n</span><span class="o">@</span><span class="kt">SubTree</span><span class="p">{</span><span class="o">..</span><span class="p">}</span> <span class="o">=</span> <span class="n">n</span><span class="p">{</span><span class="n">children</span><span class="o">=</span><span class="kt">Map</span><span class="o">.</span><span class="n">alter</span> <span class="p">(</span><span class="n">fmap</span> <span class="p">(</span><span class="n">go</span> <span class="n">xs</span><span class="p">)</span> <span class="o">.</span> <span class="n">maybe</span> <span class="p">(</span><span class="kt">Just</span> <span class="n">empty</span><span class="p">)</span> <span class="kt">Just</span><span class="p">)</span> <span class="n">x</span> <span class="n">children</span><span class="p">}</span>
</code></pre></div></div>
<p>We start by splitting the filter topic on <code class="language-plaintext highlighter-rouge">/</code> so we have the segments
and then we walk the tree. If the remaining topic is <code class="language-plaintext highlighter-rouge">[]</code> then we’ve
arrived at the topic we’re looking for and we just run the
transformation function and we’re done. Otherwise, we walk the tree
using <code class="language-plaintext highlighter-rouge">alter</code> which will create any necessary subtrees as we go.</p>
<p>Note that this would be <em>slightly</em> simpler if we required <code class="language-plaintext highlighter-rouge">a</code> to be
monoidal, but fewer constraints are possible, so we did the broadest
thing here.</p>
<p>It’s a little awkward to use, though, so we also have <code class="language-plaintext highlighter-rouge">addWith</code>:</p>
<div class="language-haskell highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">addWith</span> <span class="o">::</span> <span class="kt">Monoid</span> <span class="n">a</span> <span class="o">=></span> <span class="kt">Filter</span> <span class="o">-></span> <span class="p">(</span><span class="n">a</span> <span class="o">-></span> <span class="n">a</span> <span class="o">-></span> <span class="n">a</span><span class="p">)</span> <span class="o">-></span> <span class="n">a</span> <span class="o">-></span> <span class="kt">SubTree</span> <span class="n">a</span> <span class="o">-></span> <span class="kt">SubTree</span> <span class="n">a</span>
<span class="n">addWith</span> <span class="n">top</span> <span class="n">f</span> <span class="n">i</span> <span class="o">=</span> <span class="n">modify</span> <span class="n">top</span> <span class="p">(</span><span class="n">fmap</span> <span class="p">(</span><span class="n">f</span> <span class="n">i</span><span class="p">)</span> <span class="o">.</span> <span class="n">maybe</span> <span class="p">(</span><span class="kt">Just</span> <span class="n">mempty</span><span class="p">)</span> <span class="kt">Just</span><span class="p">)</span>
</code></pre></div></div>
<p><code class="language-plaintext highlighter-rouge">addWith</code> assumes <code class="language-plaintext highlighter-rouge">a</code> is monoidal and gives us a far simpler
transformation by just allowing us to add a specific value with a
collision function to deal with existing cases.</p>
<p>e.g., the most simple case, <code class="language-plaintext highlighter-rouge">add</code>:</p>
<div class="language-haskell highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">add</span> <span class="o">::</span> <span class="kt">Monoid</span> <span class="n">a</span> <span class="o">=></span> <span class="kt">Filter</span> <span class="o">-></span> <span class="n">a</span> <span class="o">-></span> <span class="kt">SubTree</span> <span class="n">a</span> <span class="o">-></span> <span class="kt">SubTree</span> <span class="n">a</span>
<span class="n">add</span> <span class="n">top</span> <span class="o">=</span> <span class="n">addWith</span> <span class="n">top</span> <span class="p">(</span><span class="o"><></span><span class="p">)</span>
</code></pre></div></div>
<p><code class="language-plaintext highlighter-rouge">add</code> does the thing you’d expect when adding a new value. e.g., if
you have a <code class="language-plaintext highlighter-rouge">SubTree [Int]</code> that has subscribers at <code class="language-plaintext highlighter-rouge">a/b/c</code> of <code class="language-plaintext highlighter-rouge">[1,2]</code>
and you add <code class="language-plaintext highlighter-rouge">[3]</code> at that path, you’ll have <code class="language-plaintext highlighter-rouge">[1,2,3]</code>.</p>
<p>This is also how we build <code class="language-plaintext highlighter-rouge">fromList</code>:</p>
<div class="language-haskell highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">fromList</span> <span class="o">::</span> <span class="kt">Monoid</span> <span class="n">a</span> <span class="o">=></span> <span class="p">[(</span><span class="kt">Filter</span><span class="p">,</span> <span class="n">a</span><span class="p">)]</span> <span class="o">-></span> <span class="kt">SubTree</span> <span class="n">a</span>
<span class="n">fromList</span> <span class="o">=</span> <span class="n">foldr</span> <span class="p">(</span><span class="n">uncurry</span> <span class="n">add</span><span class="p">)</span> <span class="n">mempty</span>
</code></pre></div></div>
<h3 id="searching">Searching</h3>
<p>And now, the entire reason this thing exists: finding subscribers.</p>
<p>The most general function we have for this is <code class="language-plaintext highlighter-rouge">findMap</code> which has a
fairly simple signature:</p>
<div class="language-haskell highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">findMap</span> <span class="o">::</span> <span class="kt">Monoid</span> <span class="n">m</span> <span class="o">=></span> <span class="kt">Topic</span> <span class="o">-></span> <span class="p">(</span><span class="n">a</span> <span class="o">-></span> <span class="n">m</span><span class="p">)</span> <span class="o">-></span> <span class="kt">SubTree</span> <span class="n">a</span> <span class="o">-></span> <span class="n">m</span>
</code></pre></div></div>
<p>I’m going to omit all the code here since it’s about 8 lines long
because of all the weird expansion rules, but the signature tells us
really everything we need to know. It looks a lot like <code class="language-plaintext highlighter-rouge">foldMap</code>.
Given a topic and a function that converts whatever <code class="language-plaintext highlighter-rouge">a</code> is found for
that topic to a monoidal value, you get a monoidal value.</p>
<p>e.g., if the <code class="language-plaintext highlighter-rouge">a</code> is <em>already</em> a monoid, you get this function:</p>
<div class="language-haskell highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">find</span> <span class="o">::</span> <span class="kt">Monoid</span> <span class="n">a</span> <span class="o">=></span> <span class="kt">Topic</span> <span class="o">-></span> <span class="kt">SubTree</span> <span class="n">a</span> <span class="o">-></span> <span class="n">a</span>
<span class="n">find</span> <span class="n">top</span> <span class="o">=</span> <span class="n">findMap</span> <span class="n">top</span> <span class="n">id</span>
</code></pre></div></div>
<p>So when I store my subscriptions as a list, <code class="language-plaintext highlighter-rouge">find "some/topic" st</code>
gives me a list of all the things subscribed as <code class="language-plaintext highlighter-rouge">some/topic</code> or
<code class="language-plaintext highlighter-rouge">+/topic</code> or <code class="language-plaintext highlighter-rouge">topic/+</code> or <code class="language-plaintext highlighter-rouge">topic/#</code> or <code class="language-plaintext highlighter-rouge">#</code>.</p>
<h2 id="in-practice">In Practice</h2>
<p>In my original implementation of my mqtt broker, I implemented
subscriptions as a dumb list. It seemed like it was going to be a
hard problem, so I punted until I could do something better. Every
message that was published had to look at every subscription for every
client and see which ones matched before redistributing stuff. For my
little server at home, I have around 250 subsriptions at any point in
time and get about one message per second on average. That’s on the
verge of gross.</p>
<p>But it turned out to be very easy to implement something efficient
that worked quite well. I just have a <a href="https://hackage.haskell.org/package/stm-2.5.0.0/docs/Control-Concurrent-STM-TVar.html">TVar</a> that holds a
<code class="language-plaintext highlighter-rouge">SubTree (Map SessionID SubOptions)</code> and just use <code class="language-plaintext highlighter-rouge">stm</code> to do the
reads and writes. The <em>semantics</em> are still quite complicated as
there are private and shared subscriptions and there’s the session
vs. client separation and being able to have concurrent deliveries of
messages sent by one client to any relevant subscribers while new
clients are concurrently modifying the subscription <code class="language-plaintext highlighter-rouge">SubTree</code>.</p>
<p>Being able to express data structures like this and test them
thoroughly against well-established laws makes me avoid having to
think of large swaths of bugs I’d write in most other languages.</p>
<p>I replaced mosquitto with my own <a href="https://github.com/dustin/mqttd">mqttd</a> project a year or so
ago due to a couple strange bugs I’d encounter occasionally and some
missing features of MQTT v5 I wanted to use. I’m around 1,400 lines
in with a solid broker I’m relying on.</p>
<p>But <a href="https://github.com/dustin/mqttd/blob/master/src/MQTTD/SubTree.hs">SubTree.hs</a> is one of my favorite pieces of code.</p>
GoPro Plus2020-04-29T00:00:00+00:00http://dustin.github.com/2020/04/29/gopro-plus<p>This is the story of a project I started (almost exactly) two months
ago. Not much of it is about the code itself (links at the bottom),
but you can read what I did and why and stuff. I got to write a bunch
of stuff in my favorite language and learn a new language along the
way. I also got a little bit of control over my media.</p>
<h1 id="gopro-plus">GoPro Plus</h1>
<div>
<img src="/images/beach-small.jpg" alt="beach" title="sunset at the beach" class="floatleft" width="400" height="300" />
</div>
<p>I got a new GoPro Hero 8 for a trip I took recently. There were lots
of exciting things to look at. The picture to the left was taken on
January 20 at a resort in Siquijor.</p>
<p>The resort had quite good connectivity, and since I bought a new
GoPro, I figured I’d try out the GoPro Plus 30 day trial thing they
were offering. I could plug in my camera and my video and pictures
and stuff would magicaly be stored safely in The Cloud™.</p>
<h2 id="gopros">GoPros</h2>
<p>GoPro Plus ends up being something like $5/mo (or I think $50/year if
you do it annually). For this, you get unlimited storage and up to
two camera replacements per year. It’s a pretty good deal to not have
to worry about asset tracking, etc… And the camera replacement is a
good incentive to actually try to use the thing.</p>
<p>The mobile app lets you browse around in the cloud-stored data and
find things of interest to assemble into videos locally. It does a
pretty good job of thinking up the edits and stuff for you.</p>
<p>Also, this isn’t my first GoPro, but you can upload video through
their web UI. I uploaded a bunch of my old clips so I could store all
my footage in one cloudy place.</p>
<p>Except, there were a few things that annoyed me about this.</p>
<h2 id="gocons">GoCons</h2>
<p>There were a few things that annoyed me about GoPro Plus early on (in
no particular order):</p>
<ol>
<li>The web site made it quite difficult to navigate anything but your
most recent media.</li>
<li>While you could upload media and the camera could upload time lapse
photos as a single medium, the web UI didn’t let you do this.</li>
<li>Sometimes, things would just not upload and I couldn’t figure out
why.</li>
<li>Downloading in bulk (e.g., a day’s worth of stuff) is nearly
impossible.</li>
<li>Sharing is painful and broken.</li>
</ol>
<p>As I became more of a power user, I found weirder, more obscure bugs
(e.g., items stuck on their end in a particular state, or incorrectly
recognized media).</p>
<p>But perhaps the biggest problem of all: I wanted to be able to make
sure I could tell what media I’d uploaded and retrieve it all at
will. I don’t know if this service will last forever, but I do know I
would like my media to, so if they decide to shut down or something, I
want to know I can get all my media out quickly and easily.</p>
<p>This is a fairly big flaw, as GoPro Plus doesn’t have an API. Well,
officially…</p>
<h2 id="getting-my-data">Getting My Data</h2>
<div>
<img src="/images/dive-small.jpg" alt="rope swing flop" title="flawless half backflip from a rope swing" class="floatright" width="250" height="384" />
</div>
<p>I finally decided to dive in on February 26. I found a little bit of
stuff online where people had managed to get a listing from the media
service, but these were all quite incomplete and didn’t meet any of my
goals.</p>
<p>My goto language these days is <a href="https://www.haskell.org/">Haskell</a> as
it’s consistently the easiest language I work in so I started
exploring the API endpoint in ghci with
<a href="https://hackage.haskell.org/package/wreq">wreq</a> and
<a href="http://hackage.haskell.org/package/aeson">aeson</a>. As it turns out,
their web app is just a javascript interface to the same API the
mobile devices use, so it was relatively easy to just watch what it
does and do the same thing.</p>
<p>I spent a bit of time over the weekend just making essential bits.
The first thing I figured I should do was capture all metadata from
the search results into a local sqlite database, along with the
thumbnails I could get from their image servers. I built my <code class="language-plaintext highlighter-rouge">gopro</code>
commandline tool to manage authentication tokens (and refreshes) and
then built a little web service that could give me my data back.</p>
<p>The GoPro Plus media-browser web site has endless scrolling of a few
tens of items at a time. If I want to find the oldest of my 2,229
items I currently have stored, it might take me ten minutes just to
scroll to the bottom (go to bottom of page, wait for load, repeat).
Just getting to footage from the above trip is an excursion in
itself.</p>
<p>Initially, I built an interface using
<a href="http://crossfilter.github.io/crossfilter/">crossfilter</a> to let me do
multi-dimensional filtering of my data quickly and easily, but I don’t
much like working in JavaScript, so I decided to learn
<a href="https://elm-lang.org/">elm</a> as part of this project. My new UI in elm
was much more pleasant to work with, but there’s no crossfilter and
javascript FFI is awkward even when you’re not trying to do something
that fancy. But all I really needed was a collection of filter
functions to apply in succession, so writing my own functional
equivalent of crossfilter in elm ended up being basically something
like this:</p>
<div class="language-elm highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="kt">List</span><span class="o">.</span><span class="n">filter</span> <span class="p">(</span><span class="o">\</span><span class="n">m</span> <span class="o">-></span> <span class="kt">List</span><span class="o">.</span><span class="n">all</span> <span class="p">(</span><span class="o">\</span><span class="n">f</span> <span class="o">-></span> <span class="n">f</span> <span class="n">model</span> <span class="n">m</span><span class="p">)</span> <span class="n">allFilters</span><span class="p">)</span> <span class="n">media</span>
</code></pre></div></div>
<p>Given that I know when all of the media was captured and from what
device, and what the type of media it was, I could easily build myself
a date-based filter and camera filter and type filter and suddenly
could ~instantly.</p>
<h2 id="need-more-data">Need More Data</h2>
<p>However, the GoPro Hero 8 captures some really rich telemetry. They
open-sourced a <a href="https://github.com/gopro/gpmf-parser">parser</a> for this
which is super cool, but I ran into two problems with it:</p>
<ol>
<li>Their code seems very difficult to use (see the hundreds of lines
of their basic
<a href="https://github.com/gopro/gpmf-parser/tree/master/demo">demo</a>).</li>
<li>The demo didn’t even display very relevant data for most of my
media.</li>
</ol>
<p>That was a little unfortunate. I want access to all of the metadata,
but after playing with their C code for a bit, I finally realized
that’s quite an uphill battle.</p>
<p>So I finally just read their spec and wrote a <a href="https://github.com/dustin/gpmf">gpmf
parser</a> from scratch. My core GPMF
parser is under 100 lines of code and gave me complete data from every
sample I ran into. The low-level data is a bit too low-level, so I
added another ~176 SLOC to cover higher level concepts like GPS
location and speed, face detection, etc…</p>
<p>Now I just needed to extract all the metadata from my ~700GB library.</p>
<p>However, you don’t need the entire content for this. It turns out
that as part of processing your video uploads, GoPro Plus produces a
few different variants of your content. This typically includes a
remarkably small one called <code class="language-plaintext highlighter-rouge">mp4_low</code> which usually carries the full
metadata. There are lots of qualifiers in there because things aren’t
<em>entirely</em> consistent, but it’s good enough:</p>
<div class="language-haskell highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">ms</span> <span class="o"><-</span> <span class="n">asum</span> <span class="p">[</span>
<span class="n">fv</span> <span class="s">"mp4_low"</span> <span class="p">(</span><span class="n">fn</span> <span class="s">"low"</span><span class="p">),</span>
<span class="n">fv</span> <span class="s">"high_res_proxy_mp4"</span> <span class="p">(</span><span class="n">fn</span> <span class="s">"high"</span><span class="p">),</span>
<span class="n">fv</span> <span class="s">"source"</span> <span class="p">(</span><span class="n">fn</span> <span class="s">"src"</span><span class="p">),</span>
<span class="n">pure</span> <span class="kt">Nothing</span><span class="p">]</span>
</code></pre></div></div>
<p><code class="language-plaintext highlighter-rouge">fv</code> here is a function that locally caches a stream from GoPro by
name and then attempts to extract a GPMF stream from it. Often,
<code class="language-plaintext highlighter-rouge">mp4_low</code> has it and we’re done quickly and easily. If not, try the
higher res variant used for streaming in the browser, then the source,
then give up and declare there’s no GPMF (e.g., my old Hero 4 footage
doesn’t have any).</p>
<p>I added geographical areas to my database, a web handler to retreive
them, summarized location in my metadata and added a bit of
point-in-box Elm code. At this point, “Show me all of my media taken
in the Philippines in the year 2020” is two clicks, roughly instant,
and I see <code class="language-plaintext highlighter-rouge">Showing 426 (49.88 GB) out of 2,229 items (946.81 GB)</code>
along with all the nicely grouped thumbnails in my web UI.</p>
<h2 id="syncing">Syncing</h2>
<div>
<img src="/images/cutting.jpg" alt="syncing" title="cutting wood with amos" class="floatright" width="350" height="263" />
</div>
<p>Syncing is relatively straightforward. I run my commandline tool
periodically. It asks for paginated media ordered by upload date
(descending) and keeps asking for pages until it sees a media ID it
already has. Subtract the existing data and I can pull down
whatever’s leftover.</p>
<p>I keep metadata, thumbnails, and downloadable variant metadata (minus
URLs as they expire) in my media table. So grab these things for each
missing record in concurrent batches (as not to do everything at once)
and commit them to my local DB.</p>
<p>I have this <code class="language-plaintext highlighter-rouge">retrieve</code> function that, given a medium ID returns the
list of downloadable artifacts. The URLs themselves are signed s3
URLs that expire after a bit, so they’re only really interesting when
you’re actualy downloading an artifact. There are a few different
types of artifacts, some of which have one URL and some have
multiple. The resulting JSON includes <code class="language-plaintext highlighter-rouge">url</code> as a key for when there’s
one URL, and <code class="language-plaintext highlighter-rouge">urls</code> when there’s more than one. This sort of makes
sense, but it causes a lack of uniformity. Each one <em>also</em> has a
<code class="language-plaintext highlighter-rouge">head</code>/<code class="language-plaintext highlighter-rouge">heads</code> variation of the URL for issuing a <code class="language-plaintext highlighter-rouge">HEAD</code> request
instead of a <code class="language-plaintext highlighter-rouge">GET</code> request.</p>
<p>This detail sounds boring, but one interesting thing here a fun
aeson-lens tip. If you have an arbitrarily complex <a href="https://gist.github.com/dustin/081e6fae77a540aed5279f8962cadf55">chunk of
JSON</a>
without much uniformity, but for any object anywhere within the
structure, you want to remove a couple of keys, you could write
something like this:</p>
<div class="language-haskell highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">fetchSansURLs</span> <span class="o">::</span> <span class="p">(</span><span class="kt">HasGoProAuth</span> <span class="n">m</span><span class="p">,</span> <span class="kt">MonadIO</span> <span class="n">m</span><span class="p">)</span> <span class="o">=></span> <span class="kt">MediumID</span> <span class="o">-></span> <span class="n">m</span> <span class="p">(</span><span class="kt">Maybe</span> <span class="kt">Value</span><span class="p">)</span>
<span class="n">fetchSansURLs</span> <span class="o">=</span> <span class="n">fmap</span> <span class="p">(</span><span class="n">_Just</span> <span class="o">.</span> <span class="n">deep</span> <span class="n">values</span> <span class="o">.</span> <span class="n">_Object</span> <span class="o">%~</span> <span class="n">sans</span> <span class="s">"url"</span> <span class="o">.</span> <span class="n">sans</span> <span class="s">"head"</span><span class="p">)</span> <span class="o">.</span> <span class="n">retrieve</span>
</code></pre></div></div>
<p>(the actual version excludes <code class="language-plaintext highlighter-rouge">urls</code> and <code class="language-plaintext highlighter-rouge">heads</code>, but that’s not an
important detail and this line’s long enough).</p>
<h3 id="web-syncing">Web Syncing</h3>
<p>One seemingly unnecessary annoyance is that I had to reload my web
page every time I did a commandline sync. I made this slightly better
by just making a data sync (soft reload) button, but I thought it
might be better to sync from the web UI itself.</p>
<p>The biggest problem with this was knowing what’s going on while this
is happening. I’m using an elm library called
<a href="https://package.elm-lang.org/packages/pablen/toasty">toasty</a> to put
little ephemeral popups on the screen. I’m also using <code class="language-plaintext highlighter-rouge">MondaLogger</code>
in the commandline tool. Combining these seems like a good idea.</p>
<p>Initially, I was asking the UI to poll the server to look for items in
the database that should be displayed on the UI. This was gross for
lots of reasons. I decided this might be a good time to learn a bit
about websockets.</p>
<p>I’d “used” websockets once in the past when I added websocket support
for my <a href="http://hackage.haskell.org/package/net-mqtt">mqtt library</a>,
but this was mostly theoretical and all client-side. I figured
server-side websockets in Haskell would be a bit of a challenge and
client-side in Elm would be easy. My expectations were totally
backwards.</p>
<p>I wrote a custom stderr logging function and a logging function that
writes to an <a href="https://hackage.haskell.org/package/stm/docs/Control-Concurrent-STM-TChan.html">STM broadcast
TChan</a>.
The latter is what I ended up using for websockets. I initially would
do something fancy during the sync to redirect the log using a local
logger mutation, but I figured I might as well just always log to the
websocket channel in case any browsers are attached. It’s slightly
less code.</p>
<p>My logger looks like this:</p>
<div class="language-haskell highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">notificationLogger</span> <span class="o">::</span> <span class="kt">TChan</span> <span class="kt">Notification</span>
<span class="o">-></span> <span class="p">(</span><span class="kt">Loc</span> <span class="o">-></span> <span class="kt">LogSource</span> <span class="o">-></span> <span class="kt">LogLevel</span> <span class="o">-></span> <span class="kt">LogStr</span> <span class="o">-></span> <span class="kt">IO</span> <span class="nb">()</span><span class="p">)</span>
<span class="n">notificationLogger</span> <span class="n">ch</span> <span class="kr">_</span> <span class="kr">_</span> <span class="n">lvl</span> <span class="n">str</span> <span class="o">=</span> <span class="kr">case</span> <span class="n">lvl</span> <span class="kr">of</span>
<span class="kt">LevelDebug</span> <span class="o">-></span> <span class="n">pure</span> <span class="nb">()</span>
<span class="kt">LevelInfo</span> <span class="o">-></span> <span class="n">note</span> <span class="kt">NotificationInfo</span>
<span class="kr">_</span> <span class="o">-></span> <span class="n">note</span> <span class="kt">NotificationError</span>
<span class="kr">where</span> <span class="n">note</span> <span class="n">t</span> <span class="o">=</span> <span class="n">atomically</span> <span class="o">$</span> <span class="n">writeTChan</span> <span class="n">ch</span> <span class="p">(</span><span class="kt">Notification</span> <span class="n">t</span> <span class="s">"GoPro"</span> <span class="n">lstr</span><span class="p">)</span>
<span class="n">lstr</span> <span class="o">=</span> <span class="kt">TE</span><span class="o">.</span><span class="n">decodeUtf8</span> <span class="o">$</span> <span class="n">fromLogStr</span> <span class="n">str</span>
</code></pre></div></div>
<p>…and that’s hooked up to the web like this:</p>
<div class="language-haskell highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">wsapp</span> <span class="o">::</span> <span class="kt">Env</span> <span class="o">-></span> <span class="kt">WS</span><span class="o">.</span><span class="kt">ServerApp</span>
<span class="n">wsapp</span> <span class="kt">Env</span><span class="p">{</span><span class="n">noteChan</span><span class="p">}</span> <span class="n">pending</span> <span class="o">=</span> <span class="kr">do</span>
<span class="n">ch</span> <span class="o"><-</span> <span class="n">atomically</span> <span class="o">$</span> <span class="n">dupTChan</span> <span class="n">noteChan</span>
<span class="n">conn</span> <span class="o"><-</span> <span class="kt">WS</span><span class="o">.</span><span class="n">acceptRequest</span> <span class="n">pending</span>
<span class="kt">WS</span><span class="o">.</span><span class="n">withPingThread</span> <span class="n">conn</span> <span class="mi">30</span> <span class="p">(</span><span class="n">pure</span> <span class="nb">()</span><span class="p">)</span> <span class="o">$</span>
<span class="n">forever</span> <span class="p">(</span><span class="kt">WS</span><span class="o">.</span><span class="n">sendTextData</span> <span class="n">conn</span> <span class="o">.</span> <span class="kt">J</span><span class="o">.</span><span class="n">encode</span> <span class="o">=<<</span> <span class="p">(</span><span class="n">atomically</span> <span class="o">.</span> <span class="n">readTChan</span><span class="p">)</span> <span class="n">ch</span><span class="p">)</span>
</code></pre></div></div>
<p>At this point, my entire web-based sync on the server-side looks like
this:</p>
<div class="language-haskell highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">post</span> <span class="s">"/api/sync"</span> <span class="kr">do</span>
<span class="kr">_</span> <span class="o"><-</span> <span class="n">lift</span> <span class="o">.</span> <span class="n">async</span> <span class="o">$</span> <span class="kr">do</span>
<span class="n">runFullSync</span>
<span class="n">sendNotification</span> <span class="p">(</span><span class="kt">Notification</span> <span class="kt">NotificationReload</span> <span class="s">""</span> <span class="s">""</span><span class="p">)</span>
<span class="n">status</span> <span class="n">noContent204</span>
</code></pre></div></div>
<p>The <code class="language-plaintext highlighter-rouge">NotificationReload</code> that gets sent is a special message that
isn’t displayed by the browser, but just indicates that it should
reload the media catalog.</p>
<p>So, now that that’s done, let’s do the elm side. I won’t dump all
that code out here (it was a lot). The short story is that Elm
doesn’t have native websockets support and wiring it up was a bit of a
pain. It’s not too bad after that, though. <code class="language-plaintext highlighter-rouge">Notification</code> messages
come in and are dispatched easily enough:</p>
<div class="language-elm highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">doNotifications</span> <span class="p">:</span> <span class="kt">Model</span> <span class="o">-></span> <span class="kt">List</span> <span class="kt">Notification</span> <span class="o">-></span> <span class="p">(</span> <span class="kt">Model</span><span class="o">,</span> <span class="kt">Cmd</span> <span class="kt">Msg</span> <span class="p">)</span>
<span class="n">doNotifications</span> <span class="n">model</span> <span class="o">=</span>
<span class="k">let</span> <span class="n">reload_</span> <span class="p">(</span><span class="n">m</span><span class="o">,</span> <span class="n">a</span><span class="p">)</span> <span class="o">=</span>
<span class="p">(</span><span class="n">m</span><span class="o">,</span> <span class="kt">Cmd</span><span class="o">.</span><span class="n">batch</span> <span class="p">[</span><span class="n">a</span><span class="o">,</span> <span class="n">reload</span><span class="p">])</span>
<span class="n">doNote</span> <span class="n">n</span> <span class="n">m</span> <span class="o">=</span>
<span class="k">case</span> <span class="n">n</span><span class="o">.</span><span class="n">typ</span> <span class="k">of</span>
<span class="kt">NotificationInfo</span> <span class="o">-></span> <span class="n">toastSuccess</span> <span class="n">n</span><span class="o">.</span><span class="n">title</span> <span class="n">n</span><span class="o">.</span><span class="n">msg</span> <span class="n">m</span>
<span class="kt">NotificationError</span> <span class="o">-></span> <span class="n">toastError</span> <span class="n">n</span><span class="o">.</span><span class="n">title</span> <span class="n">n</span><span class="o">.</span><span class="n">msg</span> <span class="n">m</span>
<span class="kt">NotificationReload</span> <span class="o">-></span> <span class="n">reload_</span> <span class="n">m</span>
<span class="kt">NotificationUnknown</span> <span class="o">-></span> <span class="n">toastError</span> <span class="s">"</span><span class="s2">Unhandled Notification"</span> <span class="p">(</span><span class="n">n</span><span class="o">.</span><span class="n">msg</span><span class="p">)</span> <span class="n">m</span>
<span class="k">in</span>
<span class="kt">List</span><span class="o">.</span><span class="n">foldr</span> <span class="n">doNote</span> <span class="p">(</span><span class="n">model</span><span class="o">,</span> <span class="kt">Cmd</span><span class="o">.</span><span class="n">none</span><span class="p">)</span>
</code></pre></div></div>
<p><br /></p>
<h2 id="uploading">Uploading</h2>
<div>
<img src="https://raw.githubusercontent.com/dustin/gopro-plus/master/upload.png" alt="uploading" title="upload process" class="floatleft" width="185" height="210" />
</div>
<p>The upload process is
<a href="https://github.com/dustin/gopro-plus/wiki/Upload">complicated</a>. I’ve
not documented the whole in English, but my
<a href="http://hackage.haskell.org/package/gopro-plus/docs/GoPro-Plus-Upload.html">GoPro.Plus.Upload</a>
module does all of the things I’ve figured out how to do.</p>
<p>It’s quite facinating. A single medium upload can consist of multiple
files (e.g., a timelapse photo series, or a video that was so large it
broke into multiple files on its own). Additionally, they’re
leveraging S3 chunked uploads such that when you define an upload and
specify your chunk size (I’m using 6MB which is what they use in the
web app), they give you a collection of pre-signed URLs to do the
upload. The order you process these things doesn’t matter.</p>
<p>In the end, this means you can get massive parallelism for most
uploads. Some of my uploads have consisted of hundreds of parts. I
don’t have great connectivity where I am currently, but I can rsync my
media to a location with good connectivity without any care for how
long it takes. Once it arrives, I can use my commandline tool to
upload with all the cores and network I can throw at the problem and
then tie it up in the end.</p>
<p>I end up doing a lot of fun parallelism in this project, which often
gets too large. I get a lot of use out of this function:</p>
<div class="language-haskell highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">mapConcurrentlyLimited</span> <span class="o">::</span> <span class="p">(</span><span class="kt">MonadMask</span> <span class="n">m</span><span class="p">,</span> <span class="kt">MonadUnliftIO</span> <span class="n">m</span><span class="p">,</span> <span class="kt">Traversable</span> <span class="n">f</span><span class="p">)</span>
<span class="o">=></span> <span class="kt">Int</span>
<span class="o">-></span> <span class="p">(</span><span class="n">a</span> <span class="o">-></span> <span class="n">m</span> <span class="n">b</span><span class="p">)</span>
<span class="o">-></span> <span class="n">f</span> <span class="n">a</span>
<span class="o">-></span> <span class="n">m</span> <span class="p">(</span><span class="n">f</span> <span class="n">b</span><span class="p">)</span>
<span class="n">mapConcurrentlyLimited</span> <span class="n">n</span> <span class="n">f</span> <span class="n">l</span> <span class="o">=</span> <span class="n">liftIO</span> <span class="p">(</span><span class="n">newQSem</span> <span class="n">n</span><span class="p">)</span> <span class="o">>>=</span> <span class="nf">\</span><span class="n">q</span> <span class="o">-></span> <span class="n">mapConcurrently</span> <span class="p">(</span><span class="n">b</span> <span class="n">q</span><span class="p">)</span> <span class="n">l</span>
<span class="kr">where</span> <span class="n">b</span> <span class="n">q</span> <span class="n">x</span> <span class="o">=</span> <span class="n">bracket_</span> <span class="p">(</span><span class="n">liftIO</span> <span class="p">(</span><span class="n">waitQSem</span> <span class="n">q</span><span class="p">))</span> <span class="p">(</span><span class="n">liftIO</span> <span class="p">(</span><span class="n">signalQSem</span> <span class="n">q</span><span class="p">))</span> <span class="p">(</span><span class="n">f</span> <span class="n">x</span><span class="p">)</span>
</code></pre></div></div>
<p>My commandline tool has two different upload commands because in one
case you want to just blast a bunch of photos and clips up and in the
other case, you want to sequence a series of parts into a single item
of a specific type. This was a huge win for me because before I
figured out how to do this, I had a couple folders of hundreds photos
each that were GoPro time lapses that I couldn’t upload to GoPro using
any of their tools.</p>
<p>Actually, it’s got a third upload command because I was able to
recover a failed upload, but that was a super complicated process and
their API doesn’t <em>quite</em> let you do this in a usefully automatable
way.</p>
<h2 id="updating">Up…dating?</h2>
<p>One issue I found in my bulk uploads is that I had a large number of
items for which GoPro couldn’t recognize the camera. This was
especially weird because <em>I</em> could. More specifically, in many cases,
my own GPMF parser correctly identified the camera used, but GoPro
didn’t know what it was.</p>
<p>I had found a URL from which I could <code class="language-plaintext highlighter-rouge">GET</code> the details of a medium by
its ID and I had observed in the process that it would mutate its
state by using a simple <code class="language-plaintext highlighter-rouge">PUT</code>. Can I do that?</p>
<p>I wrote a <code class="language-plaintext highlighter-rouge">fixup</code> command that lets me do bulk edits on the <em>GoPro</em>
side using SQL queries. e.g., consider the following query:</p>
<div class="language-sql highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">select</span> <span class="n">m</span><span class="p">.</span><span class="n">media_id</span><span class="p">,</span> <span class="k">g</span><span class="p">.</span><span class="n">camera_model</span>
<span class="k">from</span> <span class="n">media</span> <span class="n">m</span> <span class="k">join</span> <span class="n">meta</span> <span class="k">g</span> <span class="k">on</span> <span class="p">(</span><span class="n">m</span><span class="p">.</span><span class="n">media_id</span> <span class="o">=</span> <span class="k">g</span><span class="p">.</span><span class="n">media_id</span><span class="p">)</span>
<span class="k">where</span> <span class="n">m</span><span class="p">.</span><span class="n">camera_model</span> <span class="k">is</span> <span class="k">null</span>
<span class="k">and</span> <span class="k">g</span><span class="p">.</span><span class="n">camera_model</span> <span class="k">is</span> <span class="k">not</span> <span class="k">null</span>
</code></pre></div></div>
<p>This query emits one row per record where I know the camera type from
my metadata extraction, but GoPro does not. The
<a href="https://github.com/dustin/gopro/blob/master/src/GoPro/Commands/Fixup.hs">fixup</a>
code will run that query, look up each item based on the returned
column <code class="language-plaintext highlighter-rouge">media_id</code> and update the JSON fields corresponding to the
column names returned from the SQL query (e.g., <code class="language-plaintext highlighter-rouge">camera_model</code> above)
and row values/types. I fixed up something like 800 bad items with a
single, simple command.</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>gopro fixup "select m.media_id, g.camera_model ..."
</code></pre></div></div>
<p>I also used this for individual edits since I could craft literal rows
for edits easily enough and didn’t otherwise need to make a special
tool. Not all fields are editable, though.</p>
<h2 id="extraction">Extraction</h2>
<div>
<img src="/images/retrieve.jpg" alt="retrieving" title="extracting all the things" class="floatright" width="285" height="225" />
</div>
<p>But what if I want to move everything out of GoPro Plus <em>right now</em>?</p>
<p>I have a <code class="language-plaintext highlighter-rouge">gopro backup</code> command that does that. It uses local
credentials and my previously mentioned <code class="language-plaintext highlighter-rouge">retrieve</code> command to request
all the (signed) media URLs from GoPro Plus and then pushes them into
an <a href="https://aws.amazon.com/sqs/">SQS</a> queue which delivers to an <a href="https://aws.amazon.com/lambda/">AWS
Lambda</a> function whose job is to copy
the data into my own S3 bucket. Since GoPro itself stores all of its
data in S3, I figured moving the data from their bucket to mine using
AWS technologies made sense.</p>
<p>I did write <a href="https://github.com/dustin/gopro/blob/master/lambda/download-to-s3.js">some
javascript</a>
for the lambda function, but it’s small and boring.</p>
<p>In my current, low-bandwidth, unreliable network state, I can move
~1TB of media out of GoPro cloud into bits I control with a small
command that takes a couple minutes to run. Most of the time on my
end is just making a couple thousand calls (though I’m using
<code class="language-plaintext highlighter-rouge">mapConcurrentlyLimited</code> here, of course). Then it’s just however
fast AWS Lambda wants to feed my function.</p>
<p>There’s really no reason I couldn’t just convert this to my own GoPro
Plus from here. I’ve got browsing, media persistence, etc… I just
need to do a bit of scaling, but I’m already using ffmpeg…</p>
<h2 id="resources--current-status">Resources / Current Status</h2>
<p>Relevant github repos:</p>
<ul>
<li><a href="http://hackage.haskell.org/package/gopro-plus">GoPro Plus client library</a></li>
<li><a href="https://github.com/dustin/gpmf">GPMF Parser</a></li>
<li><a href="https://github.com/dustin/gopro">GoPro Plus CLI/Web UI</a></li>
</ul>
<p>The GoPro Plus client API is fairly complete. There are parts I
understand, but haven’t really used because I haven’t figured out what
I want to do with them. e.g., I couldn’t figure out how to retrieve
the contents of a GoPro Plus collection (for sharing), so they’re not
super useful to me if they’re write-only. Also, device lists and
stuff aren’t that interesting, but this code is pretty easy to work
on. I’ve considered <code class="language-plaintext highlighter-rouge">fixup</code>-style editing into this library because
it’d be relatively easy.</p>
<p>The GPMF Parser low-level bits seemed to work with every sample I
could find, but I do think there are some theoretical things I didn’t
implement. I can’t find anything that uses them, though. The <code class="language-plaintext highlighter-rouge">DEVC</code>
stuff for higher-level abstraction has plenty of room for expansion in
areas that aren’t that interesting to my current project.</p>
<p>The GoPro CLI and Web UI are good enough for me right now, which isn’t
saying that much since I’m not much of a web designer.</p>
<p><img src="/images/gopro-ui.png" alt="gopro ui" /></p>
<h2 id="in-summary">In Summary</h2>
<p>With a bit of reverse engineering, learning a new programming
language, delving into parsing video telemetry extracting poring
through a silly number of third-party libraries to do all the things I
want (e.g., exif, websockets, amazonka, etc…), I made a cloud
service personally useful.</p>
<p>Hopefully someone else finds this work useful, too, but if I’m the
only GoPro Plus user out there, then at least I’ve got awesome tools. :)</p>
Playing with LEDs2015-06-28T00:00:00+00:00http://dustin.github.com/2015/06/28/ws2182<h1 id="playing-with-ws2812-leds">Playing with WS2812 LEDs</h1>
<div>
<img src="/images/led.jpg" alt="led" title="a WS2812 multicolor LED" class="floatright" width="300" height="307" />
</div>
<p>I’ve been playing with WS2812 addressable LEDs for a few days now.
They’re kind of neat. You can read a lot about these in
<a href="https://learn.adafruit.com/adafruit-neopixel-uberguide/overview">Adafruit’s Neopixel Überguide</a>, but there are a wide variety of
these things available and lots of interesting applications.</p>
<p>I ordered a handful from Adafruit last time I needed some stuff from
there and finally got around to playing with them.</p>
<p>Of course, the first thing I did was wire them up and try to play with
existing sketches. That was neat, but I wanted to figure out how hard
it would be to deploy them into something else.</p>
<p>I came across this really great <a href="https://github.com/TauLabs/TauLabs/wiki/PicoC-Project:-RGB-LEDs">PicoC project for Tau Labs</a>
that embedded an attiny85 into a servo wire and used it to provider a
higher level abstraction to interacting with the LEDs. I built one of
these real quick (though changed the wire protocol slightly to allow
me to address one pixel at a time).</p>
<h2 id="fridays-projects">Friday’s Projects</h2>
<p><img src="https://github.com/dustin/logic-ws2812/raw/master/docs/ws2812.png" alt="logic screenshot" title="Logic Plugin for WS2812" width="640" class="centered" /></p>
<p>Josh.com had a <a href="http://wp.josh.com/2014/05/13/ws2812-neopixels-are-not-so-finicky-once-you-get-to-know-them/">nice informative post</a> explaining a lot of
details about other ways to play with them, which got me very
interested in sniffing stuff out and see what’s going on should I try
out a few of these alternative libraries.</p>
<p>This inspired me to write friday’s first project of the evening – a
<a href="https://github.com/dustin/logic-ws2812">WS2812 Saleae Logic plugin</a>. As seen in the screenshot above,
this detects which lights should be which color at exactly what time
offset as things go across the wire, both interactively above, and in
an export allowing further processing.</p>
<p>So the obvious second project for Friday night was to write a playback
tool for these recordings. The nice side-effect is that I was able to
see what a particular program would look like on a longer strand of
LEDs without having a long strand. I could just tell an arduino I had
N LEDs and record the signal it sent out to the phantom LEDs (without
even having to plug a single one in).</p>
<p>e.g., I only have 5, but my simulator let me see what 13
would look like on strandtest by running the actual program on my
actual arduino. The result looks like this:</p>
<div id="wssim"></div>
<script src="https://cdnjs.cloudflare.com/ajax/libs/d3/3.5.5/d3.min.js" charset="utf-8"></script>
<script src="/static/ws2812.js" charset="utf-8"></script>
<p>Initially, I was running the default configured for 60 LEDs. I
certainly don’t have that many, but it makes a <a href="http://bl.ocks.org/dustin/e7207542c69ecbb53ca9">nicer demo</a>.</p>
<h2 id="saturday">Saturday</h2>
<p>Saturday, I had dance recitals and other stuff, so I didn’t get much
done. My biggest accomplishment was building a bit of code I could
run on my attiny85 to allow PWM-based light mode selection. I coded
up a couple of modes and had a simple table to select which mode
based on the minimum PWM level (in μS) like this:</p>
<figure class="highlight"><pre><code class="language-c" data-lang="c"><span class="k">struct</span> <span class="p">{</span>
<span class="kt">unsigned</span> <span class="kt">long</span> <span class="n">minVal</span><span class="p">;</span>
<span class="kt">void</span> <span class="p">(</span><span class="o">*</span><span class="n">mode</span><span class="p">)();</span>
<span class="p">}</span> <span class="n">modes</span><span class="p">[]</span> <span class="o">=</span> <span class="p">{</span>
<span class="p">{</span><span class="mi">1500</span><span class="p">,</span> <span class="n">flash</span><span class="p">},</span>
<span class="p">{</span><span class="mi">900</span><span class="p">,</span> <span class="n">pulse</span><span class="p">},</span>
<span class="p">{</span><span class="mi">0</span><span class="p">,</span> <span class="n">emergency</span><span class="p">},</span>
<span class="p">};</span></code></pre></figure>
<p>This is a dumber version of the Tau Labs thing above, but it means it
can work with dumber things, which is nice.</p>
<h2 id="sunday">Sunday</h2>
<p><img src="/images/livermore.jpg" alt="livermore" title="Livermore" class="centered" /></p>
<p>Sunday, I ended up out driving most of the day. It was nice out. Not
a lot of time again, but I did get to miniaturize the bits from
yesterday.</p>
<p>I had to hack together a 500Ω resistor out of two 1kΩ 0603 resistors I
soldered together to attach the output signal wire. You can see the
progress of this as it went from prototype on a breadboard, through
the tiny soldering to the ready-for-consumer product (almost):</p>
<p><img src="/images/ws2812-wire.jpg" alt="ws2812 wire" title="WS2812 Wire" class="centered" /></p>
<p>In the end, I have my mode selection on a wire, demoed below. Note
“emergency” mode is when there’s no PWM signal at all, as in the
beginning of this video (flash red). Then I switched to the servo
tester to try varying the PWM level to see it hit different modes. As
shown in the snippet above, “pulse” is on the lower end and “flash” is
on the upper end. The names might not be useful, but you can see the
difference in the demo.</p>
<iframe width="560" height="315" src="https://www.youtube.com/embed/FuRvBpahmBI" frameborder="0" allowfullscreen="1"></iframe>
<p>I haven’t fully figured out what I want to do with the lights yet, but
I’m having fun.</p>
Opensky2015-06-10T00:00:00+00:00http://dustin.github.com/2015/06/10/opensky<h1 id="opensky">Opensky</h1>
<div>
<img src="/images/openskyrx.jpg" alt="opensky" title="an opensky rx" class="floatright" width="300" height="300" />
</div>
<p>I’ve been doing a lot hardware stuff recently. I didn’t know much
about hardware when I started and I like learning, so this is a lot of
fun for me. Let me introduce you to a project I’ve been doing some
work on lately called Opensky.</p>
<p>Opensky is an open source hardware and software RC receiver that
speaks the Frsky D8 protocol, so it interoperates with
<a href="http://www.frsky-rc.com/product/pro.php?pro_id=137">Frsky Taranis</a>. To the right is an Opensky receiver I
built and use on one of my smaller flying things.</p>
<p>My contributions to Opensky so far have been small. I documented the
<a href="https://github.com/Brotronics/OpenSky">hardware requirements (BOM)</a> in case someone else wanted
to build one and added a few firmware features. I made it easier to
bind to a transmitter after flashing the firmware and added RSSI PPM
injection since I like the way that works in <a href="http://openlrsng.org/">OpenLRSng</a>.</p>
<p>Failsafe is critical for a remote control system, and Opensky’s
failsafe works reasonably well, but can be a bit tricky to set up
correctly. The existing failsafe detects a loss of radio link and
sets all channels to 1000μS and continues these pulses. The problem
is that I’m getting about 995μS pulses at minimum, so the failsafe
value is actually higher than my lowest transmitted value. The flight
controller can’t know that the connection is down without some odd
tweaking, so the failsafe ends up not being very safe in this case.</p>
<p>Last week’s project was writing a couple <a href="https://www.saleae.com/">Saleae Logic</a>
analyzer plugins for helping me analyze this sort of thing. I didn’t
actually have an application at the time, but I figured it’d be fun.
As it turns out, my <a href="https://github.com/dustin/logic-cppm">CPPM</a> analyzer was helpful in figuring
out how to make a better failsafe last night.</p>
<p>First, I took at look at what the <a href="http://www.frsky-rc.com/product/pro.php?pro_id=24">d4r-ii</a> “no pulses” failsafe
looked like on the wire. It’s not exactly what I expected, but
shortly after turning off my transmitter, the signal line went high
and stayed there:</p>
<div>
<img src="/images/d4r-failsafe.png" width="100%" class="centered" alt="d4r failsafe" title="D4R Failsafe" />
</div>
<p>Getting the Opensky to do that was a <a href="https://github.com/dustin/OpenSkyRX/commit/7715d6eb60944557a1661f55530ae8639f6b8ae0">small bit of code</a>,
but I’d kind of screwed myself over on the programming aspect.</p>
<p>The Opensky RX is based on Arduino in the sense that it uses at
atmega328p microcontroller and uses the arduino toolkit for building
the firmware images. The receiver itself has a UART exposed and can
be programmed directly from the Arduino software if you include the
boot loader.</p>
<div>
<img src="/images/programming-opensky.jpg" alt="programming-opensky" title="my awesome ICSP setup" class="floatleft" width="300" height="222" />
</div>
<p>But of course, I didn’t do that when I made mine. So I had to rig up
an ICSP setup with the soldering of a bunch of tiny bits of wire and
metal, some chip clips, and a bunch of other crap that ended up
looking like the thing to the left.</p>
<p>You can’t really see all the detail in the mess, but the one wire mast
that has to be dropped approximately into the center of the board is
non-trivial. But it worked. As a favor to my future self, I went
ahead and added the arduino bootloader.</p>
<p>Adding the arduino bootloader provides a couple major advantages.
First, I can program the thing without all that wiring difficulty.
But also, I can actually interact with the thing’s serial port for
debugging and stuff. I figured I’d give this a go.</p>
<div>
<img src="/images/opensky-devkit.jpg" alt="programming-opensky v2" title="Opensky Dev Kit V2" class="floatright" width="300" height="222" />
</div>
<p>To the right, you can see the current version of my opensky dev kit.
I installed a single right angle pin to get to DTR off the Opensky RX
itself, and then just used a six pin female connector to plug it in.
I added male connectors to access all the individual pins, but ran a
common ground across the board.</p>
<p>The board also has a 3v3 regulator to run the RX off a battery as well
as another six pin female connector (and breakout male pins) that
allow me to plug a CP2102 (USB/UART adaptor) breakout board in. Of
course, the serial port and DTR are already wired up. You just need
one jumper wire to specify the desired power source (battery or the
CP2102’s 3v3 output).</p>
<p>Hopefully this allows me to do some more dev on this thing since there
are more things I’d like to do.</p>
<p>In the meantime, here are a couple of pictures of how I’m using one of
these in real life:</p>
<p>I attached the Opensky RX directly to an <a href="http://www.multirotorsuperstore.com/flight-controllers/afromini-32.html">afromini</a>. The RX
itself requires 3V3 (any more and you kill the radio), so I removed
the input voltage pin I had an ran a bit of 30AWG silicone wire to the
output of the LDO that provides power to the main processor on the
flight controller. This worked out pretty well:</p>
<div>
<img src="/images/opensky-afromini.png" width="100%" class="centered" alt="opensky on afromini" title="Opensky on Afromini" />
</div>
<p>The above is the control system for my 190mm quad which has been quite
fun to fly. The RSSI over PPM lets me verify I’m not getting a
weakened control signal (on OSD) as I fly around.</p>
<div>
<img src="/images/sparrow.png" width="100%" class="centered" alt="my sparrow" title="Completed Sparrow (with opensky)" />
</div>
<p>If you want to make an Opensky RX yourself, check out the
[hardware][openskyrx] page for the list of parts to get started along
with a couple boards from <a href="https://oshpark.com/shared_projects/p9XDsm1u">oshpark</a>. There’s more
documentation to write, but I found enough to get stuff working quite
well.</p>
<p>This stuff keeps me busy. I feel like have no idea what I’m doing,
but in the end, the things I make fly (and help others make things
fly). That’s very rewarding.</p>
Taranis and the Nano QX2014-12-25T00:00:00+00:00http://dustin.github.com/2014/12/25/taranis-and-the-nano-qx<h1 id="whats-this">What’s This?</h1>
<div>
<img src="/images/qav400.jpg" alt="qav400" title="My QAV400 in a test flight" class="floatright" width="406" height="295" />
</div>
<p>So, this isn’t my typical programming post, but I wanted to write
about stuff I learn in my hobby time. I fly things.</p>
<p>Building and flying is a fun, but I have a specific topic that might
be interesting.</p>
<p>To the right is a picture I took during an early test flight of a quad
I built. Building and flying is pretty fun, but you need a lot of
space, big batteries, good weather, etc…</p>
<p>I want to write about some of my “development” work in the meantime,
though. It’s Christmas today. The weather’s not too bad, but it’s a
bit windy. Since it’s winter, we also don’t have as much light, so I
want something I can fly indoors.</p>
<div>
<img src="/images/fpvnanoqx.jpg" alt="fpv nano qx" title="The FPV Blade Nano QX is great for indoor training" class="floatleft" width="210" height="158" />
</div>
<p>So I practice a lot indoors on my <a href="http://www.bladehelis.com/Products/Default.aspx?ProdID=BLH7280">Blade Nano QX</a> for both
<abbr title="line of sight">LoS</abbr> and
<abbr title="first person view">FPV</abbr> flying. I slam it into
walls and people and what-not with minimal harm.</p>
<p>If you pick one of these up with one of the
<abbr title="ready to fly">RTF</abbr> kits, you get a <em>terrible</em>
radio controller. It’s OK to learn some basics, but won’t get you
very far (literally, my tiny apartment exceeds the range of this
thing).</p>
<h1 id="better-control">Better Control</h1>
<div>
<img src="/images/taranis.png" alt="taranis+" title="This is my radio, there are many like it, but this one is mine" class="floatright" width="240" height="250" />
</div>
<p>My weapon of choice is the <a href="http://www.frsky-rc.com/product/pro.php?pro_id=137">Taranis+</a>. It’s basically a
weird computer whose interface is a bunch of switches and knobs and
joysticks that you can use to control things over RF.</p>
<p>Note that the Nano QX requires <a href="https://www.spektrumrc.com/Technology/DSM2.aspx">DSM2</a> or <a href="https://www.spektrumrc.com/Technology/DSMX.aspx">DSMX</a> for
control and the Taranis won’t do that natively, but I got an
<a href="http://www.hobbyking.com/hobbyking/store/__51704__OrangeRX_DSMX_DSM2_Compatible_2_4Ghz_Transmitter_Module_JR_Turnigy_compatible_US_Warehouse_.html?strSearch=orangerx">OrangeRx</a> module that speaks the right protocols and
plugged it in as an external radio.</p>
<p>In order to get basic flight control of the Nano QX, you plug in the
module, and set up the following:</p>
<ul>
<li>Thrust on Channel 1</li>
<li>Aileron on Channel 2</li>
<li>Elevator on Channel 3</li>
<li>Rudder on Channel 4</li>
<li>Flight Mode on Channel 6</li>
</ul>
<p>Note that flight mode is a toggle, so I mix it in from a momentary
switch (<code class="language-plaintext highlighter-rouge">SH</code>) on the Taranis. This is a bit unfortunate, because it
means the radio has no way to tell what flight mode you’re in – you
have to just look at the stupid lights to see what it’s doing.</p>
<p>Also note that aileron and rudder are reversed, so in the “servos”
config, you’ll need to mark them as inverted. e.g.:</p>
<div>
<img src="/images/taranis-qx-servo.png" width="424" height="128" class="centered" alt="inversed servos" title="Note inverted servos" />
</div>
<p>At this point, you should have basic flight operations.</p>
<h1 id="advanced-mixing">Advanced Mixing</h1>
<p>But that’s all background. The main thing I wanted to write about is
how I use <a href="http://www.open-tx.org/">OpenTX</a> to actually do interesting things with this
model.</p>
<h2 id="taming-acro-mode">Taming Acro Mode</h2>
<p>First, the acro mode on the Nano has been described as “fidgety.” In
that mode, rather than auto-leveling when you let off the sticks, the
pitch and roll are basically held such that aileron and elevator
control change the rate at which it rotates in the specified
direction. Tiny, tiny adjustments will just about flip the thing.
This wasn’t fun for me, so the first thing I did was make a curve I
could apply to pitch and roll to give me subtle controls in the
middle, but still allow me to do flips and stuff.</p>
<div>
<img src="/images/taranis-qx-curvy.png" width="424" height="128" class="centered" alt="control curve" title="Real controls have curves" />
</div>
<p>To apply this, I made a flight mode on the Taranis controlled by <code class="language-plaintext highlighter-rouge">SA</code>.
I only change ail and ele control in mixes. For example, existing ele
control would be set for only flight mode 0. The new mode (2 at this
point) gets a new mix that’s only for flight mode 2 and works pretty
much like the default, but applies the curve defined above. Repeat
for ail.</p>
<p>Now you jump into the right flight mode on the taranis and flip <code class="language-plaintext highlighter-rouge">SH</code>
until the light turns red and bam – it’s flyable.</p>
<h2 id="auto-banking-via-super-advanced-mixing">Auto Banking (via Super Advanced Mixing)</h2>
<p>That was fun and practical, but I wanted see if I could automatically
bank the craft while flying when I try to turn only using yaw.</p>
<p><strong>Goal</strong>: In isolation, yaw, pitch, and roll should all work normally.
But when pitched forward, yaw should proportionally also apply roll to
bank the aircraft.</p>
<p><strong>Secondary goal</strong>: I have no idea how much to do this, so I want to
make the amount configurable on the fly via one of the sliders (I used
<code class="language-plaintext highlighter-rouge">LS</code>).</p>
<p>This was non-obvious enough to make me want to post about it.</p>
<h3 id="flight-mode-setup">Flight mode Setup</h3>
<p>We create <code class="language-plaintext highlighter-rouge">FM1</code> for this new flight mode. For the initial goal, we’re
not adjusting anything, so nothing special just yet.</p>
<h3 id="limit-rudder-optional">Limit Rudder (optional)</h3>
<p>For <code class="language-plaintext highlighter-rouge">FM1</code>, I modify my <code class="language-plaintext highlighter-rouge">CH4</code> (Rud) mix to a weight of 50%. This is
optional, but since my goal is to fly around and do circles and stuff,
I thought slowing down the pirouettes was helpful.</p>
<p>If you do this, this should override the existing rudder control.</p>
<h3 id="make-the-curve">Make the Curve</h3>
<p>We need a curve that takes us from zero in the middle to 100 at either
size. A smoothed curve is good here. I use the following:</p>
<div>
<img src="/images/taranis-qx-arcurve.png" width="424" height="128" class="centered" alt="autobank control curve" title="curving the roll" />
</div>
<p>This curve is used to grab an absolute distance from zero in elevator
control, as we want to bank proportionally to the pitch regardless of
the direction we’re going.</p>
<h3 id="configure-aileron-channel">Configure Aileron Channel</h3>
<p>So here’s the kind of tricky part. For the FM3 aileron channel
mixer, we want three inputs.</p>
<ol>
<li>Rudder at 50%</li>
<li>Elevator at 50% with the above curve applied <em>multiplied</em> in</li>
<li>Aileron input <em>added</em> in</li>
</ol>
<p>This allows us to mix the rudder and elevator to compute an aileron
while still giving us relatively normal aileron control, though it’ll
be a bit exaggerated at speed if you try to bank manually and/or
difficult to flatten.</p>
<h3 id="dynamically-changing-bank-amount">Dynamically Changing Bank Amount</h3>
<p>The 50% up there was an arbitrary number. I don’t actually know what
a good number is, and it sucks to reconfigure everything when you want
to experiment. That’s why we have knobs and sliders and stuff.</p>
<p>We can adjust weights with a global variable, so we just need a way to
adjust the global variable. Firstly, have the global <code class="language-plaintext highlighter-rouge">GV1</code> variable
owned by the flight mode, as shown here, defaulting to 0.</p>
<div>
<img src="/images/taranis-qx-gvar.png" width="424" height="128" class="centered" alt="global variables" title="setting the global vars" />
</div>
<p>Because we want to adjust the weight from 0 to 100 (unless you want
counter-banking), we need a curve to adjust the slider input linearly
over that amount.</p>
<p>So basically this curve:</p>
<div>
<img src="/images/taranis-qx-adjcurve.png" width="424" height="128" class="centered" alt="adjustment curve" title="linear 0-100 curve for adjustment" />
</div>
<p>Applied to a new input with <code class="language-plaintext highlighter-rouge">LS</code> as a source (I used input slot 10
here just to separate from the radio channels):</p>
<div>
<img src="/images/taranis-qx-adjinput.png" width="424" height="128" class="centered" alt="adjustment input" title="adjustment input" />
</div>
<p>Then it’s a simple matter of having a special function set the value
of <code class="language-plaintext highlighter-rouge">GV1</code> to the cooked output of this input (<code class="language-plaintext highlighter-rouge">IAAdj</code>).</p>
<p>Then just swap out the <code class="language-plaintext highlighter-rouge">50</code> weight above with the variable <code class="language-plaintext highlighter-rouge">GV1</code> and
you’re done!</p>
<p>Rudder example:</p>
<div>
<img src="/images/taranis-qx-adjrud.png" width="424" height="128" class="centered" alt="adjusted rudder" title="adjusted rudder" />
</div>
<p>Repeat the same for elevator and enjoy the magic.</p>
<h1 id="but-im-lazy">But I’m Lazy!</h1>
<p>You can look over the OpenTX <a href="/static/nanoqx-printed.html">printout</a> of the
model, or just <a href="https://raw.githubusercontent.com/dustin/taranis/master/nanoqx.eepe">download the model</a> directly to play with
it.</p>
Pools in Go2014-04-25T00:00:00+00:00http://dustin.github.com/2014/04/25/chan-pool<div>
<img src="/images/pool.jpg" alt="poolin' gophers" title="Gophers in a Pool" class="center" width="560" height="374" />
</div>
<h1 id="pooling-in-go">Pooling in Go</h1>
<p>A few people in irc have asked how to build an object pool. I’ve had
a need for this in the past and came up with a design I rather like,
so I’ve walked people through the code a few times.</p>
<p>Since I’m at <a href="http://www.gophercon.com/">Gophercon</a> and I had a similar discussion
today, I figured I’d write up a detailed description of how my code
works, how it progressed and why it is the way it is now.</p>
<h2 id="channels-as-a-pool">Channels as a Pool</h2>
<p>You can think of a buffered channel in go as a thread-safe queue. It
also has this nice property of blocking if it’s too full or too
empty.</p>
<p>Channels practically do all the work for you.</p>
<p>For example, the core of the first version of the pool I wrote for
<a href="https://github.com/couchbaselabs/go-couchbase">go-couchbase</a> follows. <code class="language-plaintext highlighter-rouge">cp.connections</code> is a buffered
channel of <code class="language-plaintext highlighter-rouge">*memcached.Client</code>. <code class="language-plaintext highlighter-rouge">cp.mkConn</code> will build a new
connection whenever one is needed.</p>
<figure class="highlight"><pre><code class="language-go" data-lang="go"><span class="k">func</span> <span class="p">(</span><span class="n">cp</span> <span class="o">*</span><span class="n">connectionPool</span><span class="p">)</span> <span class="n">Get</span><span class="p">()</span> <span class="p">(</span><span class="o">*</span><span class="n">memcached</span><span class="o">.</span><span class="n">Client</span><span class="p">,</span> <span class="kt">error</span><span class="p">)</span> <span class="p">{</span>
<span class="k">select</span> <span class="p">{</span>
<span class="k">case</span> <span class="n">rv</span> <span class="o">:=</span> <span class="o"><-</span><span class="n">cp</span><span class="o">.</span><span class="n">connections</span><span class="o">:</span>
<span class="c">// Existing connection</span>
<span class="k">return</span> <span class="n">rv</span><span class="p">,</span> <span class="no">nil</span>
<span class="k">case</span> <span class="o"><-</span><span class="n">time</span><span class="o">.</span><span class="n">After</span><span class="p">(</span><span class="n">time</span><span class="o">.</span><span class="n">Millisecond</span><span class="p">)</span><span class="o">:</span>
<span class="c">// No existing connection, let's make one</span>
<span class="k">return</span> <span class="n">cp</span><span class="o">.</span><span class="n">mkConn</span><span class="p">(</span><span class="n">cp</span><span class="o">.</span><span class="n">host</span><span class="p">,</span> <span class="n">cp</span><span class="o">.</span><span class="n">name</span><span class="p">)</span>
<span class="p">}</span>
<span class="p">}</span>
<span class="k">func</span> <span class="p">(</span><span class="n">cp</span> <span class="o">*</span><span class="n">connectionPool</span><span class="p">)</span> <span class="n">Return</span><span class="p">(</span><span class="n">c</span> <span class="o">*</span><span class="n">memcached</span><span class="o">.</span><span class="n">Client</span><span class="p">)</span> <span class="p">{</span>
<span class="k">select</span> <span class="p">{</span>
<span class="k">case</span> <span class="n">cp</span><span class="o">.</span><span class="n">connections</span> <span class="o"><-</span> <span class="n">c</span><span class="o">:</span>
<span class="k">default</span><span class="o">:</span>
<span class="c">// Overflow connection</span>
<span class="n">c</span><span class="o">.</span><span class="n">Close</span><span class="p">()</span>
<span class="p">}</span>
<span class="p">}</span></code></pre></figure>
<p>The first time you try to get a connection from this pool, the channel
is empty, so it hits the delay path, opens a connection, and then
returns the newly created connection. While that’s in use, a second
would also do the same thing.</p>
<p>Returning a connection is simply a non-blocking send back into the
channel. The reason it’s non-blocking is subtle, but should hopefully
be obvious – if there’s space in the channel, we keep the
connection. If there’s no space, it was overflow and more than we
want to keep in the pool, so we immediately close it.</p>
<h2 id="problems">Problems?</h2>
<p>This served me well for quite a while, but one of my users had an
application in which hundreds of goroutines simultaneously wanted to
perform a quick DB operation. All of them timed out instantly, all of
them opened new connections and then all but the pool size (~5?)
closed them because they were overflow.</p>
<p>So how can we limit the total number of connections? I dreaded fixing
this problem for a long time. I was thinking about having a counter
for the number of connections outstanding, which would need a lock,
and then a condition for notifying waiters availability of space
whenever a new slot became available and then loop. But mixing that
with the existing channel op would be painfully tedious.</p>
<p>Turns out, you just need another channel to control connections. This
brings us to go’s most amazing feature: <code class="language-plaintext highlighter-rouge">select</code>. This issue quickly
degraded from something I was trying to push back to the application
author to work around to basically the following code:</p>
<figure class="highlight"><pre><code class="language-go" data-lang="go"><span class="k">func</span> <span class="p">(</span><span class="n">cp</span> <span class="o">*</span><span class="n">connectionPool</span><span class="p">)</span> <span class="n">Get</span><span class="p">()</span> <span class="p">(</span><span class="n">rv</span> <span class="o">*</span><span class="n">memcached</span><span class="o">.</span><span class="n">Client</span><span class="p">,</span> <span class="n">err</span> <span class="kt">error</span><span class="p">)</span> <span class="p">{</span>
<span class="c">// Try to grab an available connection within 1ms</span>
<span class="k">select</span> <span class="p">{</span>
<span class="k">case</span> <span class="n">rv</span> <span class="o">:=</span> <span class="o"><-</span><span class="n">cp</span><span class="o">.</span><span class="n">connections</span><span class="o">:</span>
<span class="k">return</span> <span class="n">rv</span><span class="p">,</span> <span class="no">nil</span>
<span class="k">case</span> <span class="o"><-</span><span class="n">time</span><span class="o">.</span><span class="n">After</span><span class="p">(</span><span class="n">time</span><span class="o">.</span><span class="n">Millisecond</span><span class="p">)</span><span class="o">:</span>
<span class="c">// No connection came around in time, let's see</span>
<span class="c">// whether we can get one or build a new one first.</span>
<span class="k">select</span> <span class="p">{</span>
<span class="k">case</span> <span class="n">rv</span> <span class="o">:=</span> <span class="o"><-</span><span class="n">cp</span><span class="o">.</span><span class="n">connections</span><span class="o">:</span>
<span class="k">return</span> <span class="n">rv</span><span class="p">,</span> <span class="no">nil</span>
<span class="k">case</span> <span class="n">cp</span><span class="o">.</span><span class="n">createsem</span> <span class="o"><-</span> <span class="no">true</span><span class="o">:</span>
<span class="c">// Room to make a connection</span>
<span class="n">rv</span><span class="p">,</span> <span class="n">err</span> <span class="o">:=</span> <span class="n">cp</span><span class="o">.</span><span class="n">mkConn</span><span class="p">(</span><span class="n">cp</span><span class="o">.</span><span class="n">host</span><span class="p">,</span> <span class="n">cp</span><span class="o">.</span><span class="n">auth</span><span class="p">)</span>
<span class="k">if</span> <span class="n">err</span> <span class="o">!=</span> <span class="no">nil</span> <span class="p">{</span>
<span class="c">// On error, release our create hold</span>
<span class="o"><-</span><span class="n">cp</span><span class="o">.</span><span class="n">createsem</span>
<span class="p">}</span>
<span class="k">return</span> <span class="n">rv</span><span class="p">,</span> <span class="n">err</span>
<span class="p">}</span>
<span class="p">}</span>
<span class="p">}</span>
<span class="k">func</span> <span class="p">(</span><span class="n">cp</span> <span class="o">*</span><span class="n">connectionPool</span><span class="p">)</span> <span class="n">Return</span><span class="p">(</span><span class="n">c</span> <span class="o">*</span><span class="n">memcached</span><span class="o">.</span><span class="n">Client</span><span class="p">)</span> <span class="p">{</span>
<span class="k">select</span> <span class="p">{</span>
<span class="k">case</span> <span class="n">cp</span><span class="o">.</span><span class="n">connections</span> <span class="o"><-</span> <span class="n">c</span><span class="o">:</span>
<span class="k">default</span><span class="o">:</span>
<span class="c">// Overflow connection.</span>
<span class="o"><-</span><span class="n">cp</span><span class="o">.</span><span class="n">createsem</span>
<span class="n">c</span><span class="o">.</span><span class="n">Close</span><span class="p">()</span>
<span class="p">}</span>
<span class="p">}</span></code></pre></figure>
<p>Notice the new channel <code class="language-plaintext highlighter-rouge">cp.createsem</code> which is the semaphore we use
for opening connections. The buffer size controls how many total
connections we can possibly have outstanding. Each time we establish
a new connection, we place an item in that channel. When we close a
connection, we remove it again.</p>
<p><code class="language-plaintext highlighter-rouge">Return</code> is roughly the same, save the overflow semaphore management.</p>
<p>The <code class="language-plaintext highlighter-rouge">Get</code> method is trickier. There’s a <code class="language-plaintext highlighter-rouge">select</code> block nested within
another <code class="language-plaintext highlighter-rouge">select</code> block. The outer one only attempts to wait for an
existing connection. If one becomes available relatively soon (one
millisecond), we just use it. It’s generally better to use an
existing connection than open a new one.</p>
<p>If the pool wait times out, we go into the inner <code class="language-plaintext highlighter-rouge">select</code>. It should
be noted that this is similar, but is different in an important way.
We still wait for an available connection, but we also wait for the
ability to create a new connection. Waiting for this connection
semaphore in the outer <code class="language-plaintext highlighter-rouge">select</code> block would cause us to open
connections prematurely.</p>
<p>Rolling them into a single <code class="language-plaintext highlighter-rouge">select</code> is perfectly valid if the objects
and their creation are cheap, but when they’re not, it’s easy enough
to prefer reuse over creation as I’ve done here.</p>
<h2 id="final-version">Final Version</h2>
<p>The above shows an approximate evolution of my pool. Check out the
<a href="https://github.com/couchbaselabs/go-couchbase/blob/master/conn_pool.go">production code</a> if you want to see the final hardened
version. There are a few differences between the above code code and
what’s in production:</p>
<ol>
<li>nil receivers are supported (will close a connection if there’s no pool)</li>
<li>returning nil is supported (does nothing)</li>
<li>there are health checks on the connection objects</li>
<li>some tracing exists to help understand which path was used to
establish connections.</li>
<li>connection pool shutdown is supported via closing and clearing the
connection channel</li>
<li>there’s a short-circuit connection pool that avoids allocating a
timer</li>
<li>there’s always an overall timeout on acquiring a connection</li>
</ol>
<p>Otherwise, the theory of operation is as described above.</p>
<p>Also, for a good time, notice that I’ve got 100% <a href="https://github.com/couchbaselabs/go-couchbase/blob/master/conn_pool_test.go">test coverage</a>
on this pool. That’s really quite nice considering how much crap it
has to deal with.</p>
<h2 id="bonus--connectionpoolclose">Bonus: (*connectionPool).Close()</h2>
<p>This is only mildly related, but I wanted to describe the connection
pool shutdown mechanism. It’s slightly clever and super dense, but
I’m keeping it.</p>
<p>The purpose of this method is to close all connections available in
the pool and signal to anything waiting for the pool that it’s closed
and should error immediately (these are all the <code class="language-plaintext highlighter-rouge">errPoolClosed</code> paths
you can see in <a href="https://github.com/couchbaselabs/go-couchbase/blob/master/conn_pool.go">the code</a>).</p>
<p>This is the code in its entirety:</p>
<figure class="highlight"><pre><code class="language-go" data-lang="go"><span class="k">func</span> <span class="p">(</span><span class="n">cp</span> <span class="o">*</span><span class="n">connectionPool</span><span class="p">)</span> <span class="n">Close</span><span class="p">()</span> <span class="p">(</span><span class="n">err</span> <span class="kt">error</span><span class="p">)</span> <span class="p">{</span>
<span class="k">defer</span> <span class="k">func</span><span class="p">()</span> <span class="p">{</span> <span class="n">err</span><span class="p">,</span> <span class="n">_</span> <span class="o">=</span> <span class="nb">recover</span><span class="p">()</span><span class="o">.</span><span class="p">(</span><span class="kt">error</span><span class="p">)</span> <span class="p">}()</span>
<span class="nb">close</span><span class="p">(</span><span class="n">cp</span><span class="o">.</span><span class="n">connections</span><span class="p">)</span>
<span class="k">for</span> <span class="n">c</span> <span class="o">:=</span> <span class="k">range</span> <span class="n">cp</span><span class="o">.</span><span class="n">connections</span> <span class="p">{</span>
<span class="n">c</span><span class="o">.</span><span class="n">Close</span><span class="p">()</span>
<span class="p">}</span>
<span class="k">return</span>
<span class="p">}</span></code></pre></figure>
<p>So, ignore the <code class="language-plaintext highlighter-rouge">defer</code> for a second. We close the connections channel
and then iterate the channel closing each connection within it. Then
we return. The return is naked, so it’s just going to return the
current value of <code class="language-plaintext highlighter-rouge">err</code>, which is <code class="language-plaintext highlighter-rouge">nil</code> (could also <code class="language-plaintext highlighter-rouge">return nil</code> –
makes no difference here).</p>
<p>Pretty straightforward.</p>
<p>OK, now stop ignoring the <code class="language-plaintext highlighter-rouge">defer</code>. Why is that there? Well, if you
call <code class="language-plaintext highlighter-rouge">Close</code> twice for some reason, the channel close will panic.
This <code class="language-plaintext highlighter-rouge">recover</code> will catch that panic and convert it to an error and
store it in <code class="language-plaintext highlighter-rouge">err</code>, the method’s return value. The first call
returns nil and the second returns an error. Admittedly, it’s not an
awesome error that is obvious what you’ve done wrong, but it doesn’t
panic.</p>
<p>But what happens in the first call case? In the first call, <code class="language-plaintext highlighter-rouge">recover</code>
returns something that isn’t an error (<code class="language-plaintext highlighter-rouge">nil</code>). We do a two-value type
assertion which returns a nil value and false. Then we assign this
new <code class="language-plaintext highlighter-rouge">nil</code> to our <code class="language-plaintext highlighter-rouge">err</code> return variable. The method returns nil, but
for a different reason.</p>
<p>This method actually just returns whatever the first result of a
two-result type assertion on the <code class="language-plaintext highlighter-rouge">recover</code> is.</p>
Big Data2014-02-04T00:00:00+00:00http://dustin.github.com/2014/02/04/bigdata<div>
<img src="/images/bigdata.png" alt="big data" title="I like big data and I cannot lie." class="center" width="560" height="363" />
</div>
<h1 id="thanks-for-the-memories">Thanks for the memories</h1>
<p>A while back, there was a leak of a LinkedIn password data in the form
of a list of unsalted SHA-1 hashes. A few sites had password check
tools up such that you could provide your password or a hash of it and
it’d tell you whether it was found in the leaked information.</p>
<p>These sites were all really slow, and would sometimes report database
errors. I found it curious that anyone would even consider a database
for a fixed-size single record lookup of a small amount of immutable
data.</p>
<p>I downloaded the data set and played with it during a meeting. In
about a half hour, I had <a href="https://gist.github.com/dustin/2885182">a small server</a> that could load the data
set into memory in a couple seconds and serve responses from memory
stupidly fast an with perfect horizontal scalability.</p>
<h1 id="capacity-planning">Capacity planning</h1>
<p>I think in the LinkedIn case, there were 6.5 million hashes leaked.
SHA-1 hashes are 20 bytes.</p>
<p>That’s 130MB of data.</p>
<p>I have tabs in Chrome right now that are using more memory than this,
yet people deployed multi-tier infrastructure to answer simple
presence queries. They’re fragile, complex and slow.</p>
<h1 id="what-database-do-i-need">What database do I need?</h1>
<p>This brings me to my motivation for writing this. People have gone a
little bit overboard with thinking up big solutions to small problems.
I’m not trying to pick on any particular user, but I did have an
example that helps make the point pretty clearly.</p>
<p>I picked up a Stack Overflow question yesterday about
<a href="http://stackoverflow.com/questions/21479025/">checking for hash presence</a> that was almost an identical problem.
Note that the question is tagged <code class="language-plaintext highlighter-rouge">bigdata</code>. In this case, it was “a
few million” SHA-256es.</p>
<p>Initially, the user attempted to use both MySQL and Couchbase to solve
the problem. Apparently Couchbase used too much memory (presumably
using hex encoded keys) and MySQL was too slow, so he tried sharding
the table by first nibble, but it still was too slow, so he asked for
help.</p>
<p>Two of the answers were (reasonably sensible) suggestions for MySQL.
Some schema suggestions and configuration parameters that will help
with efficiency.</p>
<p>Another was suggesting some combination of hbase, redis, and
cassandra. That’s just… overkill. This is a super small scale
problem.</p>
<p>The spec said “several million” of these hashes. I wrote a small test
with 50 million hashes. 50e6 * 32 is about a gig and a half of RAM.
It’d be unusual to find a computer that couldn’t spare 1.5GB of RAM
for such processing. You have to get up to about a billion hashes
before it starts to get a little harder.</p>
<p>“But that won’t scale!” you say? An EC2 instance that can hold about
8 billion such hashes in memory costs about $3.50 per hour. By the
time you get to that level, you can think about something better
anyway.</p>
<h1 id="but-i-dont-want-to-write-a-lot-of-code">But I don’t want to write a lot of code!</h1>
<p>I pointed to <a href="https://gist.github.com/dustin/2885182">the code</a> I’d written in that meeting as an example
to get started. It contains both the text -> binary format convert
thingy as well as a web server that loads that file into memory and
returns an HTTP status that indicates presence. That was a distracted
half hour of work.</p>
<p>However, I realized that I’d since written <a href="https://github.com/dustin/go-hashset">go-hashset</a>. That
makes this type of problem <em>much</em> easier.</p>
<p>Below is a complete server using <code class="language-plaintext highlighter-rouge">go-hashset</code> and <code class="language-plaintext highlighter-rouge">net/http</code> that will
return <code class="language-plaintext highlighter-rouge">HTTP 204</code> on a hit and <code class="language-plaintext highlighter-rouge">HTTP 410</code> on a miss given a GET
request to <code class="language-plaintext highlighter-rouge">/[sha-256]</code>.</p>
<figure class="highlight"><pre><code class="language-go" data-lang="go"><span class="k">package</span> <span class="n">main</span>
<span class="k">import</span> <span class="p">(</span>
<span class="s">"encoding/hex"</span>
<span class="s">"log"</span>
<span class="s">"net/http"</span>
<span class="s">"os"</span>
<span class="s">"github.com/dustin/go-hashset"</span>
<span class="p">)</span>
<span class="k">const</span> <span class="p">(</span>
<span class="n">hashSize</span> <span class="o">=</span> <span class="m">32</span>
<span class="n">listenAddr</span> <span class="o">=</span> <span class="s">":8080"</span>
<span class="p">)</span>
<span class="k">func</span> <span class="n">loadFile</span><span class="p">(</span><span class="n">fn</span> <span class="kt">string</span><span class="p">)</span> <span class="o">*</span><span class="n">hashset</span><span class="o">.</span><span class="n">Hashset</span> <span class="p">{</span>
<span class="n">f</span><span class="p">,</span> <span class="n">err</span> <span class="o">:=</span> <span class="n">os</span><span class="o">.</span><span class="n">Open</span><span class="p">(</span><span class="n">fn</span><span class="p">)</span>
<span class="k">if</span> <span class="n">err</span> <span class="o">!=</span> <span class="no">nil</span> <span class="p">{</span>
<span class="n">log</span><span class="o">.</span><span class="n">Fatalf</span><span class="p">(</span><span class="s">"Error opening hash file: %v"</span><span class="p">,</span> <span class="n">err</span><span class="p">)</span>
<span class="p">}</span>
<span class="k">defer</span> <span class="n">f</span><span class="o">.</span><span class="n">Close</span><span class="p">()</span>
<span class="n">hs</span><span class="p">,</span> <span class="n">err</span> <span class="o">:=</span> <span class="n">hashset</span><span class="o">.</span><span class="n">Load</span><span class="p">(</span><span class="n">hashSize</span><span class="p">,</span> <span class="n">f</span><span class="p">)</span>
<span class="k">if</span> <span class="n">err</span> <span class="o">!=</span> <span class="no">nil</span> <span class="p">{</span>
<span class="n">log</span><span class="o">.</span><span class="n">Fatalf</span><span class="p">(</span><span class="s">"Error loading hashes: %v"</span><span class="p">,</span> <span class="n">err</span><span class="p">)</span>
<span class="p">}</span>
<span class="k">return</span> <span class="n">hs</span>
<span class="p">}</span>
<span class="k">func</span> <span class="n">main</span><span class="p">()</span> <span class="p">{</span>
<span class="n">hs</span> <span class="o">:=</span> <span class="n">loadFile</span><span class="p">(</span><span class="s">"hashes.bin"</span><span class="p">)</span>
<span class="n">http</span><span class="o">.</span><span class="n">HandleFunc</span><span class="p">(</span><span class="s">"/"</span><span class="p">,</span> <span class="k">func</span><span class="p">(</span><span class="n">w</span> <span class="n">http</span><span class="o">.</span><span class="n">ResponseWriter</span><span class="p">,</span> <span class="n">req</span> <span class="o">*</span><span class="n">http</span><span class="o">.</span><span class="n">Request</span><span class="p">)</span> <span class="p">{</span>
<span class="n">b</span><span class="p">,</span> <span class="n">_</span> <span class="o">:=</span> <span class="n">hex</span><span class="o">.</span><span class="n">DecodeString</span><span class="p">(</span><span class="n">req</span><span class="o">.</span><span class="n">URL</span><span class="o">.</span><span class="n">Path</span><span class="p">[</span><span class="m">1</span><span class="o">:</span><span class="p">])</span>
<span class="k">if</span> <span class="nb">len</span><span class="p">(</span><span class="n">b</span><span class="p">)</span> <span class="o">==</span> <span class="n">hashSize</span> <span class="o">&&</span> <span class="n">hs</span><span class="o">.</span><span class="n">Contains</span><span class="p">(</span><span class="n">b</span><span class="p">)</span> <span class="p">{</span>
<span class="n">w</span><span class="o">.</span><span class="n">WriteHeader</span><span class="p">(</span><span class="m">204</span><span class="p">)</span>
<span class="p">}</span> <span class="k">else</span> <span class="p">{</span>
<span class="n">w</span><span class="o">.</span><span class="n">WriteHeader</span><span class="p">(</span><span class="m">410</span><span class="p">)</span>
<span class="p">}</span>
<span class="p">})</span>
<span class="n">log</span><span class="o">.</span><span class="n">Printf</span><span class="p">(</span><span class="s">"Listening on %v"</span><span class="p">,</span> <span class="n">listenAddr</span><span class="p">)</span>
<span class="n">log</span><span class="o">.</span><span class="n">Fatal</span><span class="p">(</span><span class="n">http</span><span class="o">.</span><span class="n">ListenAndServe</span><span class="p">(</span><span class="n">listenAddr</span><span class="p">,</span> <span class="no">nil</span><span class="p">))</span>
<span class="p">}</span></code></pre></figure>
<p>Half the code is loading the file and the other half is specific to
this HTTP API. It’s easy enough to imagine another protocol if this
doesn’t work for you.</p>
<h1 id="conclusion">Conclusion</h1>
<p>“big data” isn’t all that clearly defined, but as a rule of thumb,
here are indicators that you’re definitely not working with big data:</p>
<ul>
<li>If it fits comfortably in your phone, it’s not big data</li>
<li>If it fits comfortably in your computer’s RAM, it’s not big data</li>
<li>If it fits on your laptop’s SSD, it’s not big data</li>
<li>If it fits on a single hard drive, it’s not big data</li>
</ul>
<p>One might even argue that if you can fit the data into a single
computer, it’s not worth calling it big data, though big data
processing tools can benefit even on smaller scale.</p>
<p>In the meantime, enjoy the smaller data in life. It’s fun and easy.</p>
2013 Contributions2013-12-31T00:00:00+00:00http://dustin.github.com/2013/12/31/2013<p>2013 was a reasonably productive year. The days I didn’t commit any
open source were, of course, a Tuesday and a Thursday. Boo.</p>
<div>
<img src="/images/2013.png" width="587" height="230" alt="2013" title="2013" />
</div>
<p>But I’ll focus on the positive. I really learned a lot and hopefully
made at least a few things that helped people. Based on bug reports
and pull requests, at least a couple people got some use out of my
time.</p>
<p>One thing I definitely did too little was blog. I find interesting
things and I write them into code and bring it out when it comes up in
conversation. I’ll never get good at it if I don’t start practicing
more.</p>
<p>Today, I wrote
<a href="https://gist.github.com/dustin/8202528">a little program</a> to dig
through all the github public events of 2013 (about 82 million) I
store in <a href="http://dustin.sallings.org/2012/09/27/cbfs.html">cbfs</a> and figure out which ones were of my doing. That
yielded 2,483 push events, which I reviewed in a google spreadsheet to
find 2,110 of them that were actually my fault.</p>
<p>Notice that github shows 2,229 contributions this year. This is
because I managed to open source some of the stuff I was working on.
yay. I’ll get into more of that later. In the meantime, let’s do a
quick breakdown of a few interesting things that happened each month.</p>
<h1 id="january">January</h1>
<p>In January, there were <em>two</em> days I didn’t do any code. But a lot I
did.</p>
<div>
<a href="/images/2013-cbugg.png"><img src="/images/2013-cbugg.png" width="250" height="164" alt="cbugg" title="cbugg" class="floatright" /></a>
</div>
<p>I started writing <a href="https://github.com/couchbaselabs/cbugg">cbugg</a> as the perfect bug tracker for me. There
were a couple of goals here. One of them was to not have to touch
Jira since everything I did there just consumed entirely too much
time. We wanted to create workflows that streamlined things as much
as possible.</p>
<p>Perhaps more importantly, we wanted to exercise
<a href="http://www.couchbase.com/">Couchbase Server</a> (and <a href="https://github.com/couchbaselabs/go-couchbase">go-couchbase</a>) and get a better
understanding of what it was like to throw an app together really
quick with this. <a href="http://crate.im/">Aaron Miller</a> saw the initial version and
made a new UI for it in in <a href="http://angularjs.org/">AngularJS</a> that was really great.</p>
<p><a href="https://github.com/mschoch">Marty Schoch</a> came along and did some really awesome search
integration. Overall, it’s been pretty great.</p>
<p>I did some work on <a href="http://dustin.sallings.org/2012/09/27/cbfs.html">cbfs</a> since I keep increasingly more critical
things there (e.g. <a href="https://www.youtube.com/watch?v=V0QpTnKaNE8">attachments in cbfs</a>, builds, etc…)</p>
<p>I did lots of work on <a href="https://github.com/dustin/gitmirror">gitmirror</a>. gitmirror is great for keeping
local copies of everything you push up to github, updated all the
time. It’s also a good way to have a single integration point of all
of your repositories. It’s got a tool, <a href="https://github.com/dustin/gitmirror/tree/master/setuphooks">setuphooks</a>, that allows
you to use patterns to configure any type of hook event on a repo, or
across all repos you own or all repos in an organization to which you
have appropriate access. It feeds any change anyone does to any repo
at Couchbase into cbugg, for example.</p>
<p>I still don’t have a complete <a href="http://en.wikipedia.org/wiki/Automatic_Packet_Reporting_System">APRS</a> stack that both receives and
transmits, but I did get more work done on <a href="https://github.com/dustin/go-aprs">go-aprs</a> that uses
<a href="https://github.com/dustin/go-nma">go-nma</a> to cause my phone and tablet to go off if anyone mentions
my callsign on the radio.</p>
<p>I started <a href="https://github.com/dustin/sallingshome">sallingshome</a> to manage some of the goings on around my
house. Primarily, it just has the chore management of the house.
Kids go there to see what tasks are available, they do the things, the
things get marked done and unavailable for their respective repeat
periods and I have a lot of things to pay for. It was my first
from-scratch Angular project which <em>really</em> helped my understanding,
but it’s also used a lot around my house. If you think it’ll help
you, let me know and I’ll try to document it enough to let someone
else run it.</p>
<h1 id="february">February</h1>
<p>February saw a lot more cbugg work. It was addictive, so we kept
going.</p>
<div>
<img src="/images/2013-cbgb.png" alt="cbgb" title="cbgb" class="floatleft" />
</div>
<p>But we also figured out what we were doing with <a href="http://cbgb.io/">cbgb</a>.
<a href="https://github.com/steveyen">Steve Yen</a> and I made a pretty useful working clone of
Couchbase Server in pure go. In particular, I did a lot of work to
scale it up to large numbers of buckets. That was some fun work that
led to me extracting <a href="https://github.com/dustin/go-broadcast">go-broadcast</a> and making a heavily
multiplexing implementation of it I used when I had hundreds of
thousands of things that could be wanting ticks and what-not.</p>
<p>These projects took most of my time, but I also did a little work on
<a href="https://github.com/dustin/frames">frames</a>, <a href="https://github.com/couchbaselabs/go-couchbase">go-couchbase</a>, <a href="https://github.com/dustin/seriesly">seriesly</a>, <a href="https://github.com/dustin/location">location</a>,
and a few other things.</p>
<h1 id="march">March</h1>
<p>March was tons more <a href="http://cbgb.io/">cbgb</a>. We had a mission of supporting any
application that could run against Couchbase, but in a single binary
download. We were evaluating this by running <a href="http://dustin.sallings.org/2012/09/27/cbfs.html">cbfs</a>, <a href="https://github.com/couchbaselabs/cbugg">cbugg</a>,
and whatever other apps we could find against it. Fixing bugs as we
found them.</p>
<p>I fixed a few bugs in a <a href="https://github.com/igm/sockjs-go/">sockjs</a> library we were using in cbugg.
Marty had originally written some magical websocket code, but we run a
lot of our web services through an Apache frontend, which eats
websockets, so he chose sockjs as a fallback. It was decent, but had
a few broken channel idioms and would hang and/or panic if things
didn’t go perfectly. Those got contributed back upstream.</p>
<p><a href="https://github.com/dustin/go-jsonpointer">go-jsonpointer</a> is built on <a href="https://github.com/dustin/gojson">my fork</a> of go’s JSON
package. In addition to updating the fork, I ended up plaguing Marty
with a few bugs in edge cases I wasn’t handling well. It’s got a
pretty good test suite, though, so maintaining it isn’t too bad.</p>
<p>I have a tool called <a href="https://github.com/couchbaselabs/pktreplay">pktreplay</a> I use for taking memcached packet
captures and playing them back against a machine. We’ve used this to
reproduce bugs from customer production situations that were difficult
to simulate from descriptions. I did a little bit of maintenance on
this for another customer engagement.</p>
<h1 id="april">April</h1>
<p>I worked on cbgb a lot more in April. Just more stuff.</p>
<div>
<img src="/images/2013-latency.png" width="250" height="188" alt="latency" title="passive latency" class="floatright" />
</div>
<p>I also took a trip to visit a customer who was having some
unpredictable latency in production. I built a tool I called
<a href="https://github.com/couchbaselabs/pktlatency">pktlatency</a> that passively measures latency from packet traces and
then dumps them into data I could process with R. After juggling it
around a bit, I created the plot you see to the right showing
distributions of slow responses by server node. It seems obvious in
retrospect, but one machine was consistently the source of all the
issues. Got the customers netops involved and just hung around in
Baltimore with <a href="http://trondn.blogspot.com/">Trond</a>.</p>
<p>I also had been working on a robot that played some games for me
online at the time. This involved automatically moving transactions
around in bitcoind, so I worked a lot on
<a href="https://github.com/GeertJohan/go.bitcoin">a go bitcoin interface</a>. I eventually released my game
bot code since all the games went away (boo). It’s no longer useful,
but there were some interesting bits of code in there that someone
will find useful someday.</p>
<p>There are a few packages called goquery, but I found
<a href="https://github.com/opesun/goquery">this one</a> and fixed up and implemented some of the parts
that I needed and got that back upstream.</p>
<p>I wrote a really neat backup mechanism for cbfs that backs up into
itself. I should really do a post on this, but I’ve made use of it
several times. On catastrophic failure of Couchbase, I can restore
terabytes of data in minutes. I’ve made use of this a few times
(though generally because I just decide to destroy my database).</p>
<h1 id="may">May</h1>
<p>In May I rewrote one of my photo album incarnations on AngularJS.
That was exciting.</p>
<p>I also did a tiny bit of work on <a href="https://github.com/couchbase/sync_gateway">sync_gateway</a>. My go infection
spread to co-workers who built some pretty awesome software from idea
to deployment in go.</p>
<p>cbfs’ backup tool was pretty great, but older backups could reference
objects that no longer existed since GC was only rooted by current
file references. I wanted to make sure that everything that existed
in backups was always available, so I decided to consider objects in
backups gc roots as well. I wrote <a href="https://github.com/dustin/go-hashset">go-hashset</a> as an efficient way
to maintain and operate on sets of hashes used as object references.
At the time, I had somewhere around 200k distinct live objects, but I
was building for billions and it was fun.</p>
<h1 id="june">June</h1>
<p>I mostly fixed bugs in June. I worked on <a href="https://github.com/steveyen">Steve’s</a> slabber
some, a lot of go-couchbase, finished off my hashset, and a few other
things. 168 commits across 22 projects as far as the github public
feed saw.</p>
<p>One thing that was fun in June, though, was the way I capture the
public feed data. I wrote a tool that syncs it up with a chunk of
cbfs so I’ve always got recent, replicated copies of the data
locally. That was very handy when I wanted to research this blog
post. :)</p>
<h1 id="july">July</h1>
<div>
<img src="/images/cbfsperf.png" alt="cbfs perf" title="cbfs perf" class="floatleft" />
</div>
<p>In July, I actually blogged a little. I had a post about
<a href="/2013/07/04/siginfo.html">using SIGINFO</a> to ask for interactive process information
The Unix Way (unless you’re on Linux).</p>
<p>That helped me understand download problems I was running into which
led to my writing <a href="/2013/07/17/saturate.html">go-saturate</a>.</p>
<p>Now my downloads can kill networks again.</p>
<p>This month, I also started working on <a href="http://www.couchbasecloud.com/">Couchbase Cloud</a>. It’s
a self-service frontend to a usable sync_gateway you can use as a sync
point for <a href="https://github.com/couchbase/mobile">Couchbase Mobile</a>. It’s AngularJS, go, and
cbgb. It was one of the biggest things I worked on in July, but only
accounted for about 12% of my commits.</p>
<h1 id="august">August</h1>
<p>Although I worked on 30 projects in August, most were relatively
small. I extracted the logger used in sync_gateway into a project
called <a href="https://github.com/couchbaselabs/clog">clog</a>. Not because the world needed another logger, but
because it helped meet Marty’s requirements for something more easily
than the rest of them.</p>
<p>I built <a href="https://github.com/dustin/papertrails">papertrails</a> to roll up my logs that get dumped into S3
from <a href="http://papertrailapp.com/">papertrail</a> monthly. It’s small, but really quite useful (and
I’m about to run it for all of December’s logs). That’s slightly
exciting for me. These replicate through <a href="http://www.bittorrent.com/sync">btsync</a> into cbfs and all
that fun stuff.</p>
<p>I also did some work on <a href="https://github.com/dustin/go-coap">go-coap</a> as I began using it in Couchbase
Cloud as a cheap and lossy means of reporting some DB events.
<a href="http://tools.ietf.org/html/draft-ietf-core-coap-18">CoAP</a> is like a really lightweight HTTP over UDP. I POST DB events
such as opening and closing of DBs with their sizes and stuff to the
management system. If it’s busy or dropped, it’s not an issue. Most
importantly it’s not polling and the code delivering the events isn’t
burdened with lots of file descriptors, a slow or down server, etc…</p>
<h1 id="september">September</h1>
<p>September is when I started having actual users on Couchbase Cloud, so
we got the necessary features in for it to go by itself, logging,
monitoring, and supervision.</p>
<p>This means I worked on supporting tools such as <a href="https://github.com/dustin/logexec">logexec</a> which
makes it super easy to send arbitrary programs’ output to syslog
(i.e. papertrail), go-couchbase, cbgb, go-coap.</p>
<p>I also designed a new circuitboard for my <a href="/2012/09/16/wash.html">washer project</a>, though
when I got around to hardware procurement, I think I found something
even better. It’s in front of me, not assembled.</p>
<h1 id="october">October</h1>
<div>
<a href="http://cbvis.west.spy.net/static/vbuckets.html?cluster=http%3A//mango.hq.couchbase.com%3A8091/"><img src="/images/2013-vbmap.png" width="250" height="150" alt="vbmap" title="vbmap" class="floatright" /></a>
</div>
<p>My birthday is in October. But later in the month, I got to go visit
a customer in Montréal. They had a decent amount of data and traffic,
so I got to pull out an old tool I’d built for watching our clusters
rebalance.</p>
<p>The link to the right will take you to a live visualization of a
fairly boring cluster I’ve got. Just trust me, it’s super-exciting
when it’s not stable. I’ve got record and playback tools to let me
see what this looked after the fact and with variable time.</p>
<p>I started using <a href="http://docker.io/">docker</a> a lot more and creating tools like
<a href="https://github.com/dustin/confsed">confsed</a> to help dynamically rewrite JSON APIs that try to
magically discover their addresses to proper external addresses that
aren’t known until instance run time.</p>
<p>I wrote <a href="https://github.com/dustin/bindir/blob/master/go-manifest">go-manifest</a> (last month, really) and <a href="https://github.com/dustin/bindir/blob/master/go-set-versions">go-set-versions</a>
as an example of how trivial it is to manage go packages if you try.
I wrote the first one while eating lunch at my desk just to show how
to easily determine the revisions of all of your dependencies as your
project is built. The latter just to reverse it since I had a couple
people not believing the process could be reversed… somehow.</p>
<p>But I don’t use such things myself. I like progress, so I wish
everyone to run the latest everything. The fear, of course, is that
things will break and you won’t know about them for long periods of
time. For this, I wrote <a href="https://github.com/dustin/gadzooks">gadzooks</a>. I have an instance of this
running on GAE that sees every public change that goes through github,
as well as acting like a github hook receiver (which I populate with
<a href="https://github.com/dustin/gitmirror/tree/master/setuphooks">setuphooks</a>). You configure sets of dependencies and a
build to trigger to <a href="http://drone.io/">drone.io</a> or your favorite CI system and any
time anything that <em>might</em> affect your build changes, your build gets
triggered.</p>
<h1 id="november">November</h1>
<div>
<a href="//raw.github.com/dustin/randbo/master/randbo.png"><img src="//raw.github.com/dustin/randbo/master/randbo.png" width="250" height="200" alt="randbo" title="randbo" class="floatleft" /></a>
</div>
<p><a href="https://github.com/dustin/randbo">Randbo</a> was one of those projects that just had to be written
because of the name. Someone wanted a way to grab arbitrary random
<code class="language-plaintext highlighter-rouge">[]byte</code>s in go, which to me, means you want an <code class="language-plaintext highlighter-rouge">io.Reader</code>. I’d
written something like it before, so I threw it together. I’m sure I
spent more time on the image.</p>
<p>I published my first CRAN package: <a href="http://cran.r-project.org/web/packages/humanFormat/index.html">humanFormat</a>. This is similar
to my <a href="https://github.com/dustin/go-format">go-format</a> package, but for R. I was doing some R plots and
got tired of pasting in the same format functions and changing them
slightly depending on my data. I must say, it’s a lot more difficult
to get a package available to people in R than in go. They really vet
it. You have to pass all the tests (of course it failed on Windows
the first time, because Windows can’t spell μs properly), document all
the things, etc…</p>
<p>I pretty much rewrote <a href="https://github.com/dustin/go-couch">go-couch</a> tests from scratch. I wanted to
get
<img src="https://coveralls.io/repos/dustin/go-couch/badge.png?branch=master" alt="coverage" />
and not have to hit an actual CouchDB every time.</p>
<p><a href="http://coveralls.io/">coveralls</a> is a great tool which I started using a lot more. This
led to major work on <a href="https://github.com/mattn/goveralls">goveralls</a> including adding support for go
1.2’s built-in cover tool, offline executions of coverage and several
fixes for issues I ran into.</p>
<p>I added go template support to <a href="http://docker.io/">docker</a> inspect so you can
more easily script things on your docker hosts.</p>
<p>My <a href="https://github.com/dustin/gosh">gosh</a> server I use for doing builds and deployments
from webhooks in such a way that doesn’t excessively use resources
seemed like it deserved a proper repo instead of the gist I’d been
keeping it in.</p>
<div>
<img src="http://west-spy.appspot.com/house/" width="272" height="193" alt="house" title="house" class="floatright" />
</div>
<p>And look to your right.</p>
<p><a href="https://github.com/dustin/westspy">This</a> is actually rendering and serving from GAE in a neat
batch processing thing that gets data from my house and processes it
on demand.</p>
<p>That was a fun challenge for which I learned pull queues and generally
learned how to be really lazy with resources on GAE.</p>
<h1 id="december">December</h1>
<div>
<a href="http://ph.couchbase.net/"><img src="/images/2013-ph.png" width="250" height="200" alt="phone home" title="watch phone home events" class="floatleft" /></a>
</div>
<p>I finally got to open source the <a href="https://github.com/couchbaselabs/statstore">stat collector</a> we use to collect
anonymous statistics from field units at Couchbase. It was mostly
closed because I had some passwords and stuff in it. Opening it is
really great because I abstractly talked about it when trying to help
people out on Google App Engine apps, but couldn’t show them real code
I’d written because I did dumb stuff like store passwords and junk it.</p>
<p>Click the image to see people (who opted in for anonymous stat
collection) using Couchbase right now.</p>
<p><a href="https://github.com/dustin/yellow">Yellow</a> is which is a tiny go library that helps you raise
awareness of bits of your code that are executing more slowly than you
expect in production. It’s another thing I’ve pulled out of a few
applications and wanted to get some reuse out of it.</p>
<div>
<a href="http://camlistore.org/"><img src="/images/2013-tardis.png" width="250" height="244" alt="time travel" title="time travel" class="floatright" /></a>
</div>
<p>In the last half of of December, I started getting involved in
<a href="http://camlistore.org/">camlistore</a> which is really quite fun. I <a href="https://plus.google.com/105229686595945792364/posts/449d7ohG1aH">wrote a little</a>
about my adventures in extended attributes over on google plus, but
suffice it to say I’ve been well over my head for a while. I went
from not knowing anything about FUSE or camlistore to wanting to
implement a means of looking at any point in the history of the
filesystem by timestamp.</p>
<p>And then I thought it’d be a good idea to make a Mac OS X GUI for all
the things camlistore.</p>
<p>I do plan on working on it a lot more, though. It’s a really great project.</p>
Wherefore go-saturate2013-07-17T00:00:00+00:00http://dustin.github.com/2013/07/17/saturate<h1 id="wherefore-go-saturate">Wherefore go-saturate</h1>
<div>
<img src="/images/saturated.png" alt="saturated" title="saturated" class="floatleft" />
</div>
<p>In my <a href="../04/siginfo.html">previous blog post</a>, I wrote about a bottleneck I ran
into that caused my application to pause when I felt it should’ve been
working hard.</p>
<p>My workers were all stuck waiting for data from one source while other
sources of the same data were available.</p>
<p>This is how I fixed it.</p>
<h1 id="problems-illustrated">Problems Illustrated</h1>
<div>
<img src="https://raw.github.com/dustin/go-talks/master/channels/hot.png" alt="hotness" title="the new hotness" class="floatright" />
</div>
<p>First, it’s good to see exactly what the problem looked like.</p>
<p>There are four workers and they all need a mix of data of type <code class="language-plaintext highlighter-rouge">a</code> or
<code class="language-plaintext highlighter-rouge">b</code>. Both <code class="language-plaintext highlighter-rouge">a</code> and <code class="language-plaintext highlighter-rouge">b</code> data sets are replicated so any given record
can come from one of two of these servers.</p>
<p>If there are equal tasks for <code class="language-plaintext highlighter-rouge">a</code> and <code class="language-plaintext highlighter-rouge">b</code> and servers are all generally
available, but one of the servers is slower than the others, then you
will invariably end up with all of the workers stuck retrieving data
from <code class="language-plaintext highlighter-rouge">a rep 1</code> even though <code class="language-plaintext highlighter-rouge">a rep 2</code> contains the same data.</p>
<p>But why is it blocked on that node? Because that node is slow.</p>
<p>When randomly selecting workers and one worker takes considerably
longer to complete its work, you will inevitably be blocked waiting
for slower workers to complete their tasks.</p>
<p>If this isn’t intuitive to you, think about it this way:</p>
<div>
<img src="/images/slowrand.png" alt="random slowness" title="random slowness" class="floatleft" />
</div>
<p>Imagine just two servers containing the same information from which
you randomly choose for any given request. Let’s say one server is 8x
slower than the other.</p>
<p>Assume a reasonable random number generator such that there’s a 50%
chance that any given request will be issued against the fast one, and
a 50% chance it will be issued against the slow one.</p>
<p>Now imagine you’ve got two clients that are wanting to grab a bunch of
data from the two servers.</p>
<p>The scenario illustrated above will occur frequently:</p>
<ol>
<li>Client 1 hits the fast server.</li>
<li>Client 2 hits the fast server.</li>
<li>Client 1 hits the slow server - is blocked for a while.</li>
<li>Client 2 hits the fast server again.</li>
<li>Client 2 hits the slow server - all workers are blocked while the
fast server is idle.</li>
</ol>
<p>Boo.</p>
<h1 id="what-does-go-saturate-do-about-this">What Does go-saturate Do About This?</h1>
<p>If you can express your work in the form of <code class="language-plaintext highlighter-rouge">(task, []resources)</code>
pairs, you can use <a href="//github.com/dustin/go-saturate">go-saturate</a> to help resolve this
type of problem with a double-fanout as illustrated and discussed
below.</p>
<p>Internally, the resources are each represented by a channel that has
one or more goroutines servicing it (user-specified). A resource is
“available” when a worker is idle and just waiting for new work on a
channel. If the resource is slow, the worker spends more time working
and less time accepting new work (see the red in the sequence diagram
above).</p>
<div>
<img src="https://raw.github.com/dustin/go-talks/master/channels/twotier.png" alt="double fanout" title="double fanout" class="floatright" />
</div>
<p>The diagram to the right represents a producer feeding into two tiers
of workers. The first tier workers, e.g. <code class="language-plaintext highlighter-rouge">worker 1</code> are higher level
(e.g. copy a file from the internet to local disk) while the second
tier workers, e.g. <code class="language-plaintext highlighter-rouge">a rep 1</code> are lower-level (e.g. read this file from
this location).</p>
<p>It’s important to understand that the first-tier workers are in the
same process as the second tier workers, the distinction being that
the second tier workers are operating on a resource identified by the
first tier workers and the duration of that work isn’t necessarily
predictable.</p>
<p>In the illustrated case, workers 1 and 3 are concurrently performing
tasks that need data from <code class="language-plaintext highlighter-rouge">a</code>.</p>
<p>Let’s break down the specific example shown in that diagram.</p>
<ol>
<li>The producer submits two tasks that both need information from <code class="language-plaintext highlighter-rouge">a</code></li>
<li>Workers 1 and 3 pick up this work.</li>
<li>They both identify <code class="language-plaintext highlighter-rouge">a rep 1</code> and <code class="language-plaintext highlighter-rouge">a rep 2</code> workers as being
possible sources of <code class="language-plaintext highlighter-rouge">a</code>.</li>
<li>The <code class="language-plaintext highlighter-rouge">select</code> from worker 1 chooses <code class="language-plaintext highlighter-rouge">a rep 2</code></li>
<li>The <code class="language-plaintext highlighter-rouge">select</code> from worker 3 chooses <code class="language-plaintext highlighter-rouge">a rep 1</code></li>
</ol>
<p>Note that steps 4 and 5 can occur at approximately the same time, but
the <code class="language-plaintext highlighter-rouge">a rep 1</code> worker can <em>only</em> pick up one job and the first-tier
worker (e.g. <code class="language-plaintext highlighter-rouge">worker 1</code>) can <em>only</em> have one worker pick up its task.</p>
<p>By asking for one of either and choosing the most available resource,
they’re able to do their work without blocking on each other. For
example, if <code class="language-plaintext highlighter-rouge">a rep 1</code> is slow, tasks requiring <code class="language-plaintext highlighter-rouge">a</code> will continue to
get their <code class="language-plaintext highlighter-rouge">a</code>s from <code class="language-plaintext highlighter-rouge">a rep 2</code> as fast as that replica can keep up.</p>
<p>In a sense, the resources are self-selecting.</p>
<div>
<img src="/images/cbfsperf.png" alt="cbfs perf" title="cbfs perf" class="floatright" />
</div>
<p>Here is a concrete example that illustrates go-saturate in the real
world. It’s being used in the <a href="http://labs.couchbase.com/cbfs/">cbfs</a> client code. (cbfs is a
distributed blob store for Couchbase Server).</p>
<p>As you can see in the right (a cbfs scenario where one server has
100Mbps ethernet and the others have gig-e), slow resources are still
used, but when a slow resource is being slow and a fast resource is
available, we won’t choose the slow one.</p>
<p>In long-running tasks, this is great – if I can keep all the fast
nodes <em>and</em> the slow nodes busy, then I get a little more out of the
network as a whole.</p>
<h1 id="other-resources">Other Resources</h1>
<p>You can read the <a href="http://godoc.org/github.com/dustin/go-saturate">go-saturate docs</a> to get an overview of the API.</p>
<p>Also, I gave a presentation at work earlier this week on this topic
which goes a lot further into basic go channel semantics interactively
including a range of information from basics like how to send and
receive on a channel to how send a message to exactly one of a
dynamically built set of recipients.</p>
<p>That presentation is captured in the following 20 minute video:</p>
<iframe width="560" height="315" src="//www.youtube.com/embed/QDO5YOrKSiQ" frameborder="0" allowfullscreen="1">
</iframe>
<p>Both the <a href="http://talks.godoc.org/github.com/dustin/go-talks/channels.slide">slides</a> and the <a href="http://talks.godoc.org/github.com/dustin/go-talks/channels.article">long-form</a> contents
of the material used in that presentation are available.</p>
Need the INFO2013-07-04T00:00:00+00:00http://dustin.github.com/2013/07/04/siginfo<div>
<img src="/images/notwork.png" alt="notwork" title="What are you doing?" class="floatright" />
</div>
<h1 id="why-isnt-my-program-working-harder">Why Isn’t my Program Working Harder?</h1>
<p>In my efforts to saturate my network with <a href="/2012/09/27/cbfs.html">cbfs</a>, I kept
noticing lulls in my graphs – both the <a href="/2012/09/09/seriesly.html">seriesly</a> and the
<a href="http://bjango.com/mac/istatmenus/">OSX widget</a> I use showing me what my computer’s doing. I
decided to figure out what the client is doing when it’s supposed to
be breaking my networks.</p>
<p>The cbfsclient download tool does three passes against the cluster to
figure out what to do:</p>
<ol>
<li>The first translates the path you requested to a tree of filenames
and object IDs (typically sha1 hashes).</li>
<li>The second finds current locations of all of those objects.</li>
<li>The third just starts flipping through all of those objects and
asking one of the origin servers at random to stream it down.</li>
</ol>
<p>As all requests are against origin servers, it should almost always
just be a straight up <code class="language-plaintext highlighter-rouge">sendfile</code>.</p>
<p>So how do I find out what’s going on? <code class="language-plaintext highlighter-rouge">netstat</code> wasn’t really useful –
it just told me that my client didn’t have anything to do, but had a
few connections open.</p>
<p>The thing I really want to know is exactly what HTTP requests are
currently in flight and how long they’ve been in flight. However, I
only want to know this when I observe behavior to be weird.</p>
<h2 id="enter-siginfo">Enter SIGINFO</h2>
<p><code class="language-plaintext highlighter-rouge">SIGINFO</code> is awesome. On BSD systems, <code class="language-plaintext highlighter-rouge">^T</code> sends <code class="language-plaintext highlighter-rouge">SIGINFO</code> to the
process currently attached to the terminal. A few programs
(e.g. <code class="language-plaintext highlighter-rouge">dd</code>) have built-in <code class="language-plaintext highlighter-rouge">SIGINFO</code> handlers that give you useful
information on long-lived processes.</p>
<p><code class="language-plaintext highlighter-rouge">^T</code> doesn’t work on Linux. I don’t know why and explaining that is
beyond the scope of this post, but I’m not developing on Linux, so
back to the lecture at hand.</p>
<p>Signals in UNIX are essentially messages delivered to the process, but
the UNIX APIs for signal handling involve registering a function to be
called when the signal is available for processing. This is
unfortunate because most things you’d be tempted to do in a signal
handler are <a href="https://www.securecoding.cert.org/confluence/display/seccode/SIG30-C.+Call+only+asynchronous-safe+functions+within+signal+handlers">unsafe</a>.</p>
<p>In go, signals are delivered to a channel. A goroutine reading from
that channel can safely do anything any other goroutine can do.</p>
<p>The most simple example of <code class="language-plaintext highlighter-rouge">SIGINFO</code>, at least on OS X, is as follows:</p>
<figure class="highlight"><pre><code class="language-go" data-lang="go"><span class="k">package</span> <span class="n">main</span>
<span class="k">import</span> <span class="p">(</span>
<span class="s">"log"</span>
<span class="s">"os"</span>
<span class="s">"os/signal"</span>
<span class="s">"syscall"</span>
<span class="p">)</span>
<span class="c">// SIGINFO isn't part of the stdlib, but it's 29 on most systems</span>
<span class="k">const</span> <span class="n">SIGINFO</span> <span class="o">=</span> <span class="n">syscall</span><span class="o">.</span><span class="n">Signal</span><span class="p">(</span><span class="m">29</span><span class="p">)</span>
<span class="k">func</span> <span class="n">main</span><span class="p">()</span> <span class="p">{</span>
<span class="n">ch</span> <span class="o">:=</span> <span class="nb">make</span><span class="p">(</span><span class="k">chan</span> <span class="n">os</span><span class="o">.</span><span class="n">Signal</span><span class="p">,</span> <span class="m">1</span><span class="p">)</span>
<span class="n">signal</span><span class="o">.</span><span class="n">Notify</span><span class="p">(</span><span class="n">ch</span><span class="p">,</span> <span class="n">SIGINFO</span><span class="p">)</span>
<span class="k">go</span> <span class="k">func</span><span class="p">()</span> <span class="p">{</span>
<span class="k">for</span> <span class="n">_</span> <span class="o">=</span> <span class="k">range</span> <span class="n">ch</span> <span class="p">{</span>
<span class="n">log</span><span class="o">.</span><span class="n">Printf</span><span class="p">(</span><span class="s">"You pressed ^T"</span><span class="p">)</span>
<span class="p">}</span>
<span class="p">}()</span>
<span class="k">select</span> <span class="p">{}</span>
<span class="p">}</span></code></pre></figure>
<p>You can read more on go signal handling either in <a href="http://golang.org/pkg/os/signal/">the docs</a>
or <a href="https://code.google.com/p/go-wiki/wiki/SignalHandling">wiki</a>.</p>
<h2 id="now-do-something-useful">Now Do Something Useful</h2>
<p>I spent a few minutes replacing the <a href="http://golang.org/pkg/net/http/#RoundTripper">http RoundTripper</a> for
the default client for my program with one that would keep track of
the beginning of every HTTP request through the <code class="language-plaintext highlighter-rouge">Close()</code> of the
response body.</p>
<p>Then I ran my program again. Once I saw another lull, I pressed <code class="language-plaintext highlighter-rouge">^T</code>
and, <em>ah ha</em>!</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>load: 1.45 cmd: cbfsclient 14765 running 47.14u 145.92s
In-flight HTTP requests:
servicing "http://slownode:8484/.cbfs/blob/[oid]" for 1m34.00s
servicing "http://slownode:8484/.cbfs/blob/[oid]" for 1m31.67s
servicing "http://slownode:8484/.cbfs/blob/[oid]" for 1m32.08s
servicing "http://slownode:8484/.cbfs/blob/[oid]" for 1m33.67s
servicing "http://slownode:8484/.cbfs/blob/[oid]" for 1m34.06s
servicing "http://slownode:8484/.cbfs/blob/[oid]" for 47.26s
</code></pre></div></div>
<p>This program was only allowing 6 concurrent requests and they were all
stuck doing requests against the same slow node. It’s so obvious when
you can see it.</p>
<h2 id="code">Code?</h2>
<p>I threw this together in about fifteen minutes just to debug this current
situation, but I’ve got it down to <code class="language-plaintext highlighter-rouge">initHttpMagic()</code> in <code class="language-plaintext highlighter-rouge">cbfsclient</code>’s
<a href="https://github.com/couchbaselabs/cbfs/blob/master/tools/cbfsclient/httpmagic.go">httpmagic.go</a>.</p>
<p>It’s not well documented at this point because, well, I spent ~15
minutes on it to solve my problem. The basic theory is pretty
straightforward, though:</p>
<ol>
<li>Every time we start a request, record the URL and timestamp.</li>
<li>Every time the response body is closed, forget about that URL.</li>
<li>When <code class="language-plaintext highlighter-rouge">^T</code> is pressed, dump out the current map.</li>
</ol>
<p>(Do note that I don’t ever have two requests to the same URL, so I’m
not worried about losing that information.)</p>
<p>Good luck, out there.</p>
CBFS DNS Service2012-10-05T00:00:00+00:00http://dustin.github.com/2012/10/05/cbfsdns<div>
<img src="/images/stupid.png" alt="stupid" title="Warning: Stupid" class="floatright" />
</div>
<h1 id="cbfs-dns-service">CBFS DNS Service</h1>
<p><strong>Warning</strong>: This is kind of a silly idea and not necessarily a
recommendation for how you should do things.</p>
<p>Also, this is not a replacement for <a href="http://www.dns-sd.org/">DNS-SD</a> or <a href="http://www.multicastdns.org/">mDNS</a> or
any such things. But it’s a fun toy I got working in a couple of
hours, so I’m playing with it.</p>
<h2 id="dns-for-humans">DNS for Humans</h2>
<p>If you’re looking at this web page, you’ve probably interacted in some
way with the <a href="http://en.wikipedia.org/wiki/Domain_Name_System">domain name system</a>. It’s pretty convenient as a
human to ask for <code class="language-plaintext highlighter-rouge">dustin.github.com</code> and not think about what that
means.</p>
<p>There are tons of descriptions of this service out there, how it
works, etc… I’m not going to get into that as much as a small bit
on relevant parts and how they’re generally used.</p>
<h3 id="a-records">A Records</h3>
<p>Many of the DNS queries that are tossed about are for <code class="language-plaintext highlighter-rouge">A</code>, or address
records. e.g., I ask my browser for <code class="language-plaintext highlighter-rouge">dustin.github.com</code> which does a
magical DNS dance around looking for an <code class="language-plaintext highlighter-rouge">A</code> record for
<code class="language-plaintext highlighter-rouge">dustin.github.com.</code> and get the following:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>dustin.github.com. 41033 IN A 204.232.175.78
</code></pre></div></div>
<p>From that point on, the browser has an IP address it can talk to.
Typically, this is put in place by the human who owns the IP address
<code class="language-plaintext highlighter-rouge">204.232.175.78</code>. This is most likely a wildcard for <code class="language-plaintext highlighter-rouge">*.github.com.</code>,
but most importantly, the management direction is <em>usually</em> “thing I
want to provide” -> “resource on which I can provide it.”</p>
<p>i.e. you want to provide pages, you configure up your machines for it
and point the service at the machines.</p>
<h3 id="ns-records">NS Records</h3>
<p>Part of the above requires a lookup of an <code class="language-plaintext highlighter-rouge">NS</code> record to find out
where to even ask for the <code class="language-plaintext highlighter-rouge">A</code> record. The <code class="language-plaintext highlighter-rouge">NS</code> record is the way that
a domain in DNS can delegate responsibility to another system. In
this case, someone who administers <code class="language-plaintext highlighter-rouge">com.</code> delegates to someone who
adminsters <code class="language-plaintext highlighter-rouge">github.com.</code> (TLDs are a little more complicated, but this
is roughly the idea)</p>
<p>Most of that magic happens in the DNS server, but basically, assuming
I know who serves <code class="language-plaintext highlighter-rouge">com.</code>, I ask who serves <code class="language-plaintext highlighter-rouge">github.com.</code> to ask where
<code class="language-plaintext highlighter-rouge">dustin.github.com.</code> is. Such an <code class="language-plaintext highlighter-rouge">NS</code> query returns the following:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>;; ANSWER SECTION:
github.com. 81150 IN NS ns4.p16.dynect.net.
github.com. 81150 IN NS ns3.p16.dynect.net.
github.com. 81150 IN NS ns1.p16.dynect.net.
github.com. 81150 IN NS ns2.p16.dynect.net.
;; ADDITIONAL SECTION:
ns4.p16.dynect.net. 81150 IN A 204.13.251.16
ns3.p16.dynect.net. 81150 IN A 208.78.71.16
ns2.p16.dynect.net. 81150 IN A 204.13.250.16
ns1.p16.dynect.net. 81150 IN A 208.78.70.16
</code></pre></div></div>
<p>This lists both the names of the nameservers that I asked about and
was kind enough to also senda long their IP addresses so I don’t have
to make another trip to figure out where they are.</p>
<h3 id="srv-records">SRV Records</h3>
<p><code class="language-plaintext highlighter-rouge">SRV</code> records are kind of neat. They tell the address(es) of something,
but also on which doors to knock. They also provide concepts of
“weight” and “priority.”</p>
<p>Jumping really quick into an example, let’s say you want to IM
someone. You’re logged into gmail and you want to talk to
<code class="language-plaintext highlighter-rouge">example@jabber.org</code>. Well, the first thing google’s going to want to
know is how to connect to this service. Specifically, the XMPP
service over TCP. To find that out, it issues an <code class="language-plaintext highlighter-rouge">SRV</code> query against
<code class="language-plaintext highlighter-rouge">_xmpp-server._tcp.jabber.org.</code> and gets this:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>_xmpp-server._tcp.jabber.org. 900 IN SRV 31 31 5269 fallback.jabber.org.
_xmpp-server._tcp.jabber.org. 900 IN SRV 30 30 5269 hermes.jabber.org.
_xmpp-server._tcp.jabber.org. 900 IN SRV 30 30 5269 hermes6.jabber.org.
[...and in the extras section]
hermes.jabber.org. 900 IN A 208.68.163.221
hermes6.jabber.org. 900 IN A 208.68.163.221
hermes6.jabber.org. 900 IN AAAA 2605:da00:5222:5269::2:1
fallback.jabber.org. 900 IN A 208.68.163.218
</code></pre></div></div>
<p>There’s an equally low priority for <code class="language-plaintext highlighter-rouge">hermes</code> and <code class="language-plaintext highlighter-rouge">hermes6</code>, so google
will try one of those first. <code class="language-plaintext highlighter-rouge">hermes6</code> has two IP addresses, so it
may try all three of those addresses before trying <code class="language-plaintext highlighter-rouge">fallback.</code></p>
<p>These lookups are done magically not only by servers communicating in
XMPP, but also clients that want to talk to XMPP. Someone publishes
the connection details and we’re all good to go.</p>
<h2 id="now-you-fully-understand-global-dns">Now You Fully Understand Global DNS</h2>
<p>OK, if you came here not knowing much about DNS, you still don’t, but
that’s OK. My main point is that often when people who think about
DNS think about DNS, they are thinking about what they want to publish
and how things are going to find them.</p>
<p>The exceptions here are in <a href="http://www.dns-sd.org/">DNS-SD</a> and, to a degree,
<a href="http://www.multicastdns.org/">mDNS</a>. You’ve probably interacted with both of these when you
ask your computer to find a printer or someone tells you to look at
something on his laptop (in my case, that’s <code class="language-plaintext highlighter-rouge">dustinnmb.local.</code>).
These magical discovery protocols are pretty awesome for ad-hoc
services, and with properly administered DNS-SD, even globally
advertised services.</p>
<p>But I came here to talk about something I’m doing that’s just a little
bit different. Probably not different enough to justify the hour or
two I spent today trying to make it work, but interesting for me.</p>
<h2 id="dns-for-self-organizing-services">DNS for Self-Organizing Services</h2>
<p><a href="http://dustin.github.com/2012/09/27/cbfs.html">cbfs</a> is a storage service that, if we get it all right, blurs
the line between administered and magic. The servers need a bit of
configuration to know where to coordinate, but after that, clients can
pretty much pick any one of them to work with.</p>
<p>At home, I have a couple of nodes that are going to remain
“permanent,” but intend to have a few others coming and going as I
experiment with things.</p>
<p>The thing that’s a little difficult is figuring out which node I
should talk to when things go wrong. And if I want to use a service
name (as opposed to just always hitting the same host I know is
running the service), what do I point it to? And what do I do when
that host goes down? And even when everything’s mostly stable, what’s
the best machine to talk to do the thing I want to do <em>right now</em>?</p>
<p>Because of these questions, I had the absolutely ridiculous idea to
make cbfs its own DNS server.</p>
<p>*sigh*</p>
<p>It’s useful, though. cbfs is actively monitoring the cluster, knows
what nodes are in it, out of it, when nodes start to die, it can
respond instantly, etc… If I plug in a node, I want clients to find
it instantly, and I use my web browser and curl as clients as lot, so
I’d like it to work there, too.</p>
<p>For this, I did two things:</p>
<h3 id="srv-records-1">SRV Records</h3>
<p>Firstly, you can make an <code class="language-plaintext highlighter-rouge">SRV</code> request as a “smart” client for
<code class="language-plaintext highlighter-rouge">_cbfs._tcp.[domain].</code> to get the current list of nodes <em>and</em> a
recommendation for which node to talk to at that point in time.</p>
<p>Here’s an example from my network at home (I’m abbreviating the
queries name to <code class="language-plaintext highlighter-rouge">$q</code> just to keep the line short enough to read):</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>;; ANSWER SECTION:
$q 5 IN SRV 2 0 8484 dustinnmb.cbfs.west.spy.net.
$q 5 IN SRV 3 5 8484 bigdell.cbfs.west.spy.net.
$q 5 IN SRV 0 1 8484 z.cbfs.west.spy.net.
$q 5 IN SRV 1 1 8484 menudo.cbfs.west.spy.net.
;; AUTHORITY SECTION:
cbfs.west.spy.net. 3600 IN NS ns.west.spy.net.
;; ADDITIONAL SECTION:
z.cbfs.west.spy.net. 60 IN A 192.168.1.38
menudo.cbfs.west.spy.net. 60 IN A 192.168.1.97
dustinnmb.cbfs.west.spy.net. 60 IN A 192.168.1.113
bigdell.cbfs.west.spy.net. 60 IN A 192.168.1.135
ns.west.spy.net. 3600 IN A 192.168.1.40
</code></pre></div></div>
<p>This begs a bit of explanation.</p>
<p>Firstly, <code class="language-plaintext highlighter-rouge">ns.west.spy.net.</code> is my primary name server at home. It’s
an off-the-shelf <a href="https://www.isc.org/software/bind">bind</a> instance running on OpenBSD (at least,
some strange embedded OS I based off of OpenBSD at some point in the
past). This is an administration point where I go and enter RRs for
services I want to offer. It serves both my internal and external
<code class="language-plaintext highlighter-rouge">west.spy.net.</code> domains (which are different).</p>
<p>Internally, I want to provide service for <code class="language-plaintext highlighter-rouge">cbfs.west.spy.net.</code>, but I
want it to be magical and dynamic. I also don’t want to run cbfs as
root, so I have DNS bound to a different port. No problem at all, I
just forward to a couple of known cbfs servers with the cbfs DNS
service running using the following config:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>zone "cbfs.west.spy.net." {
type forward;
forwarders {
192.168.1.38 port 8453;
192.168.1.97 port 8453;
};
};
</code></pre></div></div>
<p>This should be obvious, but it basically just causes DNS queries for
that zone to proxy through and hit my cbfs server.</p>
<p>The rest of the stuff from above is all dynamic and coming out of
cbfs’ internal state. You’ll notice the priorties in the answer
section are different (these are the sequential, but not ordered
numbers). These priorities are the same priorities cbfs would use for
data distribution internally. They’re approximately (but
intentionally not exactly) prioritized by heartbeat recency.</p>
<p>Currently, the weight is unused as the priorities order the node usage
absolutely, but since I just hacked this thing together, I’m likely to
do something different after I play with it a bit.</p>
<p>One thing to note is that those are not “hostnames” in the
conventional sense, but just the things I passed to the <code class="language-plaintext highlighter-rouge">-nodeID</code>
parameter to cbfs. cbfs itself creates the hostname glue and does all
that magic.</p>
<h3 id="a-records-1">A Records</h3>
<p>“But wait!”, you say, “I thought you wanted this to work with your
browser and curl. They won’t do these SRV lookups!”</p>
<p>Correct, so cbfs also responds to <code class="language-plaintext highlighter-rouge">A</code> or <code class="language-plaintext highlighter-rouge">ANY</code> queries be returning a
handful of <code class="language-plaintext highlighter-rouge">A</code> records to make these other clients happy. Example:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>;; QUESTION SECTION:
;cbfs.west.spy.net. IN A
;; ANSWER SECTION:
cbfs.west.spy.net. 5 IN A 192.168.1.38
cbfs.west.spy.net. 5 IN A 192.168.1.135
cbfs.west.spy.net. 5 IN A 192.168.1.97
</code></pre></div></div>
<p>In this case, I just asked for “the address” of <code class="language-plaintext highlighter-rouge">cbfs.west.spy.net.</code>
and it gave me three, any one of which is expected to be happy to
answer any query I might throw at it.</p>
<p>The keen-eyed reader notes this is fewer than the four listed above.
I arbitrarily decided to kill a node while writing one of the
paragraphs above. Things kept going fine.</p>
<h2 id="in-conclusion">In Conclusion</h2>
<p>I probably could (and will) accomplish much of the same with mDNS and
I don’t think providing name resolution services with DNS is
particularly novel, but this was a fun hack I did for my birthday and
I hope it inspires someone to do something better.</p>
CBFS2012-09-27T00:00:00+00:00http://dustin.github.com/2012/09/27/cbfs<h1 id="cbfs---a-couchbase-large-object-store">cbfs - a couchbase large object store</h1>
<div>
<img src="/images/toys.jpg" alt="where do i get all these wonderful toys?" title="Where Do I Get all these Wonderful Toys?" class="floatright" />
</div>
<p>Thirteen days ago, <a href="http://hexeditreality.com/">Marty Schoch</a> and I created an empty
directory called <code class="language-plaintext highlighter-rouge">cbfs</code> and started typing some <a href="http://golang.org/">go</a> code into
it. The idea was to create an answer to the frequent question, “how
do I store large items in <a href="http://www.couchbase.com/">Couchbase</a>?”</p>
<p>I think we’re both pretty pleased with the results and would like to
share what we’ve made a bit more broadly.</p>
<p><a href="http://github.com/couchbaselabs/cbfs">cbfs</a> is essentially a read/write HTTP server with a minimal
<a href="https://github.com/couchbaselabs/cbfs/wiki/Protocol">RESTful API</a> for getting a bit more meta information out of
it and serving apps on couchbase. We had a few goals and were able to
demonstrate almost all of them within a week of birth. Things like:</p>
<ul>
<li>No single point of failure.</li>
<li>No single point of contention.</li>
<li>No limits to the size or volume of data that can be served.</li>
<li>No invalid states to reconcile.</li>
<li>Adding and removing nodes and other management tasks should be
as easy as possible.</li>
</ul>
<p>Also, we wanted to be able to bring back something along the spirit of
<a href="http://couchapp.org/">couchapps</a> (but much easier!).</p>
<p>We haven’t written up enough on it yet, but I did do a demonstration
to our team today in a google hangout. You can see that quick intro
in the first half of the following video.</p>
<iframe width="560" height="315" src="http://www.youtube.com/embed/YLTXdrvYITA" frameborder="0" allowfullscreen="1">
</iframe>
<p>The <a href="http://labs.couchbase.com/cbfs/">slides</a> are available as well. They are meant to be
served from a cbfs instance doing the demo, but I made the demos also
work when offline.</p>
<p>Some neat stuff has been built around this since we started.</p>
<p>Trond Norbye wrote a <a href="http://trondn.blogspot.com/2012/09/yanfs-yet-another-network-file-system.html">FUSE interface</a> so we could mount it
locally. I’m synchronizing our <a href="http://db.tt/0bYIeqqB">Dropbox</a> stuff into cbfs and
watched Trond browse around in a terminal and interact with the files
as if they were local.</p>
<p>Marty’s written a pretty awesome <a href="https://github.com/couchbaselabs/cbfs-admin">admin console</a>.
I’ve done some demos of other stuff built on the API as well
(including the built-in cluster monitoring console and the commandline
tool).</p>
<p><img src="/images/cbfs-admin-600.png" alt="cbfs admin" class="centered" /></p>
<p>The <a href="https://github.com/couchbaselabs/cbfs/wiki">wiki page</a> was mostly written in the first four days, but
describes what the idea was pretty well. It’s a lot more complete
now, though the <a href="https://github.com/couchbaselabs/cbfs/issues">issues list</a> shows where we want to add more
polish.</p>
<div>
<img src="/images/cbfs-small-monitor.png" alt="monitor view" class="floatleft" />
</div>
<p>I’m running a cluster at home and in the office and serving content
and apps out of it, so it’s definitely self-hosting and stuff.</p>
<p>Come join us, help finish tasks, think of new ones, find ways you
think it could be better…</p>
<p>But, as a reminder, this isn’t two weeks old yet, so if you want
something that’s actually been in production with PBs of data, look at
<a href="http://code.google.com/p/mogilefs/">mogilefs</a>. We wrote this because we wanted it to exist and
wanted to have some answers for specific questions we’d been asked.</p>
Airing My Dirty Laundry2012-09-16T00:00:00+00:00http://dustin.github.com/2012/09/16/wash<h1 id="on-laundry">On Laundry</h1>
<p>Of the various housework I have to do, laundry is actually not that
bad. I have this great machine I bought a long time ago that does
most of the work. I just have to remember to put things in it and
we’re good. I have another machine that takes the clean things and
transforms them into dry things. It’s a magical experience, tainted
only by short attention span that leads me to forget to transfer the
laundry from the cleaning machine to the drying machine.</p>
<p>I set out to solve this problem using my only skills, but I have a
problem. How do I program an old washing machine?</p>
<h1 id="feeling-it">Feeling It</h1>
<div>
<img src="/images/vibsense.png" alt="vibration sensor" title="Vibration Sensor" class="floatright" />
</div>
<p>My first idea was to use a <a href="https://www.sparkfun.com/products/9196">vibration sensor</a> attached to the
laundry devices to detect when they’re doing something.</p>
<p>This is more complicated than it sounds because the vibration can be a
bit subtle during some of the cycles and getting reliable signal out
of these sensors at the levels I needed seemed just a little too
indirect.</p>
<h1 id="power-measurement">Power Measurement</h1>
<p>“I know, I’ll just measure the power it’s using.”</p>
<div>
<img src="/images/ct.png" alt="current transformer" title="Current Transformer" class="floatleft" />
</div>
<p>I have awesome tools like the “<a href="http://www.p3international.com/products/special/p4400/p4400-ce.html">Kill A Watt</a>” that do much
of what I need. I just need to get the data out. Adafruit has
<a href="http://www.ladyada.net/make/tweetawatt/">Tweet-a-Watt</a>, which is a great concept, but really
expensive, and still not doing <em>exactly</em> what I want. So I started
looking into building my own.</p>
<p>I looked around for the most basic thing I could find and found this
cheap <a href="http://www.amazon.com/013-030-Output-Non-invasive-Current-Transformer/dp/B005CTWE8A/">current transformer</a> on Amazon. Of course, I had no clue
whatsoever how to use a current transformer. I read a lot about them
and there were lots of documents that made me feel like I was going to
tear a hole in space and time if I didn’t properly cross the pins.
This device is like the anti-Gozer.</p>
<p>As an uneducated software hacker, it took me a lot of trial and error
to get the right circuit designed for this thing. At the time, I did
a lot of the work on my iPad, using <a href="http://icircuitapp.com/">iCircuit</a> which I
strongly recommend. I used that to design (and test) the following
circuit:</p>
<p><img src="/images/ct-circuit.png" alt="circuit" title="Current Transformer Circuit" class="centered" /></p>
<p>I can’t stress how useful iCircuit was. My notes are filled with
readings from scopes where I had attempted something and didn’t quite
get the right result. The cases that mattered most were sudden
spikes, limits and endurance. Some of the circuits didn’t drain well,
so they’d just kind of float up after a while.</p>
<p>I don’t use my iPad much since I got my Nexus 7. iCircuit was one of
the tools I hoped I could replace. There were rumors of an Android
version coming, but I couldn’t find anything concrete. I did,
however, find <a href="https://play.google.com/store/apps/details?id=com.everycircuit">EveryCircuit</a>. This is a pretty great
piece of software. It’s missing a couple of features I hope make it,
but it also does a few things much better than iCircuit. It’s a grand
age for making things.</p>
<div>
<img src="/images/ct-scope.png" alt="" title="Current Transformer Scoped" class="floatleft" />
</div>
<p>Once I got a circuit that was good in theory, it was time to
breadboard it and try it with some real load.</p>
<p>As I mentioned above, I found that jolting the CT with a sudden load
would call it to bounce way off the charts in early testing. This is
sort of the electronic equivalent of a stack overflow, except instead
of crashing my program, it burns down my house.</p>
<p>I had incentive to get this right.</p>
<p>There was a lot of simulated testing, then a lot of breadboard testing
and then I wanted something I could actually deploy in my garage. I
went over to <a href="http://www.halted.com/">halted</a> and found some decent prototyping boards
and ended up with something that was a bit more rigid than the
breadboard.</p>
<p><img src="/images/ct-final.jpg" alt="final build" title="Current Transformer Circuit Build" class="centered" /></p>
<p>I don’t have a picture of the actual final product which is
unfortunate. I went through a few different debugging strategies.
First, I would look at things just through the console. That’s very
inconvenient as the device is hooked up in my garage. The radio
signal wasn’t always great, and I’d affect it by getting close to it
(insert obvious Heisenbug joke). I added a light so I wouldn’t have
to get too close and hook up a computer and stuff. Then I wanted to
know more than one thing, so I’d have the light blink at different
rates.</p>
<p>In the end, I hooked up a 2x16 character LCD so I could just print out
whatever the sensor and radio states were. That was <em>really</em> helpful,
but I damaged it in the final installation so that it’s basically
useless now. It was good enough to get it going, though.</p>
<h1 id="building-a-reader">Building a Reader</h1>
<div>
<img src="/images/rfm12b.jpg" alt="rfm12b" title="RFM 12b" class="floatleft" />
</div>
<p>At this point, I have hardware that converts the magnetic field
observed by the appliance’s power draw to 0-n volts. Originally I was
aiming for 5V, but after looking at wireless options, I decided to
give <a href="http://shop.moderndevice.com/products/jeenode-kit">JeeNode</a> a shot. They run at 3.3V, but are otherwise
pretty much <a href="http://www.arduino.cc/">Arduino</a> compatible, but featuring a small, low
power and most interestingly, cheap radio - the <a href="http://shop.moderndevice.com/products/rfm12b-radio">rfm12b</a>.</p>
<p>Getting things up and running was pretty easy. One problem I found in
using these radios is that they’re pretty low-level. Robust protocols
don’t come for free.</p>
<p>For this, my needs were pretty simple. The project evolved just a
but, but essentially I send a packet out with an 8-bit sequence ID and
I keep sending the same sequence ID until the other end responds. I
really only want to know if the device has been on since the last time
I looked, so I send the most recent reading off of the sensor and the
highest value I’ve read since I received an ACK. Every transmission
requests an ACK, but there’s often tons of interference (in both
directions) so each side has to transmit and be heard by the other
before the needle is moved.</p>
<p>Internally, the sensor updates every second. It transmits at least
once every 10s and once on every change since the last ACKd value.</p>
<h1 id="making-it-useful">Making it Useful</h1>
<div>
<img src="/images/jeelink.jpg" alt="jeelink" title="JeeLink" class="floatright" />
</div>
<p>On the reader side, I have a really simple read-only protocol that
converts the stuff in the air to RS232 through a USB interface using a
simple <a href="http://golang.org/">go</a> program that does a few things:</p>
<ol>
<li>Serves the readings up over HTTP.</li>
<li>Lights up stuff in my living room telling me the state of the
laundry.</li>
<li>Sends out alerts with <a href="https://www.notifymyandroid.com/">NotifyMyAndroid</a> so my phone and Nexus
7 start beeping when things change.</li>
</ol>
<p>I had to start by creating an <a href="https://github.com/dustin/rs232.go">RS232</a> interface for go. I’ve
been able to use this for a couple of projects now (hopefully I can
write about one of the others, because it’s pretty awesome).</p>
<p>The <a href="https://github.com/dustin/nma.go">NotifyMyAndroid interface for go</a> alone has kept me from
having to re-wash laundry that sat in the washer too long.</p>
<p><a href="https://github.com/dustin/washer">The source</a> to the sensor, reader firmware and go parts are
all available, though I can’t guarantee they’re 100% ready to deploy
for anyone who isn’t me. If anyone wants to try something similar,
I’ll gladly help, though.</p>
Seriesly Internals Tour2012-09-13T00:00:00+00:00http://dustin.github.com/2012/09/13/inside-seriesly<h1 id="seriesly-internals-tour">Seriesly Internals Tour</h1>
<p>My previous <a href="/2012/09/09/seriesly.html">a blog post</a> introduces a documented-oriented time
series database I wrote. It’s introductory material that somewhat
describes why the software exists and a bit about how to use it.</p>
<p>Here I describe how it works. As mentioned before, the bulk of the
software was in a useful state in two weeks. I attribute much of this
success to <a href="http://golang.org/">go</a>. <a href="https://github.com/dustin/seriesly">The code</a> itself contains the details of
my knowledge, but I’ll highlight some of the parts that helped the
most in the coming paragraphs.</p>
<p>The process described in the following paragraphs <em>generally</em> occurs
in under a millisecond, in my installations. On a sufficiently large,
cold query, it could theoretically take several minutes.</p>
<h1 id="logical-overview">Logical Overview</h1>
<div>
<img src="/images/seriesly-logical.png" alt="logical" title="Logical flow of a query" class="floatright" />
</div>
<p>The query model is as simple as I could meet all of my current use
cases with.</p>
<p>The diagram to the right represents a logical overview of a query
flowing through the system.</p>
<h2 id="understanding-the-query">Understanding the Query</h2>
<p>First, the HTTP query is parsed and put into a struct that represents
the information that’s needed. Here we validate and parse the <code class="language-plaintext highlighter-rouge">from</code>
and <code class="language-plaintext highlighter-rouge">to</code> timestamps, the grouping, the reducers, etc…</p>
<p>Also, we place a <code class="language-plaintext highlighter-rouge">before</code> timestamp that is the latest possible time
for us to do any work on this query (queries that would run longer
than the server’s configured max query time are aborted even if
partially processed).</p>
<p>Most importantly, there’s some output data in the struct. The
following example should aid in understanding. Start with an input
query:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>/testdb/_query?from=1347469924&to=2013&group=3600000&ptr=/a&reducer=min
</code></pre></div></div>
<p>That will map to a <code class="language-plaintext highlighter-rouge">queryIn</code> struct that looks like this:</p>
<figure class="highlight"><pre><code class="language-go" data-lang="go"><span class="n">queryIn</span><span class="p">{</span>
<span class="n">dbname</span><span class="o">:</span> <span class="s">"testdb"</span><span class="p">,</span>
<span class="n">from</span><span class="o">:</span> <span class="n">time</span><span class="o">.</span><span class="n">Date</span><span class="p">(</span><span class="m">2012</span><span class="p">,</span> <span class="m">9</span><span class="p">,</span> <span class="m">12</span><span class="p">,</span> <span class="m">10</span><span class="p">,</span> <span class="m">12</span><span class="p">,</span> <span class="m">4</span><span class="p">,</span> <span class="m">0</span><span class="p">,</span> <span class="n">time</span><span class="o">.</span><span class="n">UTC</span><span class="p">},</span>
<span class="n">to</span><span class="o">:</span> <span class="n">time</span><span class="o">.</span><span class="n">Date</span><span class="p">{</span><span class="m">2013</span><span class="p">,</span> <span class="m">1</span><span class="p">,</span> <span class="m">1</span><span class="p">,</span> <span class="m">0</span><span class="p">,</span> <span class="m">0</span><span class="p">,</span> <span class="m">0</span><span class="p">,</span> <span class="m">0</span><span class="p">,</span> <span class="n">time</span><span class="o">.</span><span class="n">UTC</span><span class="p">},</span>
<span class="n">group</span><span class="o">:</span> <span class="m">3600000</span><span class="p">,</span>
<span class="n">ptrs</span><span class="o">:</span> <span class="p">[]</span><span class="kt">string</span><span class="p">{</span><span class="s">"/a"</span><span class="p">},</span>
<span class="n">reds</span><span class="o">:</span> <span class="p">[]</span><span class="kt">string</span><span class="p">{</span><span class="s">"min"</span><span class="p">},</span>
<span class="c">// Control, logging, timeout, etc...</span>
<span class="n">start</span><span class="o">:</span> <span class="n">time</span><span class="o">.</span><span class="n">Now</span><span class="p">(),</span>
<span class="n">before</span><span class="o">:</span> <span class="n">time</span><span class="o">.</span><span class="n">Now</span><span class="p">()</span><span class="o">.</span><span class="n">Add</span><span class="p">(</span><span class="n">queryTimeout</span><span class="p">),</span>
<span class="n">started</span><span class="o">:</span> <span class="m">0</span><span class="p">,</span>
<span class="c">// Output channels</span>
<span class="n">out</span><span class="o">:</span> <span class="nb">make</span><span class="p">(</span><span class="k">chan</span> <span class="o">*</span><span class="n">processOut</span><span class="p">),</span>
<span class="n">cherr</span><span class="o">:</span> <span class="nb">make</span><span class="p">(</span><span class="k">chan</span> <span class="kt">error</span><span class="p">),</span>
<span class="p">}</span></code></pre></figure>
<p>The <code class="language-plaintext highlighter-rouge">started</code> int is a counter that’s incremented (atomically) each
time a grouping is located. All of the final responses are fed into
the <code class="language-plaintext highlighter-rouge">out</code> channel. Once the query worker completes walking the tree,
it will send its return value (generally <code class="language-plaintext highlighter-rouge">nil</code>) into <code class="language-plaintext highlighter-rouge">cherr</code>,
informing the HTTP handler that all groupings have been found and
<code class="language-plaintext highlighter-rouge">started</code> will no longer increase.</p>
<h2 id="locating-the-relevant-documents">Locating the Relevant Documents</h2>
<p>The query processor server is asynchronously walking the docs to
locate the ones that apply to the query. Once the query is submitted,
the HTTP handler no longer pays any attention to the query directly.</p>
<p>But the actual query processing is where we start needing to tighten
resource controls. In theory, we don’t need more queries digging
through storage than we have the resources to process. In practice
(it’s all configurable), oversubscribing doesn’t seem to make anything
faster, but leads to a lot of memory bloat as more things are stuck in
progress.</p>
<p>Without delving too deeply into <a href="http://en.wikipedia.org/wiki/Queueing_theory">queueing theory</a>, let’s just
imagine you’re at a well-organized bank where you’ve got a single line
and four identical tellers.</p>
<div style="clear: both; margin: 5px;">
<img src="/images/cashier.png" alt="cashier" class="floatleft" />
<img src="/images/cashier.png" alt="cashier" class="floatleft" />
<img src="/images/cashier.png" alt="cashier" class="floatleft" />
<img src="/images/cashier.png" alt="cashier" class="floatleft" />
</div>
<p>The number of workers is fixed throughout the lifetime of a process
and each has one simple job:</p>
<blockquote>
<p>to transform a query to a collection of grouped document IDs to be
handled by a doc processor on behalf of a client</p>
</blockquote>
<p>That’s it. One of the workers takes the request off the queue, starts
digging through documents between the <code class="language-plaintext highlighter-rouge">from</code> and <code class="language-plaintext highlighter-rouge">to</code> range of the
query emitting a batch when it crosses every boundary defined by
<code class="language-plaintext highlighter-rouge">group</code> and passes it down to a doc processor.</p>
<p>It’s important to note that both the queues <em>into</em> and <em>out of</em> the
query processor are <strong>blocking</strong>, at least, eventually. By default,
an HTTP client will block on submitting his query until a worker is
available and a query worker will block on submitting a batch to doc
workers until a doc worker is available. Both of these things are
configurable, but in practice, letting work build up between these
areas increases memory overhead without getting more work done sooner.</p>
<h2 id="processing-documents">Processing Documents</h2>
<div>
<img src="/images/seriesly-queryfrag.png" alt="query fragment" class="floatright" />
</div>
<p>The document processors are another group of identical workers with
another fairly simple job:</p>
<blockquote>
<p>to read a collection of documents, extract relevant information from
them, and reduce the extracted information to an output value</p>
</blockquote>
<p>Extracting a portion of the query example document from <a href="/2012/09/09/seriesly.html">the intro
blog post</a>, you can see the role it performs to the right.</p>
<p>Before it begins pulling documents, it creates one goroutine for each
reduction that was requested for the result. I’ll describe this
pattern more below, but at this point it’s important to understand
that we’re going to be streaming data through the reducers and not
accumulating and batch processing results.</p>
<p>The most expensive things it does are also least interesting, so I’m
going to be a bit hand-wavy. It fetches the document from disk (which
is pretty much <code class="language-plaintext highlighter-rouge">pread</code>) and parses it as a <code class="language-plaintext highlighter-rouge">json</code> document.</p>
<p>Ignoring all the gory details of coercion, error handling, etc… that
gives us a loop that looks like this:</p>
<figure class="highlight"><pre><code class="language-go" data-lang="go"><span class="c">// Build the slice of reducers</span>
<span class="n">reducers</span><span class="p">,</span> <span class="n">outputs</span> <span class="o">:=</span> <span class="n">makeReducers</span><span class="p">(</span><span class="n">reds</span><span class="p">)</span>
<span class="c">// Pull each document from storage</span>
<span class="k">for</span> <span class="n">_</span><span class="p">,</span> <span class="n">data</span> <span class="o">:=</span> <span class="k">range</span> <span class="n">docs</span> <span class="p">{</span>
<span class="n">jsondoc</span> <span class="o">:=</span> <span class="n">parseJson</span><span class="p">(</span><span class="n">data</span><span class="p">)</span>
<span class="k">for</span> <span class="n">i</span><span class="p">,</span> <span class="n">jsonpointer</span> <span class="o">:=</span> <span class="k">range</span> <span class="n">ptrs</span> <span class="p">{</span>
<span class="n">reducers</span><span class="p">[</span><span class="n">i</span><span class="p">]</span> <span class="o"><-</span> <span class="n">extractValue</span><span class="p">(</span><span class="n">jsondoc</span><span class="p">,</span> <span class="n">jsonpointer</span><span class="p">)</span>
<span class="p">}</span>
<span class="p">}</span>
<span class="c">// Close reducer input channels so they complete</span>
<span class="n">closeAll</span><span class="p">(</span><span class="n">chans</span><span class="p">)</span>
<span class="c">// Read the output of the channels and send it all back</span>
<span class="n">results</span> <span class="o"><-</span> <span class="n">recvItemFromAllChannels</span><span class="p">(</span><span class="n">chans</span><span class="p">)</span></code></pre></figure>
<p>Due to the way this is streamed, we never require more than one
document in memory per document worker. A typical reducer for gauge
values doesn’t need more than its return value resident.</p>
<p>Closing the reducer input channels causes them to emit their results
on an output channel. Harvesting all these results for a “row” joins
all the goroutines and allows us to emit the value back to the HTTP
worker so it can transmit the results (described in further detail
below).</p>
<p><em>“But wait,”</em> you say, <em>“what are these goroutines that are being
joined?”</em></p>
<p><strong>Every column in every row is a goroutine</strong>. e.g. a 2,000 row result
with 100 pointers and reducers applied requires <em>at least</em> 200,000
short-lived goroutines to be started to model this concurrency.</p>
<p>Now, I imagine many of you are wondering why I’d make reducers be
functions that are run in separate goroutines that keep their state on
the stack and then consume a channel of input. Why wouldn’t I just
make them structs where their state is held as a field and then just
have a method called on the input to modify the state?</p>
<p>Because that’s hard. Consider the <code class="language-plaintext highlighter-rouge">c_min</code> reducer to understand why:</p>
<figure class="highlight"><pre><code class="language-go" data-lang="go"><span class="k">func</span><span class="p">(</span><span class="n">input</span> <span class="k">chan</span> <span class="n">ptrval</span><span class="p">)</span> <span class="k">interface</span><span class="p">{}</span> <span class="p">{</span>
<span class="n">rv</span> <span class="o">:=</span> <span class="n">math</span><span class="o">.</span><span class="n">NaN</span><span class="p">()</span>
<span class="k">for</span> <span class="n">v</span> <span class="o">:=</span> <span class="k">range</span> <span class="n">convertTofloat64Rate</span><span class="p">(</span><span class="n">input</span><span class="p">)</span> <span class="p">{</span>
<span class="k">if</span> <span class="n">v</span> <span class="o"><</span> <span class="n">rv</span> <span class="o">||</span> <span class="n">math</span><span class="o">.</span><span class="n">IsNaN</span><span class="p">(</span><span class="n">rv</span><span class="p">)</span> <span class="p">{</span>
<span class="n">rv</span> <span class="o">=</span> <span class="n">v</span>
<span class="p">}</span>
<span class="p">}</span>
<span class="k">return</span> <span class="n">rv</span>
<span class="p">}</span></code></pre></figure>
<p>The <code class="language-plaintext highlighter-rouge">c_min</code> reducer computes the minimum growth rate (per second) of a
stream of pointer values on the given channel. The trick is that in
order to know the rate at a given instance, I need to know both the
<em>next</em> value and the timestamp of the next value so I can divide the
growth rate over the time delta.</p>
<p>It’s all obvious once you consider how <code class="language-plaintext highlighter-rouge">convertTofloat64Rate</code> must
work. This is a function that takes a stream of <code class="language-plaintext highlighter-rouge">ptrval</code>s and emits a
string of <code class="language-plaintext highlighter-rouge">float64</code>s that are flattened rates over that range. It
does all of the hard work around finding the first suitable value
(skipping <code class="language-plaintext highlighter-rouge">nil</code> values where the pointer didn’t point to valid data)
and computing the delta from each previous value to emit. By
definition, this always consumes more values than it emits.</p>
<p>Perhaps my brain is twisted by decades of living in unix shells, but
I do the same thing processing data in the shell:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>% grep field fromfile | ./computerate | ./min
</code></pre></div></div>
<p>Except goroutines are way cheaper than threads (which are way cheaper
than processes), so the way that’s natural to me also performs quite
well. This is certainly <em>possible</em> to do with OO-style state
management, but I don’t want to write that code.</p>
<p>As a bonus, <code class="language-plaintext highlighter-rouge">convertTofloat64Rate</code> (and its older brother
<code class="language-plaintext highlighter-rouge">convertTofloat64</code>) creates a goroutine that consumes the input
channel in order to emit over its output channel. That means that if
all 200,000 of the results were numeric, it’d require 400,000
goroutines.</p>
<h2 id="transmitting-results">Transmitting Results</h2>
<p><em>Meanwhile, back in the HTTP handler…</em></p>
<p>After it submitted the request, the HTTP handler started a <code class="language-plaintext highlighter-rouge">select</code>
loop across the two channels built at query time. We’ll call them the
<code class="language-plaintext highlighter-rouge">error</code> channel and <code class="language-plaintext highlighter-rouge">results</code> channel.</p>
<p>When the error channel has received its completion message and there
has been one grouping output for every grouping found by the walker
(the counter mentioned above), the query processing is complete.</p>
<p>The HTTP layer currently emits docs as soon as they come out of the
results channel, this brings two great benefits:</p>
<p><strong>The first result latency is as low as possible.</strong> Once a result is
known, we immediately transmit it. Early versions batched up the
results and would get similar throughput, but I decided it was
unnecessary.</p>
<p><strong>Memory usage is as low as possible.</strong> Since I’m not batching up the
results (and more importantly, not serializing them into a in a single
encoder call), memory usage is down to the minimum required to pass
the necessary information around and encode it for the wire.</p>
<p>However, these benefits come with two consequences:</p>
<p><strong>Groupings are not returned to the HTTP requester in any defined
order.</strong> In fact, if you run the same query twice, you will get the
same results, but with different time windows appearing in different
locations in the result. This is especially strange if you stream
compressed data from the server and see it coming back as different
sizes with the same canonical document representation.</p>
<p><strong>There’s no clean way to report an error in an HTTP stream.</strong> After I
send HTTP <code class="language-plaintext highlighter-rouge">200</code> and start streaming data, it’s too late to realize
there’s a problem and call it a <code class="language-plaintext highlighter-rouge">500</code>. The best I can do is hang up
and leave you with an invalid document.</p>
<p>I struggled with this one a bit, but not having to buffer into memory,
reducing latency, etc… while not having a great answer for large
broken queries seemed like a good trade-off. Most ways I could figure
to send someone an error in-stream in the rare case where one happens
would complicate the user experience. With this approach, even
completely cold queries return results pretty much immediately.</p>
<h1 id="about-storage">About Storage</h1>
<div>
<img src="/images/seriesly-io.png" alt="query fragment" class="floatright" />
</div>
<p>Seriesly uses <a href="https://github.com/couchbase/couchstore">couchstore</a> for the backend. Depending on
how intimately you know <a href="http://couchdb.apache.org/">CouchDB</a>, you can think of it as a C
implementation of the core storage of CouchDB. Except, of course, I’m
using my <a href="https://github.com/dustin/go-couchstore">go bindings</a>.</p>
<p>Couchstore provides an on-disk append-only b-tree. This gives me a
durable format I can write to and read from at the same time. Neither
readers nor writers pay any attention to each other whatsoever. Each
query or doc worker opens a database for read access whenever it’s
necessary and just starts reading. If a write is happening at the
time, the reader just seeks backwards a bit in the file until it finds
a valid previous header and carry on.</p>
<p>Writes are batched by time and/or size. That is, a write targeting a
database doesn’t immediately hit the filesytem. This is mostly
beneficial when loading lots of data in bulk. Obviously both of these
parameters are configurable.</p>
<p>If most of the writes are small, you’ll want to compact the database
periodically. Compaction never blocks reads and almost never blocks
writes. There’s a buffered channel between the HTTP handler for
storing a new document and the actual DB writes. If you exceed this
buffer size, the put will block. Otherwise, writes are flushed,
compaction occurs, the DB is atomically replaced and then the
accumulated writes are completed.</p>
<h1 id="now-with-memcached">Now with Memcached</h1>
<p>The above all assumes no cache. Throwing memcached in the mix makes
things a lot more interesting. Let’s take the first diagram and fit a
cache into it:</p>
<div>
<img src="/images/seriesly-flow.png" alt="query fragment" class="center" />
</div>
<p>Now, conceptually, the cache just interposes between the query workers
and the document workers. The code is very close to that, in fact.
If you don’t have memcached enabled, the channel that the query worker
places its requests into is the document worker input itself. When
you have memcached enabled, the query worker output goes to the cache
workers (and cache misses go to the document workers).</p>
<div>
<img src="/images/seriesly-memcached.png" alt="query fragment" class="floatright" />
</div>
<p>What’s unusual about the cache usage here is that I’m not using any
patterns I’ve used elsewhere with memcached. It’s a little similar to
how the internals of <a href="http://code.google.com/p/spymemcached/">spymemcached</a> were implemented, but I did
some interesting binary-protocol-only stuff.</p>
<p>The diagram to the right shows the approximate anatomy of the
memcached workers and their interaction with the rest of the system.
The orange box delineates the conceptual memcached worker from the
rest of the system. Note that a memcached worker is <em>two</em> goroutines
sharing a connection to memcached. One <em>only</em> reads and one <em>only</em>
writes. I labeled the one that writes as <code class="language-plaintext highlighter-rouge">main</code> as it’s also our
interface to the cache.</p>
<p>Unlike most uses of memcached, we don’t do any “multi” type operations
like a “quiet” <code class="language-plaintext highlighter-rouge">get</code> at all. Instead, any time we need to send data
to memcached, we construct a packet and lob it over. The reader is
reading the output of that and after parsing the result into a packet
and determining that the <code class="language-plaintext highlighter-rouge">main</code> goroutine might be interested in it
(basically, this means it’s a successful <code class="language-plaintext highlighter-rouge">get</code> request), it sends it
back. These two messages are stitched together using the memcached
binary protocol <code class="language-plaintext highlighter-rouge">opaque</code> field – a 32-bit number that is application
specific and designed to enable this type of thing.</p>
<p>Pseudocode of our memcached main looks like the following:</p>
<figure class="highlight"><pre><code class="language-go" data-lang="go"><span class="k">for</span> <span class="p">{</span>
<span class="k">select</span> <span class="p">{</span>
<span class="k">case</span> <span class="n">req</span> <span class="o">:=</span> <span class="o"><-</span><span class="n">cacheRequestChannel</span><span class="o">:</span>
<span class="c">// Generate a unique identifier for this request</span>
<span class="n">opaque</span> <span class="o">:=</span> <span class="n">nextOpaque</span><span class="p">()</span>
<span class="n">opaqueMap</span><span class="p">[</span><span class="n">opaque</span><span class="p">]</span> <span class="o">=</span> <span class="n">req</span>
<span class="n">mc</span><span class="o">.</span><span class="n">transmitGetRequest</span><span class="p">(</span><span class="n">req</span><span class="o">.</span><span class="n">key</span><span class="p">,</span> <span class="n">opaque</span><span class="p">)</span>
<span class="k">case</span> <span class="n">res</span> <span class="o">:=</span> <span class="o"><-</span><span class="n">cacheResponseChannel</span><span class="o">:</span>
<span class="c">// Map the response back to the request by identifier</span>
<span class="n">in</span> <span class="o">:=</span> <span class="n">opaqueMap</span><span class="p">[</span><span class="n">res</span><span class="o">.</span><span class="n">Opaque</span><span class="p">]</span>
<span class="nb">delete</span><span class="p">(</span><span class="n">opaqueMap</span><span class="p">,</span> <span class="n">res</span><span class="o">.</span><span class="n">Opaque</span><span class="p">)</span>
<span class="k">if</span> <span class="n">in</span><span class="o">.</span><span class="n">err</span> <span class="p">{</span>
<span class="n">docProcessor</span> <span class="o"><-</span> <span class="n">in</span>
<span class="p">}</span> <span class="k">else</span> <span class="p">{</span>
<span class="n">in</span><span class="o">.</span><span class="n">response</span> <span class="o"><-</span> <span class="n">res</span>
<span class="p">}</span>
<span class="k">case</span> <span class="n">toSet</span> <span class="o">:=</span> <span class="o"><-</span><span class="n">setRequestChannel</span><span class="o">:</span>
<span class="n">mc</span><span class="o">.</span><span class="n">transmitQuietSetRequest</span><span class="p">(</span><span class="n">toSet</span><span class="o">.</span><span class="n">key</span><span class="p">,</span> <span class="n">toSet</span><span class="o">.</span><span class="n">value</span><span class="p">)</span>
<span class="p">}</span>
<span class="p">}</span></code></pre></figure>
<p>Where a typical “multiget” type strategy would require you to batch up
a bunch of requests, send them all, get all the responses and infer
the misses, we process the results with minimal state – only keeping
up with what we’ve got in-flight. If a response is a hit, we send it
back up to the http handler. If it’s a miss (or there was any other
error), we send it to the doc worker. Done.</p>
<p>Of course, the actual loop is a bit more complicated as it deals with
fetch cancellations, connection failures which result in dumping all
in-flight cache requests directly into the document worker queues
while asynchronously starting a timed reconnection loop and a few
other things. The above is pretty much the golden path, though.</p>
<p>At the layer the cache is installed, there are a lot of benefits I
don’t get, but when the site was pretty popular on hacker news, I was
seeing queries like this flying by:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>Completed query processing in 69.528ms, 9,394 keys, 1,920 chunks
</code></pre></div></div>
<p>That means we pulled 9,394 document IDs from the on-disk index,
chopped them up into 1,920 groups, computed hash keys for them and
then passed through the above <code class="language-plaintext highlighter-rouge">select</code> loop 3,840 times (once each for
1,920 requests and again for the responses) in 69.528 milliseconds.
Under load. That’s an absolute ceiling of 35µs cache round trip time,
again, ignoring all other queries running at this time.</p>
Seriesly - Document Oriented Time Series DB2012-09-09T00:00:00+00:00http://dustin.github.com/2012/09/09/seriesly<h1 id="why-so-seriesly">Why so, Seriesly?</h1>
<div>
<img src="/images/seriesly-compare.png" alt="comparison" title="Comparing Time Series DAta" class="floatleft" />
</div>
<p>So, I started writing a document-oriented <a href="http://en.wikipedia.org/wiki/Time_series_database">time-series database</a>
in <a href="http://golang.org/">go</a> two weeks ago. It’s really nothing groundbreaking, but
it’s been quite fun.</p>
<p>My purpose is to have a system that allows me to store arbitrary
performance data captured as it’s seen in the wild, and then later
come up with ways to look at it.</p>
<p>Check out this <a href="https://gist.github.com/3683423">real life example</a> to get a feel for the kinds of
captures we’re working with. This sample represents a single point in
time, there may be many of these occuring at any frequency (well,
nothing more frequent than once per nanosecond). From this, we can do
arbitrary queries and report groupings at millisecond granularity.</p>
<h1 id="usage">Usage</h1>
<p>The <a href="https://github.com/dustin/seriesly/wiki">seriesly wiki</a> describes in detail all that can be done
with it, but the general strategy is the following:</p>
<ol>
<li>Capture lots of data.</li>
<li>Query the data.</li>
</ol>
<p>An important goal is to make it this simple in practice.</p>
<h2 id="capturing-data">Capturing Data</h2>
<p>Captured data is submitted to a database and recorded with a
timestamp. The timestamp may be generated by the system, or may be
supplied at request time (which is useful for backfilling data).</p>
<h2 id="querying-data">Querying Data</h2>
<p>Querying captured data is about taking data from a specific time
range, grouping it into specific time chunks (e.g. 5 minutes worth of
data at a time) selecting keys from the data (using
<a href="http://tools.ietf.org/html/draft-ietf-appsawg-json-pointer-03">json-pointer</a>) to query on and performing reductions
over the values selected using those keys.</p>
<p>The following diagram shows an example query and how the fields are
selected and values are returned (conceptually). For this query, I’ve
asked for the <code class="language-plaintext highlighter-rouge">count</code> of <code class="language-plaintext highlighter-rouge">/a</code>, the <code class="language-plaintext highlighter-rouge">avg</code> of <code class="language-plaintext highlighter-rouge">/c/e</code> and the <code class="language-plaintext highlighter-rouge">min</code> of <code class="language-plaintext highlighter-rouge">/b/2</code>
grouped in 5 minute windows (<code class="language-plaintext highlighter-rouge">300</code> seconds).</p>
<p><img src="//github.com/dustin/seriesly/wiki/serieslyquery.png" alt="query" /></p>
<p>(The key in the result does not represent the actual key that will be
emitted. I used a human-readable time representation for illustration
purposes only. Had this been an actual query, the timestamps would’ve
all been absolute and emitted as the number of milliseconds since UNIX
epoch.)</p>
<h2 id="web-stuff">Web Stuff</h2>
<p>I use <a href="http://square.github.com/cubism/">cubism</a> for a lot of my time series fun. You can see
an example of this backed by seriesly at <a href="http://bleu.west.spy.net/~dustin/seriesly/">my thermometers page</a>
at home. Seriesly is designed after building query interfaces to
cubism for other systems. There’s no impedene mismatch between what
cubism wants to know and the query language provided by seriesly.</p>
<p>Other than the normal cubism configuration and styling, the code to
fetch data for display is pretty straightforward:</p>
<figure class="highlight"><pre><code class="language-javascript" data-lang="javascript"><span class="c1">// First, point it to your data</span>
<span class="kd">var</span> <span class="nx">baseUrl</span> <span class="o">=</span> <span class="dl">"</span><span class="s2">http://my.seriesly.host/</span><span class="dl">"</span><span class="p">,</span>
<span class="nx">dbname</span> <span class="o">=</span> <span class="dl">"</span><span class="s2">mydb</span><span class="dl">"</span><span class="p">,</span>
<span class="nx">pointer</span> <span class="o">=</span> <span class="dl">"</span><span class="s2">/some/json/pointer</span><span class="dl">"</span><span class="p">,</span>
<span class="nx">reducer</span> <span class="o">=</span> <span class="dl">"</span><span class="s2">avg</span><span class="dl">"</span><span class="p">,</span>
<span class="nx">lbl</span> <span class="o">=</span> <span class="dl">"</span><span class="s2">Label for my Metric</span><span class="dl">"</span><span class="p">;</span>
<span class="c1">// Then get a seriesly metric source:</span>
<span class="kd">var</span> <span class="nx">sr</span> <span class="o">=</span> <span class="nx">context</span><span class="p">.</span><span class="nx">seriesly</span><span class="p">(</span><span class="nx">baseURL</span><span class="p">);</span>
<span class="c1">// And then your specific metric you want to plot:</span>
<span class="kd">var</span> <span class="nx">myMetric</span> <span class="o">=</span> <span class="nx">sr</span><span class="p">.</span><span class="nx">metric</span><span class="p">(</span><span class="nx">dbname</span><span class="p">,</span> <span class="nx">pointer</span><span class="p">,</span> <span class="nx">reducer</span><span class="p">,</span> <span class="nx">lbl</span><span class="p">);</span>
<span class="c1">// Then plot it like you would any other metric:</span>
<span class="nx">d3</span><span class="p">.</span><span class="nx">select</span><span class="p">(</span><span class="nx">here</span><span class="p">).</span><span class="nx">selectAll</span><span class="p">(</span><span class="dl">"</span><span class="s2">.horizon</span><span class="dl">"</span><span class="p">)</span>
<span class="p">.</span><span class="nx">data</span><span class="p">(</span><span class="nx">things</span><span class="p">.</span><span class="nx">map</span><span class="p">(</span><span class="kd">function</span><span class="p">(</span><span class="nx">x</span><span class="p">)</span> <span class="p">{</span> <span class="k">return</span> <span class="nx">myMetric</span><span class="p">;</span> <span class="p">}))</span>
<span class="p">.</span><span class="nx">enter</span><span class="p">().</span><span class="nx">insert</span><span class="p">(</span><span class="dl">"</span><span class="s2">div</span><span class="dl">"</span><span class="p">,</span> <span class="dl">"</span><span class="s2">.bottom</span><span class="dl">"</span><span class="p">)</span>
<span class="p">.</span><span class="nx">attr</span><span class="p">(</span><span class="dl">"</span><span class="s2">class</span><span class="dl">"</span><span class="p">,</span> <span class="dl">"</span><span class="s2">horizon</span><span class="dl">"</span><span class="p">)</span>
<span class="p">.</span><span class="nx">call</span><span class="p">(</span><span class="nx">context</span><span class="p">.</span><span class="nx">horizon</span><span class="p">());</span></code></pre></figure>
<p>A comparison of a metric against itself is just like you’d expect:</p>
<figure class="highlight"><pre><code class="language-javascript" data-lang="javascript"><span class="c1">// Get a metric and compare it to the same data a week ago.</span>
<span class="kd">var</span> <span class="nx">primary</span> <span class="o">=</span> <span class="nx">sr</span><span class="p">.</span><span class="nx">metric</span><span class="p">(</span><span class="nx">dbname</span><span class="p">,</span> <span class="nx">pointer</span><span class="p">,</span> <span class="nx">reducer</span><span class="p">),</span>
<span class="nx">secondary</span> <span class="o">=</span> <span class="nx">primary</span><span class="p">.</span><span class="nx">shift</span><span class="p">(</span><span class="o">-</span><span class="mi">7</span> <span class="o">*</span> <span class="mi">24</span> <span class="o">*</span> <span class="mi">60</span> <span class="o">*</span> <span class="mi">60</span> <span class="o">*</span> <span class="mi">1000</span><span class="p">);</span>
<span class="c1">// Then do the compare thing.</span>
<span class="nx">d3</span><span class="p">.</span><span class="nx">select</span><span class="p">(</span><span class="nx">here</span><span class="p">).</span><span class="nx">selectAll</span><span class="p">(</span><span class="dl">"</span><span class="s2">.comparison</span><span class="dl">"</span><span class="p">)</span>
<span class="p">.</span><span class="nx">data</span><span class="p">([[</span><span class="nx">primary</span><span class="p">,</span> <span class="nx">secondary</span><span class="p">]])</span>
<span class="p">.</span><span class="nx">enter</span><span class="p">().</span><span class="nx">append</span><span class="p">(</span><span class="dl">"</span><span class="s2">div</span><span class="dl">"</span><span class="p">)</span>
<span class="p">.</span><span class="nx">attr</span><span class="p">(</span><span class="dl">"</span><span class="s2">class</span><span class="dl">"</span><span class="p">,</span> <span class="dl">"</span><span class="s2">comparison</span><span class="dl">"</span><span class="p">)</span>
<span class="p">.</span><span class="nx">call</span><span class="p">(</span><span class="nx">comparison</span><span class="p">);</span></code></pre></figure>
<p>For complete examples, check out my <a href="http://bleu.west.spy.net/~dustin/seriesly/">temperature</a> plots and
view source. <a href="http://bleu.west.spy.net/~dustin/seriesly/series.js">series.js</a> has all the details.</p>
<h1 id="using-it">Using It</h1>
<p>Within a few days of starting the project, I had around fifteen
million data points loaded and querying. Today marks the first
fortnight birthday of the system and I’ve moved from the “make it work
correctly” phase to the “make it fast” phase.</p>
<p>I think I’ve been doing pretty well. When I started, every query like
this was taking over two minutes:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>Completed query processing in 5.7s, 1,894,473 keys, 1,396 chunks
</code></pre></div></div>
<p>I’ve still got a couple things I want to do that I haven’t yet, but
for the most part, I’m just looking for more experience with it.</p>
<h1 id="further-reading">Further Reading…</h1>
<p>I hope to write more about how some of the implementation since
<a href="http://golang.org/">go</a> made it possible to do some things that are quite difficult
in other languages. Particularly, modeling the query execution
concurrency and parallelism made it a lot easier to take the working
software and make it run faster and efficiently use all the resources
I could throw at it.</p>
<p>In the meantime, check out <a href="https://github.com/dustin/seriesly">the seriesly project page</a> to get
the software, file bugs (including new ideas), contribute changes and
all that.</p>
Incremental Mapreduce for Analytics with R2012-04-16T00:00:00+00:00http://dustin.github.com/2012/04/16/couchr<h1 id="incremental-mapreduce-for-analytics-with-r">Incremental Mapreduce for Analytics with R</h1>
<p>I’ve been wanting to describe some of my work with using R to help me
understand data I’m collecting in Couchbase Server<sup>†</sup>
because I find it quite interesting, useful and easy. However, it’s
been difficult for me to figure out a good starting point because I
don’t know who the audience would be. That is, finding the right set
of assumptions to get going has been quite hard.</p>
<p>Last week, however, I spoke to a really awesome guy in a media company
who had a specific question: “How can my analyists report on all the
wonderful data I’m storing in <a href="http://www.couchbase.com/">Couchbase</a>?” I dug deeper.
Who are these analysts? What tools do they use?</p>
<p style="font-size: smaller"><sup>†</sup> the incremental Map
Reduce Views are identical to Apache CouchDB views, so everything will
also work with CouchDB</p>
<h2 id="my-audience">My Audience</h2>
<p>Turns out, the analysists pretty close to what I would imagine. They
often use some kind of data warehousing tools from Oracle that do all
kinds of great magic, and then fall over really hard if you drift
outside of bounds they’re comfortable with. This sounded like
something I could ignore. But then he said something that gave me a
pretty solid foothold. While they’re not programmers, they do use
<a href="http://www.r-project.org/">R</a> as part of their data analysis.</p>
<p>Because this question was asked by a Couchbase user who wanted to know
how to get his data out, I’m going to assume anyone reading this knows
R a bit better than Couchbase.</p>
<h2 id="about-views">About Views</h2>
<p>There are a lot of things you can read if you want to understand the
couch view concept. The <a href="http://www.couchbase.com/docs/couchbase-manual-2.0/couchbase-views.html">view chapter</a> of the
<em>Couchbase Server Manual</em> covers the concept pretty well. If you want
to know everything you can know, then dig through that, but for most
of my uses, it really comes down to three things:</p>
<ol>
<li>Extract the useful information.</li>
<li>Sort it, putting like things together.</li>
<li>Do some basic aggregation.</li>
</ol>
<p>That’s how I take a lot of data and turn it into useful information
most of the time. Hopefully the examples that follow will help you do
the same.</p>
<h2 id="the-data">The Data</h2>
<p>The hardest part of any data grokking tutorial is that it’s never
about your data. This simultaneously makes it less interesting to the
reader and often makes it a bit harder to apply to your own problems.</p>
<p>Unfortunately, the most interesting data I regularly extract for
reporting is somewhat sensitive, so I can’t share the things that I’ve
got the most use out of, but I’m hoping this will help lead you to
something interesting.</p>
<p>The data I’ve chosen to work with is the SFPD Reported incidents set
from the <a href="https://data.sfgov.org/">SF Data Website</a> web site. It’s pretty much
everything that the SFPD has reported since 2003.</p>
<p>These documents are pretty regular and flat. Your data may be more
complicated, but the techniques are the same. Let’s begin by looking
at an example document from the SFPD data set:</p>
<figure class="highlight"><pre><code class="language-js" data-lang="js"><span class="p">{</span>
<span class="dl">"</span><span class="s2">category</span><span class="dl">"</span><span class="p">:</span> <span class="dl">"</span><span class="s2">Prostitution</span><span class="dl">"</span><span class="p">,</span>
<span class="dl">"</span><span class="s2">incident_id</span><span class="dl">"</span><span class="p">:</span> <span class="mi">90096348</span><span class="p">,</span>
<span class="dl">"</span><span class="s2">district</span><span class="dl">"</span><span class="p">:</span> <span class="dl">"</span><span class="s2">Tenderloin</span><span class="dl">"</span><span class="p">,</span>
<span class="dl">"</span><span class="s2">timestamp</span><span class="dl">"</span><span class="p">:</span> <span class="dl">"</span><span class="s2">2009-01-27T04:03:00</span><span class="dl">"</span><span class="p">,</span>
<span class="dl">"</span><span class="s2">lon</span><span class="dl">"</span><span class="p">:</span> <span class="o">-</span><span class="mf">122.416261836834</span><span class="p">,</span>
<span class="dl">"</span><span class="s2">lat</span><span class="dl">"</span><span class="p">:</span> <span class="mf">37.7853750846376</span><span class="p">,</span>
<span class="dl">"</span><span class="s2">location</span><span class="dl">"</span><span class="p">:</span> <span class="dl">"</span><span class="s2">Ofarrell St / Hyde St</span><span class="dl">"</span><span class="p">,</span>
<span class="dl">"</span><span class="s2">time</span><span class="dl">"</span><span class="p">:</span> <span class="dl">"</span><span class="s2">04:03</span><span class="dl">"</span><span class="p">,</span>
<span class="dl">"</span><span class="s2">date</span><span class="dl">"</span><span class="p">:</span> <span class="dl">"</span><span class="s2">2009-01-27</span><span class="dl">"</span><span class="p">,</span>
<span class="dl">"</span><span class="s2">resolution</span><span class="dl">"</span><span class="p">:</span> <span class="dl">"</span><span class="s2">Arrest, Cited</span><span class="dl">"</span><span class="p">,</span>
<span class="dl">"</span><span class="s2">day</span><span class="dl">"</span><span class="p">:</span> <span class="dl">"</span><span class="s2">Tuesday</span><span class="dl">"</span><span class="p">,</span>
<span class="dl">"</span><span class="s2">desc</span><span class="dl">"</span><span class="p">:</span> <span class="dl">"</span><span class="s2">Solicits To Visit House Of Prostitution</span><span class="dl">"</span>
<span class="p">}</span></code></pre></figure>
<p>I think I can understand what all these things are, so let’s get to
work.</p>
<h2 id="getting-data-into-r">Getting Data into R</h2>
<p>There are a few packages I’ll be using here, so let’s make sure we get
those into your R before we go:</p>
<figure class="highlight"><pre><code class="language-r" data-lang="r"><span class="n">install.packages</span><span class="p">(</span><span class="nf">c</span><span class="p">(</span><span class="s1">'rjson'</span><span class="p">,</span><span class="w"> </span><span class="s1">'ggplot2'</span><span class="p">,</span><span class="w"> </span><span class="s1">'reshape'</span><span class="p">),</span><span class="w">
</span><span class="n">dependencies</span><span class="o">=</span><span class="kc">TRUE</span><span class="p">)</span><span class="w">
</span><span class="n">require</span><span class="p">(</span><span class="n">rjson</span><span class="p">)</span><span class="w">
</span><span class="n">require</span><span class="p">(</span><span class="n">reshape</span><span class="p">)</span><span class="w">
</span><span class="n">require</span><span class="p">(</span><span class="n">RColorBrewer</span><span class="p">)</span><span class="w">
</span><span class="n">require</span><span class="p">(</span><span class="n">ggplot2</span><span class="p">)</span></code></pre></figure>
<p>As R likes “square” data, I tend to have the output of my views be
very regular, which also means I can have very simple functions for
taking a view and pulling it back out. For this purpose, I have some
basic common setup in my R scripts that looks like this:</p>
<figure class="highlight"><pre><code class="language-r" data-lang="r"><span class="c1"># Pointer to your couchbase view base. This is where you find your</span><span class="w">
</span><span class="c1"># own data</span><span class="w">
</span><span class="n">urlBase</span><span class="w"> </span><span class="o"><-</span><span class="w"> </span><span class="s1">'http://couchbase.example.com/sfpd'</span><span class="w">
</span><span class="c1"># This is your basic GET request -> parsed JSON.</span><span class="w">
</span><span class="n">getData</span><span class="w"> </span><span class="o"><-</span><span class="w"> </span><span class="k">function</span><span class="p">(</span><span class="n">subpath</span><span class="p">)</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="n">fromJSON</span><span class="p">(</span><span class="n">file</span><span class="o">=</span><span class="n">paste</span><span class="p">(</span><span class="n">urlBase</span><span class="p">,</span><span class="w"> </span><span class="n">subpath</span><span class="p">,</span><span class="w"> </span><span class="n">sep</span><span class="o">=</span><span class="s1">''</span><span class="p">))</span><span class="o">$</span><span class="n">rows</span><span class="w">
</span><span class="p">}</span><span class="w">
</span><span class="c1"># And this flattens it into a data frame, optionaly naming the</span><span class="w">
</span><span class="c1"># columns.</span><span class="w">
</span><span class="n">getFlatData</span><span class="w"> </span><span class="o"><-</span><span class="w"> </span><span class="k">function</span><span class="p">(</span><span class="n">sub</span><span class="p">,</span><span class="w"> </span><span class="n">n</span><span class="o">=</span><span class="kc">NULL</span><span class="p">)</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="n">b</span><span class="w"> </span><span class="o"><-</span><span class="w"> </span><span class="n">plyr</span><span class="o">::</span><span class="n">ldply</span><span class="p">(</span><span class="n">getData</span><span class="p">(</span><span class="n">sub</span><span class="p">),</span><span class="w"> </span><span class="n">unlist</span><span class="p">)</span><span class="w">
</span><span class="k">if</span><span class="w"> </span><span class="p">(</span><span class="o">!</span><span class="nf">is.null</span><span class="p">(</span><span class="n">n</span><span class="p">))</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="nf">names</span><span class="p">(</span><span class="n">b</span><span class="p">)</span><span class="w"> </span><span class="o"><-</span><span class="w"> </span><span class="n">n</span><span class="w">
</span><span class="p">}</span><span class="w">
</span><span class="n">b</span><span class="w">
</span><span class="p">}</span><span class="w">
</span><span class="c1"># Also, I'm going to be working with days of week, so I need these:</span><span class="w">
</span><span class="n">dow</span><span class="w"> </span><span class="o"><-</span><span class="w"> </span><span class="nf">c</span><span class="p">(</span><span class="s1">'Sunday'</span><span class="p">,</span><span class="w"> </span><span class="s1">'Monday'</span><span class="p">,</span><span class="w"> </span><span class="s1">'Tuesday'</span><span class="p">,</span><span class="w"> </span><span class="s1">'Wednesday'</span><span class="p">,</span><span class="w">
</span><span class="s1">'Thursday'</span><span class="p">,</span><span class="w"> </span><span class="s1">'Friday'</span><span class="p">,</span><span class="w"> </span><span class="s1">'Saturday'</span><span class="p">)</span><span class="w">
</span><span class="n">shortdow</span><span class="w"> </span><span class="o"><-</span><span class="w"> </span><span class="nf">c</span><span class="p">(</span><span class="s1">'Sun'</span><span class="p">,</span><span class="w"> </span><span class="s1">'Mon'</span><span class="p">,</span><span class="w"> </span><span class="s1">'Tue'</span><span class="p">,</span><span class="w"> </span><span class="s1">'Wed'</span><span class="p">,</span><span class="w"> </span><span class="s1">'Thu'</span><span class="p">,</span><span class="w"> </span><span class="s1">'Fri'</span><span class="p">,</span><span class="w"> </span><span class="s1">'Sat'</span><span class="p">)</span></code></pre></figure>
<h2 id="overall-crime-report-count">Overall Crime Report Count</h2>
<p>As with most data sets, I don’t actually even know where to start, so
first let’s just see what kinds of crimes we’ve got. I’m interested
in total counts and counts by day of week. The nice thing is that
with couch views, I can build a single view that will tell me either.
Let’s look at the view source:</p>
<figure class="highlight"><pre><code class="language-js" data-lang="js"><span class="kd">function</span><span class="p">(</span><span class="nx">doc</span><span class="p">)</span> <span class="p">{</span>
<span class="nx">emit</span><span class="p">([</span><span class="nx">doc</span><span class="p">.</span><span class="nx">category</span><span class="p">,</span> <span class="nx">doc</span><span class="p">.</span><span class="nx">day</span><span class="p">],</span> <span class="kc">null</span><span class="p">);</span>
<span class="p">}</span></code></pre></figure>
<p>Looks really simple, but combined with the <code class="language-plaintext highlighter-rouge">_count</code> built-in reducer,
this can do a lot of neat things when grouping. With <code class="language-plaintext highlighter-rouge">group_level=1</code>,
we get crime count by category. Let’s plot that and see what’s
popular. Assuming we saved that in a design document called
<code class="language-plaintext highlighter-rouge">categories</code> with the view name of <code class="language-plaintext highlighter-rouge">byday</code>, here’s what you tell R:</p>
<figure class="highlight"><pre><code class="language-r" data-lang="r"><span class="c1"># Get a dataframe containing the categories and their respective counts</span><span class="w">
</span><span class="n">cat</span><span class="w"> </span><span class="o"><-</span><span class="w"> </span><span class="n">getFlatData</span><span class="p">(</span><span class="s1">'_design/categories/_view/byday?group_level=1'</span><span class="p">,</span><span class="w">
</span><span class="nf">c</span><span class="p">(</span><span class="s1">'cat'</span><span class="p">,</span><span class="w"> </span><span class="s1">'count'</span><span class="p">))</span><span class="w">
</span><span class="c1"># The columns come back as strings and requires fixes to make it useful</span><span class="w">
</span><span class="n">cat</span><span class="o">$</span><span class="n">cat</span><span class="w"> </span><span class="o"><-</span><span class="w"> </span><span class="n">factor</span><span class="p">(</span><span class="n">cat</span><span class="o">$</span><span class="n">cat</span><span class="p">)</span><span class="w">
</span><span class="n">cat</span><span class="o">$</span><span class="n">count</span><span class="w"> </span><span class="o"><-</span><span class="w"> </span><span class="nf">as.numeric</span><span class="p">(</span><span class="n">cat</span><span class="o">$</span><span class="n">count</span><span class="p">)</span><span class="w">
</span><span class="c1"># Also, I found sorting it by count made it easier to understand</span><span class="w">
</span><span class="n">cat</span><span class="o">$</span><span class="n">cat</span><span class="w"> </span><span class="o"><-</span><span class="w"> </span><span class="n">reorder</span><span class="p">(</span><span class="n">cat</span><span class="o">$</span><span class="n">cat</span><span class="p">,</span><span class="w"> </span><span class="n">cat</span><span class="o">$</span><span class="n">count</span><span class="p">)</span><span class="w">
</span><span class="c1"># Now plot it.</span><span class="w">
</span><span class="n">ggplot</span><span class="p">(</span><span class="n">cat</span><span class="p">,</span><span class="w"> </span><span class="n">aes</span><span class="p">(</span><span class="n">cat</span><span class="p">,</span><span class="w"> </span><span class="n">count</span><span class="p">,</span><span class="w"> </span><span class="n">alpha</span><span class="o">=</span><span class="n">count</span><span class="p">))</span><span class="w"> </span><span class="o">+</span><span class="w">
</span><span class="n">geom_bar</span><span class="p">(</span><span class="n">fill</span><span class="o">=</span><span class="s1">'#333399'</span><span class="p">,</span><span class="w"> </span><span class="n">stat</span><span class="o">=</span><span class="s1">'identity'</span><span class="p">)</span><span class="w"> </span><span class="o">+</span><span class="w">
</span><span class="n">scale_alpha</span><span class="p">(</span><span class="n">to</span><span class="o">=</span><span class="nf">c</span><span class="p">(</span><span class="m">0.4</span><span class="p">,</span><span class="w"> </span><span class="m">0.9</span><span class="p">),</span><span class="w"> </span><span class="n">legend</span><span class="o">=</span><span class="kc">FALSE</span><span class="p">)</span><span class="w"> </span><span class="o">+</span><span class="w">
</span><span class="n">scale_y_continuous</span><span class="p">(</span><span class="n">formatter</span><span class="o">=</span><span class="s2">"comma"</span><span class="p">)</span><span class="w"> </span><span class="o">+</span><span class="w">
</span><span class="n">labs</span><span class="p">(</span><span class="n">x</span><span class="o">=</span><span class="s1">''</span><span class="p">,</span><span class="w"> </span><span class="n">y</span><span class="o">=</span><span class="s1">''</span><span class="p">)</span><span class="w"> </span><span class="o">+</span><span class="w">
</span><span class="n">opts</span><span class="p">(</span><span class="n">title</span><span class="o">=</span><span class="s1">'Total Crime Reports'</span><span class="p">)</span><span class="w"> </span><span class="o">+</span><span class="w">
</span><span class="n">coord_flip</span><span class="p">()</span><span class="w"> </span><span class="o">+</span><span class="w">
</span><span class="n">theme_bw</span><span class="p">()</span></code></pre></figure>
<p>Then R will give you this:</p>
<p><img src="/images/r/sfpd_cats.png" alt="all cats" /></p>
<h2 id="by-day-of-week">By Day of Week</h2>
<p>I found this to be somewhat interesting, so I wanted to know what the
distribution was by day of week. I can use the same view above with
<code class="language-plaintext highlighter-rouge">group_level=2</code>, but since the rates are tremendously different, I had
R compute the relative variance across the data frame for each
category by day of week and then plotted that. Here’s the R code:</p>
<figure class="highlight"><pre><code class="language-r" data-lang="r"><span class="c1"># Grab the same data, but separated by day of week.</span><span class="w">
</span><span class="n">cat_byday</span><span class="w"> </span><span class="o"><-</span><span class="w"> </span><span class="n">getFlatData</span><span class="p">(</span><span class="s1">'_design/categories/_view/byday?group_level=2'</span><span class="p">,</span><span class="w">
</span><span class="nf">c</span><span class="p">(</span><span class="s1">'cat'</span><span class="p">,</span><span class="w"> </span><span class="s1">'day'</span><span class="p">,</span><span class="w"> </span><span class="s1">'count'</span><span class="p">))</span><span class="w">
</span><span class="c1"># I'm doing similar fixup to the above, but with another ordering and</span><span class="w">
</span><span class="c1"># a couple views of day of week (much for playing around)</span><span class="w">
</span><span class="n">cat_byday</span><span class="o">$</span><span class="n">cat</span><span class="w"> </span><span class="o"><-</span><span class="w"> </span><span class="n">factor</span><span class="p">(</span><span class="n">cat_byday</span><span class="o">$</span><span class="n">cat</span><span class="p">)</span><span class="w">
</span><span class="n">cat_byday</span><span class="o">$</span><span class="n">cat</span><span class="w"> </span><span class="o"><-</span><span class="w"> </span><span class="n">reorder</span><span class="p">(</span><span class="n">cat_byday</span><span class="o">$</span><span class="n">cat</span><span class="p">,</span><span class="w"> </span><span class="n">cat_byday</span><span class="o">$</span><span class="n">cat</span><span class="p">)</span><span class="w">
</span><span class="n">cat_byday</span><span class="o">$</span><span class="n">count</span><span class="w"> </span><span class="o"><-</span><span class="w"> </span><span class="nf">as.numeric</span><span class="p">(</span><span class="n">cat_byday</span><span class="o">$</span><span class="n">count</span><span class="p">)</span><span class="w">
</span><span class="n">cat_byday</span><span class="o">$</span><span class="n">cat_by_count</span><span class="w"> </span><span class="o"><-</span><span class="w"> </span><span class="n">reorder</span><span class="p">(</span><span class="n">cat_byday</span><span class="o">$</span><span class="n">cat</span><span class="p">,</span><span class="w"> </span><span class="n">cat_byday</span><span class="o">$</span><span class="n">count</span><span class="p">)</span><span class="w">
</span><span class="n">cat_byday</span><span class="o">$</span><span class="n">day</span><span class="w"> </span><span class="o"><-</span><span class="w"> </span><span class="n">factor</span><span class="p">(</span><span class="n">cat_byday</span><span class="o">$</span><span class="n">day</span><span class="p">,</span><span class="w"> </span><span class="n">levels</span><span class="o">=</span><span class="n">dow</span><span class="p">)</span><span class="w">
</span><span class="n">cat_byday</span><span class="o">$</span><span class="n">shortdow</span><span class="w"> </span><span class="o"><-</span><span class="w"> </span><span class="n">factor</span><span class="p">(</span><span class="n">cat_byday</span><span class="o">$</span><span class="n">day</span><span class="p">,</span><span class="w"> </span><span class="n">levels</span><span class="o">=</span><span class="n">dow</span><span class="p">,</span><span class="w"> </span><span class="n">labels</span><span class="o">=</span><span class="n">shortdow</span><span class="p">)</span><span class="w">
</span><span class="c1"># Compute the percentage of each category by its day of week</span><span class="w">
</span><span class="n">a1</span><span class="w"> </span><span class="o"><-</span><span class="w"> </span><span class="n">aggregate</span><span class="p">(</span><span class="n">cat_byday</span><span class="o">$</span><span class="n">count</span><span class="p">,</span><span class="w"> </span><span class="n">by</span><span class="o">=</span><span class="nf">list</span><span class="p">(</span><span class="n">cat_byday</span><span class="o">$</span><span class="n">cat</span><span class="p">),</span><span class="w"> </span><span class="n">sum</span><span class="p">)</span><span class="w">
</span><span class="n">a2</span><span class="w"> </span><span class="o"><-</span><span class="w"> </span><span class="n">aggregate</span><span class="p">(</span><span class="n">cat_byday</span><span class="o">$</span><span class="n">count</span><span class="p">,</span><span class="w"> </span><span class="n">by</span><span class="o">=</span><span class="nf">list</span><span class="p">(</span><span class="n">cat_byday</span><span class="o">$</span><span class="n">day</span><span class="p">,</span><span class="n">cat_byday</span><span class="o">$</span><span class="n">cat</span><span class="p">),</span><span class="w">
</span><span class="n">sum</span><span class="p">)</span><span class="w">
</span><span class="n">total_per_day</span><span class="w"> </span><span class="o"><-</span><span class="w"> </span><span class="nf">rep</span><span class="p">(</span><span class="n">a1</span><span class="o">$</span><span class="n">x</span><span class="p">,</span><span class="w"> </span><span class="nf">rep</span><span class="p">(</span><span class="m">7</span><span class="p">,</span><span class="nf">length</span><span class="p">(</span><span class="n">a1</span><span class="o">$</span><span class="n">x</span><span class="p">)))</span><span class="w">
</span><span class="n">cat_byday</span><span class="o">$</span><span class="n">perc</span><span class="w"> </span><span class="o"><-</span><span class="w"> </span><span class="n">a2</span><span class="o">$</span><span class="n">x</span><span class="w"> </span><span class="o">/</span><span class="w"> </span><span class="n">total_per_day</span><span class="w">
</span><span class="c1"># Let's see what this looks like</span><span class="w">
</span><span class="n">ggplot</span><span class="p">(</span><span class="n">cat_byday</span><span class="p">,</span><span class="w"> </span><span class="n">aes</span><span class="p">(</span><span class="n">shortdow</span><span class="p">,</span><span class="w"> </span><span class="n">cat</span><span class="p">,</span><span class="w"> </span><span class="n">size</span><span class="o">=</span><span class="n">perc</span><span class="p">,</span><span class="w"> </span><span class="n">alpha</span><span class="o">=</span><span class="n">perc</span><span class="p">))</span><span class="w"> </span><span class="o">+</span><span class="w">
</span><span class="n">geom_point</span><span class="p">(</span><span class="n">color</span><span class="o">=</span><span class="s1">'#333399'</span><span class="p">)</span><span class="w"> </span><span class="o">+</span><span class="w">
</span><span class="n">scale_alpha</span><span class="p">(</span><span class="n">legend</span><span class="o">=</span><span class="kc">FALSE</span><span class="p">)</span><span class="w"> </span><span class="o">+</span><span class="w">
</span><span class="n">scale_size</span><span class="p">(</span><span class="n">legend</span><span class="o">=</span><span class="kc">FALSE</span><span class="p">)</span><span class="w"> </span><span class="o">+</span><span class="w">
</span><span class="n">scale_y_discrete</span><span class="p">(</span><span class="n">limits</span><span class="o">=</span><span class="n">rev</span><span class="p">(</span><span class="n">levels</span><span class="p">(</span><span class="n">cat_byday</span><span class="o">$</span><span class="n">cat</span><span class="p">)))</span><span class="w"> </span><span class="o">+</span><span class="w">
</span><span class="n">labs</span><span class="p">(</span><span class="n">x</span><span class="o">=</span><span class="s1">''</span><span class="p">,</span><span class="w"> </span><span class="n">y</span><span class="o">=</span><span class="s1">''</span><span class="p">)</span><span class="w"> </span><span class="o">+</span><span class="w">
</span><span class="n">theme_bw</span><span class="p">()</span></code></pre></figure>
<p>That seems like a lot of setup, but it was mostly just type setup and
stuff. We’ll reuse some below. At this point, we’ve got something to
look at, though:</p>
<p><img src="/images/r/sfpd_all_by_day.png" alt="all cats by day" /></p>
<h2 id="focused-subset">Focused Subset</h2>
<p>There’s a lot of data here and it’s all relative making it hard to
kind of see how to compare things. I wanted to really look at a
couple of areas and figure out what kinds of correlations existed. As
I already had the data loaded, I figured I’d just grab a subset of
what was already requested and facet plot it.</p>
<figure class="highlight"><pre><code class="language-r" data-lang="r"><span class="c1"># Pick a few categories of interest</span><span class="w">
</span><span class="n">interesting</span><span class="w"> </span><span class="o"><-</span><span class="w"> </span><span class="nf">c</span><span class="p">(</span><span class="s1">'Drug/narcotic'</span><span class="p">,</span><span class="w">
</span><span class="s1">'Prostitution'</span><span class="p">,</span><span class="w">
</span><span class="s1">'Drunkenness'</span><span class="p">,</span><span class="w">
</span><span class="s1">'Disorderly Conduct'</span><span class="p">)</span><span class="w">
</span><span class="c1"># Extract just this subset and refactor the categories</span><span class="w">
</span><span class="n">sex_and_drugs</span><span class="w"> </span><span class="o"><-</span><span class="w"> </span><span class="n">cat_byday</span><span class="p">[</span><span class="n">cat_byday</span><span class="o">$</span><span class="n">cat</span><span class="w"> </span><span class="o">%in%</span><span class="w"> </span><span class="n">interesting</span><span class="p">,]</span><span class="w">
</span><span class="n">sex_and_drugs</span><span class="o">$</span><span class="n">cat</span><span class="w"> </span><span class="o"><-</span><span class="w"> </span><span class="n">factor</span><span class="p">(</span><span class="nf">as.character</span><span class="p">(</span><span class="n">sex_and_drugs</span><span class="o">$</span><span class="n">cat</span><span class="p">))</span><span class="w">
</span><span class="n">ggplot</span><span class="p">(</span><span class="n">sex_and_drugs</span><span class="p">,</span><span class="w"> </span><span class="n">aes</span><span class="p">(</span><span class="n">shortdow</span><span class="p">,</span><span class="w"> </span><span class="n">count</span><span class="p">))</span><span class="w"> </span><span class="o">+</span><span class="w">
</span><span class="n">facet_wrap</span><span class="p">(</span><span class="o">~</span><span class="n">cat</span><span class="p">,</span><span class="w"> </span><span class="n">scales</span><span class="o">=</span><span class="s1">'free_y'</span><span class="p">)</span><span class="w"> </span><span class="o">+</span><span class="w">
</span><span class="n">geom_bar</span><span class="p">(</span><span class="n">fill</span><span class="o">=</span><span class="s1">'#333399'</span><span class="p">,</span><span class="w"> </span><span class="n">stat</span><span class="o">=</span><span class="s1">'identity'</span><span class="p">)</span><span class="w"> </span><span class="o">+</span><span class="w">
</span><span class="n">scale_y_continuous</span><span class="p">(</span><span class="n">formatter</span><span class="o">=</span><span class="s2">"comma"</span><span class="p">)</span><span class="w"> </span><span class="o">+</span><span class="w">
</span><span class="n">labs</span><span class="p">(</span><span class="n">x</span><span class="o">=</span><span class="s1">''</span><span class="p">,</span><span class="w"> </span><span class="n">y</span><span class="o">=</span><span class="s1">''</span><span class="p">)</span><span class="w"> </span><span class="o">+</span><span class="w">
</span><span class="n">opts</span><span class="p">(</span><span class="n">title</span><span class="o">=</span><span class="s1">'Select Crime Reports by Day'</span><span class="p">)</span><span class="w"> </span><span class="o">+</span><span class="w">
</span><span class="n">theme_bw</span><span class="p">()</span></code></pre></figure>
<p>And that should show me a lot more detail on these individual
categories.</p>
<p><img src="/images/r/sfpd_sex_and_drugs.png" alt="sex and drugs" /></p>
<p>Personally, I found the lack of correlation between alcohol related
incidents and others quite interesting. Alcohol seems to be the
anti-drug. Maybe prostitutes don’t like drunks. Who knows…</p>
<h2 id="over-time">Over Time</h2>
<p>At this point, I realized the data’s going back to 2003 and I haven’t
even considered whether things are getting better or worse. I didn’t
really explore this very much, but wanted to get a quick feel for
whether things are getting better or worse. Here’s a view that will
tell us incident rates by year and category:</p>
<figure class="highlight"><pre><code class="language-js" data-lang="js"><span class="kd">function</span><span class="p">(</span><span class="nx">doc</span><span class="p">)</span> <span class="p">{</span>
<span class="kd">var</span> <span class="nx">ymd</span> <span class="o">=</span> <span class="nx">doc</span><span class="p">.</span><span class="nx">date</span><span class="p">.</span><span class="nx">split</span><span class="p">(</span><span class="dl">'</span><span class="s1">-</span><span class="dl">'</span><span class="p">);</span>
<span class="nx">emit</span><span class="p">([</span><span class="nb">parseInt</span><span class="p">(</span><span class="nx">ymd</span><span class="p">[</span><span class="mi">0</span><span class="p">],</span> <span class="mi">10</span><span class="p">),</span> <span class="nx">doc</span><span class="p">.</span><span class="nx">category</span><span class="p">],</span> <span class="kc">null</span><span class="p">);</span>
<span class="p">}</span></code></pre></figure>
<p>As in all these examples, I combine this with the <code class="language-plaintext highlighter-rouge">_count</code> built-in
reduce. Let’s just chart up the yearly rates with the following R:</p>
<figure class="highlight"><pre><code class="language-r" data-lang="r"><span class="n">byyear</span><span class="w"> </span><span class="o"><-</span><span class="w"> </span><span class="n">getFlatData</span><span class="p">(</span><span class="s1">'_design/categories/_view/by_year?group_level=1'</span><span class="p">,</span><span class="w">
</span><span class="nf">c</span><span class="p">(</span><span class="s1">'year'</span><span class="p">,</span><span class="w"> </span><span class="s1">'count'</span><span class="p">))</span><span class="w">
</span><span class="n">byyear</span><span class="o">$</span><span class="n">year</span><span class="w"> </span><span class="o"><-</span><span class="w"> </span><span class="nf">as.numeric</span><span class="p">(</span><span class="n">byyear</span><span class="o">$</span><span class="n">year</span><span class="p">)</span><span class="w">
</span><span class="c1"># There's not enough 2012 here, so let's ignore that for this chart.</span><span class="w">
</span><span class="n">ggplot</span><span class="p">(</span><span class="n">byyear</span><span class="p">[</span><span class="n">byyear</span><span class="o">$</span><span class="n">year</span><span class="w"> </span><span class="o"><</span><span class="w"> </span><span class="m">2012</span><span class="p">,],</span><span class="w"> </span><span class="n">aes</span><span class="p">(</span><span class="n">year</span><span class="p">,</span><span class="w"> </span><span class="n">count</span><span class="p">))</span><span class="w"> </span><span class="o">+</span><span class="w">
</span><span class="n">stat_smooth</span><span class="p">(</span><span class="n">fill</span><span class="o">=</span><span class="s1">'#333399'</span><span class="p">)</span><span class="w"> </span><span class="o">+</span><span class="w">
</span><span class="n">labs</span><span class="p">(</span><span class="n">y</span><span class="o">=</span><span class="s1">''</span><span class="p">,</span><span class="w"> </span><span class="n">x</span><span class="o">=</span><span class="s1">''</span><span class="p">)</span><span class="w"> </span><span class="o">+</span><span class="w">
</span><span class="n">scale_y_continuous</span><span class="p">(</span><span class="n">formatter</span><span class="o">=</span><span class="n">comma</span><span class="p">)</span><span class="w"> </span><span class="o">+</span><span class="w">
</span><span class="n">opts</span><span class="p">(</span><span class="n">title</span><span class="o">=</span><span class="s2">"Total Incident Reports by Year"</span><span class="p">)</span><span class="w"> </span><span class="o">+</span><span class="w">
</span><span class="n">theme_bw</span><span class="p">()</span></code></pre></figure>
<p>What’s this tell us?</p>
<p><img src="/images/r/sfpd_byyear.png" alt="by year" /></p>
<p>Looks like things are getting better (or police are getting lazier).
I could dig into this a bit more to find out whether it’s true for all
categories, but I’m not that interested, so let’s look at something
else.</p>
<h2 id="crime-by-area">Crime by Area</h2>
<p>I was interested in knowing whether certain crimes were more popular
in some areas than others. I’m using the doc’s <code class="language-plaintext highlighter-rouge">district</code> property
for this (rather than the built-in coordinates) and thought it might
be a good use case for a heatmap.</p>
<p>One thing I noticed is that some reports don’t have a district
associated with them. I chose to ignore those for this report, but
you can quite easily see how you might substitute it with a custom
value if you wanted to specifically consider it. Let’s begin with the
following view code:</p>
<figure class="highlight"><pre><code class="language-js" data-lang="js"><span class="kd">function</span><span class="p">(</span><span class="nx">doc</span><span class="p">)</span> <span class="p">{</span>
<span class="k">if</span> <span class="p">(</span><span class="nx">doc</span><span class="p">.</span><span class="nx">district</span> <span class="o">!=</span> <span class="kc">null</span><span class="p">)</span> <span class="p">{</span>
<span class="nx">emit</span><span class="p">([</span><span class="nx">doc</span><span class="p">.</span><span class="nx">category</span><span class="p">,</span> <span class="nx">doc</span><span class="p">.</span><span class="nx">district</span><span class="p">],</span> <span class="kc">null</span><span class="p">);</span>
<span class="p">}</span>
<span class="p">}</span></code></pre></figure>
<p>Of course, we’ll use the <code class="language-plaintext highlighter-rouge">_count</code> built-in again. One thing I should
note about this is that while I did originally plot <em>all</em> data, I
later decided that I wasn’t interested in any area that had less than
1,000 crimes reported. As this is the <em>output</em> of the filter, I
needed to apply that in R as we have no means of requesting that from
couch a view (since the views are materialized and the map function
did not include a filter before the reduce was applied). Ideally,
we’d support this in the actual view request, but in the meantime, we
can extract it easily in post:</p>
<figure class="highlight"><pre><code class="language-r" data-lang="r"><span class="n">by_region</span><span class="w"> </span><span class="o"><-</span><span class="w"> </span><span class="n">getFlatData</span><span class="p">(</span><span class="s1">'_design/region/_view/by_cat_region?group_level=2'</span><span class="p">,</span><span class="w">
</span><span class="nf">c</span><span class="p">(</span><span class="s1">'cat'</span><span class="p">,</span><span class="w"> </span><span class="s1">'region'</span><span class="p">,</span><span class="w"> </span><span class="s1">'count'</span><span class="p">))</span><span class="w">
</span><span class="n">by_region</span><span class="o">$</span><span class="n">count</span><span class="w"> </span><span class="o"><-</span><span class="w"> </span><span class="nf">as.numeric</span><span class="p">(</span><span class="n">by_region</span><span class="o">$</span><span class="n">count</span><span class="p">)</span><span class="w">
</span><span class="n">by_region</span><span class="o">$</span><span class="n">region</span><span class="w"> </span><span class="o"><-</span><span class="w"> </span><span class="n">factor</span><span class="p">(</span><span class="n">by_region</span><span class="o">$</span><span class="n">region</span><span class="p">)</span><span class="w">
</span><span class="c1"># Ignore anything that doesn't have at least 1,000 incidents</span><span class="w">
</span><span class="n">pop_regions</span><span class="w"> </span><span class="o"><-</span><span class="w"> </span><span class="n">by_region</span><span class="p">[</span><span class="n">by_region</span><span class="o">$</span><span class="n">count</span><span class="w"> </span><span class="o">></span><span class="w"> </span><span class="m">1000</span><span class="p">,]</span><span class="w">
</span><span class="n">pop_regions</span><span class="o">$</span><span class="n">cat</span><span class="w"> </span><span class="o"><-</span><span class="w"> </span><span class="n">factor</span><span class="p">(</span><span class="nf">as.character</span><span class="p">(</span><span class="n">pop_regions</span><span class="o">$</span><span class="n">cat</span><span class="p">))</span><span class="w">
</span><span class="c1"># And have the hottest crimes float to the top</span><span class="w">
</span><span class="n">pop_regions</span><span class="o">$</span><span class="n">cat</span><span class="w"> </span><span class="o"><-</span><span class="w"> </span><span class="n">reorder</span><span class="p">(</span><span class="n">pop_regions</span><span class="o">$</span><span class="n">cat</span><span class="p">,</span><span class="w"> </span><span class="n">pop_regions</span><span class="o">$</span><span class="n">count</span><span class="p">)</span><span class="w">
</span><span class="n">ggplot</span><span class="p">(</span><span class="n">pop_regions</span><span class="p">,</span><span class="w"> </span><span class="n">aes</span><span class="p">(</span><span class="n">x</span><span class="o">=</span><span class="n">region</span><span class="p">,</span><span class="w"> </span><span class="n">y</span><span class="o">=</span><span class="n">cat</span><span class="p">,</span><span class="w"> </span><span class="n">fill</span><span class="o">=</span><span class="n">count</span><span class="p">,</span><span class="w"> </span><span class="n">alpha</span><span class="o">=</span><span class="n">count</span><span class="p">))</span><span class="w"> </span><span class="o">+</span><span class="w">
</span><span class="n">geom_tile</span><span class="p">()</span><span class="w"> </span><span class="o">+</span><span class="w">
</span><span class="n">scale_fill_continuous</span><span class="p">(</span><span class="s1">'Incidents'</span><span class="p">,</span><span class="w">
</span><span class="n">formatter</span><span class="o">=</span><span class="k">function</span><span class="p">(</span><span class="n">x</span><span class="p">)</span><span class="w">
</span><span class="n">sprintf</span><span class="p">(</span><span class="s2">"%dk"</span><span class="p">,</span><span class="w"> </span><span class="n">x</span><span class="w"> </span><span class="o">/</span><span class="w"> </span><span class="m">1000</span><span class="p">))</span><span class="w"> </span><span class="o">+</span><span class="w">
</span><span class="n">scale_alpha_continuous</span><span class="p">(</span><span class="n">legend</span><span class="o">=</span><span class="kc">FALSE</span><span class="p">,</span><span class="w"> </span><span class="n">to</span><span class="o">=</span><span class="nf">c</span><span class="p">(</span><span class="m">0.7</span><span class="p">,</span><span class="w"> </span><span class="m">1.0</span><span class="p">))</span><span class="w"> </span><span class="o">+</span><span class="w">
</span><span class="n">labs</span><span class="p">(</span><span class="n">x</span><span class="o">=</span><span class="s1">''</span><span class="p">,</span><span class="w"> </span><span class="n">y</span><span class="o">=</span><span class="s1">''</span><span class="p">)</span><span class="w"> </span><span class="o">+</span><span class="w">
</span><span class="n">theme_bw</span><span class="p">()</span><span class="w"> </span><span class="o">+</span><span class="w">
</span><span class="n">opts</span><span class="p">(</span><span class="n">title</span><span class="o">=</span><span class="s1">'Crime Types by District'</span><span class="p">,</span><span class="w">
</span><span class="n">axis.text.x</span><span class="o">=</span><span class="n">theme_text</span><span class="p">(</span><span class="n">angle</span><span class="o">=</span><span class="m">-90</span><span class="p">),</span><span class="w">
</span><span class="n">legend.position</span><span class="o">=</span><span class="s1">'right'</span><span class="p">)</span></code></pre></figure>
<p>That gives us the following heatmap:</p>
<p><img src="/images/r/sfpd_regions.png" alt="regions" /></p>
<p>The blank areas didn’t have 1,000 incidents of the specified type of
crime in the indicated area since 2003. The lighter blue areas have
had some incidents. The bright red have the most. Looks like I want
to avoid the southern district.</p>
<h2 id="how-many-does-the-da-refuse">How Many Does the DA Refuse?</h2>
<p>As an example of pulling a server-side aggregate on part of the data,
I found the “District Attorney Refuses To Prosecute” resolution type
particularly interesting, so I wanted to know how often this happens.
Again, we start with a simple view:</p>
<figure class="highlight"><pre><code class="language-js" data-lang="js"><span class="kd">function</span><span class="p">(</span><span class="nx">doc</span><span class="p">)</span> <span class="p">{</span>
<span class="nx">emit</span><span class="p">([</span><span class="nx">doc</span><span class="p">.</span><span class="nx">resolution</span><span class="p">,</span> <span class="nx">doc</span><span class="p">.</span><span class="nx">category</span><span class="p">],</span> <span class="kc">null</span><span class="p">);</span>
<span class="p">}</span></code></pre></figure>
<p>Then we do our normal <code class="language-plaintext highlighter-rouge">_count</code> thing. However, the difference here is
that when I do the request, I want to use the <code class="language-plaintext highlighter-rouge">start_key</code> and
<code class="language-plaintext highlighter-rouge">end_key</code> parameters to find only things that were resolved in this
way. I happen to know that the list of resolutions goes from
“District Attorney Refuses To Prosecute” to “Exceptional Clearance”,
so I can just look for things that start with “Di” and end with things
that start with “Dj”. These are also arrays I’m emitting, so it’s
really based on the first element of the array. The R code then looks
like this:</p>
<figure class="highlight"><pre><code class="language-r" data-lang="r"><span class="n">by_resolution</span><span class="w"> </span><span class="o"><-</span><span class="w"> </span><span class="n">getFlatData</span><span class="p">(</span><span class="n">paste</span><span class="p">(</span><span class="s1">'_design/resolution/_view/by_res_cat'</span><span class="p">,</span><span class="w">
</span><span class="s1">'?group_level=2&start_key=["Di]'</span><span class="p">,</span><span class="w">
</span><span class="s1">'&end_key=["Dj"]'</span><span class="p">,</span><span class="w"> </span><span class="n">sep</span><span class="o">=</span><span class="s2">""</span><span class="p">),</span><span class="w">
</span><span class="nf">c</span><span class="p">(</span><span class="s1">'resolution'</span><span class="p">,</span><span class="w"> </span><span class="s1">'cat'</span><span class="p">,</span><span class="w"> </span><span class="s1">'count'</span><span class="p">))</span><span class="w">
</span><span class="n">by_resolution</span><span class="o">$</span><span class="n">count</span><span class="w"> </span><span class="o"><-</span><span class="w"> </span><span class="nf">as.numeric</span><span class="p">(</span><span class="n">by_resolution</span><span class="o">$</span><span class="n">count</span><span class="p">)</span><span class="w">
</span><span class="n">by_resolution</span><span class="o">$</span><span class="n">cat</span><span class="w"> </span><span class="o"><-</span><span class="w"> </span><span class="n">factor</span><span class="p">(</span><span class="n">by_resolution</span><span class="o">$</span><span class="n">cat</span><span class="p">)</span><span class="w">
</span><span class="n">by_resolution</span><span class="o">$</span><span class="n">cat</span><span class="w"> </span><span class="o"><-</span><span class="w"> </span><span class="n">reorder</span><span class="p">(</span><span class="n">by_resolution</span><span class="o">$</span><span class="n">cat</span><span class="p">,</span><span class="w"> </span><span class="n">by_resolution</span><span class="o">$</span><span class="n">count</span><span class="p">)</span><span class="w">
</span><span class="n">ggplot</span><span class="p">(</span><span class="n">by_resolution</span><span class="p">,</span><span class="w"> </span><span class="n">aes</span><span class="p">(</span><span class="n">cat</span><span class="p">,</span><span class="w"> </span><span class="n">count</span><span class="p">,</span><span class="w"> </span><span class="n">alpha</span><span class="o">=</span><span class="n">count</span><span class="p">))</span><span class="w"> </span><span class="o">+</span><span class="w">
</span><span class="n">scale_alpha</span><span class="p">(</span><span class="n">to</span><span class="o">=</span><span class="nf">c</span><span class="p">(</span><span class="m">0.4</span><span class="p">,</span><span class="w"> </span><span class="m">0.9</span><span class="p">),</span><span class="w"> </span><span class="n">legend</span><span class="o">=</span><span class="kc">FALSE</span><span class="p">)</span><span class="w"> </span><span class="o">+</span><span class="w">
</span><span class="n">coord_flip</span><span class="p">()</span><span class="w"> </span><span class="o">+</span><span class="w">
</span><span class="n">geom_bar</span><span class="p">(</span><span class="n">fill</span><span class="o">=</span><span class="s1">'#333399'</span><span class="p">,</span><span class="w"> </span><span class="n">stat</span><span class="o">=</span><span class="s1">'identity'</span><span class="p">)</span><span class="w"> </span><span class="o">+</span><span class="w">
</span><span class="n">labs</span><span class="p">(</span><span class="n">x</span><span class="o">=</span><span class="s1">''</span><span class="p">,</span><span class="w"> </span><span class="n">y</span><span class="o">=</span><span class="s1">''</span><span class="p">)</span><span class="w"> </span><span class="o">+</span><span class="w">
</span><span class="n">opts</span><span class="p">(</span><span class="n">title</span><span class="o">=</span><span class="s1">'Crimes the DA Refused to Prosecute'</span><span class="p">)</span><span class="w"> </span><span class="o">+</span><span class="w">
</span><span class="n">theme_bw</span><span class="p">()</span></code></pre></figure>
<p>R then gives us the following:</p>
<p><img src="/images/r/sfpd_da_refused.png" alt="regions" /></p>
<p>Do note that these are <em>absolute</em> numbers. Don’t call up SF and
complain because they don’t care about assault as they care about
vandalism. There are simply more of those cases. I’ll leave as an
exercise to the reader evaluating resolution types by category and
deciding what to think about them.</p>
<h2 id="in-conclusion">In Conclusion</h2>
<p>I could obviously keep going with this for days, but just wanted to
help people understand my process. In most places I use this, the
patterns are similar. Data sets may grow very large, but the
aggregations remain small. Incremental processing of the views means
my sorted and aggregated answers continue to arrive quickly and
processing remains cheap.</p>
What I've Been Up To (2011-09-20)2011-09-20T00:00:00+00:00http://dustin.github.com/2011/09/20/catching-up<h1 id="what-ive-been-up-to---a-virtual-blog-digest">What I’ve been Up To - A Virtual Blog Digest</h1>
<p>I’ve been busy doing lots and lots of things. Much of it is work
related, but I figured I’d quickly dump out a semi-structured list of
things I’ve been doing since my last post (over three months!)</p>
<p>Many of these things deserve their own blog posts, but since I’ve
clearly not been doing that, this digest will at least provide
pointers for those interested.</p>
<h2 id="mccouch">mccouch</h2>
<p>My primary job is to build a better couchbase server.
<a href="https://github.com/couchbase/mccouch">mccouch</a> is a key part of that by providing a memcached
binary protocol interface to CouchDB. Bypassing HTTP and using
memcached multi-set semantics gets us really good throughput for
putting data on disk.</p>
<h2 id="house-temperature-readers">house temperature readers</h2>
<div>
<img src="http://bleu.west.spy.net/house/" alt="house" title="I live here." class="floatright" />
</div>
<p>My thermometers live on a <a href="https://github.com/dustin/ibutton">1-wire bus</a> I wrote a stack and
collector for that sends multicast data around the LAN I pick up from
various things like the app that generates the image to your right.</p>
<p>I wanted to get that data stored into CouchDB, so I wrote a quick
<a href="https://gist.github.com/1028791">multicast temperature -> couchdb</a> thing in
<a href="http://coffeescript.org/">coffeescript</a> for <a href="http://nodejs.org/">node.js</a>.</p>
<p>There were a couple of aspects of that I didn’t like (I think the
primary thing was the VSS on the Linux box I had it deployed), so I
ended up <a href="https://gist.github.com/1088300">rewriting it in go</a>. That’s been serving me pretty
well.</p>
<h2 id="couchdb-duplicator">couchdb duplicator</h2>
<p>In early testing of replicator DB of CouchDB (i.e. playing with it
around the house), I wrote <a href="https://gist.github.com/1033557">a quick tool</a> to have one
CouchDB be a full mirror of another for all existing databases. I’ve
got a few big databases. That was fun.</p>
<h2 id="complex-data-driven-couchdb-partial-replication">complex data-driven couchdb partial replication</h2>
<p>I thought it might be fun to replicate my temperature database (~11M
docs at the time) to another database filtered down to only the
documents that met certain criteria specified by the replication
request. <a href="https://gist.github.com/1034925">That worked</a> via a replication document
specification, but backfilling wasn’t as fast as I’d like. I’ll need
to do more work here.</p>
<h2 id="location-project">location project</h2>
<div>
<img src="/images/locations-small.png" alt="locations" title="Where I go." class="floatright" />
</div>
<p>I created my <a href="https://github.com/dustin/location">location project</a> as a repository of location
data from <a href="http://www.google.com/latitude/">google latitude</a> and <a href="http://www.tripit.com/">tripit</a>. It
automatically populates itself with data from these sources and will
let me explore my many travels interactively thanks to
<a href="https://github.com/couchbase/geocouch">geocouch</a> (my first time using it) and google maps.</p>
<p>As you can see, my many travels are typically to Tahoe and home.</p>
<h2 id="code-review">code review</h2>
<div>
<img src="/images/gerrit-small.png" alt="gerrit" title="Our code." class="floatright" />
</div>
<p>I did some work here and there on our <a href="http://dustinphoto.iriscouch.com/gerrit/_design/app/index.html">gerrit dashboard</a> thing
that shows what we’ve been working on.</p>
<p>One somewhat notable thing was switching from gravatar to
<a href="https://www.libravatar.org/">libravatar</a> for the avatars. Most people probably didn’t
notice, but libravatar is pretty cool, and it’s got pass-through
compatability.</p>
<p>I’ve got one of these for <a href="http://dustinphoto.iriscouch.com/android/_design/app/index.html">android</a> as well, though
kernel.org’s outage has made it pretty stale.</p>
<h2 id="photo-app-work">photo app work</h2>
<p>I did a bunch of work on my <a href="https://github.com/dustin/photo-couch">photo couchapp</a>. I use this to
store and replicate all of my photos. It’s probably not exciting to
most people (though everyone’s free to use it for a great way to use
the cloud to benefit without dataloss or availability risk).</p>
<p>I made a bulk edit screen and built myself a proxy for accessing the
full-size image I have mirrored onto S3 via a signing proxy redirector
thing to my house. This is a service I wrote in (I think) python. I
originally tried to write it in node.js, but I found a bug in the S3
API I was trying to use and just wanted it to work.</p>
<h2 id="readme">readme</h2>
<p>I wrote <a href="https://github.com/dustin/readme">this web app</a> for our support guys to track
discussions of interest from around the web. Instead of them having
to have duplicate RSS reader config, lots of mail setups and stuff,
they just have one screen they can look at and share and track state
of all the things they need to respond to.</p>
<p>(and sorry, the app doesn’t have a README, if anyone’s interested, I
can start documenting it)</p>
<h2 id="reddit-pics">reddit pics</h2>
<p>Another tiny app I wrote just for me, my <a href="https://github.com/dustin/rpics">reddit pics app</a>
presents me with a stream of images from reddit and lets me mark the
ones I want to keep, while deleting the rest.</p>
<p>(well, deleting in this case means removing from my local DB which is
a replicate of an upstream master DB that keeps All The Things)</p>
<h2 id="gomemcached-update">gomemcached update</h2>
<p>Someone filed a bug report against my <a href="https://github.com/dustin/gomemcached">go memcached
server</a> that involved little more than an update for a
newer compiler, but that was fun to play with again.</p>
<h2 id="gotap">gotap</h2>
<p>I don’t even remember why, but I wrote an implementation of the
memcached tap protocol <a href="https://github.com/dustin/gotap">in go</a>. This is the protocol upon
which we build replication, failover, ETL, etc… in membase.</p>
<h2 id="html5-page-idle-stuff">html5 page idle stuff</h2>
<p>HTML5 added a feature that allows you to detect when the browser
renders your page hidden (most commonly by switching tabs). This is
really awesome for things like <a href="http://dustinphoto.iriscouch.com/gerrit/_design/app/index.html">my code review</a> and
<a href="https://github.com/dustin/rpics">reddit</a> apps since I can disable the realtime data stream to
pages that aren’t even being looked at (and enable it when someone
looks again!)</p>
<h2 id="web-de-auth-proxy">web de-auth proxy</h2>
<p>I did some private use map visualizations use google maps (which
<a href="http://vmx.cx/">Volker Mische</a> ported to the jquery openmap API).
Unfortunately, they also used private databases and running a web
screen saver or full-time web browser that requires auth can be a
minor annoyance.</p>
<p>That sounded like a great job for <a href="http://nodejs.org/">node.js</a>, so I wrote a
quick <a href="https://gist.github.com/bf62443ce52ae3e8604f">deauthing proxy in node.js</a> using the built-in http
server, client, and a pipe. It was a bit more complicated than I
initially assumed it would be, but that was basically it.</p>
<p>Unfortunately, it was very unreliable. Not sure if my bug, or
node.js, but after a few minutes, it’d stop answering any HTTP
requests, or even acknowledging that it received one. That kind of
sucked.</p>
<p>So I wrote <a href="https://gist.github.com/c510c603dabfdc13ce53">a new one in go</a>. It’s the same number of lines
(once I added commandline parsing and stuff), but has worked
flawlessly since I deployed it even with a bunch of concurrent clients
grabbing lots of data and long-polling and what-not.</p>
<h2 id="fprof-visualization">fprof visualization</h2>
<div>
<img src="/images/fprof-dot-small.png" alt="fprof visualization" title="There's more where this came from." class="floatright" />
</div>
<p>Part of our ongoing effort of performancing everything involves
understanding everything. I did a little bit of stuff with a <a href="https://gist.github.com/1091684">tiny
module</a> or <a href="https://gist.github.com/1096713">two</a> showing me the current state
of things, but that really didn’t help understanding things moving
forward.</p>
<p>My first pass at this was to understand the data produced by <a href="http://www.erlang.org/doc/man/fprof.html">Erlang’s
fprof</a> and organize it into a giant document we can pan and
trace and see where things are unnecessarily hot and why. I wrote <a href="https://gist.github.com/0cd51b5a97b0569bc250">my
fprof dot</a> thing to do that work. It’s not quite
productionalized, but it’s worked well for me.</p>
<h2 id="web-pipe">web pipe</h2>
<p>Turns out, graphviz from homebrew is broken on Lion (one of the
relatively few issues I’ve run into), and I wanted to be able to look
at stuff I was building, so I made a little <a href="https://gist.github.com/1205139">web pipe</a> tool
that let me have processes that run on remote machines take data from
stdin and pass it back to stdout on my local machine using curl and
node.js. That worked pretty well.</p>
<h2 id="erlang-dtrace">erlang dtrace</h2>
<p>This definitely deserves a post by itself, but in the meantime, I’ve
been working to add proper dtrace support to erlang (once and for
all!)</p>
<p>I mentioned my first attempt with fprof above. There was a second
attempt at digging out enough information where I used the low-level
tracing facilities of erlang itself to really understand where
everything was happening. It will really tell you a lot, but you have
to write a lot of tools and really just end up doing it yourself.</p>
<p>So then came <a href="http://en.wikipedia.org/wiki/DTrace">dtrace</a>.</p>
<p>Anyone who knows dtrace will understand this, and anyone who knows me
will have heard it all (all!) before, but dtrace answers all the
questions. Life, the universe, everything.</p>
<p>A co-worker asked me how long it takes for a message sent from one
erlang process to be picked up in another process. Without as much as
picking up an editor, I blasted out <a href="https://gist.github.com/6f73df27f67a7d123e63">this commandline</a> and
was able to say it’s around 8µs ± a few nanoseconds for clock skew
across cores and processes that are scheduled independently on other
threads regardless of my message having been sent.</p>
<p>My <a href="https://github.com/dustin/otp/wiki/DTrace">DTrace wiki page</a> is a starting point for getting the
stuff going and knowing where all the probes are (basically
function/bif/nif entry/return, process scheduler events, gc events,
allocation events, message sending, hibernate, probably more).</p>
<p>It makes it really easy to test things that are just invisible to
erlang itself – like knowing which erlang functions cause heap growth
that directly causes the OS itself to actually <code class="language-plaintext highlighter-rouge">mmap</code>/<code class="language-plaintext highlighter-rouge">sbrk</code> in more
memory since it all correlates.</p>
<p>You, too, can know it all.</p>
<h2 id="lua-again">lua (again)</h2>
<p>Way back in October of last year I started experimenting with
server-side scripting in membase. I spent a bit of time a couple of
weeks ago and actually got a pretty rich set of APIs and stuff built
and was able to demo complex server-side manipulation and batch
processing with a dynamically extensible client.</p>
<p>There were lots of fun challenges inside of this (e.g. multiple
threads operating on lua concurrently while being able to define
global functions available to all future sessions while blocking
accidental global variable definition). I’m hoping to get it into
some products as a way to enhance testing and more quickly extend
client functionality (though ideally users won’t even know about it).</p>
<p>Glue code can be a lot of fun. I’ve really got a lot more to say
about this one, too.</p>
<h2 id="couchconf">couchconf</h2>
<p>I did a CouchConf talk on jQuery in San Francisco. There are a <a href="http://www.couchbase.com/news-and-events/events">lot
more coming</a>, so I might be doing more of this on
different topics very soon.</p>
<p>For this one, I did a bunch of prep work, thought about what I was
going to do, etc… Then I threw it all away and finalized my
presentation with <a href="http://prezi.com/eix3nsat9kt8/couchbase-jquery-and-you/">prezi</a>.</p>
<p>I wanted to demo some of the stuff I was talking about, so I thought
of something new and literally <a href="https://github.com/dustin/couch-bitcoin">wrote the app</a> on the ride
up to the show (and kept enhancing it until my talk).</p>
<p>The app itself isn’t that exciting – it shows realtime trades of
bitcoin across all exchanges (data populated by <a href="https://gist.github.com/1093582">a little go
app</a> that I threw together to replace a python one I didn’t
like).</p>
<h2 id="git-tree-hash-based-positive-test-result-memoization">git tree hash based positive test result memoization</h2>
<p>I can’t remember what I was doing, but it involved running a bunch of
tests with <a href="http://dustin.github.com/2010/03/28/git-test-sequence.html">git test sequence</a> and they weren’t fast enough.
Someone had given me some memoization code a while back, but it was
commit based and used refs and I think it lacked a bit of perfection.</p>
<p>To speed up my testing on an evolving tree, I <a href="https://github.com/dustin/bindir/compare/8026fd8435...b036ad02c5">updated
it</a> to record successful test results by the test issued
and the tree hash into the object store directly. This means there
are dangling objects that will eventually be cleaned up, but this is
going to happen around the time that you don’t care about those tests
results anyway, so it’s perfect.</p>
<p>Everyone: abuse the git object store to make your lives better.</p>
<h2 id="leveldb---first-impressions">leveldb - first impressions</h2>
<p>Somewhere along the way, I wrote a backend for membase that stores
data in <a href="http://code.google.com/p/leveldb/">leveldb</a>. It’s a lot of fun to work with, and as
you can see in <a href="http://prezi.com/yg1igorplxii/leveldb-first-impressions/">my results</a>, it’s up to 10x faster than
our highly tuned SQLite on inserts on SSD (less on EC2, but that’s
OK). It’s certainly more consistent.</p>
<p>It’s not <em>perfect</em>, though. It’s not really any faster on fetches or
commits (sometimes a lot slower) and I’ve got <a href="http://code.google.com/p/leveldb/issues/detail?id=34">one crashing
bug in leveldb</a>, that I need to get out of the way before
I can even show it to anyone.</p>
<h2 id="the-programming-challenge">the programming challenge</h2>
<p>I got to review some code from a candidate a few times for a fairly
simple “let me see your handwriting” kind of coding question we pass
around.</p>
<p>The basic problem involves reading a file with a bunch (~1M) usernames
into a structure in memory and providing a function to tell you
whether a given user is in it.</p>
<p>Candidates invariably want to build something like a trie, but often
fall into this trap where they believe pointers don’t take up space
(even arrays of pointers, apparently). I wanted to give it a go, so I
did a plain C implementation that ended up being fewer lines of code
than what we’d got from our candidates, used about 10% of the memory
and ran many times faster.</p>
<p>I sucked some other people’s time into this as well. If you’d like to
offer a suggestion, <a href="https://gist.github.com/1189242">here’s a tool</a> that generates 100M
usernames for you to try out. (finally got to use a bloom filter,
yay! (and no, the answer doesn’t involve a bloom filter unless you
want to do a times/space tradeoff thing)</p>
<p>If you want to try it, note that my program is < 100 lines of plain C
(no external libraries) and uses 1.26GB of RAM on my macbook for
100,000,000 usernames. It loads and indexes them in 84 seconds and
spends another 31 seconds verifying it can find each of the
100,000,000 users within that list (+ 4 more that are known to not be
there). Please do better than that.</p>
<h2 id="program-like-a-pirate">program like a pirate</h2>
<p>I’ve written a couple thousand lines of <a href="http://www.r-project.org/">R</a> in the last month or
so.</p>
<p>Here are some examples taken completely out of context. I also owe a
blog post on how incredibly easy it is to just throw all your data
into CouchDB and then look at it with R.</p>
<p><img src="/images/r/events.png" alt="events" /> <img src="/images/r/ram.png" alt="ram" />
<img src="/images/r/rand.png" alt="rand" /> <img src="/images/r/time.png" alt="time" />
<img src="/images/r/world.png" alt="world" /></p>
<h2 id="python-heatmap">python heatmap</h2>
<p>Some of my visualization work wasn’t just couchdb -> R to pdf or png
or something, but I wanted to do some slightly different stuff.</p>
<p>As part of my upcoming <a href="http://py.codeconf.com/">pycodeconf</a> talk on
<a href="/2010/10/27/breakdancer.html">breakdancer</a>, I wanted to come up with some
visualization on what it looked like for a cluster of tests to fail
out of a 130k test suite in a way that might be consumable by a
human. For this, I grabbed a <a href="https://github.com/chucknthem/heatmap">python heatmap</a>
implementation and hacked on it a bit to do what I wanted.</p>
<p>Well, that didn’t work for me all that well, but it did get me
interested in doing an interactive heatmap animation of geographical
density data changing over time, so I ended up doing that instead and
updating the library a little bit to do it. That was more
successful.</p>
<h2 id="couchdb-wikipedia">couchdb wikipedia</h2>
<p>Over the weekend, I loaded all of wikipedia (2011-09-01) into a
couchdb and added a geo index over all of the features I could find
that reference a “place.” It’s pretty fun to look around and find
articles by bounding box. It’s also pretty decently fast to see the
r-tree go with a bit over 300,000 points across the world.</p>
<p>Hopefully, I’ll end up making an offline wikipedia at some point, but
I’ve got a lot of projects lined up ahead of this.</p>
<h1 id="but-mostly">…but mostly</h1>
<p>It’s been customer and product work. I’ve got a variety of really
interesting problems at customer sites (e.g. a 100 microsecond SLA for
one, high volume realtime data analysis across many dimensions for
another).</p>
<p>We’re continuing to define and build <a href="http://www.unqlspec.org/">UnQL</a> for everybody and
attending and hosting lots of talks all over the place. I hope to see
you at some of them.</p>
<p>(and if you program in C, C++, objective C, java, go, erlang,
javascript, R, ruby, and/or python and want to help use and build some
awesome technologies, I could use some help)</p>
New Operations in Membase2011-06-07T00:00:00+00:00http://dustin.github.com/2011/06/07/new-ops<h1 id="new-operations-in-membase">New Operations in Membase</h1>
<p>We built a couple of new protocol operations for people building
applications. The general goal of adding an operation is to keep it
orthogonal to other commands while enhancing the functionality in a
way that lets you do things that couldn’t be done before, or at least
were common and difficult to do efficiently.</p>
<p>Here is a description of the new commands and an idea of how they
might be used.</p>
<div>
<img class="floatright" src="/images/synchronize.png" alt="synchronize!" />
</div>
<h2 id="sync">Sync</h2>
<p>The first new concept we introduced is a <code class="language-plaintext highlighter-rouge">sync</code> command for providing
a barrier where you wait for an application’s data to change state in
specific ways such as having an item change from a known value or
achieve a specified level of durability.</p>
<p>Quick background on how this works in membase (for which we
implemented <code class="language-plaintext highlighter-rouge">sync</code> to begin with): Membase’s engine has what is
effectively an air-gap between the network interface and the disk.
Operations are almost all processed from and to RAM and then
asynchronously replicated and persisted. Incoming items are available
for request immediately upon return from your mutation command
(i.e. the next request for a given key will return the item that was
just set), but replication and persistence will be happening soon.</p>
<p>The membase <code class="language-plaintext highlighter-rouge">sync</code> command is somewhat analagous to <a href="http://linux.die.net/man/2/fsync">fsync</a> or
perhaps <a href="http://linux.die.net/man/2/msync">msync</a> in that you can first freely lob items at
membase and verify that it’s accepted them at the lowest level of
availability. When you have stored a set of critical items, you can
then issue a <code class="language-plaintext highlighter-rouge">sync</code> command with the set of your critical keys and
required durability level and the server will block until this level
is achieved (or something happens that prevents us from doing so).</p>
<p>There were discussions about different semantics (such as a
fully-sync’d mode or a specific <code class="language-plaintext highlighter-rouge">set+sync</code> type command). While a
single <code class="language-plaintext highlighter-rouge">set+sync</code> command would be one fewer round trip than doing a
separate <code class="language-plaintext highlighter-rouge">set</code> and <code class="language-plaintext highlighter-rouge">sync</code>, it makes little difference in practice
since the typical effect of a <code class="language-plaintext highlighter-rouge">sync</code> command is a delay. This,
however, comes at the cost of making it very difficult to do any sort
of practical batching or pipelining. One can sync after every
command, after a large batch, or on select items from within a large
batch.</p>
<h3 id="what-can-you-sync-on">What can you Sync On?</h3>
<p>The specification permits a given set of keys to be monitored for one
of the following state changes:</p>
<ol>
<li>Wait for Replication</li>
<li>Wait for Persistence</li>
<li>Wait for Replication <em>and</em> Persistence</li>
<li>Wait for Replication <em>or</em> Persistence</li>
<li>Wait for Mutation</li>
</ol>
<p>There’s also space for a lightly discussed “any vs. all” flag for the
keys where you can hand the server a set of keys and be informed as
soon as any one of them changes to the desired state instead of
waiting for all of them.</p>
<h3 id="example">Example</h3>
<p>Given a giant sack of items, with a mix of important items (want
stored) and really important items (must guarantee are stored before
returning), let’s do the right thing.</p>
<figure class="highlight"><pre><code class="language-python" data-lang="python"><span class="k">def</span> <span class="nf">store_stuff</span><span class="p">(</span><span class="n">items</span><span class="p">):</span>
<span class="s">"""Store a collection of items.
Items will be stored asynchronously, then important items
will be synchronized on before returning."""</span>
<span class="n">important</span> <span class="o">=</span> <span class="p">[]</span>
<span class="k">for</span> <span class="n">i</span> <span class="ow">in</span> <span class="n">items</span><span class="p">:</span>
<span class="n">mc</span><span class="p">.</span><span class="nb">set</span><span class="p">(</span><span class="n">i</span><span class="p">.</span><span class="n">key</span><span class="p">,</span> <span class="n">i</span><span class="p">.</span><span class="n">exp</span><span class="p">,</span> <span class="n">i</span><span class="p">.</span><span class="n">flags</span><span class="p">,</span> <span class="n">i</span><span class="p">.</span><span class="n">value</span><span class="p">)</span>
<span class="k">if</span> <span class="n">i</span><span class="p">.</span><span class="n">important</span><span class="p">:</span>
<span class="n">important</span><span class="p">.</span><span class="n">append</span><span class="p">(</span><span class="n">i</span><span class="p">)</span>
<span class="n">mc</span><span class="p">.</span><span class="n">sync_replication_or_persistence</span><span class="p">(</span><span class="n">important</span><span class="p">)</span></code></pre></figure>
<p>(note that a python client supports these features, but not <em>exactly</em>
with this API, but this should give you the basic idea)</p>
<p>Similarly, one can rate limit inserts such that items don’t go in
faster than they can be written to disk.</p>
<figure class="highlight"><pre><code class="language-python" data-lang="python"><span class="k">def</span> <span class="nf">store_stuff_slowly</span><span class="p">(</span><span class="n">items</span><span class="p">,</span> <span class="n">sync_every</span><span class="o">=</span><span class="mi">1000</span><span class="p">):</span>
<span class="s">"""Store a collection of items without building a large
replication backlog."""</span>
<span class="k">for</span> <span class="n">n</span><span class="p">,</span> <span class="n">i</span> <span class="ow">in</span> <span class="nb">enumerate</span><span class="p">(</span><span class="n">items</span><span class="p">,</span> <span class="mi">1</span><span class="p">):</span>
<span class="n">mc</span><span class="p">.</span><span class="nb">set</span><span class="p">(</span><span class="n">i</span><span class="p">.</span><span class="n">key</span><span class="p">,</span> <span class="n">i</span><span class="p">.</span><span class="n">exp</span><span class="p">,</span> <span class="n">i</span><span class="p">.</span><span class="n">flags</span><span class="p">,</span> <span class="n">i</span><span class="p">.</span><span class="n">value</span><span class="p">)</span>
<span class="k">if</span> <span class="p">(</span><span class="n">n</span> <span class="o">%</span> <span class="n">sync_every</span><span class="p">)</span> <span class="o">==</span> <span class="mi">0</span><span class="p">:</span>
<span class="n">mc</span><span class="p">.</span><span class="n">sync_replication</span><span class="p">(</span><span class="n">i</span><span class="p">)</span></code></pre></figure>
<p>Every <code class="language-plaintext highlighter-rouge">sync_every</code> item (default 1000) waits for synchronization to
catch up. Setting <code class="language-plaintext highlighter-rouge">sync_every</code> to one would cause us fully
synchronize every item.</p>
<h2 id="touch">Touch</h2>
<p>We have heard from quite a few projects owners that they’d like the
ability to have items with a sliding window of expiration. For
example, instead of having an item expire after five minutes of
mutating (which is how you specify an object’s time-to-live today),
we’d like it to expire after five minutes of inactivity.</p>
<p>If you’re familiar with LRU caches (such as memcached), you should
note that this is semantically quite different from LRU. With an LRU,
we effectively don’t care about old data. The use cases for <code class="language-plaintext highlighter-rouge">touch</code>
require us to actively disable access to inactive data on a
user-defined schedule.</p>
<p>The <code class="language-plaintext highlighter-rouge">touch</code> command can be used to adjust expiration on an existing
key without touching the value. It uses the same type of expiration
definition all mutation commands use, but doesn’t actually touch the
data.</p>
<p>Similar to <code class="language-plaintext highlighter-rouge">touch</code> we added a <code class="language-plaintext highlighter-rouge">gat</code> (<code class="language-plaintext highlighter-rouge">get-and-touch</code>) command that
returns the data and adjusts the expiration at the same time. For
most use cases, <code class="language-plaintext highlighter-rouge">gat</code> is probably more appropriate than <code class="language-plaintext highlighter-rouge">touch</code>, but
it really depends on how you build your application.</p>
<h3 id="example-usage">Example Usage</h3>
<p>Usage of <code class="language-plaintext highlighter-rouge">touch</code> and <code class="language-plaintext highlighter-rouge">gat</code> are pretty straightfoward. A really common
pattern might be storing session data where we want “idle” data to be
removed quickly, but active data to stick around as long as it’s
active.</p>
<figure class="highlight"><pre><code class="language-python" data-lang="python"><span class="k">def</span> <span class="nf">get_session</span><span class="p">(</span><span class="n">session_id</span><span class="p">,</span> <span class="n">max_session_age</span><span class="o">=</span><span class="mi">300</span><span class="p">):</span>
<span class="s">"""Get a valid session object for the given session ID.
Sessions will only live for five minutes.
Unauthenticated will be thrown if the session
can not be loaded."""</span>
<span class="n">s</span> <span class="o">=</span> <span class="n">mc</span><span class="p">.</span><span class="n">gat</span><span class="p">(</span><span class="n">session_id</span><span class="p">,</span> <span class="n">max_session_age</span><span class="p">)</span>
<span class="k">if</span> <span class="ow">not</span> <span class="n">s</span><span class="p">:</span>
<span class="n">throw</span> <span class="n">Unauthenticated</span><span class="p">()</span>
<span class="k">return</span> <span class="n">s</span></code></pre></figure>
<p>This example showed a simple session loader that keeps the session
alive and signals mission sessions to another part of the application
stack that can deal with logins and stuff.</p>
<h1 id="availability">Availability</h1>
<p>We’ve been using this stuff, but we haven’t yet achieved universal
availability.</p>
<h2 id="servers">Servers</h2>
<p><a href="http://www.couchbase.org/products/membase/1-7-beta">Membase 1.7</a> provides this full <code class="language-plaintext highlighter-rouge">touch</code> and <code class="language-plaintext highlighter-rouge">gat</code>
functionality and partial <code class="language-plaintext highlighter-rouge">sync</code> functionality.</p>
<p>For <code class="language-plaintext highlighter-rouge">sync</code>, only waiting for replication is supported, and only a
single replica. The protocol allows for the tracking of up to 16
replicas, but membase as a cluster uses transitive replication so it’s
not possible to track when the second replica is complete from the
primary host (much less the sixteenth!).</p>
<p>Similarly, we’ve written most of the code for syncing on persistence,
but before our 2.0 storage strategies, we thing it could be more
harmful than useful in most applications. Even with our 2.0
strategies, it’s likely that it’s not as appropriate as replication
tracking for all but the absolutely most important data.</p>
<p>Memcached 1.6(ish) has support for <code class="language-plaintext highlighter-rouge">touch</code> and <code class="language-plaintext highlighter-rouge">gat</code> in the default
engine (which also ships in Membase).</p>
<h2 id="clients">Clients</h2>
<p>In addition to <code class="language-plaintext highlighter-rouge">mc_bin_client.py</code> (which is a sort of
reference/playground client that ships with membase and we write many
of the tools with), we’ve got support in two clients yet, but we’re
considering the feature “evolving” as we’re trying to find the best
way to do it. Feedback is far more than welcome!</p>
<h3 id="java">Java</h3>
<p>spymemcached 2.7 has support for <code class="language-plaintext highlighter-rouge">touch</code>, <code class="language-plaintext highlighter-rouge">gat</code>, and <code class="language-plaintext highlighter-rouge">sync</code>.</p>
<h3 id="c-sharp">C Sharp</h3>
<p>The enyim C# client for memcached has support for <code class="language-plaintext highlighter-rouge">touch</code>, <code class="language-plaintext highlighter-rouge">gat</code>, and
<code class="language-plaintext highlighter-rouge">sync</code> in a release that should hit the shelves quite soon.</p>
Couchbase OSX2011-04-04T00:00:00+00:00http://dustin.github.com/2011/04/04/mac-couchbase<div>
<img src="/images/membase-server-osx.png" alt="membase/osx" title="The Status Menu" class="floatright" />
</div>
<p>About a year ago, <a href="http://trondn.blogspot.com/">Trond</a> asked me to build him a GUI tool for
running membase on his Mac. I finally got around to it and we liked
it enough that we’re making it <a href="http://www.couchbase.com/press-releases/Membase-for-Mac-OS-X">available</a> to everyone.</p>
<p>I’ve been on a cocoa development kick lately after doing some work
<a href="http://twitter.com/janl">Jan’s</a> CouchDBX for our <a href="http://www.couchbase.com/products-and-services/couchbase-server">Couchbase Server 1.1</a>
release. It was really quite awesome and easy to get people going. I
recommended it to all of my friends who were interested in CouchDB,
but it was not something I ever ran myself.</p>
<p>After releasing 1.1, I started thinking about what would really make
it better and put a bunch of time into something I’d run on my own
(both for development and my home production instance). The biggest
features I wanted for myself were the following:</p>
<ul>
<li>Minimal UI in the app (much prefer the status bar only)</li>
<li>Autorestart on failure (I like to kill my daemons randomly)</li>
<li>Easy start at login</li>
</ul>
<p>The minimal UI includes everything you need and nothing you don’t.</p>
<p>Firstly, it no longer brings its own web browser. You have one you
like, that’s the one I want you to use.</p>
<p>Also, I run on a headless server with minimal resources. I kill my
CouchDB randomly when it gets big, or whenever else I feel like doing
it. It’s a crash-only design, so why not let that happen and just
restart? If you have a persistent failure (i.e. it can’t run for at
least ten seconds), the server will pop up an alert box letting you
know, stop automatically restarting and wait for you to tell it to
retry or just give up.</p>
<p>Similarly, whenever my machine finishes booting, I want it running my
server. Instead of hand-crafting a launchd config like I normally do,
I just check a box. Done!</p>
<p>“But wait,” you say, “what does this have to do with membase?” After
estabilishing the process monitoring framework, the desired
interaction, and the build system (really the worst part), it was
pretty obvious how to get membase running a similar way. The build
process isn’t totally straightforward (lots of weird library stuff
requiring me to learn all about <a href="http://developer.apple.com/library/mac/#documentation/Darwin/Reference/ManPages/man1/install_name_tool.1.html">install_name_tool</a> and magic
incantations of automatically discovery of development time
dependencies and packaging them up) and when you’re done, you either
have to write a launchd plist or just sit in a terminal with the
thing running and manage its logs and all that kind of stuff. That’s
tedious.</p>
<p>In the end, there are two free packages ready for you that should work
exactly as you’d expect software to work on your Mac.</p>
<p>Get <a href="http://www.couchbase.com/products-and-services/couchbase-server">Couchbase Server</a> and <a href="http://www.couchbase.com/products-and-services/membase-server">Membase Server</a> and
instantly make all of your friends envious of how easily you can set
up scalable databases on your Mac.</p>
Using Dropbox as a Work Queue2011-02-27T00:00:00+00:00http://dustin.github.com/2011/02/27/dropbox-queue<div>
<a href="http://dustinphoto.couchone.com/photo-public/_design/app/index.html">
<img src="http://dustinphoto.couchone.com/photo-public/ebcf8d8b7e6c1a9680e2b108d2b0e1de/thumb.jpg" alt="photo" title="My dorkboard" class="floatright" />
</a>
</div>
<p>I’ve been writing myself a web-based photo album for over a decade
now. It’s gone through many different technologies over the years as
I used it to learn new ways of doing stuff.</p>
<p>I’m finally shedding the years of java from it to do something that’s
not only more lightweight, but also distributed. The purpose of this
post isn’t to describe the photo album, but a little bit of background
will help to understand the point of what’s going on.</p>
<p>Most recently, my photo album is implemented as a <a href="http://couchapp.org/">couchapp</a>
which has been huge fun.</p>
<h2 id="my-photo-album">My Photo Album</h2>
<div>
<img src="https://github.com/dustin/photo-couch/wiki/photo-ng.png" alt="architecture" title="how the photo album works" class="floatleft" />
</div>
<p>The <a href="http://github.com/dustin/photo-couch">current version</a> has a basic <a href="http://couchdb.apache.org/">CouchDB</a> backend
that lets me set up replication as shown in this image to the left
here. Photos can be added or edited on any server (with the exception
of the <a href="http://dustinphoto.couchone.com/photo-public/_design/app/index.html">public instance</a>, which is a replication target only)
and these adds and edits will eventually make their way to all of the
other servers.</p>
<p>Previously, pictures were added to the photo album by the means of a
<a href="https://github.com/dustin/photoupload">standalone OS X app</a> that batches them into my local DB
which I then replicate out to others.</p>
<p>However, I haven’t written such a thing for Android, so getting photos
I take on my phone while I’m out stored safely has been a bit more
difficult. This is where Dropbox comes in.</p>
<h2 id="dropbox-as-a-networked-spool">Dropbox As a Networked Spool</h2>
<div>
<img src="/images/photo-dropbox-rep.png" alt="submitting through dropbox" class="floatright" title="From my phone to you." />
</div>
<p>The Android app for Dropbox is really well done. Apps that can share
files get a share button that will place the content directly into
Dropbox folders.</p>
<p>Dropbox itself delivers the files into their locations
<a href="http://forums.dropbox.com/topic.php?id=21246">atomically</a> – that is, if a file doesn’t sync up to
dropbox in its entirety, it won’t be delivered, and a file will not
show up on any client machine until it’s properly fully down and can
be moved in atomically (via <code class="language-plaintext highlighter-rouge">rename(2)</code>).</p>
<p>This seems like something worth taking advantage of, and the most
obvious way for me was to create a <a href="http://en.wikipedia.org/wiki/Spooling">spool</a> within Dropbox.</p>
<h3 id="the-queue-processor">The Queue Processor</h3>
<p>I wrote a simple uploader that grabs as much info as it can from the
photos that appear via dropbox, uses <a href="http://www.pythonware.com/products/pil/">PIL</a> to do some scaling and
stuff and then sticks it in the local DB.</p>
<p>The recipe for safely and reliably processing items from a
<a href="http://en.wikipedia.org/wiki/Spooling">spool</a> is relatively simple and well-known. It is the
way print and mail processing systems have worked for decades. The
wikipedia overview does a better job of explaining the concept, but
I’ll go over the code I wrote for mine since I find it easier to
understand concepts with code than with English.</p>
<p>The actual queue processor based on the uploader is easy to understand
(almost the entire thing is shown below), but does have to take care
of failure modes, new things coming in while it’s processing, etc…
For this, I end up with three directories:</p>
<ol>
<li>The incoming directory</li>
<li>The work directory</li>
<li>The complete directory</li>
</ol>
<p>The processor atomically moves an item from the incoming directory
(inside Dropbox) to the work directory. It works on the item from
that location, then atomically moves it to the done directory once
it’s done.</p>
<p>Here’s what my spool processor looks like in python:</p>
<figure class="highlight"><pre><code class="language-python" data-lang="python"><span class="k">def</span> <span class="nf">process</span><span class="p">(</span><span class="n">basename</span><span class="p">,</span> <span class="n">processing</span><span class="p">,</span> <span class="n">done</span><span class="p">):</span>
<span class="n">workfile</span> <span class="o">=</span> <span class="n">os</span><span class="p">.</span><span class="n">path</span><span class="p">.</span><span class="n">join</span><span class="p">(</span><span class="n">processing</span><span class="p">,</span> <span class="n">basename</span><span class="p">)</span>
<span class="n">donefile</span> <span class="o">=</span> <span class="n">os</span><span class="p">.</span><span class="n">path</span><span class="p">.</span><span class="n">join</span><span class="p">(</span><span class="n">done</span><span class="p">,</span> <span class="n">basename</span><span class="p">)</span>
<span class="c1"># Move from the incoming directory to the work directory
</span> <span class="n">os</span><span class="p">.</span><span class="n">rename</span><span class="p">(</span><span class="n">basename</span><span class="p">,</span> <span class="n">workfile</span><span class="p">)</span>
<span class="c1"># Actual work happens here.
</span> <span class="n">doSomethingImportantWith</span><span class="p">(</span><span class="n">workfile</span><span class="p">)</span>
<span class="c1"># Move from the work directory to the complete directory
</span> <span class="n">os</span><span class="p">.</span><span class="n">rename</span><span class="p">(</span><span class="n">workfile</span><span class="p">,</span> <span class="n">donefile</span><span class="p">)</span>
<span class="k">if</span> <span class="n">__name__</span> <span class="o">==</span> <span class="s">'__main__'</span><span class="p">:</span>
<span class="n">incoming</span><span class="p">,</span> <span class="n">processing</span><span class="p">,</span> <span class="n">done</span> <span class="o">=</span> <span class="n">sys</span><span class="p">.</span><span class="n">argv</span><span class="p">[</span><span class="mi">1</span><span class="p">:]</span>
<span class="n">os</span><span class="p">.</span><span class="n">chdir</span><span class="p">(</span><span class="n">incoming</span><span class="p">)</span>
<span class="k">for</span> <span class="n">basename</span> <span class="ow">in</span> <span class="n">os</span><span class="p">.</span><span class="n">listdir</span><span class="p">(</span><span class="s">'.'</span><span class="p">):</span>
<span class="k">try</span><span class="p">:</span>
<span class="n">process</span><span class="p">(</span><span class="n">basename</span><span class="p">,</span> <span class="n">processing</span><span class="p">,</span> <span class="n">done</span><span class="p">)</span>
<span class="k">except</span><span class="p">:</span>
<span class="n">traceback</span><span class="p">.</span><span class="n">print_exc</span><span class="p">()</span></code></pre></figure>
<p>I’m catching all exceptions at the toplevel loop and just logging
them. If anything goes wrong at any stage, the file will stay
wherever it was when it broke (usually the work directory).</p>
<p>Note that it <em>could</em> delete it when it’s done instead of just moving
it to a <code class="language-plaintext highlighter-rouge">done</code> directory, but I don’t want to automatically delete
stuff.</p>
<h3 id="invoking-the-processor">Invoking the Processor</h3>
<p>My script runs on Mac OS X, so I wrote a quick <a href="http://en.wikipedia.org/wiki/Launchd">launchd</a>
plist to read a sync dir and operate like a mail spool. Basically, if
nothing’s coming in, nothing happens.</p>
<p>My monitor plist looks something like the following:</p>
<figure class="highlight"><pre><code class="language-xml" data-lang="xml"><span class="cp"><?xml version="1.0" encoding="UTF-8"?></span>
<span class="cp"><!DOCTYPE plist PUBLIC -//Apple Computer//DTD PLIST 1.0//EN
http://www.apple.com/DTDs/PropertyList-1.0.dtd ></span>
<span class="nt"><plist</span> <span class="na">version=</span><span class="s">"1.0"</span><span class="nt">></span>
<span class="nt"><dict></span>
<span class="nt"><key></span>Label<span class="nt"></key></span>
<span class="nt"><string></span>net.spy.photoupload<span class="nt"></string></span>
<span class="nt"><key></span>ProgramArguments<span class="nt"></key></span>
<span class="nt"><array></span>
<span class="nt"><string></span>/path/to/script<span class="nt"></string></span>
<span class="nt"><string></span>/Users/me/Dropbox/incoming-photos<span class="nt"></string></span>
<span class="nt"><string></span>/Users/me/spool/work<span class="nt"></string></span>
<span class="nt"><string></span>/Users/me/spool/done<span class="nt"></string></span>
<span class="nt"></array></span>
<span class="nt"><key></span>WorkingDirectory<span class="nt"></key></span>
<span class="nt"><string></span>/<span class="nt"></string></span>
<span class="nt"><key></span>QueueDirectories<span class="nt"></key></span>
<span class="nt"><array></span>
<span class="nt"><string></span>/Users/me/Dropbox/incoming-photos<span class="nt"></string></span>
<span class="nt"></array></span>
<span class="nt"></dict></span>
<span class="nt"></plist></span></code></pre></figure>
<p>Place that in <code class="language-plaintext highlighter-rouge">~/Library/LaunchAgents</code> and have load it with
<code class="language-plaintext highlighter-rouge">launchctl load ~/Library/LaunchAgents/net.spy.photoupload.plist</code> and
we’re up and monitoring (note that it’ll automatically load on boot).</p>
<p>If you wanted to run something similar on a system other than Mac OS
X, you could easily write a queue manager that monitors the directory
using <a href="http://en.wikipedia.org/wiki/Kqueue">kqueue</a> or <a href="http://en.wikipedia.org/wiki/Inotify">inotify</a> or even just a cron job
poking around looking for new stuff.</p>
Maintaining a Set in Memcached2011-02-17T00:00:00+00:00http://dustin.github.com/2011/02/17/memcached-set<h1 id="maintaining-a-set-in-memcached">Maintaining a Set in Memcached</h1>
<div>
<img src="/images/simple.png" alt="simple" title="I found this in some old military archives." class="floatright" />
</div>
<p>This is something that comes up every once in a while. I usually
describe a means of doing it that I think makes sense, but I don’t
think I’ve ever described it <em>quite</em> well enough. People tend to
think it’s complicated or slow or things like that. I’m going to try
to try to solve that problem here.</p>
<h2 id="constraints">Constraints</h2>
<p>In order to be useful for enough applications, we’re going to work
under the following assumptions:</p>
<ul>
<li>must minimize round trips to the servers</li>
<li>O(1) add (for both current size and new items coming in)</li>
<li>O(1) remove (for both current size and items being removed)</li>
<li>O(1) fetch</li>
<li>lock and wait free</li>
<li>easy to use</li>
<li>easy to understand</li>
<li>no required explicit maintenance</li>
</ul>
<p>And, of course, it has to be web scale!</p>
<h2 id="ingredients">Ingredients</h2>
<p>The concept is simple and makes use of three memcached operations with
atomicity guarantees.</p>
<p>An index is created with <code class="language-plaintext highlighter-rouge">add</code>. This should be pretty obvious.</p>
<p>Whenever we need to add or remove items, we use <code class="language-plaintext highlighter-rouge">append</code>. For this to
work, we need to encode the items in such a way as to have them
represent either positive or negative items. I created a simple
sample encoding of <code class="language-plaintext highlighter-rouge">+key</code> to represent the addition of <code class="language-plaintext highlighter-rouge">key</code> to the
set and <code class="language-plaintext highlighter-rouge">-key</code> to represent the removal of <code class="language-plaintext highlighter-rouge">key</code> from the set. I then
use spaces to separate multiple items. Example: <code class="language-plaintext highlighter-rouge">+a +b +c -b</code>
represents <code class="language-plaintext highlighter-rouge">{a, c}</code>. The sequence is, of course important.</p>
<p>A set that has members coming and going frequently enough may need to
be compacted. For that, we reencode the set and use <code class="language-plaintext highlighter-rouge">cas</code> to ensure
we can add it back without stepping on another client.</p>
<h2 id="walk-me-through-it">Walk Me Through It</h2>
<p>I’m using python for this example. Ideally this gets implemented in
your client and everything’s good to go.</p>
<p>First, we need encoders and decoders. This is actually the hard
part and it’s really trivial when it comes down to it.</p>
<h3 id="the-encoder">The Encoder</h3>
<p>We start with the most basic representation of data within our sets.</p>
<figure class="highlight"><pre><code class="language-python" data-lang="python"><span class="k">def</span> <span class="nf">encodeSet</span><span class="p">(</span><span class="n">keys</span><span class="p">,</span> <span class="n">op</span><span class="o">=</span><span class="s">'+'</span><span class="p">):</span>
<span class="s">"""Encode a set of keys to modify the set.
>>> encodeSet(['a', 'b', 'c'])
'+a +b +c '
"""</span>
<span class="k">return</span> <span class="s">''</span><span class="p">.</span><span class="n">join</span><span class="p">(</span><span class="n">op</span> <span class="o">+</span> <span class="n">k</span> <span class="o">+</span> <span class="s">' '</span> <span class="k">for</span> <span class="n">k</span> <span class="ow">in</span> <span class="n">keys</span><span class="p">)</span></code></pre></figure>
<p>This is more documentation than code, but it’s pretty clear. If you
want a set of JPEGs instead, you could create a simple binary encoding
with a length and a body instead of having it be whitespace separated.</p>
<h3 id="modifying-a-set">Modifying a Set</h3>
<p>Modification is append-only with the only difference between adding
and removing being an encoding op. This is useful because we can
write the same code for both cases.</p>
<figure class="highlight"><pre><code class="language-python" data-lang="python"><span class="k">def</span> <span class="nf">modify</span><span class="p">(</span><span class="n">mc</span><span class="p">,</span> <span class="n">indexName</span><span class="p">,</span> <span class="n">op</span><span class="p">,</span> <span class="n">keys</span><span class="p">):</span>
<span class="n">encoded</span> <span class="o">=</span> <span class="n">encodeSet</span><span class="p">(</span><span class="n">keys</span><span class="p">,</span> <span class="n">op</span><span class="p">)</span>
<span class="k">try</span><span class="p">:</span>
<span class="n">mc</span><span class="p">.</span><span class="n">append</span><span class="p">(</span><span class="n">indexName</span><span class="p">,</span> <span class="n">encoded</span><span class="p">)</span>
<span class="k">except</span> <span class="nb">KeyError</span><span class="p">:</span>
<span class="c1"># If we can't append, and we're adding to the set,
</span> <span class="c1"># we are trying to create the index, so do that.
</span> <span class="k">if</span> <span class="n">op</span> <span class="o">==</span> <span class="s">'+'</span><span class="p">:</span>
<span class="n">mc</span><span class="p">.</span><span class="n">add</span><span class="p">(</span><span class="n">indexName</span><span class="p">,</span> <span class="n">encoded</span><span class="p">)</span>
<span class="k">def</span> <span class="nf">add</span><span class="p">(</span><span class="n">mc</span><span class="p">,</span> <span class="n">indexName</span><span class="p">,</span> <span class="o">*</span><span class="n">keys</span><span class="p">):</span>
<span class="s">"""Add the given keys to the given set."""</span>
<span class="n">modify</span><span class="p">(</span><span class="n">mc</span><span class="p">,</span> <span class="n">indexName</span><span class="p">,</span> <span class="s">'+'</span><span class="p">,</span> <span class="n">keys</span><span class="p">)</span>
<span class="k">def</span> <span class="nf">remove</span><span class="p">(</span><span class="n">mc</span><span class="p">,</span> <span class="n">indexName</span><span class="p">,</span> <span class="o">*</span><span class="n">keys</span><span class="p">):</span>
<span class="s">"""Remove the given keys from the given set."""</span>
<span class="n">modify</span><span class="p">(</span><span class="n">mc</span><span class="p">,</span> <span class="n">indexName</span><span class="p">,</span> <span class="s">'-'</span><span class="p">,</span> <span class="n">keys</span><span class="p">)</span></code></pre></figure>
<p>I allow a side-effect of <code class="language-plaintext highlighter-rouge">add</code> to create the index if it doesn’t exist.</p>
<p>In an actual application, there’s a non-zero chance that the <code class="language-plaintext highlighter-rouge">append</code>
would fail because the item is missing and the immediately subsequent
<code class="language-plaintext highlighter-rouge">add</code> would fail due to a race condition. I didn’t write the code to
cover that here, but it’s pretty simple. If it matters to you, just
loop the entire modify method as long as both fail. You’d have to be
trying to get it to fail more than once.</p>
<h3 id="the-decoder">The Decoder</h3>
<p>In order to use the data, we’re going to need to decode it, so let’s
put together a quick decoder that can reverse what the above encoder
does (including the appends for add and remove).</p>
<figure class="highlight"><pre><code class="language-python" data-lang="python"><span class="k">def</span> <span class="nf">decodeSet</span><span class="p">(</span><span class="n">data</span><span class="p">):</span>
<span class="s">"""Decode an item from the cache into a set impl.
Returns a dirtiness indicator (compaction hint) and the set
>>> decodeSet('+a +b +c -b -x')
(2, set(['a', 'c']))
"""</span>
<span class="n">keys</span> <span class="o">=</span> <span class="nb">set</span><span class="p">()</span>
<span class="n">dirtiness</span> <span class="o">=</span> <span class="mi">0</span>
<span class="k">for</span> <span class="n">k</span> <span class="ow">in</span> <span class="n">data</span><span class="p">.</span><span class="n">split</span><span class="p">():</span>
<span class="n">op</span><span class="p">,</span> <span class="n">key</span> <span class="o">=</span> <span class="n">k</span><span class="p">[</span><span class="mi">0</span><span class="p">],</span> <span class="n">k</span><span class="p">[</span><span class="mi">1</span><span class="p">:]</span>
<span class="k">if</span> <span class="n">op</span> <span class="o">==</span> <span class="s">'+'</span><span class="p">:</span>
<span class="n">keys</span><span class="p">.</span><span class="n">add</span><span class="p">(</span><span class="n">key</span><span class="p">)</span>
<span class="k">elif</span> <span class="n">op</span> <span class="o">==</span> <span class="s">'-'</span><span class="p">:</span>
<span class="n">keys</span><span class="p">.</span><span class="n">discard</span><span class="p">(</span><span class="n">key</span><span class="p">)</span>
<span class="n">dirtiness</span> <span class="o">+=</span> <span class="mi">1</span>
<span class="k">return</span> <span class="n">dirtiness</span><span class="p">,</span> <span class="n">keys</span></code></pre></figure>
<p>This is the most complicated part.</p>
<h3 id="retrieving-the-items">Retrieving the Items</h3>
<p>Now that we can encode, set, and modify our data, retrieval should be
quite trivial. A basic pass would look like this:</p>
<figure class="highlight"><pre><code class="language-python" data-lang="python"><span class="k">def</span> <span class="nf">items</span><span class="p">(</span><span class="n">mc</span><span class="p">,</span> <span class="n">indexName</span><span class="p">):</span>
<span class="s">"""Retrieve the current values from the set."""</span>
<span class="n">flags</span><span class="p">,</span> <span class="n">cas</span><span class="p">,</span> <span class="n">data</span> <span class="o">=</span> <span class="n">mc</span><span class="p">.</span><span class="n">get</span><span class="p">(</span><span class="n">indexName</span><span class="p">)</span>
<span class="n">dirtiness</span><span class="p">,</span> <span class="n">keys</span> <span class="o">=</span> <span class="n">decodeSet</span><span class="p">(</span><span class="n">data</span><span class="p">)</span>
<span class="k">return</span> <span class="n">keys</span></code></pre></figure>
<p>That’s pretty much it. However, this is a pretty good time to do
compaction. <code class="language-plaintext highlighter-rouge">dirtiness</code> above measures how many removal tokens are in
the set. If there are too many, we want to kill them.</p>
<p>Imagine a <code class="language-plaintext highlighter-rouge">DIRTINESS_THRESHOLD</code> number set that decides where we want
to do autocompaction. If we have more dirtiness than this, we
compact upon retrieval (making a single get into a single get and a
single <code class="language-plaintext highlighter-rouge">CAS</code>.</p>
<p>For this use case, we don’t actually care whether the <code class="language-plaintext highlighter-rouge">CAS</code> succeeds
most of the time, so we just fire and forget. It’s safe (i.e. won’t
destroy any data), but not guaranteed to work.</p>
<p>So here’s a modified <code class="language-plaintext highlighter-rouge">items</code> function conditionally compacting:</p>
<figure class="highlight"><pre><code class="language-python" data-lang="python"><span class="k">def</span> <span class="nf">items</span><span class="p">(</span><span class="n">mc</span><span class="p">,</span> <span class="n">indexName</span><span class="p">,</span> <span class="n">forceCompaction</span><span class="o">=</span><span class="bp">False</span><span class="p">):</span>
<span class="s">"""Retrieve the current values from the set.
This may trigger a compaction if you ask it to or the encoding is
too dirty."""</span>
<span class="n">flags</span><span class="p">,</span> <span class="n">casid</span><span class="p">,</span> <span class="n">data</span> <span class="o">=</span> <span class="n">mc</span><span class="p">.</span><span class="n">get</span><span class="p">(</span><span class="n">indexName</span><span class="p">)</span>
<span class="n">dirtiness</span><span class="p">,</span> <span class="n">keys</span> <span class="o">=</span> <span class="n">decodeSet</span><span class="p">(</span><span class="n">data</span><span class="p">)</span>
<span class="k">if</span> <span class="n">forceCompaction</span> <span class="ow">or</span> <span class="n">dirtiness</span> <span class="o">></span> <span class="n">DIRTINESS_THRESHOLD</span><span class="p">:</span>
<span class="n">compacted</span> <span class="o">=</span> <span class="n">encodeSet</span><span class="p">(</span><span class="n">keys</span><span class="p">)</span>
<span class="n">mc</span><span class="p">.</span><span class="n">cas</span><span class="p">(</span><span class="n">indexName</span><span class="p">,</span> <span class="n">casid</span><span class="p">,</span> <span class="n">compacted</span><span class="p">)</span>
<span class="k">return</span> <span class="n">keys</span></code></pre></figure>
<p>And we’re done.</p>
<h2 id="in-summary">In Summary</h2>
<p><strong>Worst case add to set</strong>: 2 round trips (when the set doesn’t exist and
needs to be created, but we don’t have to know that).</p>
<p><strong>Normal add to set</strong>: 1 round trip regardless of the number of
members being added. (You don’t even need to retrieve the current
value to correctly add or remove items in bulk, much less transfer it
all back).</p>
<p><strong>Worst case retrieval</strong>: 2 round trips (when compaction is a
side-effect).</p>
<p><strong>Normal retrieval</strong>: 1 round trip (just fetch the one key).</p>
<h3 id="caveats">Caveats</h3>
<p>While the number of sets is roughly unlimited, there’s a practical
size of a single set with this implementation. It’d be trivial to
“shard” the set across multiple keys (thus across multiple servers)
if one needed very large sets (more than, say 4,000 250 byte items).</p>
<p>Since compaction is done on read in this implementation, a case where
you’re modifying very heavily but reading rarely might not be a
perfect for this code. In that case, I’d start compacting on random
writes (making worst case add/remove take about three hops where it
would’ve otherwise been one).</p>
Presenting - La Brea2010-12-03T00:00:00+00:00http://dustin.github.com/2010/12/03/labrea<h1 id="representing-la-brea">(re)Presenting La Brea</h1>
<div>
<img src="/images/labrea-bubble.jpg" alt="La Brea" class="floatright" />
</div>
<p>I realize it’s a little lame to have two posts in a row on the same
topic. I’ve done a lot of work here and the first one wasn’t very
well understood, so let’s just pretend like it never happened.</p>
<p>I thought perhaps I can make up for it slightly by using a medium I’m
not particularly accustomed to – a video slideshow thingy.</p>
<p>In my previous post, I described La Brea as a great development tool
for when your computer is too fast. There I demonstrated a use case
where I injected some calls and made stuff go slowly. I hand-waved
out some “future” direction of the project and thought people would
get it.</p>
<p>I got a lot of feedback, mostly telling me I should buy more, older
computers and carry them around, or that I can’t get a meaningful test
with a fault injection framework. This is clearly a failing on my
part of communicating my vision.</p>
<p>But we’ve already established that didn’t happen, so allow me to
introduce you to <a href="https://github.com/dustin/labrea">La Brea</a>.</p>
<iframe src="http://player.vimeo.com/video/17460485?title=0&byline=0&portrait=0&color=FF7700" width="640" height="480" frameborder="0"><p>:( frame no work</p></iframe>
<p>Feedback welcome.</p>
La Brea - Because Your Computer is Too Fast2010-11-12T00:00:00+00:00http://dustin.github.com/2010/11/12/labrea<h1 id="la-brea---because-your-computer-is-too-fast">La Brea - Because Your Computer is Too Fast</h1>
<div>
<img src="/images/labrea.jpg" alt="La Brea" class="floatright" />
</div>
<p>I’ve often thought that developers have machines that are entirely too
fast. In my case, I’ve got a relatively recent dual core machine with
an SSD. It’s awesome.</p>
<p>Except I find that some people running my software are running on
machines that are doing disk IO with considerably less capable disks.</p>
<p>In <a href="http://www.membase.org/">membase</a>, disk performance differences can be noticeable.
When I’m testing with an SSD and a customer is running with a 7200rpm
disk (it happens), I can’t see the kinds of situations they run into
from my development machine.</p>
<h2 id="how-do-i-slow-down">How Do I Slow Down?</h2>
<p>There are several options out there.</p>
<p>I played with <a href="http://cpulimit.sourceforge.net/">cpulimit</a> for a bit, but it was too coarse
and really did awful things on my mac since it is basically strobing
the process with <code class="language-plaintext highlighter-rouge">SIGSTOP</code> and <code class="language-plaintext highlighter-rouge">SIGCONT</code> on an interval.</p>
<p>I experimented with a <a href="http://linux.die.net/man/2/ptrace">ptrace</a>-based solution to allow me to
more granularly slow things down, but it both didn’t help and it turns
out that <a href="http://uninformed.org/index.cgi?v=4&a=3&p=14">ptrace is kind of broken on OS X</a> anyway, so it’s
neither a portable thing to do, or really all that useful.</p>
<p>So I wrote a <a href="https://github.com/dustin/labrea">library interposer</a>.</p>
<p>It was pretty cool to see it do stuff, but I didn’t want to tell
people to recompile the whole thing every time they wanted to change a
delay or something. I decided to toss in <a href="http://lua.org/">lua</a>.</p>
<h2 id="example">Example</h2>
<p>For example, what if you wanted a seek to take a full second 1% of the
time (and for the fun of it, log that it did). You can write the
following and feed it to <code class="language-plaintext highlighter-rouge">labrea</code>:</p>
<figure class="highlight"><pre><code class="language-lua" data-lang="lua"><span class="k">function</span> <span class="nf">before_lseek</span><span class="p">(</span><span class="n">fd</span><span class="p">,</span> <span class="n">offset</span><span class="p">,</span> <span class="n">whence</span><span class="p">)</span>
<span class="k">if</span> <span class="nb">math.random</span><span class="p">(</span><span class="mi">1</span><span class="p">,</span> <span class="mi">100</span><span class="p">)</span> <span class="o">==</span> <span class="mi">13</span> <span class="k">then</span>
<span class="nb">io.write</span><span class="p">(</span><span class="nb">string.format</span><span class="p">(</span><span class="s2">"Slowing a seek on fd=%d to %d (%d)\n"</span><span class="p">,</span>
<span class="n">fd</span><span class="p">,</span> <span class="n">offset</span><span class="p">,</span> <span class="n">whence</span><span class="p">))</span>
<span class="n">usleep</span><span class="p">(</span><span class="mi">1000000</span><span class="p">)</span>
<span class="k">end</span>
<span class="k">end</span></code></pre></figure>
<p>Now I can remember what it was like to have a rotating disk in my
laptop again.</p>
<h2 id="direction">Direction</h2>
<p>Right now, my immediate need is solved, but it’s pretty easy to add
functionality, so I’m thinking about making it be a full-on fault
injection framework. I looked at <a href="http://blitiri.com.ar/p/libfiu/">fiu</a> briefly along my path and
found that it was pretty interesting, but didn’t work on MacOS and was
still a bit too invasive for where I wanted to be (which includes
doing random stuff to third-party apps).</p>
<p>In addition to the <code class="language-plaintext highlighter-rouge">before_lseek</code> as above, I would imagine an
<code class="language-plaintext highlighter-rouge">after_lseek</code> and perhaps even an <code class="language-plaintext highlighter-rouge">around_lseek</code> allowing for full
<a href="http://en.wikipedia.org/wiki/Aspect-oriented_programming">AOP</a> on your deployed C programs.</p>
<p>But for now, it just slows stuff down.</p>
<p>(<a href="https://github.com/dustin/labrea">source here</a>)</p>
How to Test Everything2010-10-27T00:00:00+00:00http://dustin.github.com/2010/10/27/breakdancer<h1 id="how-to-test-everything">How to Test Everything</h1>
<p>I recently had a <a href="http://www.membase.org/">membase</a> user point out a sequence of
operations that led to an undesirable state. I’ve got a lot of really
good engine tests I’ve written, but not <em>this</em> case:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>add with timeout -> wait for timeout -> add with timeout
</code></pre></div></div>
<p>The bug is pretty straightforward – expiry is lazy and it turns out
I’m not checking for expiry in this case. It was pretty easy to write
this test, but immediately made me think about what <em>other</em> cases
weren’t being run.</p>
<p>Now, I know there are countless tools out there to aid in testing.
I’ve written another one. I probably spent an hour or so writing a
framework to write and run all of the tests I needed. The difference
between what I’m describing here and, for example, <a href="http://www.haskell.org/haskellwiki/Introduction_to_QuickCheck">quick
check</a> is that I want something very simple to express
actions that expect their environment to be in a particular state and
will leave the environment in another state. Then I want to hit every
possible arrangement of these actions to ensure they don’t interfere
with each other in unexpected ways.</p>
<div>
<img src="/images/permutations.png" alt="Permutations" class="floatright" />
</div>
<p>This blows up very quickly – specifically the number of tests
generated for a test sequence of <code class="language-plaintext highlighter-rouge">n</code> actions from <code class="language-plaintext highlighter-rouge">a</code> possible actions
is approximately <code>a<sup>n</sup></code>.</p>
<p>Consider three defined actions permuted into sequences of two. That
blows out to nine possibilites as shown in the diagram on the right.</p>
<p>The actions in the diagram are defined with memcached semantics on a
single key, so <code class="language-plaintext highlighter-rouge">add</code> has a prerequisite that the item <em>must not</em> exist
and <code class="language-plaintext highlighter-rouge">del</code> has a prerequisite that the item <em>must</em> exist.</p>
<p>The generated test expects success at each white box, failure at each
red box, and tracks the expected state mutations to build assertions.</p>
<div>
<img src="/images/breakdancer-exponentiality.png" alt="BreakDancer Growth" class="floatleft" />
</div>
<p>My first test… um, test ran with <code class="language-plaintext highlighter-rouge">11</code> actions in sequences of <code class="language-plaintext highlighter-rouge">4</code>
actions. I have more actions to go, but <code class="language-plaintext highlighter-rouge">4</code> is a pretty good length,
so the chart at the left is going to demonstrate my growth rate.</p>
<p>The awesome part is that it pointed out the original bug quite easily
and another couple of bugs with limited effort.</p>
<h2 id="how-do-i-use-this">How Do I Use This?</h2>
<div>
<img src="/images/action-life.png" alt="action lifecycle" class="floatright" />
</div>
<p>The API is so far pretty simple and composable. There are basically
five classes (three are shown in the image on the right).</p>
<h3 id="condition">Condition</h3>
<p>A <code class="language-plaintext highlighter-rouge">Condition</code> is a simple callable that is used for preconditions and
postconditions. A given class doesn’t care which one it’s used for,
and in many cases will be used for both.</p>
<p>For example, consider my implementation of <code class="language-plaintext highlighter-rouge">DoesNotExist</code>:</p>
<figure class="highlight"><pre><code class="language-python" data-lang="python"><span class="k">class</span> <span class="nc">DoesNotExist</span><span class="p">(</span><span class="n">Condition</span><span class="p">):</span>
<span class="k">def</span> <span class="nf">__call__</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">state</span><span class="p">):</span>
<span class="k">return</span> <span class="n">TESTKEY</span> <span class="ow">not</span> <span class="ow">in</span> <span class="n">state</span></code></pre></figure>
<p><br /></p>
<h3 id="effect">Effect</h3>
<p>An <code class="language-plaintext highlighter-rouge">Effect</code> changes our view of the state (and depending on the
driver, may actually cause something in the world to change with it).
For example, the <code class="language-plaintext highlighter-rouge">StoreEffect</code> works as follows:</p>
<figure class="highlight"><pre><code class="language-python" data-lang="python"><span class="k">class</span> <span class="nc">StoreEffect</span><span class="p">(</span><span class="n">Effect</span><span class="p">):</span>
<span class="k">def</span> <span class="nf">__call__</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">state</span><span class="p">):</span>
<span class="n">state</span><span class="p">[</span><span class="n">TESTKEY</span><span class="p">]</span> <span class="o">=</span> <span class="s">'0'</span></code></pre></figure>
<p><br /></p>
<h3 id="action">Action</h3>
<p>An <code class="language-plaintext highlighter-rouge">Action</code> brings together an <code class="language-plaintext highlighter-rouge">Effect</code> and one or more <code class="language-plaintext highlighter-rouge">Condition</code>
classes as pre and post conditions. For example, we’ll look at two
actions, an <code class="language-plaintext highlighter-rouge">Add</code> action and a <code class="language-plaintext highlighter-rouge">Set</code> action:</p>
<figure class="highlight"><pre><code class="language-python" data-lang="python"><span class="k">class</span> <span class="nc">Set</span><span class="p">(</span><span class="n">Action</span><span class="p">):</span>
<span class="n">effect</span> <span class="o">=</span> <span class="n">StoreEffect</span><span class="p">()</span>
<span class="n">postconditions</span> <span class="o">=</span> <span class="p">[</span><span class="n">Exists</span><span class="p">()]</span>
<span class="k">class</span> <span class="nc">Add</span><span class="p">(</span><span class="n">Action</span><span class="p">):</span>
<span class="n">preconditions</span> <span class="o">=</span> <span class="p">[</span><span class="n">DoesNotExist</span><span class="p">()]</span>
<span class="n">effect</span> <span class="o">=</span> <span class="n">StoreEffect</span><span class="p">()</span>
<span class="n">postconditions</span> <span class="o">=</span> <span class="p">[</span><span class="n">Exists</span><span class="p">()]</span></code></pre></figure>
<p>The interesting part of this is that <code class="language-plaintext highlighter-rouge">Set</code> and <code class="language-plaintext highlighter-rouge">Add</code> have different
semantics, but are expressed as different compositions of the same
<code class="language-plaintext highlighter-rouge">Condition</code>s and <code class="language-plaintext highlighter-rouge">Effect</code>s.</p>
<h3 id="driver">Driver</h3>
<p><code class="language-plaintext highlighter-rouge">Driver</code> is kind of a larger part (seven defined methods!). It does
enough that I can do anything from generate a C test suite for
memcached engines all the way to actually executing tests across a
remote protocol.</p>
<p>I won’t describe the entire thing here since it’s documented in the
<a href="http://github.com/dustin/BreakDancer">source</a>. I will however, close the loop by showing you
some example code that it generated that demonstrated the error we
failed to find in the first place:</p>
<figure class="highlight"><pre><code class="language-c" data-lang="c"><span class="k">static</span> <span class="k">enum</span> <span class="n">test_result</span> <span class="nf">test_add_add_delay_add</span><span class="p">(</span><span class="n">ENGINE_HANDLE</span> <span class="o">*</span><span class="n">h</span><span class="p">,</span>
<span class="n">ENGINE_HANDLE_V1</span> <span class="o">*</span><span class="n">h1</span><span class="p">)</span> <span class="p">{</span>
<span class="n">add</span><span class="p">(</span><span class="n">h</span><span class="p">,</span> <span class="n">h1</span><span class="p">);</span>
<span class="n">assertHasNoError</span><span class="p">();</span> <span class="c1">// value is "0"</span>
<span class="n">add</span><span class="p">(</span><span class="n">h</span><span class="p">,</span> <span class="n">h1</span><span class="p">);</span>
<span class="n">assertHasError</span><span class="p">();</span> <span class="c1">// value is "0"</span>
<span class="n">delay</span><span class="p">(</span><span class="n">expiry</span><span class="o">+</span><span class="mi">1</span><span class="p">);</span>
<span class="n">assertHasNoError</span><span class="p">();</span> <span class="c1">// value is not defined</span>
<span class="n">add</span><span class="p">(</span><span class="n">h</span><span class="p">,</span> <span class="n">h1</span><span class="p">);</span>
<span class="n">assertHasNoError</span><span class="p">();</span> <span class="c1">// value is "0"</span>
<span class="n">checkValue</span><span class="p">(</span><span class="n">h</span><span class="p">,</span> <span class="n">h1</span><span class="p">,</span> <span class="s">"0"</span><span class="p">);</span>
<span class="k">return</span> <span class="n">SUCCESS</span><span class="p">;</span>
<span class="p">}</span></code></pre></figure>
<p>That demonstrates how much information you know at each step of the
way. From there, we can do all kinds of stuff with our stubs (delay
above is implemented with the memcached testapp “time travel” feature,
for example).</p>
<p>From here, it’s less exciting. We provide constraints, it writes
tests, and makes sure that there’s another area that it’s impossible
for users to encounter something we haven’t seen before.</p>
Memcached Security2010-08-08T00:00:00+00:00http://dustin.github.com/2010/08/08/memcached-security<div>
<img src="/images/joe-psa.jpg" alt="Stay in School" title="Put reflectors on your bike or be runover" class="floatright" />
</div>
<p>Memcached security is a hot topic since the sensepost guys released
<a href="http://www.sensepost.com/blog/4873.html">go-derper</a> at blackhat.</p>
<p>The presentation was pretty good and informative, but it seems like
the hype around it has left a bunch of people confused. Although much
of this was covered in the presentation, it needs to be restated as
much as possible.</p>
<h2 id="first-and-always-firewall">First and Always, Firewall</h2>
<p>This is really part of the sysadmin placement test and has nothing to
do with memcached in particular, but I’m going to go ahead and mention
it anyway.</p>
<p>You always start by firewalling <em>everything</em> and then allowing only
stuff you need to pass through to the places you need it to pass
through.</p>
<p>I won’t teach you how to use your firewall, but start with the setting
that disables all connectivity to your box.</p>
<p>If you’re running a web server, allow connections to port <code class="language-plaintext highlighter-rouge">443</code>. If
you also want non-ssl connections, allow port <code class="language-plaintext highlighter-rouge">80</code>. If that’s the
only service you’re providing, then your firewalling is now complete!</p>
<p>I’d like to note that <a href="http://aws.amazon.com/ec2/">Amazon EC2</a> does this <em>by default</em>, yet
enough firewalls are misconfigured that they felt the need to send out
a form mail to many of their users to let them know that they “have at
least one security group that allows the whole internet to have access
to the port most commonly used by memcached (11211)”.</p>
<h2 id="check-your-bindings">Check Your Bindings</h2>
<p>If your application only runs on one server (with the app and
memcached on the same box), you can bind it to localhost by adding</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>-l 127.0.0.1
</code></pre></div></div>
<p>to the memcached flags. Now even though you’ve firewalled access to
memcached, you have to be <em>on</em> the machine to even contact the cache
when someone breaks your firewall settings.</p>
<h2 id="if-you-need-it-use-sasl">If You Need It, Use SASL</h2>
<p>The latest versions of memcached support <a href="http://en.wikipedia.org/wiki/Simple_Authentication_and_Security_Layer">SASL</a> authentication.</p>
<p>Although you’ve already firewalled your memcached services off, you
can require clients to perform strong authentication before using the
service.</p>
<p>You can read more about setting this up in
<a href="http://code.google.com/p/memcached/wiki/SASLHowto">the SASL howto page of the wiki</a>.</p>
<h2 id="please-please-do-not-run-as-root">Please, <em>Please</em> Do Not Run as Root</h2>
<p>memcached <em>does not</em> want to run as root. It tries hard to prevent
this. Yet many people have a “workaround” that allows memcached to
start as root (which I will not repeat) just for the sake of making
their infrastructure less secure.</p>
<p>If someone somehow bypasses the firewall you have set up preventing
access to memcached and somehow manages to find a security hole in
memcached allowing code execution, do you <em>really</em> want to just hand
over root access?</p>
<p>There are no such known issues, but we don’t audit the code to ensure
it’s safe to run as root. That’s OK, though, because no responsible
sysadmin would ever run a service as root without very strong
justification, and probably a lot of work in creating a jailed
environment.</p>
<h2 id="check-your-firewall-settings">Check Your Firewall Settings</h2>
<p>Look, I’m not doubting that you know how to set up your firewall, but
just bear with me.</p>
<div>
<a href="http://nerduo.com/thebattle/"><img src="/images/thebattle.png" alt="Knowing" title="Knowing is Half the Battle" class="floatleft" /></a>
</div>
<p>Grab <a href="http://nmap.org/">nmap</a> or similar. Run a full port scan across your box –
one from a trusted system, one from a semi-trusted system, and one
from a completely untrusted system.</p>
<p>If there’s any response for any service you cannot justify running,
you now know about it and can fix it.</p>
<p>That’s not just memcached – that’s gearman, beanstalkd, snmpd, a mail
server, a DNS server, LDAP server, etc…</p>
<p>For any service you <em>do</em> have running and publicly available, make
sure you <em>completely</em> understand the security implications of running
this service.</p>
<p>Do not be embarrassed to ask if you don’t understand everything. It’s
a lot better than being an example in a presentation at the next black
hat because you’re running a service you didn’t intend to and you
leaked important information.</p>
Scaling Memcached with vBuckets2010-06-29T00:00:00+00:00http://dustin.github.com/2010/06/29/memcached-vbuckets<p>For years, people have used memcached to scale large sites.
Originally, there was a simple modulo selection hash algorithm that
was used. It still is used quite a bit actually and it’s quite easy
to understand (although, it’s shown regularly that some people don’t
truly understand it when applied to their full system). The algorithm
is basically this:</p>
<figure class="highlight"><pre><code class="language-python" data-lang="python"><span class="n">servers</span> <span class="o">=</span> <span class="p">[</span><span class="s">'server1:11211'</span><span class="p">,</span> <span class="s">'server2:11211'</span><span class="p">,</span> <span class="s">'server3:11211'</span><span class="p">]</span>
<span class="n">server_for_key</span><span class="p">(</span><span class="n">key</span><span class="p">)</span> <span class="o">=</span> <span class="n">servers</span><span class="p">[</span><span class="nb">hash</span><span class="p">(</span><span class="n">key</span><span class="p">)</span> <span class="o">%</span> <span class="n">servers</span><span class="p">.</span><span class="n">length</span><span class="p">]</span></code></pre></figure>
<p>That is, given a hash algorithm, you hash the key and map it to a
position in the server list and contact that server for that key.
This is <em>really</em> easy to understand, but leads to a few problems.</p>
<ol>
<li>Having some servers have greater capacity than others.</li>
<li>Having cache misses skyrocket when a server dies.</li>
<li>Brittle/confusing configuration (broken things can appear to work)</li>
</ol>
<p>Ignoring weighting (which can basically be “solved” by adding the same
server multiple times to the list), the largest problem you’ve got is
what to do when a server dies, or you want to add a new one, or you
even want to <em>replace</em> one.</p>
<p>In 2007, <a href="http://www.metabrew.com/">Richard Jones</a> and crew over at <a href="http://www.last.fm/">last.fm</a>
created a new way to solve some of these problems called
<a href="https://web.archive.org/web/20120404050051/http://www.audioscrobbler.net/development/ketama/">ketama</a>. This was a library and method for “consistent
hashing” – that is, a way to greatly lower the probability of hashing
to a server that does not have the data you seek when the server list
changes.</p>
<p>It’s an awesome system, but I’m not here to write about it, so I won’t
get into the details. It still has a flaw that makes it unsuitable
for projects like <a href="http://www.membase.org/">membase</a>: it’s only probabilistically
more likely to get you to the server with your data. Looking at it
another way, it’s almost guaranteed to get you to the wrong server
sometimes, just less frequently than the modulus method described
above.</p>
<h1 id="a-new-hope">A New Hope</h1>
<p>In early 2006, Anatoly Vorobey introduced <a href="http://github.com/memcached/memcached/commit/7a308025661a49a5e19f98d2c5b8df04d96b4642">some
code</a> to create something he referred to as
“managed buckets.” This code lived there until late 2008. <a href="http://github.com/memcached/memcached/commit/04319dddabaa06d15407ab6f793b160d3b1c5edb">It was
removed</a> because it was never quite complete, not
understood at all, and we had created a newer protocol that made it
easier build such things.</p>
<p>We’ve been bringing that back, and I’m going to tell you why it exists
and why you want it.</p>
<p>First, a quick summary of what we wanted to accompish:</p>
<ol>
<li>Never service a request on the wrong server.</li>
<li>Allow scaling up <em>and</em> down at will.</li>
<li>Servers refuse commands that they should not service, <em>but</em></li>
<li>Servers still do not know about each other.</li>
<li>We can hand data sets from one server another atomically, <em>but</em></li>
<li>There are no temporal constraints.</li>
<li>Consistency is guaranteed.</li>
<li>Absolutely no network overhead is introduced in the normal case.</li>
</ol>
<p>To expand a bit on the last point relative to other solutions we
looked at, there are no proxies, location services, server-to-server
knowledge, or any other magic things that require overhead. A vbucket
aware request requires no more network operations to find the data
than it does to perform the operation on the data (it’s not even a
single byte larger).</p>
<p>There are other more minor goals such as “you should be able to add
servers while under peak load,” but those just sort of fall out for
free.</p>
<h1 id="introducing--the-vbucket">Introducing: The VBucket</h1>
<p>A vbucket is conceptually a computed subset of all possible keys.</p>
<div>
<img src="/images/vbucket/vbucket.png" alt="vbucket visualized" class="floatright" />
</div>
<p>If you’ve ever implemented a hash table, you can think of it as a
virtual hash table bucket that is the first level of hashing for all
node lookups. Instead of mapping keys directly to servers, we map
vbuckets to servers statically and have a consistent key →
vbucket computation.</p>
<p>The number of vbuckets in a cluster remains constant regardless of
server topology. This means that key <code class="language-plaintext highlighter-rouge">x</code> always maps to the same
vbucket given the same hash.</p>
<p>Client configurations have to grow a bit for this concept. Instead of
being a plain sequence of servers, the config now also has the
explicit vbucket to server mapping.</p>
<p>In practice, we model the configuration as a server sequence, hash
function, and vbucket map. Given three servers and six vbuckets (a
very small number for illustration), an example of how this works in
relation to the modulus code above would be as follows:</p>
<figure class="highlight"><pre><code class="language-python" data-lang="python"><span class="n">servers</span> <span class="o">=</span> <span class="p">[</span><span class="s">'server1:11211'</span><span class="p">,</span> <span class="s">'server2:11211'</span><span class="p">,</span> <span class="s">'server3:11211'</span><span class="p">]</span>
<span class="n">vbuckets</span> <span class="o">=</span> <span class="p">[</span><span class="mi">0</span><span class="p">,</span> <span class="mi">0</span><span class="p">,</span> <span class="mi">1</span><span class="p">,</span> <span class="mi">1</span><span class="p">,</span> <span class="mi">2</span><span class="p">,</span> <span class="mi">2</span><span class="p">]</span>
<span class="n">server_for_key</span><span class="p">(</span><span class="n">key</span><span class="p">)</span> <span class="o">=</span> <span class="n">servers</span><span class="p">[</span><span class="n">vbuckets</span><span class="p">[</span><span class="nb">hash</span><span class="p">(</span><span class="n">key</span><span class="p">)</span> <span class="o">%</span> <span class="n">vbuckets</span><span class="p">.</span><span class="n">length</span><span class="p">]]</span></code></pre></figure>
<p>It should be obvious from reading that code how the introduction of
vbuckets provides tremendous power and flexibility, but I’ll go on in
case it’s not.</p>
<h2 id="terminology">Terminology</h2>
<p>Before we get into too many details, let’s look at the terminology
that’s going to be used here.</p>
<dl>
<dt>Cluster</dt>
<dd>A collection of collaborating servers.</dd>
<dt>Server</dt>
<dd>An individual machine within a cluster.</dd>
<dt>vbucket</dt>
<dd>A subset of all possible keys.</dd>
</dl>
<p>Also, any given vbucket will be in one of the following states on any
given server:</p>
<div>
<img src="/images/vbucket/states.png" alt="VBucket States" class="floatright" />
</div>
<dl>
<dt>Active</dt>
<dd>This server is servicing all requests for this vbucket.</dd>
<dt>Dead</dt>
<dd>This server is not in any way responsible for this vbucket</dd>
<dt>Replica</dt>
<dd>
No client requests are handled for this vbucket, but it can
receive replication commands.
</dd>
<dt>Pending</dt>
<dd>This server will block all requests for this vbucket.</dd>
</dl>
<h1 id="client-operations">Client Operations</h1>
<p>Each request must include the vbucket id as computed by the hashing
algorithm. We made use of the reserved fields in the <a href="http://code.google.com/p/memcached/wiki/MemcacheBinaryProtocol">binary
protocol</a> allowing for up to 65,536 vbuckets to be created
(which is really quite a lot).</p>
<p>Since all that’s needed to consistently choose the right vbucket is
for clients to agree on the hashing algorithm and number of vbuckets,
it’s significantly harder to misconfigure a server such that you’re
communicating with the wrong server for a given vbucket.</p>
<p>Additionally, with <a href="http://github.com/northscale/libvbucket">libvbucket</a> we’ve made distributing
configurations and distributing configuration, agreeing on mapping
algorithms, and reacting to misconfigurations a problem that doesn’t
have to be solved repeatedly. Work is under way to get ports of
libvbucket to java and .net, and in the meantime <a href="https://github.com/couchbase/moxi">moxi</a> will
perform all of the translations for you if you have a non-persistent
clients or can’t wait for your favorite client to catch up.</p>
<h2 id="one-active-server">One Active Server</h2>
<div>
<img src="/images/vbucket/oneserver.png" alt="One server, six vbuckets" class="floatright" />
</div>
<p>While deployments typically have 1,024 or 4,096 vbuckets, we’re going
to continue with this model with six because it’s a lot easier to
think about and draw pictures of.</p>
<p>In the image to the right, there is one server running with six active
buckets. All requests with all possible vbuckets go to this server,
and it answers for all of them.</p>
<h2 id="one-active-server-one-new-server">One Active Server, One New Server</h2>
<div>
<img src="/images/vbucket/one-quiesc.png" alt="One active server, one quiescent server" class="floatright" />
</div>
<p>Now let us add a new server. Here’s the first bit of magic: Adding a
server does not destabilize the tree (as seen on the right).</p>
<p>Adding a server to the cluster, and even pushing it out in the
configuration to all of the clients, does not imply it will be used
immediately. Mapping is a separate concept, and all vbuckets are
still exclusively mapped to the old server.</p>
<div>
<img src="/images/vbucket/transfer.gif" alt="Transfering vbuckets from one server to two" style="clear: right; padding-top: 10px;" class="floatleft" />
</div>
<p>In order to make this server useful, we will transfer vbuckets from
one server to another. To effect a transfer, you select a set of the
vbuckets that you want the new server to own and set them all to the
pending state on the receiving server. Then we begin pulling the data
out and placing it in the new server.</p>
<p>By performing the steps in this exact order, are able to guarantee no
more than one server is active for any given vbucket at any given
point in time <em>without</em> any regard to actual chronology. That is, you
can have hours of clock skew and vbucket transfers taking several
minutes and never fail to be consistent. It’s also guaranteed that
clients will never receive <em>incorrect</em> answers.</p>
<div>
<a href="/images/vbucket/flow.png" title="Sequence diagram of a vbucket transfer."><img src="/images/vbucket/flow-small.png" alt="flow and what-not" class="floatright" /></a>
</div>
<ol>
<li>The vbucket on the new server is placed in a pending state.</li>
<li>A vbucket extract <a href="https://web.archive.org/web/20100803004713/http://blog.northscale.com/northscale-blog/2010/03/want-to-know-what-your-memcached-servers-are-doing-tap-them.html">tap</a> stream is started.</li>
<li>The vbucket tap stream atomically sets the state to dead when the
queue is in a sufficient drain state.</li>
<li>The new server only transitions from pending to active after it’s
received confirmation that the old server is no longer servicing
requests.</li>
</ol>
<p>Since subsections are being transferred indepenently, you no longer
have to limit yourself to thinking of a server moving at a time, but a
tiny fraction of a server moving at a time. This allows you to start
slowly migrating traffic from busy servers <em>at peak</em> to less busy
servers with minimal impact (with 4,096 vbuckets over 10 servers each
with 10M keys, you’d be moving about 20k keys at a time with a vbucket
transfer as you bring up your eleventh server).</p>
<p>You may notice that there is a time period where a vbucket has <em>no</em>
active server at all. This occurs at the very end of the transfer
mechanism and causes blocking to occur. In general, it should be rare
to observe a client actually blocked in the wild. This only happens
when a client gets an error from the old server indicating it’s done
prepping the transfer and can get to the new server before the new
server receives the last item. Then the new server only blocks the
client until that item is delivered and the vbucket can transition
from <code class="language-plaintext highlighter-rouge">pending</code> to <code class="language-plaintext highlighter-rouge">active</code> state.</p>
<p>Although the vbucket in the old server automatically goes into the
<code class="language-plaintext highlighter-rouge">dead</code> state when it gets far enough along, it <em>does not</em> delete data
automatically. That is explicitly done <em>after</em> confirmation that the
new node has gone <code class="language-plaintext highlighter-rouge">active</code>. If the destination node fails at any
point before we set it <code class="language-plaintext highlighter-rouge">active</code>, we can just abort the transfer and
leave the old server <code class="language-plaintext highlighter-rouge">active</code> (or set it back to <code class="language-plaintext highlighter-rouge">active</code> if we were
far enough along).</p>
<h1 id="whats-this-about-replica-state">What’s This About Replica State?</h1>
<p>HA comes up a lot, so we made sure to cover it. A <code class="language-plaintext highlighter-rouge">replica</code> vbucket
is similar to a <code class="language-plaintext highlighter-rouge">dead</code> vbucket in that from a normal client’s
perspective. That is, all requests are refused, but replication
commands are allowed. This is also similar to the <code class="language-plaintext highlighter-rouge">pending</code> state in
that records are stored, but contrasted in that clients do not block.</p>
<div>
<img src="/images/vbucket/replica1.png" alt="One replica with three servers" class="floatright" />
</div>
<p>Consider the image to the right where we have three servers, six
vbuckets, and a single replica per vbucket.</p>
<p>Like the masters, each replica is also statically mapped, so they can
be moved around at any time.</p>
<p>In this example, we replicate the vbucket to the “next” server in the
list. i.e. an <code class="language-plaintext highlighter-rouge">active</code> vbucket on <code class="language-plaintext highlighter-rouge">S1</code> replicates to a <code class="language-plaintext highlighter-rouge">replica</code>
bucket on <code class="language-plaintext highlighter-rouge">S2</code> – same for <code class="language-plaintext highlighter-rouge">S2</code> → <code class="language-plaintext highlighter-rouge">S3</code> and
<code class="language-plaintext highlighter-rouge">S3</code> → <code class="language-plaintext highlighter-rouge">S1</code>.</p>
<h2 id="multiple-replicas">Multiple Replicas</h2>
<p>We also enable strategies to have more than one copy of your data
available on nodes.</p>
<p>The diagram below shows two strategies for three servers to have one
active and two replicas of each bucket.</p>
<h2 id="1n-replication">1:n Replication</h2>
<p>The first strategy (<code class="language-plaintext highlighter-rouge">1:n</code>) refers to a master servicing multiple
slaves concurrently. The concept here is familiar to anyone who’s
dealt with data storage software that allows for multiple replicas.</p>
<div>
<img src="/images/vbucket/replica-many.png" alt="Strategies for many-child replication" style="text-align: center; margin-left: auto; margin-right: auto" />
</div>
<h2 id="chained-replication">Chained Replication</h2>
<p>The second strategy (<code class="language-plaintext highlighter-rouge">chained</code>) refers to a single master servicing
only a single slave, but having that slave have a further downstream
slave of its own. This offers the advantage of having a single stream
of mutation events coming out of a server, while still maintaining two
copies of all records. This has the disadvantage of compounding
replication latency as you traverse the chain.</p>
<p>Of course, with more than two additional copies, you could mix them
such that you do a single stream out of the master and then have the
second link of the chain V out a <code class="language-plaintext highlighter-rouge">1:n</code> stream to two further servers.</p>
<p>It’s all in how you map things.</p>
<h1 id="acknowledgments">Acknowledgments</h1>
<p>Thanks to <a href="https://dormando.me">Dormando</a> for helping decipher the original
“managed bucket” code, intent, and workflows, and <a href="http://github.com/jayesh">Jayesh
Jose</a> and the other Zynga folks for independently discovering
it and working through a lot of use cases.</p>
What We're Doing in Memcached2010-04-09T00:00:00+00:00http://dustin.github.com/2010/04/09/memcached-status<div>
<img src="/images/being-transparent.png" alt="transparency" title="So what if I like Banksy?" class="floatright" />
</div>
<p>We’ve been steadily hacking on <a href="http://memcached.org/">memcached</a>.</p>
<p>We think it’s going very well, but we do want to make sure everybody
who cares has the opportunity to see what’s going on behind the
proverbial curtain.</p>
<p>The basic theme is to build a platform that allows a company to solve
its scaling problems without preventing you from solving your own.</p>
<h2 id="extensibility">Extensibility</h2>
<p>The biggest thing we’ve been working on is getting the storage engine
interface really solid. <a href="http://trondn.blogspot.com/">Trond</a> has been thinking about this
for <a href="http://blogs.sun.com/trond/entry/memcached_and_customized_storage_engines">two years</a> and did an excellent
<a href="http://blogs.sun.com/trond/entry/presentation_at_the_mysql_users">presentation</a> on an application of it at last year’s
MySQL User Conference.</p>
<p>Since then, we’ve applied it and adapted it to handle a few real-world
scenarios and have been pretty happy with the results.</p>
<p>We’re looking forward to fewer forks of memcached to solve one-off
problems and instead making it easier to bake a solution to your
problem directly into the standard code base.</p>
<p>Our hope is that this will lead to a variety of open source solutions
to common problems. For example, <a href="http://www.northscale.com/">NorthScale</a> released the
<a href="http://github.com/northscale/bucket_engine">bucket engine</a> that allows a single memcached instance to
support multiple logical engines.</p>
<h2 id="portability">Portability</h2>
<p><a href="http://patg.net/">Patrick</a> has done a ridiculous amount of work to get us to
the point where we can officially support Windows in a maintainable
way (i.e. does as little damage to the rest of the codebase as
possible).</p>
<p>This is another area where forks have existed to solve one-off
problems, but have been unable to track bug fixes and new features.</p>
<h2 id="documentation">Documentation</h2>
<p><a href="http://consoleninja.net/">dormando</a> has essentially been doing an informed rewrite of
the documentation to make it more approachable, more comprehensive,
and just generally more better.</p>
<p>The <a href="http://memcached.org/">new site</a> was the first part of this, and has been
pretty awesome, but the <a href="http://code.google.com/p/memcached/wiki/NewStart">wiki</a> reorg is almost done and I’m
pretty excited about that.</p>
<h2 id="releases">Releases</h2>
<p>The 1.4.5 release just shipped. We have plans for a 1.4.6 maintenance
release to clear up a bit more of the problems people have seen in the
field (mostly targetting people who run operating systems that won’t
update their libraries more than once a decade).</p>
<h2 id="come-join-us">Come Join Us</h2>
<p>If you’re somewhere around Santa Clara, come join us at the <a href="http://en.oreilly.com/mysql2010/">MySQL
User Conference</a>. We’ve got a lot of stuff we’ll be finishing
up and are able to answer any questions you might have about storing
data, perhaps even some about retrieving it.</p>
<p>We’re having a <a href="http://en.oreilly.com/mysql2010/public/schedule/detail/14627">bof</a> that you are personally invited to and I
believe I may be joining <a href="http://blog.northscale.com/northscale-blog/author/matt-ingenthron">Matt</a> for a talk about some of the
work we’ve been doing.</p>
Why I Don't Use Maven2010-04-01T00:00:00+00:00http://dustin.github.com/2010/04/01/why-not-maven<h1 id="why-i-dont-use-maven-for-my-java-projects">Why I don’t Use Maven for my Java Projects</h1>
<p>(and what you can do about it)</p>
<div>
<img src="/images/maven.png" alt="maven" class="floatright" />
</div>
<p>I used to really like <a href="http://maven.apache.org/">maven</a>. A long time ago. Around version 1.x.
It had lots of great features I liked in a build system and required
very little work to do just about anything. Then there was maven 2.</p>
<p>I often try to find an image to characterize my blog posts well. In
this case, I just took the first image I saw on the maven site itself
and felt that it pretty accurately summed my experience with maven.</p>
<p>It’s a guy sitting on a table with his back to his computer looking
out a window contemplating the jump.</p>
<p>“Will it hurt?” he asks himself. “Will it hurt more than writing one
more XML element describing how my <code class="language-plaintext highlighter-rouge">.java</code> files get turned into
<code class="language-plaintext highlighter-rouge">.class</code> files?”</p>
<p>OK, so perhaps it’s not as bad as the imagery on their site makes it out
to be, but I do have actual real reasons I’m not using it.</p>
<h2 id="why-not">Why Not?</h2>
<p>I’m going to assume you know the virtues of maven. I get complaints
from users of <a href="http://code.google.com/p/spymemcached/">my memcached client</a> for not directly
supporting their build tool of choice, so I’m just going to focus on
the parts that keep me away.</p>
<h3 id="build-de-automation">Build De-automation</h3>
<p>The absolute number one reason I’ve not put any effort into converting
my build into maven is because it would <em>increase</em> my work. Not just
to do the conversion, but in an ongoing way.</p>
<div>
<img src="/images/manual.png" class="floatleft" alt="manual labor" />
</div>
<p>I’ve asked many of the people who have wanted me to run my builds with
maven if they’d do the conversion for me. I had two small requests
that I didn’t think were unreasonable: It shouldn’t increase my work
and it shouldn’t reduce the features in my software.</p>
<p>Nobody has delivered this to me, and I can’t see a way to do it myself.</p>
<p>Apparently, The Maven Way is to edit <code class="language-plaintext highlighter-rouge">pom.xml</code> for every release to
put the version number into that file (then, of course commit it in my
SCM) and then build and then do my normal tagging.</p>
<p>I have to write the version number out twice.</p>
<p>In contrast, here’s the line to figure out the version of the software
I’m producing from the <a href="http://github.com/dustin/java-memcached-client/blob/master/buildfile">buildfile</a> I use when
I build my project using <a href="http://buildr.apache.org/">apache buildr</a>:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>VERSION_NUMBER = `git describe`.strip
</code></pre></div></div>
<p>That’s how every version I’ve released has worked since I switched to
git. Before I used git with buildr, I used hg with buildr. Before I
used hg with buildr, I used hg with maven 1. Before hg and maven 1, I
used gnu arch and maven 1. Never in the history of this project have
I had to modify the build system when I change version numbers. I
think that’s an important feature.</p>
<h3 id="my-build-system-is-not-my-scm">My Build System is Not My SCM</h3>
<p>There’s a workaround for the above – you use the <a href="http://maven.apache.org/scm/plugins/index.html">maven SCM plugin</a>!</p>
<p>Except, it’s as backwards as a guy sitting on a desk facing away from
his computer.</p>
<div>
<img src="http://chart.apis.google.com/chart?cht=p&chs=250x140&chd=s:jSGBB&chl=Trond|Dustin|Sean|Patrick|Steve" class="floatright" alt="contributors" />
</div>
<p>The SCM plugin makes maven a user interface to my SCM. I cannot tell
you how much I don’t want another interface to my SCM. I can,
however, tell you that I carefully make my tags and I have tools that
do neat stuff with my carefully created tags like generate a nice
useful <a href="http://dustin.github.com/java-memcached-client/changelog.html">changelog</a>.</p>
<p>I don’t use my build tool to write my code. I certainly don’t use it
to commit my code or show me a diff.</p>
<p>What I <em>do</em> want is for it to appropriately interact with my SCM to
get the information it needs to do a build.</p>
<h3 id="missing-build-automation">Missing Build Automation</h3>
<p>One of the features they added to maven 2 was you can’t script. You
can’t script feature is great because, <a href="http://maven.apache.org/maven1.html#m1-maven-xml">according to the
site</a>, they believe it’s <em>better</em> to write a formal plugin
in java because not only is it less work for them (once), but you’ll
probably find someone else has that to do the same thing you’re doing
and you’ll want to share it!</p>
<p>One of my projects has a rather large <code class="language-plaintext highlighter-rouge">maven.xml</code> file because it
builds a compiler code generator that it uses to generate code under
the current environment. It’s <em>very</em> specific to that build and the
code that goes with it. I’m not writing and hosting a full-fledged
plugin in java to do what I used to do with a little jelly script (and
ant before that and make before that).</p>
<h3 id="missing-build-plugins">Missing Build Plugins</h3>
<p>This is a half-answer, but I’m missing features I want. One could
write them, but nobody has. This, again, was a jelly script back in
the data, and is now a buildr plugin, but for any given java library I
write, you have the ability to do this:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>dustinnmb:/tmp 794% java -jar x.jar
spy.jar on Fri Nov 13 10:47:00 PST 2009
Build platform: java 1.6.0_15 from Apple Inc. on Mac OS X version 10.6.2
Tree version: 2.5rc1
(add -c to see the recent changelog)
</code></pre></div></div>
<p>That “Tree version:” listed there is straight out of the SCM. If you
add the <code class="language-plaintext highlighter-rouge">-c</code> option, you get what is effectively my git log. You can
take a file in isolation and know which bug fixes you have and all
kinds of other junk.</p>
<p>It’s not even clear to me how one would go about doing this in maven.</p>
<h3 id="real-benefits-have-unrelated-dependencies">Real Benefits Have Unrelated Dependencies</h3>
<div>
<img src="/images/unrelated.png" class="floatleft" alt="unrelated picture" />
</div>
<p>Most of the time, when people ask me for maven support it’s not
because of how I build my software. It’s not because they’re having
trouble building my software (though that does come up).</p>
<p>Most of the time, they want to download it from the internet.</p>
<p>It was trivial to host a maven 1 repo, maven 2 repos are a bit harder
because you have to generate a descriptor and a few other magic files
and a deeper filesystem hierarchy for each build and then tell users
how to configure up your repo.</p>
<p>It’s far better to just stick them in the main, centralized
repositories, but you pretty much have to use maven itself to do
that.</p>
<p>It is of course <em>possible</em> to create artifacts using another tool
(such as <a href="http://buildr.apache.org/">apache buildr</a>) and get them uploaded into one of
the well-known maven repositories, but I don’t see any support for
this coming out of the maven community.</p>
<p>I just can’t express the absurdity of such a thing.</p>
<p>This is like someone telling you that you can’t distribute software in
ubuntu unless you use <code class="language-plaintext highlighter-rouge">bzr</code> as your revision control software. It
just doesn’t make sense.</p>
<h2 id="think-about-the-users">Think About The Users</h2>
<p>I don’t have anything against maven as a concept per se, but it’s not
right for me. Yay freedom of choice.</p>
<p>I would of course like my software to be easy to consume for people
who do choose maven. I’ve put effort into making this so, and that
effort has been slowly thwarted over time.</p>
<p>I intend to continue my efforts to make this work as easily as
possible for everyone, but if you find my software being built through
maven, it’s because of some hard work of others.</p>
git test-sequence -- Push Working Changes2010-03-28T00:00:00+00:00http://dustin.github.com/2010/03/28/git-test-sequence<h1 id="git-test-sequence-push-working-changes">git test-sequence: Push Working Changes</h1>
<div>
<img src="/images/mr-clean-small.png" alt="Herr Clean" class="floatright" />
</div>
<p>I interactively rebase my changes before I push them for two primary
reasons:</p>
<ol>
<li>The changes I push must concisely represent the changes I intended
to make.</li>
<li>Each change should be tested (i.e. don’t break the build)</li>
</ol>
<p>It’s easy enough to rewrite stuff interactively and squash and
what-not to end up with a readable history, but when you actually want
to try things out, it can be a pain.</p>
<p>Assume the upstream tree is in a working state. You want to push two
changes. The first one adds a feature and the second one uses it.
The second one also accidentally fixes a bug in the first one. That
appears OK because the tree builds before and after your push.</p>
<p>If I didn’t break the build, what’s the problem? You broke <em>a</em>
build. Next time someone runs <code class="language-plaintext highlighter-rouge">git bisect</code> he might hit it. It will
make him sad. Let us not make people sad. If we can verify every
change works, we will forever live harmonious lives.</p>
<p>I wrote a tool for git called <a href="http://github.com/dustin/bindir/blob/master/git-test-sequence">test-sequence</a> a long time ago
to support this very thing. When it is easy to verify a tree is
clean, we will at least try.</p>
<p>I’m writing about it today because it’s come up twice in conversation
this week. Once with a co-worker who was trying to prep a rather
large branch for review, and again in a <a href="http://stackoverflow.com/questions/2530015/any-tool-to-make-git-build-every-commit-to-a-branch-in-a-seperate-repository">stack overflow question</a>
where someone was eerily asking for exactly what I had written.</p>
<p>The concept is similar to an automated <code class="language-plaintext highlighter-rouge">git bisect</code> except it’s
linear. It will test <em>every</em> change between two points in the DAG.
It’ll even walk each side of a merge and test <em>those</em> changes
individually.</p>
<p>The <a href="http://stackoverflow.com/questions/2530015/any-tool-to-make-git-build-every-commit-to-a-branch-in-a-seperate-repository">stack overflow question</a> goes into a lot of details of the
why, so I’ll just talk about the how:</p>
<h2 id="using-git-test-sequence">Using git test-sequence</h2>
<p>First, put the <a href="http://github.com/dustin/bindir/blob/master/git-test-sequence">git-test-sequence</a> script somewhere in your
path.</p>
<p>Now, think about the stuff you want to test, how you want to test it,
etc…</p>
<p>The example I give in the script itself looks like this:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>git test-sequence origin/master.. 'make clean && make test'
</code></pre></div></div>
<p>Since I’m using normal range operators, that should be pretty readable
to any git user. In this case, run <code class="language-plaintext highlighter-rouge">make clean && make test</code> for
every local commit that I’ve made since I last pushed to
<code class="language-plaintext highlighter-rouge">origin/master</code>.</p>
<p>You can go the other way, too:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>git test-sequence ..origin/master 'make clean && make test'
</code></pre></div></div>
<p>…will there be any incoming changes that will break my build?</p>
<p>When combined with <a href="http://buildbot.net/">buildbot</a>, you get ludicrous power.
I’ve got a buildbot install with 21 builders currently. I’ve got 26
commits on a branch I’m moving forward. The following command will
test all 26 changes against each of the 21 builders (i.e. 546 builds
will be started):</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>git test-sequence origin/master.. 'buildbot try'
</code></pre></div></div>
<p>And no, the code didn’t work on all of the builders. Most of them in
fact. My screen was covered in growl alerts from
<a href="http://code.google.com/p/buildwatch/">buildwatch</a> letting me know that we broke something.</p>
<p>Before this is published, every build will work on every platform, and
it will be trivial to verify.</p>
conc Lists in Erlang2010-03-04T00:00:00+00:00http://dustin.github.com/2010/03/04/erlang-conc<h1 id="conc-lists-in-erlang---or-making-the-kessel-run-in-12-parsecs">conc Lists in Erlang - Or Making the Kessel Run in 12 Parsecs</h1>
<div>
<img src="/images/kesselrun.png" alt="Making the Kessel Run" class="floatright" />
</div>
<p>I saw this wonderful Guy Steele talk titled <a href="http://vimeo.com/6624203">Organizing Functional
Code for Parallel Execution</a>.</p>
<p>The talk describes how what we have learned to get us this far may be
preventing us from proceeding in the new world of many cores.</p>
<p>I don’t want to ruin the whole talk for you, you should watch it and
learn great things, but I found one thing quite inspirational. In his
description of behaviors of conc lists (which I’ll go into more
below), he described the time complexity of doing a mapreduce of his
conc list as logarithmic.</p>
<p>This blew my mind. mapreduce <code class="language-plaintext highlighter-rouge">n</code> items in <code class="language-plaintext highlighter-rouge">O(log n)</code> time. I was
fascinated. I could read the code, but I had to touch it, so I built
<a href="http://github.com/dustin/erl-conc">an implementation</a> of conc lists in erlang to try it out.</p>
<h2 id="quick-intro-to-cons-lists">Quick Intro to cons Lists</h2>
<p>conc lists are described in contrast to cons lists, so I’ll do a quick
review on those first.</p>
<p>A cons list is basically a pair. In lisp, the first element is called
a <code class="language-plaintext highlighter-rouge">car</code> and the second element is called a <code class="language-plaintext highlighter-rouge">cdr</code>, but we’re talking
about erlang here where you typically just pattern match stuff, so
I’ll just write the lists out in erlang notation and rip them apart
with pattern matching.</p>
<p>The empty list is represented as <code class="language-plaintext highlighter-rouge">[]</code>. It’s boring, but note that
it’s at the end of every list. So if you see a list that looks like
it has one element: <code class="language-plaintext highlighter-rouge">[a]</code>, the cons cell is really the pair of <code class="language-plaintext highlighter-rouge">a</code>
and <code class="language-plaintext highlighter-rouge">[]</code>.</p>
<p>In erlang, cons looks like this: <code class="language-plaintext highlighter-rouge">[SomeElement | RemainingList]</code>. You
can cons the single element above as <code class="language-plaintext highlighter-rouge">[a | []]</code>. A three element list
can be expressed as <code class="language-plaintext highlighter-rouge">[a, b, c]</code> but you can also think of the
implementation as <code class="language-plaintext highlighter-rouge">[a | [b | [c | []]]]</code>.</p>
<p>This is significant when it comes to implementing code to process cons
lists, because this is how you can model traversal operations such as
<code class="language-plaintext highlighter-rouge">map</code>, <code class="language-plaintext highlighter-rouge">foldl</code>, <code class="language-plaintext highlighter-rouge">length</code>, etc… in your head. Each splits the list
into the first element and the rest of the elements, performs an
operation on the first element and then recurses over the rest.</p>
<p>Length, for example, may be implemented as follows:</p>
<figure class="highlight"><pre><code class="language-erlang" data-lang="erlang"><span class="nb">length</span><span class="p">([])</span> <span class="o">-></span> <span class="mi">0</span><span class="p">;</span> <span class="c">% Empty list is terminal.
</span><span class="nb">length</span><span class="p">([</span><span class="nv">Hd</span><span class="p">|</span><span class="nv">Tl</span><span class="p">])</span> <span class="o">-></span> <span class="mi">1</span> <span class="o">+</span> <span class="nb">length</span><span class="p">(</span><span class="nv">Tl</span><span class="p">)</span> <span class="err">%</span> <span class="nv">Recurse</span> <span class="n">on</span> <span class="n">elements</span><span class="p">.</span></code></pre></figure>
<p>Now for the good parts…</p>
<h2 id="quick-intro-to-conc-lists">Quick Intro to conc Lists</h2>
<p>conc lists point in two directions instead of just one. They’re
really more like trees. Of course, there’s nothing stopping an
implementation from having further dimensions, but that’s an exercise
for the one doing the profiling.</p>
<p>Unlike a cons where you really only have two things (an element or the
empty list), in a conc, you have three:</p>
<div>
<img src="/images/conc-legend.png" alt="legend of the conc" class="centered" />
</div>
<p><code class="language-plaintext highlighter-rouge">null</code> is analogous to its cons empty cell. It is the end of the road
and contains no data and no pointers elsewhere.</p>
<p>A <code class="language-plaintext highlighter-rouge">singleton</code> contains exactly one element and nothing else.</p>
<p>A <code class="language-plaintext highlighter-rouge">concatenation</code> contains a left and a right pointer which each may
be to another concatenation, a singleton, or null.</p>
<p>Unlike a typical binary tree structure, these have no natural
ordering, so at this point, we consider every operation to be <code class="language-plaintext highlighter-rouge">O(n)</code>.
In a moment, we will begin violating this, but in the meantime, let’s
draw some pictures.</p>
<div>
<img src="/images/conc-unbalanced.png" alt="unbalanced conc list" class="floatright" />
</div>
<p>There are many ways to represent the same conc list. The original
presentation went over a bunch, but I’ll just illustrate two here.</p>
<p>We’ll use <code class="language-plaintext highlighter-rouge">[23, 47, 18, 11]</code> for the purpose of example, because he
did and he’s awesome.</p>
<p>To the right, we see what an unbalanced conc list representation of
our list might look like.</p>
<p>If you are familiar with trees, you can see that an in-order traversal
will visit each element in the same order as our cons representation
above.</p>
<div>
<img src="/images/conc-balanced.png" alt="balanced conc list" class="floatleft" />
</div>
<p>To our left, we see the same conc list in a well-balanced form.</p>
<p>And this is where things begin to get interesting.</p>
<p><br style="clear: both" /></p>
<h2 id="conc-list-processing">conc List Processing</h2>
<p>conc lists have two fundamental operations: <code class="language-plaintext highlighter-rouge">append</code> and <code class="language-plaintext highlighter-rouge">mapreduce</code>.</p>
<p><code class="language-plaintext highlighter-rouge">append(A, B)</code> puts can be thought of as building a concatenation of A
to B (though we do something a bit more optimal in implementation to
avoid unnecessary concatenation cells).</p>
<p><code class="language-plaintext highlighter-rouge">mapreduce</code> isn’t quite the same thing as a <code class="language-plaintext highlighter-rouge">lists:mapfoldl</code>, so don’t
let that confuse you. You can imagine a simple implementation of a
cons <code class="language-plaintext highlighter-rouge">mapreduce</code> looking like this:</p>
<figure class="highlight"><pre><code class="language-erlang" data-lang="erlang"><span class="nf">mapreduce</span><span class="p">(</span><span class="nv">MapFun</span><span class="p">,</span> <span class="nv">ReduceFun</span><span class="p">,</span> <span class="nv">Initial</span><span class="p">,</span> <span class="nv">List</span><span class="p">)</span> <span class="o">-></span>
<span class="nn">lists</span><span class="p">:</span><span class="nf">foldl</span><span class="p">(</span><span class="nv">ReduceFun</span><span class="p">,</span> <span class="nv">Initial</span><span class="p">,</span>
<span class="nn">lists</span><span class="p">:</span><span class="nf">map</span><span class="p">(</span><span class="nv">MapFun</span><span class="p">,</span> <span class="nv">List</span><span class="p">)).</span></code></pre></figure>
<p>Of course, we don’t don’t do two passes through the list. In fact,
assuming we can keep our conc lists balanced, we get to perform
linear-complexity operations in logarithmic time.</p>
<p>First, let us consider a naive implementation of a conc <code class="language-plaintext highlighter-rouge">mapreduce</code>:</p>
<figure class="highlight"><pre><code class="language-erlang" data-lang="erlang"><span class="nf">mapreduce</span><span class="p">(_</span><span class="nv">Map</span><span class="p">,</span> <span class="p">_</span><span class="nv">Reduce</span><span class="p">,</span> <span class="nv">Id</span><span class="p">,</span> <span class="n">null</span><span class="p">)</span> <span class="o">-></span>
<span class="nv">Id</span><span class="p">;</span>
<span class="nf">mapreduce</span><span class="p">(</span><span class="nv">Map</span><span class="p">,</span> <span class="p">_</span><span class="nv">Reduce</span><span class="p">,</span> <span class="p">_</span><span class="nv">Id</span><span class="p">,</span> <span class="p">{</span><span class="n">singleton</span><span class="p">,</span> <span class="nv">I</span><span class="p">})</span> <span class="o">-></span>
<span class="nv">Map</span><span class="p">(</span><span class="nv">I</span><span class="p">);</span>
<span class="nf">mapreduce</span><span class="p">(</span><span class="nv">Map</span><span class="p">,</span> <span class="nv">Reduce</span><span class="p">,</span> <span class="nv">Id</span><span class="p">,</span> <span class="p">{</span><span class="n">concatenation</span><span class="p">,</span> <span class="nv">A</span><span class="p">,</span> <span class="nv">B</span><span class="p">})</span> <span class="o">-></span>
<span class="nv">Reduce</span><span class="p">(</span><span class="nf">mapreduce</span><span class="p">(</span><span class="nv">Map</span><span class="p">,</span> <span class="nv">Reduce</span><span class="p">,</span> <span class="nv">Id</span><span class="p">,</span> <span class="nv">A</span><span class="p">),</span>
<span class="nf">mapreduce</span><span class="p">(</span><span class="nv">Map</span><span class="p">,</span> <span class="nv">Reduce</span><span class="p">,</span> <span class="nv">Id</span><span class="p">,</span> <span class="nv">B</span><span class="p">))</span></code></pre></figure>
<p>The <code class="language-plaintext highlighter-rouge">length</code> operation, for example, just maps each element to 1 and
then reduces the sum of all mapped numbers. This includes the sum of
the sums as it walks up the tree. Consider the code for <code class="language-plaintext highlighter-rouge">length</code>:</p>
<figure class="highlight"><pre><code class="language-erlang" data-lang="erlang"><span class="nb">length</span><span class="p">(</span><span class="nv">L</span><span class="p">)</span> <span class="o">-></span>
<span class="nf">mapreduce</span><span class="p">(</span><span class="k">fun</span> <span class="p">(_</span><span class="nv">X</span><span class="p">)</span> <span class="o">-></span> <span class="mi">1</span> <span class="k">end</span><span class="p">,</span>
<span class="k">fun</span><span class="p">(</span><span class="nv">X</span><span class="p">,</span><span class="nv">Y</span><span class="p">)</span> <span class="o">-></span> <span class="nv">X</span><span class="o">+</span><span class="nv">Y</span> <span class="k">end</span><span class="p">,</span>
<span class="mi">0</span><span class="p">,</span>
<span class="nv">L</span><span class="p">).</span></code></pre></figure>
<p>If you consider the balanaced list from above, the process of
computing the length is to mapreduce the left path, then the right
and then reduce the results of each, or <code class="language-plaintext highlighter-rouge">(1 + 1) + (1 + 1)</code>.</p>
<h2 id="going-parallel">Going Parallel</h2>
<div>
<img src="/images/conc-parallel.png" alt="parallel conc processing" class="floatright" />
</div>
<p>The secret is that a balanced conc list may be split in half in
constant time.</p>
<p>In the <code class="language-plaintext highlighter-rouge">mapreduce</code> implementation, each concatenation is a point where
we can parallelize the computations of the mapreduce to the left and
the mapreduce to the right and then reduce them together as the
computation completes.</p>
<p>So in a practical application, <code class="language-plaintext highlighter-rouge">length</code> will <em>concurrently</em> compute
the left list’s <code class="language-plaintext highlighter-rouge">(1 + 1)</code> and the right list’s and <code class="language-plaintext highlighter-rouge">(1 + 1)</code> and then
reduce them with the final summation of <code class="language-plaintext highlighter-rouge">(2 + 2)</code> after they finish.</p>
<p>On a small scale, this looks boring, but consider the following
example:</p>
<figure class="highlight"><pre><code class="language-erlang" data-lang="erlang"><span class="nv">C</span> <span class="o">=</span> <span class="nn">conc</span><span class="p">:</span><span class="nf">from_list</span><span class="p">(</span><span class="nn">lists</span><span class="p">:</span><span class="nf">seq</span><span class="p">(</span><span class="mi">1</span><span class="p">,</span> <span class="mi">1000</span><span class="p">)).</span>
<span class="nn">timer</span><span class="p">:</span><span class="nf">tc</span><span class="p">(</span><span class="n">conc</span><span class="p">,</span> <span class="n">mapreduce</span><span class="p">,</span>
<span class="p">[</span><span class="k">fun</span><span class="p">(_)</span> <span class="o">-></span> <span class="nn">timer</span><span class="p">:</span><span class="nf">sleep</span><span class="p">(</span><span class="mi">1000</span><span class="p">),</span> <span class="mi">1000</span> <span class="k">end</span><span class="p">,</span>
<span class="k">fun</span><span class="p">(</span><span class="nv">X</span><span class="p">,</span> <span class="nv">Acc</span><span class="p">)</span> <span class="o">-></span> <span class="nv">X</span> <span class="o">+</span> <span class="nv">Acc</span> <span class="k">end</span><span class="p">,</span>
<span class="mi">0</span><span class="p">,</span>
<span class="nv">C</span><span class="p">]).</span></code></pre></figure>
<p>Here, we take a 1,000 node conc list and perform a <code class="language-plaintext highlighter-rouge">mapreduce</code> on it
where the mapping function artificially takes 1 second (1000ms) to
complete, and then returns that same 1000 as the map result. The
reducer then adds all of those milliseconds together.</p>
<p>I’m using the <code class="language-plaintext highlighter-rouge">timer:tc</code> function here to show the magic of
auto-parallelization in the conc mapreduce. That unfortunately makes
the invocation look a bit funny, but anyone who’s programmed in erlang
has done plenty of similar <code class="language-plaintext highlighter-rouge">M,F,A</code> invocations, so I hope it’s not
terribly confusing.</p>
<p>There result of that call is <code class="language-plaintext highlighter-rouge">{4013058,1000000}</code> – that is,
4,013,058µs to perform 1,000,000ms worth of work.</p>
<p>The units are kind of confusing – complain to the erlang guys about
that. Convert them both to seconds and you’ll see that I did 1,000
seconds worth of work in 4 seconds (and you feel that when you do it
interactively).</p>
<h2 id="i-want-to-play">I Want to Play</h2>
<p>I’ve got a <a href="http://github.com/dustin/erl-conc">repo</a> up on github with my code. The mapreduce
parallelism seems about right (though it’s not depth-limited), but
rebalancing is suboptimal.</p>
<p>I look forward to your contributions to make this implementation
better, but thanks go to <a href="http://research.sun.com/people/mybio.php?uid=25706">Guy Steele</a> for the linked list of the
future.</p>
Running Processes2010-02-28T00:00:00+00:00http://dustin.github.com/2010/02/28/running-processes<div>
<img src="/images/startmeup.jpg" alt="If you start me up I'll never stop - Rolling Stones" title="I run this bitch, and I'm a keep running - Birdman and Lil' Wayne" class="floatright" />
</div>
<p>I keep suffering a lack of decent process supervision mechanisms
provided by operating systems I use.</p>
<p>Most systems and software seem to have this idea that a “start up
script” is the right approach.</p>
<p>This is horribly wrong as is demonstrated by <a href="http://god.rubyforge.org/">lots</a> and
<a href="http://mmonit.com/monit/">lots</a> and <a href="http://www.yolinux.com/HOWTO/Process-Monitor-HOWTO.html">lots</a> of tools that exist to do that on
top of so many really awful “ps via cron” kind of things people hack
together or process counts via <a href="http://www.nagios.org/">nagios</a> or whatever.</p>
<p>This post describes alternatives I’ve found to help relieve my
suffering, but has ended up exceedingly long. I’ll provide quick
links to specific sections:</p>
<h1 id="toc">TOC</h1>
<ul>
<li><a href="#howwrong">How Wrong Can You Get?</a></li>
<li><a href="#whatswrong">What's Wrong with monit/god/nagios/etc...</a></li>
<li><a href="#rightway">So What's the Right Way?</a>
<ul>
<li><a href="#osx">OSX</a></li>
<li><a href="#solaris">Solaris</a></li>
<li><a href="#ubuntu">Ubuntu</a></li>
<li><a href="#freebsd">FreeBSD</a></li>
<li><a href="#genericlinux">Generic Linux</a></li>
<li><a href="#thirdparty">Third Party Tools</a>
<ul>
<li><a href="#daemontools">daemontools</a></li>
</ul>
</li>
</ul>
</li>
</ul>
<p><a name="howwrong"> </a></p>
<h2 id="how-wrong-can-you-get">How Wrong Can You Get?</h2>
<p>The worst one of these I’ve personally seen was a group of contractors
who pieced a system together via <a href="http://www.tibco.com/">tibco</a>. There was a computer
that had a process that needed to be monitored. There was another
computer that ran a monitor thing that would send out messages over
tibco that would be processed by an agent on the first machine that
would run a really long shell script that would do some <code class="language-plaintext highlighter-rouge">ps</code>, <code class="language-plaintext highlighter-rouge">grep</code>,
<code class="language-plaintext highlighter-rouge">awk</code> and other stuff to search for a process and then maybe restart
it before replying back across the tibco bus.</p>
<p>It did this about once a minute and used something like 60% of the CPU
to verify the machine was still running. They demonstrated this to me
and was all ready for me to tell them how amazed I was with their rube
goldberg sysadmin skills, but my response didn’t leave them with the
good feeling they sought:</p>
<p>“Wow. You reinvented <a href="http://en.wikipedia.org/wiki/Init">init</a> <em>and</em> cron, but managed to make
them both less reliable and consume more CPU than I could’ve
imagined.”</p>
<p><a name="whatswrong"> </a></p>
<h2 id="whats-wrong-with-monitgodnagiosetc">What’s Wrong With monit/god/nagios/etc…</h2>
<p>Sometimes, these tools can be used for good, but often, they’re
variations on the same theme as above. You’re polling your process
list, or a pid file, or whatever to try to see if another process is
running and then restarting it if it fails.</p>
<p>Many people seem to grab these tools with the goal of rerunning their
start script sometime after they’ve noticed the process is not
running. This is the mentality I’m hoping to correct.</p>
<p>And this brings me to my point…</p>
<p><a name="rightway"> </a></p>
<h2 id="so-whats-the-right-way">So What’s the Right Way?</h2>
<p>Don’t <em>start</em> programs, <em>run</em> programs.</p>
<p><a href="http://en.wikipedia.org/wiki/Init">init</a> already does this. It does it <em>very</em> efficiently, with
no CPU overhead in the general case, no latency in the exceptional
case, no custom scripts to write, and with absolute reliability (that
is, it won’t forget to run your command, and it won’t crash without
taking the entire operating system with it).</p>
<p>Unfortunately, init has historically not been very easy to use use for
your own processes. I’ll break it down a bit here to help you keep
your stuff running.</p>
<p><strong>Note</strong>: In every case, it’s assumed that you have a program that
wants to run that does <em>not</em> daemonize on its own. Self-daemonizing
programs start you down the path to hell. You can’t use any sane
keepalive techniques so you have to resort to polling process lists or
checking the pid or something. Even managing that pidfile gets hard
when you combine it with things that change their own uid for safety
(because you should never run anything as root).</p>
<p><a name="osx"> </a></p>
<h3 id="mac-os-x">Mac OS X</h3>
<p>Mac OS X has <a href="http://developer.apple.com/macosx/launchd.html">launchd</a> which combines init, cron, inetd, and
a few other things rolled into one.</p>
<p>Launchd provides tremendous amounts of granularity over control of
processes and extends that to users themselves to run applications
consistently on events.</p>
<p>The following example shows how you can have a program that runs
consistently while you’re logged in. Place this in
<code class="language-plaintext highlighter-rouge">~/Library/LaunchAgents/com.example.someprogram.plist</code> and launchd
should do its magic to keep it going.</p>
<figure class="highlight"><pre><code class="language-xml" data-lang="xml"><span class="cp"><?xml version="1.0" encoding="UTF-8"?></span>
<span class="cp"><!DOCTYPE plist PUBLIC
"-//Apple Computer//DTD PLIST 1.0//EN"
"http://www.apple.com/DTDs/PropertyList-1.0.dtd"></span>
<span class="nt"><plist</span> <span class="na">version=</span><span class="s">"1.0"</span><span class="nt">></span>
<span class="nt"><dict></span>
<span class="nt"><key></span>Label<span class="nt"></key></span>
<span class="nt"><string></span>com.example.someprogram<span class="nt"></string></span>
<span class="nt"><key></span>KeepAlive<span class="nt"></key></span>
<span class="nt"><true/></span>
<span class="nt"><key></span>RunAtLoad<span class="nt"></key></span>
<span class="nt"><true/></span>
<span class="nt"><key></span>ProgramArguments<span class="nt"></key></span>
<span class="nt"><array></span>
<span class="nt"><string></span>/path/to/someprogram<span class="nt"></string></span>
<span class="nt"></array></span>
<span class="nt"></dict></span>
<span class="nt"></plist></span></code></pre></figure>
<p>For disconnected use, you can build a similar plist and place it in
<code class="language-plaintext highlighter-rouge">/Library/LaunchDaemons/</code>, add a <code class="language-plaintext highlighter-rouge">UserName</code> attribute and have the
system maintain a program on behalf of another user for you quite
easily.</p>
<p>Much, much more can be done with launchd (such as run programs on
filesystem and network events). It’s open source and a horrible shame
it isn’t everywhere, as it really does provide a tremendous benefit.</p>
<p><a name="solaris"> </a></p>
<h3 id="solaris">Solaris</h3>
<p>Solaris has <a href="http://hub.opensolaris.org/bin/view/Community+Group+smf/WebHome">smf</a> which is similar to launchd, but provides even
more control over lifecycle, permissions, dependencies, fault
management and many other things.</p>
<p>If you’re using Solaris, you probably understand it already, otherwise
I’d recommend reading <a href="http://hub.opensolaris.org/bin/view/Community+Group+smf/WebHome">the smf docs</a> to understand how to use it
to manage your system.</p>
<p><a name="ubuntu"> </a></p>
<h3 id="ubuntu">Ubuntu</h3>
<p>Recent versions of ubuntu ship with <a href="http://upstart.ubuntu.com/">upstart</a>, which is the
least modern of the modern system facilities, but at least does allow
you to specify that you want to <em>run</em> an application instead of just
<em>start</em> it.</p>
<p>Here’s an example upstart script I wrote for a twisted app that I
needed to keep running on an ubuntu box:</p>
<figure class="highlight"><pre><code class="language-sh" data-lang="sh">description <span class="s2">"useful description"</span>
author <span class="s2">"Dustin Sallings <dustin@spy.net>"</span>
start on runlevel 2
start on runlevel 3
stop on runlevel 0
stop on runlevel 1
stop on runlevel 4
stop on runlevel 5
stop on runlevel 6
chdir /path/to/project/directory
<span class="nb">exec</span> /usr/bin/twistd <span class="nt">--uid</span><span class="o">=</span>daemonuser <span class="nt">--syslog</span> <span class="nt">-ny</span> project.tac
respawn</code></pre></figure>
<p>Note that this relies on <code class="language-plaintext highlighter-rouge">twistd</code> to change the uid since it’s more
straightforward than using <code class="language-plaintext highlighter-rouge">su</code> or <code class="language-plaintext highlighter-rouge">sudo</code> to change the userid before
invoking the start script.</p>
<p><a name="freebsd"> </a></p>
<h3 id="freebsd">FreeBSD</h3>
<p>FreeBSD, and most BSDs for that matter have an init that will
supervise processes defined in <code class="language-plaintext highlighter-rouge">/etc/ttys</code>. This is about as
primitive as it can get, but it works fine.</p>
<p>For example, if you wanted to run <code class="language-plaintext highlighter-rouge">sshd</code> on a FreeBSD box and make
sure that it can never die, you could add the following to
<code class="language-plaintext highlighter-rouge">/etc/ttys</code>:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>sshd "/usr/local/etc/sshd_tty" unknown on
</code></pre></div></div>
<p>The script, <code class="language-plaintext highlighter-rouge">/usr/local/etc/sshd_tty</code> exists primarily to eat the
implicit argument init passes to the program it runs. For this, I
used the following script:</p>
<figure class="highlight"><pre><code class="language-sh" data-lang="sh"><span class="c">#!/bin/sh</span>
<span class="nb">exec</span> /usr/sbin/sshd <span class="nt">-D</span></code></pre></figure>
<p><code class="language-plaintext highlighter-rouge">/etc/ttys</code> basically exists for <a href="http://en.wikipedia.org/wiki/Getty_(Unix)">getty</a> type services, but
it’s suitable for any other process that needs to be supervised.</p>
<p>After any modification to <code class="language-plaintext highlighter-rouge">/etc/ttys</code>, you must run <code class="language-plaintext highlighter-rouge">init q</code> for your
changes to take effect.</p>
<p><a name="genericlinux"> </a></p>
<h2 id="generic-linux">Generic Linux</h2>
<p>On any vanilla Linux system (that is, systems that don’t use a modern
init as well as other systems with similar init mechanisms), you can
do something similar to the above with <code class="language-plaintext highlighter-rouge">/etc/inittab</code>, although it has
no implicit argument, so you can directly invoke sshd.</p>
<p>For example, the following works on my RedHat 5.4 box:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>sshd:2345:respawn:/usr/sbin/sshd -D
</code></pre></div></div>
<p>After adding this entry (and, of course, making sure you don’t already
have an sshd server running), you can run <code class="language-plaintext highlighter-rouge">/sbin/telinit q</code> to get it
to reload.</p>
<p><a name="thirdparty"> </a></p>
<h2 id="third-party-tools">Third Party Tools</h2>
<p>It’s possible to bring in a third-party program to supplement an init
that’s less awesome than <a href="http://developer.apple.com/macosx/launchd.html">launchd</a> and <a href="http://hub.opensolaris.org/bin/view/Community+Group+smf/WebHome">smf</a>. I
personally have limited experience doing this, but there’s one in
particular that I’ve been using and found to do a good job.</p>
<p><a name="daemontools"> </a></p>
<h3 id="daemontools">daemontools</h3>
<p><a href="http://cr.yp.to/daemontools.html">daemontools</a> is <a href="http://cr.yp.to/djb.html">djb</a> weird-ware that serves as an
excellent UNIX process supervision framework.</p>
<p>It’s a bit weird out of the box, though. It really wishes a file
structure existed that is just not natural on any UNIX system, but
some ports overcome that a bit.</p>
<p>However, it’s very composable and has a lot of small tools that each
do one thing really well.</p>
<p>I’ve got an OpenBSD machine that’s customized to the point where I
wrote my own <code class="language-plaintext highlighter-rouge">/etc/rc</code> from scratch. This machine runs my DNS and
DHCP services as well as a few other things around the house.</p>
<p>I used to have a problem where it’d lose power or something and one of
the processes wouldn’t come up cleanly – very often DHCP, which made
things rather difficult for me. I thought this would be a great place
to try daemontools for the first time.</p>
<p>It has so far made things much easier to deal with.</p>
<p>One service I have running, for example, is sample_devices from my
<a href="http://bleu.west.spy.net/~dustin/projects/ibutton/">ibutton</a> suite (thermometers → multicast, basically).
This needs to run as a user who can access <code class="language-plaintext highlighter-rouge">/dev/tty01</code> (I don’t run
stuff as root unless absolutely required) and has a bit of init.</p>
<p>For this to work, I just have to create a <code class="language-plaintext highlighter-rouge">run</code> file in
<code class="language-plaintext highlighter-rouge">/services/sample_devices</code> which looks like this:</p>
<figure class="highlight"><pre><code class="language-sh" data-lang="sh"><span class="c">#!/bin/sh</span>
<span class="nb">mkdir</span> /tmp/sample
<span class="nb">chown </span>uucp /tmp/sample
<span class="nb">exec </span>setuidgid uucp /usr/local/sbin/sample_devices <span class="se">\</span>
<span class="nt">-b</span> /dev/tty01 <span class="nt">-c</span> /tmp/sample <span class="se">\</span>
<span class="nt">-m</span> 225.0.0.37 <span class="nt">-p</span> 6789 <span class="nt">-t</span> 64 <span class="nt">-s</span> 2121</code></pre></figure>
<p>Note that the <code class="language-plaintext highlighter-rouge">exec</code> is important, as daemontools has a lot of control
utilities which requires it to know the pid of the actual running
process, not a shell that started it. It’s good practice anyway.</p>
<p>The one thing I don’t like about daemontools is that the service
directories contain the startup scripts, control sockets, as well as
other lock states. It’d serve me better if my service definitions
lived somewhere in <code class="language-plaintext highlighter-rouge">/etc/</code> and the runtime control lived somewhere in
<code class="language-plaintext highlighter-rouge">/var/run</code>, but I’m pretty happy with the results.</p>
Countdown Project2009-12-31T00:00:00+00:00http://dustin.github.com/2009/12/31/countdown<p>OK. Last blog post of the year.</p>
<p>My TV is broken and my daughter said she wanted to celebrate the New
Year by watching some kind of ball dropping thing, so I had this sort
of “last minute” idea for something and searched around the house for
junk I could build it out of.</p>
<iframe width="420" height="315" src="//www.youtube.com/embed/Qv_ruj4lhXo" frameborder="0" allowfullscreen="1">
</iframe>
<p>A few LEDs, a servo, an <a href="http://www.moderndevice.com/products/bbb-kit">rbb</a> (<a href="http://www.arduino.cc/">arduino</a> clone) and,
um, whatever scrap cardboard I could find and I was good to go.</p>
<p>If something looks ghetto, that’s a feature. I strongly encourage
people like myself who have perfectionist tendencies to just try to
make doing things poorly a goal now and then. It helps.</p>
<p>I didn’t really consider this much of a sustainable project, so I
really kind of hacked it together. You can see the code that runs it
on <a href="http://gist.github.com/267014">github</a>.</p>
<div>
<img alt="hot mess" class="floatleft" src="/images/countdown-back-small.jpg" />
</div>
<p>The python script initializes communication, synchronizes (which has a
corresponding visual effect across the LEDs) and then calculates the
time until the nearest target time (I’ve been rounding to the nearest
15s, hour, whatever).</p>
<p>One second before the countdown sequence starts, it pauses whatever
music I have playing and then turns the volume up.</p>
<p>After the command is sent, it waits a second and starts the music.
This causes the actual music to start just about in time for the clock
to strike midnight.</p>
<p>Then it blinks erratically.</p>
<p>It’s crap, but as it says in the comment in the python script, worse
is better.</p>
<h2 id="update-2010">Update (2010)</h2>
<div>
<img alt="arted" class="floatright" src="/images/countdown-arted-small.jpg" />
</div>
<p>I made a few changes to it before we got to use it (actually, I was
playing with it until about 15 minutes before midnight :/ ). Figured
I’d update this post with the details.</p>
<p>Summary of changes since the video:</p>
<ol>
<li>It got a new face (thanks, kids).</li>
<li>I manually specified the angles instead of computing them to feel
more organic (and matched what the kids did).</li>
<li>Control side pushed sync commands through every fifteen seconds
during the last ten minutes.</li>
<li>I added some countdown vocals on the computer side to start us
going at <code class="language-plaintext highlighter-rouge">t - 10s</code>.</li>
</ol>
<p>It was a great success. It’s now dismantled.</p>
Hello World in Go -- A Memcached Server2009-11-12T00:00:00+00:00http://dustin.github.com/2009/11/12/gomemcached<h1 id="hello-world-in-go--a-memcached-server">Hello World in Go – A Memcached Server</h1>
<div>
<img src="/images/gomemcached.png" alt="go memcached" class="floatright" />
</div>
<p>I sat down last night to learn <a href="http://golang.org/">go</a>. I’ve been a fan of
concurrency-oriented languages since I wrote my <a href="http://github.com/dustin/environ">first erlang
program</a> in early 2004 (which I still use). I’ve been
itching to give <a href="http://golang.org/">go</a> a go since the announcement.</p>
<p>The first thing I thought of that could naturally be modeled as a
concurrent program was a <a href="http://memcached.org/">memcached</a> server. That has sort
of become my “hello, world” as I have implemented the <a href="http://github.com/dustin/memcached-test">first python
server</a> for the memcached binary protocol (using asyncore), the
<a href="http://memcached.org/">main server</a> (which <a href="http://blogs.sun.com/trond/">Trond</a> made beautiful), an
<a href="/2009/10/11/ememcached.html">erlang server</a>, and a <a href="http://github.com/dustin/twisted-memcached">twisted server</a>.</p>
<p>My <a href="http://github.com/dustin/gomemcached">go server</a> has a lot in common with
<a href="/2009/10/11/ememcached.html">ememcached</a> since both languages are concurrency
oriented. There is one process/goroutine for managing the actual data
storage, one for accepting TCP connections, and one for each connected
user.</p>
<p>The TCP listener just accepts and spins up new processes or goroutines
to handle the IO on the connections and then they wait for data. Once
they’ve read a single request, they dispatch to the storage mechanism
which will sequentually process each operation, regardless of input
concurrency. Very simple, very easy to understand.</p>
<p>I mostly liked what I saw. The code feels a little C-like, but in
practice is fairly intuitive and pleasant to work with. There are
many <a href="http://scienceblogs.com/goodmath/2009/11/googles_new_language_go.php">reviews</a> of the language and I’m not attempting to write
one, but I will say that I got hung up on the lack of exceptions which
required me to do <a href="http://github.com/dustin/gomemcached/blob/master/mc_conn_handler.go#L72">weird</a> error handling in sequential code.</p>
<p>The concurrency primitives remind me a bit of <a href="http://en.wikipedia.org/wiki/Alef_%28programming_language%29">alef</a> which I
remember from early <a href="http://plan9.bell-labs.com/plan9/">plan9</a> distributions, though I never
actually got a chance to work with it before it got killed off. The
one part of alef I really liked lived on in go, which further
motivated me.</p>
ZFS for MacOS X2009-10-23T00:00:00+00:00http://dustin.github.com/2009/10/23/mac-zfs<h1 id="zfs-for-macos-x">ZFS for MacOS X</h1>
<div>
<img src="/images/zfs.png" alt="ZFS ROX" class="floatright" />
</div>
<p>So I’m a pretty heavy <a href="http://en.wikipedia.org/wiki/ZFS">ZFS</a> user. I’ve got a FreeBSD box with a ZFS
root that all my important stuff is on, and a mac mini with ZFS that
all my movies are on.</p>
<p>Around Snow Leopard, Apple seemed to have not only dropped support for
it, but the download from <a href="http://macosforge.org/">macosforge</a> seemed to not work,
either.</p>
<p>I’ve been hoping someone would come around and fix this, but nothing’s
happened.</p>
<p>Then today, I read this:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>The ZFS project has been discontinued. The mailing list and
repository will also be removed shortly.
</code></pre></div></div>
<p>That made me very sad, so I decided to do something about it.</p>
<h2 id="github-project">Github Project</h2>
<p>I set up a <a href="http://github.com/dustin/mac-zfs">mac-zfs</a> project on github and started hacking. It
didn’t take long and I was able to get to my zpool that has all of my
music. (for the record, Bad Religion was my ZFS test case)</p>
<h2 id="downloads">Downloads</h2>
<div>
<a href="http://cloud.github.com/downloads/dustin/mac-zfs/ZFS-119-SnowLeopard.pkg">
<img src="/images/pkg.png" alt="Install Me" class="floatright" />
</a>
</div>
<p>I wanted to make it a bit easier to get going, so I created an
<a href="http://cloud.github.com/downloads/dustin/mac-zfs/ZFS-119-SnowLeopard.pkg">installer</a>.</p>
<p>You still need to kind of know what you’re doing to make use of it.
But I’m hoping we can work together to make it easy for everybody.</p>
<h2 id="update">Update:</h2>
<p>We’ve got a <a href="http://code.google.com/p/maczfs/">google code</a> page up for bug tracking and
what-not and a <a href="http://groups.google.com/group/zfs-macos">mailing list</a> for general discussion.</p>
<p>Come join us.</p>
Memcached Report Card2009-10-22T00:00:00+00:00http://dustin.github.com/2009/10/22/memcached-reportcard<h1 id="memcached---report-card-for-an-open-source-project">Memcached - Report Card for an Open Source Project</h1>
<p>While approaching our <a href="http://code.google.com/p/memcached/wiki/ReleaseNotes140">1.4</a> release, the memcached project slipped into
a state where we weren’t pushing out new code very frequently.</p>
<p>Some members of the community demanded releases occur more frequently,
so <a href="http://consoleninja.net/">dormando</a> published a proposal for a release cycle for
memcached that would help drive us into the direction of universal
happiness.</p>
<p>I had a feeling that his plan and efforts to keep us on it were quite
effective, but the purpose of this post is to sort of take an
objective retrospective look and see how the reality of the progress
of the project holds up to my perception.</p>
<h2 id="release-scheduling">Release Scheduling</h2>
<p><img src="/images/memcached-velocity.png" alt="Release Velocity" /></p>
<p>This chart pretty much speaks for itself. It’s clear to see that
we very quickly approached the stated goal of 30 days between
releases.</p>
<p>I should note that <a href="http://code.google.com/p/memcached/wiki/ReleaseNotes141">1.4.1</a> was released later than expected
because a user hit a real live failure scenario in a production system
that took a while to isolate. Once we did, we wrote some tests to
ensure that the entire class of bug described can’t happen again.</p>
<p>These releases are bigger than they appear, but very well tested on
our <a href="http://code.google.com/p/memcached/wiki/BuildFarm">build farm</a> and in production environments. To see
what goes into each release, check the release notes for
<a href="http://code.google.com/p/memcached/wiki/ReleaseNotes140">1.4.0</a>, <a href="http://code.google.com/p/memcached/wiki/ReleaseNotes141">1.4.1</a>, and <a href="http://code.google.com/p/memcached/wiki/ReleaseNotes142">1.4.2</a>.</p>
<h2 id="bug-status">Bug Status</h2>
<p>We’ve been taking bugs very seriously.</p>
<p>Crash bugs are resolved <em>very</em> quickly, but bugs in general are
resolved within our release cycle.</p>
<p>Please let us know via <a href="http://groups.google.com/group/memcached">our mailing list</a> or our <a href="http://code.google.com/p/memcached/issues/list">bug
tracker</a> if you’re having issues or can’t figure something
out. We fix things very quickly, but <strong>we can’t fix what we don’t hear
about</strong>. Remember that goes for the <a href="http://code.google.com/p/memcached/wiki/Start">wiki docs</a>, too.</p>
<p>The following chart shows the average age (time between when they were
filed and when they were closed) of bugs by month in memcached. The
error bars are showing the minimum and maximum age for any given
month. Issues marked as invalid, won’tfix, and had other indicators
of not being actual defects or missing features in the software were
excluded.</p>
<p>Four anomalies were identified and described briefly in the chart.
These are all considered trivial issues that were not detrimental to
the state of the server, but they do show that we’re sometimes not too
quick in cases like these. (more details on <a href="http://code.google.com/p/memcached/issues/detail?id=9">A</a>, <a href="http://code.google.com/p/memcached/issues/detail?id=42">B</a>,
<a href="http://code.google.com/p/memcached/issues/detail?id=53">C</a>, and <a href="http://code.google.com/p/memcached/issues/detail?id=59">D</a>).</p>
<p><img src="/images/memcached-issue-age.png" alt="Bug Age" /></p>
<p>I also found it useful to consider what I called the “bug load.” That
is, the number of bugs going in and out of the project by month.</p>
<p>The following chart is showing bugs activity in the memcached
project.</p>
<p>Positive numbers are bugs being reported to the project. Negative
numbers are bugs that we closed (bugs removed from the project). The
line indicates the net number of bugs at the end of the given month.</p>
<p><img src="/images/memcached-bug-load.png" alt="Bug Load" /></p>
<h2 id="so-whats-the-grade">So What’s the Grade?</h2>
<p>I’d say we’ve got a solid B. We have room for improvement and I’m
confident we’re making it.</p>
<p>If you disagree (and can articulate why specifically) or otherwise
have feedback, a warm, active community awaits.</p>
<p>Contact us on irc <a href="irc://irc.freenode.net/memcached">(freenode.net #memcached)</a>, email suggestions
or questions to our <a href="http://groups.google.com/group/memcached">list</a>, or just go <a href="http://code.google.com/p/memcached/issues/list">file a bug</a>.</p>
EMemcached2009-10-11T00:00:00+00:00http://dustin.github.com/2009/10/11/ememcached<h1 id="why-an-erlang-memcached-implementation">Why an Erlang Memcached Implementation?</h1>
<div>
<img src="/images/a-o-k.png" alt="Memcached! Do you speak it?" class="floatright" />
</div>
<p>I wrote an <a href="http://github.com/dustin/ememcached">erlang implementation</a> of a memcached server
speaking the binary protocol over the weekend. You’re all probably
wondering why.</p>
<p>It was half a learning exercise, and half a way to plug a
<a href="http://blogs.sun.com/trond/entry/memcapable">memcapable</a> interface into more applications. In its
current form, I consider it a framework to take existing erlang
backend stores (e.g dict, ets, dets, mnesia, <a href="http://riak.basho.com/">riak</a>, etc…) and
put a memcached interface on them. On its own, it’s a little boring.</p>
<p>The first implementation of the binary protocol was in my
<a href="http://github.com/dustin/memcached-test">memcached-test</a> project (which I still actively use
as a reference implementation when playing around with features and
testing clients and things). It’s a simple asyncore based server in
python (with a sample synchronous client).</p>
<p>A bit later, came the actual <a href="http://code.google.com/p/memcached/">production C</a> server version
we all know and love (which also had my <a href="http://code.google.com/p/spymemcached/">java client</a> to
talk to).</p>
<p>I wrote <a href="http://github.com/dustin/twisted-memcached">twisted memcached</a> and built an S3-backed server
one weekend.</p>
<p>That leads us back to the present <a href="http://github.com/dustin/ememcached">erlang-based server</a>.</p>
<h2 id="but-wait-theres-more">But Wait, There’s More!</h2>
<p>If you have a basic understanding of erlang, you may find this
implementation to be the best documentation of the protocol in
existence.</p>
<p>In the main loop of a connection, all I’m doing is calling
<code class="language-plaintext highlighter-rouge">process_message</code> in a loop with the connected socket, a reference to
the storage server (an erlang <code class="language-plaintext highlighter-rouge">gen_server</code> implementation), and the
result of a call to <code class="language-plaintext highlighter-rouge">gen_tcp:recv(Socket, 24)</code>. That last call will
either return <code class="language-plaintext highlighter-rouge">{ok, SomeData}</code> or <code class="language-plaintext highlighter-rouge">{error, SomeReason}</code>.</p>
<h3 id="the-fun-part">The Fun Part</h3>
<p>The only definition of <code class="language-plaintext highlighter-rouge">process_message/3</code> I have is shown below. The
<em>only</em> valid way to call this is when the third argument is the <code class="language-plaintext highlighter-rouge">{ok,
Data}</code> tuple where <code class="language-plaintext highlighter-rouge">Data</code> in this case is a binary pattern. Some of
the values are filled in (which means they <em>must</em> match for this
function body to be invoked), and some are bindings which will receive
the value.</p>
<p>In the code below (extracted from <a href="http://github.com/dustin/ememcached/blob/master/src/mc_connection.erl">mc_connection.erl</a>),
you’ll see that the first 8 bits must exactly be the defined
<code class="language-plaintext highlighter-rouge">REQ_MAGIC</code> constant, and then the next 8 bits are stored in <code class="language-plaintext highlighter-rouge">OpCode</code>,
and so on.</p>
<p>Any attempt to process a message not in this form will result in the
connection processing crashing (the effect of which is your client
being disconnected from it).</p>
<p>So you can see how erlang easily allows us to rip the bits we need out
of the header for dispatch, so the next thing is to ask for the
remaining data (extra headers, key, and body) before dispatching to
the storage server process.</p>
<figure class="highlight"><pre><code class="language-erlang" data-lang="erlang"><span class="nf">process_message</span><span class="p">(</span><span class="nv">Socket</span><span class="p">,</span> <span class="nv">StorageServer</span><span class="p">,</span>
<span class="p">{</span><span class="n">ok</span><span class="p">,</span> <span class="o"><<?</span><span class="nv">REQ_MAGIC</span><span class="p">:</span><span class="mi">8</span><span class="p">,</span> <span class="nv">OpCode</span><span class="p">:</span><span class="mi">8</span><span class="p">,</span> <span class="nv">KeyLen</span><span class="p">:</span><span class="mi">16</span><span class="p">,</span>
<span class="nv">ExtraLen</span><span class="p">:</span><span class="mi">8</span><span class="p">,</span> <span class="mi">0</span><span class="p">:</span><span class="mi">8</span><span class="p">,</span> <span class="mi">0</span><span class="p">:</span><span class="mi">16</span><span class="p">,</span>
<span class="nv">BodyLen</span><span class="p">:</span><span class="mi">32</span><span class="p">,</span>
<span class="nv">Opaque</span><span class="p">:</span><span class="mi">32</span><span class="p">,</span>
<span class="nv">CAS</span><span class="p">:</span><span class="mi">64</span><span class="o">>></span><span class="p">})</span> <span class="o">-></span>
<span class="c">% After the header is extras, key, and then body
</span> <span class="nv">Extra</span> <span class="o">=</span> <span class="nf">read_data</span><span class="p">(</span><span class="nv">Socket</span><span class="p">,</span> <span class="nv">ExtraLen</span><span class="p">),</span>
<span class="nv">Key</span> <span class="o">=</span> <span class="nf">read_data</span><span class="p">(</span><span class="nv">Socket</span><span class="p">,</span> <span class="nv">KeyLen</span><span class="p">),</span>
<span class="c">% Note that the length of the body from the header includes
</span> <span class="c">% the lengths of the key and extras.
</span> <span class="nv">Body</span> <span class="o">=</span> <span class="nf">read_data</span><span class="p">(</span><span class="nv">Socket</span><span class="p">,</span> <span class="nv">BodyLen</span> <span class="o">-</span> <span class="p">(</span><span class="nv">KeyLen</span> <span class="o">+</span> <span class="nv">ExtraLen</span><span class="p">)),</span>
<span class="c">% Dispatch the read data to a gen_server process.
</span> <span class="nv">Res</span> <span class="o">=</span> <span class="nn">gen_server</span><span class="p">:</span><span class="nf">call</span><span class="p">(</span><span class="nv">StorageServer</span><span class="p">,</span>
<span class="p">{</span><span class="nv">OpCode</span><span class="p">,</span> <span class="nv">Extra</span><span class="p">,</span> <span class="nv">Key</span><span class="p">,</span> <span class="nv">Body</span><span class="p">,</span> <span class="nv">CAS</span><span class="p">}),</span>
<span class="nf">respond</span><span class="p">(</span><span class="nv">Socket</span><span class="p">,</span> <span class="nv">OpCode</span><span class="p">,</span> <span class="nv">Opaque</span><span class="p">,</span> <span class="nv">Res</span><span class="p">).</span></code></pre></figure>
<h3 id="the-storage-server">The Storage Server</h3>
<p>A storage server process is a <code class="language-plaintext highlighter-rouge">gen_server</code> implementation whose
<code class="language-plaintext highlighter-rouge">handle_call</code> implementations look will take a tuple of <code class="language-plaintext highlighter-rouge">{OpCode,
ExtraHeader, Key, Value, CAS}</code> and return a <code class="language-plaintext highlighter-rouge">mc_response</code> record.</p>
<p>For an example storage server, consider my two <code class="language-plaintext highlighter-rouge">flush</code> implementations
in my <a href="http://github.com/dustin/ememcached/blob/master/src/mc_handler_hashtable.erl">hashtable store</a> (noting that flash has one 32-bit
integer in extra header, no key, and no value):</p>
<figure class="highlight"><pre><code class="language-erlang" data-lang="erlang"><span class="c">% Immediate flush. Ignore the current state and make a new one.
</span><span class="nf">handle_call</span><span class="p">({</span><span class="o">?</span><span class="nv">FLUSH</span><span class="p">,</span> <span class="o"><<</span><span class="mi">0</span><span class="p">:</span><span class="mi">32</span><span class="o">>></span><span class="p">,</span> <span class="o"><<>></span><span class="p">,</span> <span class="o"><<>></span><span class="p">,</span> <span class="p">_</span><span class="nv">CAS</span><span class="p">},</span>
<span class="p">_</span><span class="nv">From</span><span class="p">,</span> <span class="p">_</span><span class="nv">State</span><span class="p">)</span> <span class="o">-></span>
<span class="p">{</span><span class="n">reply</span><span class="p">,</span> <span class="nl">#mc_response</span><span class="p">{},</span> <span class="nl">#mc_state</span><span class="p">{}};</span>
<span class="c">% Delayed flush. Keep the current state and schedule a
% flush to occur (via handle_info) in the future.
</span><span class="nf">handle_call</span><span class="p">({</span><span class="o">?</span><span class="nv">FLUSH</span><span class="p">,</span> <span class="o"><<</span><span class="nv">Delay</span><span class="p">:</span><span class="mi">32</span><span class="o">>></span><span class="p">,</span> <span class="o"><<>></span><span class="p">,</span> <span class="o"><<>></span><span class="p">,</span> <span class="p">_</span><span class="nv">CAS</span><span class="p">},</span>
<span class="p">_</span><span class="nv">From</span><span class="p">,</span> <span class="nv">State</span><span class="p">)</span> <span class="o">-></span>
<span class="nn">erlang</span><span class="p">:</span><span class="nb">send_after</span><span class="p">(</span><span class="nv">Delay</span> <span class="o">*</span> <span class="mi">1000</span><span class="p">,</span> <span class="nf">self</span><span class="p">(),</span> <span class="n">flush</span><span class="p">),</span>
<span class="p">{</span><span class="n">reply</span><span class="p">,</span> <span class="nl">#mc_response</span><span class="p">{},</span> <span class="nv">State</span><span class="p">};</span>
<span class="err">%</span> <span class="nv">More</span> <span class="n">stuff</span> <span class="nv">Follows</span></code></pre></figure>
<p>That’s pretty much it. Even if nobody uses this code, it’s useful to
me as a protocol reference since it’s easier to read than even the
<a href="http://cloud.github.com/downloads/memcached/memcached/protocol-binary.txt">binary specification</a>.</p>
spymemcached Optimizations2009-09-23T00:00:00+00:00http://dustin.github.com/2009/09/23/spymemcached-optimizations<p>I got around to pushing out a new RC of <a href="http://code.google.com/p/spymemcached/">spymemcached</a>
today. It’s been a while, but I’m glad I got around to it.</p>
<p>The <a href="http://groups.google.com/group/spymemcached/browse_thread/thread/9d93e5658e813c29">announcement</a> has the release notes (also in the
tag), but there is a particular optimization I’ve been thinking about
for a while, and would like to go over here somewhere below.</p>
<p>But first, I’ll frame it with a bit of memcached protocol fundamentals.</p>
<h2 id="introduction-to-quiet-operations">Introduction to Quiet Operations</h2>
<div>
<img class="floatleft" alt="Ask, don't tell." src="/images/memcached-sparse-get.png" />
</div>
<p>Back when we were initially designing the binary protocol, we were
considering how we’d handle the multi-gets. We went through several
proposals until we realized that the actual essense of multi-get model
was really just a class of operation that allowed us to infer some of
the responses.</p>
<p>The above diagram shows a simple case of a multi-get. We ask for the
values behind four keys. The server sends us responses for two of
those keys and then says it’s done. In a client, we can safely assume
that it just didn’t have the other two. It doesn’t need to actually
tell us that.</p>
<p>So in the binary protocol model, we just made a type of command that
didn’t respond with “uninteresting responses.” It’s easy to see how
in a <code class="language-plaintext highlighter-rouge">get</code> operation, the <code class="language-plaintext highlighter-rouge">not found</code> response is uninteresting as we
can infer it.</p>
<h2 id="other-quiet-operations">Other Quiet Operations</h2>
<p>Shortly before the actual 1.4.0 release of memcached, we defined
semantics for all “quiet” operations in such a way that allowed us to
maximize efficiency without compromising correctness.</p>
<p>The <a href="http://code.google.com/p/memcached/wiki/MemcacheBinaryProtocol">binary protocol definition</a> goes through these in
tremendous detail, but those familiar with the Unix philosophy will
probably find such things intuitive.</p>
<p>For example, in Unix, the <code class="language-plaintext highlighter-rouge">rm</code> command does not print out any output
on success. If it completes and didn’t say otherwise, you can assume
it was successful.</p>
<p>Similarly, a quiet <code class="language-plaintext highlighter-rouge">delete</code> operation doesn’t need to tell us that it
successfully deleted something. That’s its job. We want to know when
it fails to do it.</p>
<h2 id="optimizing-with-a-quiet-set">Optimizing with a Quiet Set</h2>
<div>
<img class="floatright" alt="Look. Faster!" src="/images/multiset-perf.png" />
</div>
<p>The optimization I was interested in was making a multi-set type
operation that worked similarly to the multi-get functionality. After
struggling with what such an API might look like, I finally decided
that the right thing to do is not change the API at all.</p>
<p>Instead, I do something similar to <a href="http://code.google.com/p/spymemcached/wiki/Optimizations">multiget escalation</a>
– an optimization that’s been part of <a href="http://code.google.com/p/spymemcached/">spymemcached</a> for
a long time now. If many threads are pushing sets in (or even a
single-thread since the typical use-case of set is async), the
packetization of these commands escalates a sequence of similar
commands into a single sparse operation working on all of the items
together.</p>
<p>While YMMV, my cache loader test ran consistently twice as fast.</p>
<p>Previously, one million requests would require the client to process
one million responses. Now, one million requests (assume none fail)
will require the client to process one tiny response.</p>
<p>If any <em>do</em> fail, the respective callers will, of course, be notified.
The ones that don’t fail receive synthetic callbacks as the client
infers their success.</p>
<h2 id="what-you-need-to-do">What You Need to Do</h2>
<p>If you’re using <a href="http://code.google.com/p/spymemcached/">spymemcached</a>, upgrade and you get the
optimizations.</p>
<p>If you’re a client author, see how much better things are for your
users as you make broader use of quiet operations of the binary
protocol.</p>
Tornado on Twisted2009-09-12T00:00:00+00:00http://dustin.github.com/2009/09/12/tornado<div>
<a href="http://www.nataliedee.com/index.php?date=050906"><img class="floatright" alt="Twisted Tornados" src="/images/snake-tornado.png" /></a>
</div>
<p>So what’s this about <a href="http://www.tornadoweb.org/">tornado</a>?</p>
<p>The <a href="http://friendfeed.com/">friendfeed</a> guys created an awesome web site with
what was obviously (from the outside) quite awesome technology. A
couple days ago, they released the <a href="http://bret.appspot.com/entry/tornado-web-server">the technology</a>
behind the site.</p>
<h2 id="the-problems">The Problems</h2>
<p>Tornado really is two different things:</p>
<ol>
<li>A great framework for building web sites.</li>
<li>A low-level networking toolkit.</li>
</ol>
<p>Most of us who use <a href="http://twistedmatrix.com/">twisted</a> were quite surprised to find out
that rather than using twisted’s awesome networking core, they
reimplemented a bunch of it. Moreover, they kind of had really vague
negative things to say about twisted. It’s not clear what problem
they had with it, but as stated the logic kind of fails:</p>
<ol>
<li>Twisted doesn’t have a good web framework.</li>
<li>No major web sites run on twisted.</li>
<li>So… we’re going to build something completely from scratch.</li>
</ol>
<p>Now, I certainly build stuff from scratch unnecessarily all the time,
and I also don’t want to seem unthankful for the free gift brought to
us from our friendfeed friends, but in its current state, it’s a
technological island.</p>
<p>On one hand, you have low-level missing pieces. From the bottom,
you’ve got multiplexing implementations. twisted has been around a
while, so it supports native multiplexors including those for select,
poll, epoll, kqueue, CoreFoundation, wxPython, win32, gtk, glib, qt,
and probably more. While some of them are less interesting than
others, that’s a lot of catching up to do. Some user already began
adding <a href="http://github.com/rphillips/tornado/commit/41aa49a8dcfe4f5fa91dfe1da9e05797d3397d25">kqueue support</a>, but it’s more of an example of
the kinds of things that you don’t get for free.</p>
<p>At the high end, there is an incredible selection of protocol
implementations from twisted you just can’t use. If you’re building a
web site on an asynchronous framework, you don’t ever want to block to
get further data. So whatever your networking framework, you need it
to have a sane way of asychronously communicating to all of your
dependencies.</p>
<p>They wrote an http client, though there are things I do with the
twisted http client that I don’t see a way to do with the tornado
client such as <a href="http://github.com/dustin/twitty-twister/blob/master/lib/twitter.py#L331">parse infinite xml streams</a> from http
responses.</p>
<h2 id="the-solution">The Solution</h2>
<p>So instead of just generally being frustrated, I thought I’d see what
it’d take to rip out the stuff that was reimplemented and use just
core twisted.</p>
<p>The original tornado code doesn’t have much in the way of tests, so
it’s currently in the “appears to work” state, but this is what got me
there:</p>
<p><img src="/images/tornado-diffstat.png" alt="Diffstat" /></p>
<p>That is, with an offset of -1,297 lines of code, it can be observed to
work for the cases I tried, although I’m sure there are still lines
that need to be deleted before everything works.</p>
<p>The good news is that this means that you get all the richness of
twisted and the good parts of tornado combined.</p>
<p>Everybody wins.</p>
<p>So go grab <a href="http://github.com/dustin/tornado">the code</a> and see if it works in your app, or
send me some patches for parts you were able to explore a bit more
deeply.</p>
Buildbot and Git Repositories2009-09-06T00:00:00+00:00http://dustin.github.com/2009/09/06/buildbot-git<h1 id="buildbot-and-git-repositories">Buildbot and Git Repositories</h1>
<div>
<img class="floatright" src="/images/RefuseToApologize.png" alt="I refuse to apologize." />
</div>
<p>In a recent conversation with the GitHub guys, I was talking about how
my <a href="http://buildbot.net/">buildbot</a> setup was hitting GitHub and how a recent
filesystem glitch of theirs caused my screen to turn red with
<a href="http://growl.info/">growl</a> alerts from <a href="http://code.google.com/p/buildwatch/">buildwatch</a>.</p>
<p>The response was a tongue-in-cheek “I refuse to
apologize.”</p>
<p>The thing is, that response is absolutely the right one. This is
distributed revision control. Why did I have a screen full of growl
alerts because of a failure of a filesystem completely unrelated to
what I was doing?</p>
<p>I was relying on GitHub to be highly and quickly available to my
seventeen (and growing in number) buildbot slaves for this project.</p>
<p><em>Most</em> of the time, there’s nothing for them to actually get from a
centralized revision control – quite simply, they were asking for
information that they had cryptographically verifiable assurance that
they already had.</p>
<p>Today I made a <a href="http://github.com/dustin/buildbot/commit/fabad2476cebc077d58c9293ce389d465648b019">small change</a> to buildbot that prevents the
slaves from ever talking to any network service to pick out a
reference version in our most common use cases, thus realigning myself
with the thing that initially sold me on GitHub’s service: It enhances
collaboration without causing me to be dependent on the service.</p>
<p>For this failure, I am thankful. This new code will always be faster
and more reliable for the common case even when GitHub works
absolutely flawlessly.</p>
<h2 id="see-also">See Also</h2>
<p><a href="http://ozmm.org/posts/when_github_goes_down.html">When GitHub Goes Down</a>.</p>
Memcached 1.4.02009-07-16T00:00:00+00:00http://dustin.github.com/2009/07/16/memcached-1.4<h1 id="memcached-140">Memcached 1.4.0</h1>
<div>
<img class="floatright" src="/images/cream-med.png" alt="c.r.e.a.m" />
</div>
<p>I’m a bit late to the blogging party here, but we finally released
memcached 1.4.0. Check out <a href="http://code.google.com/p/memcached/wiki/ReleaseNotes140">the release notes</a> for more
details.</p>
<p>The release notes cover quite a lot of the interesting stuff, but they
don’t properly reflect the time and effort that went into making this
all happen.</p>
<p>There are a lot of bug fixes, as one might expect after some time. A
lot of testing has shown that performance is better pretty much all
around, but very few people have ever seen memcached be a performance
bottleneck in their applications, so that’s not too exciting.</p>
<p>The biggest part of this release, however, is something I’ve been
working on for about two years: <a href="http://cloud.github.com/downloads/memcached/memcached/protocol-binary.txt">the binary protocol</a>.</p>
<h2 id="the-binary-protocol">The Binary Protocol</h2>
<p>So what’s the big deal about the binary protocol?</p>
<p>The most obvious thing for protocol implementors is that it’s now
<em>really</em> easy to parse the protocol. After reading a fixed-size
header, a low-level packet processor can figure out where to dispatch
the input and split it into all of its major components (key, value,
opaque, cas, extras, etc…).</p>
<p>That’s great for the (small) number of developers who write servers
and clients, but what about random people out there who just want to
use memcached? Semantic enhancements in the new protocol allow us to
build some really cool stuff.</p>
<p>The first example of such a thing is Trond’s
<a href="http://blogs.sun.com/trond/date/20090625">replication</a> functionality for <a href="http://launchpad.net/libmemcached">libmemcached</a>.
We now have a clean fire-and-mostly-forget protocol semantics that
allows for improvements like efficient client-side replication. It
also makes it safe to make bulk-loaders (even with CAS).</p>
<h2 id="go-try-it">Go Try It</h2>
<p>We’ve run tons of tests, others have run tests, there’ve been various
deployments large and small, but if you’re running something older,
it’s your turn.</p>
<p>We work hard to make sure that the development versions work on all
platforms we can find anyone to care about. Each change is built and
tested on all supported platforms before the change is accepted into
our master branch.</p>
<p>Do note that 1.4.0 has some build issues on OpenBSD, but someone
graciously donated a builder to our buildbot farm so they’re all
cleared up for 1.4.1 (which is planned for later this month).</p>
<p>In the meantime, there are several ways to pick it up:</p>
<h3 id="packages">Packages</h3>
<p>Package systems are slow to pick up… anything it seems. If your
system’s package manager is shipping memcached 1.4.0, please let me
know.</p>
<p>In the meantime…</p>
<h3 id="use-the-source">Use the Source</h3>
<p>You can download <a href="http://memcached.googlecode.com/files/memcached-1.4.0.tar.gz">the source distribution</a> from the google code
<a href="http://code.google.com/p/memcached/downloads/list">download site</a>.</p>
<p>Building and installing is quite simple:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>./configure
sudo make install
</code></pre></div></div>
<p>Then just run <code class="language-plaintext highlighter-rouge">/usr/local/bin/memcached</code> whichever way suits your
fancy. Personally, I like upstart on Linux, launchd on OS X, smf on
Solaris, etc…</p>
<h3 id="deploying-on-amazon-ec2aws">Deploying on Amazon EC2/AWS?</h3>
<p>I’ve put together some Amazon AMIs that are production ready and
ridiculously simple to get going.</p>
<p>Each AMI allocates all but 512MB of RAM on the system to memcached and
just starts up happy and running. These images are based on Ubuntu
9.04 and have an upstart config for the actual daemon execution so if
we somehow have some kind of crashing bug, they’ll automatically and
instantly restart.</p>
<p>Depending on your needs, you can select one of the following:</p>
<h4 id="us">US</h4>
<p>I’ve assembled a 32-bit AMI (<code class="language-plaintext highlighter-rouge">ami-39c52450</code>) for small instances, and
a 64-bit AMI (<code class="language-plaintext highlighter-rouge">ami-1fc52476</code>) for large instances. They show up as
the following:</p>
<ul>
<li>ami-39c52450 - northscale/community-memcached-1.4.0-i386.manifest.xml</li>
<li>ami-1fc52476 - northscale/community-memcached-1.4.0-x86_64.manifest.xml</li>
</ul>
<p>For example, using the ec2 command-line tools can start an extra large
64-bit instance with the following invocation:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>ec2-run-instances ami-1fc52476 --instance-type m1.xlarge
</code></pre></div></div>
<p>After this instance comes up, you’ll find memcached 1.4.0 listening on
port 11211 with about 15GB of RAM at your disposal.</p>
<p>No maintenance should be necessary, but <code class="language-plaintext highlighter-rouge">ec2-run-instance</code>’s <code class="language-plaintext highlighter-rouge">-k</code>
parameter for supplying a root ssh key is still honored in case you
want to still look around.</p>
<h4 id="eu">EU</h4>
<p>There are European versions of the same images as <code class="language-plaintext highlighter-rouge">ami-818ba3f5</code>
for 32-bit and <code class="language-plaintext highlighter-rouge">ami-838ba3f7</code> for 64-bit.</p>
<ul>
<li>ami-818ba3f5 - northscale-eu/community-memcached-1.4.0-i386.manifest.xml</li>
<li>ami-838ba3f7 - northscale-eu/community-memcached-1.4.0-x86_64.manifest.xml</li>
</ul>
Project Skyscraper2009-07-11T00:00:00+00:00http://dustin.github.com/2009/07/11/skyscraper<h1 id="project-skyscraper">Project Skyscraper</h1>
<p>It occurred to me that there’s a lot of value in building xmpp
services – much like web services, but using existing connections and
xmpp instead of http.</p>
<p>In collaboration with <a href="http://github.com/ga2arch">ga2arch</a>, I launched an xmpp service
called skyscraper.im. This has actually been running for a while now,
but I’ve been too caught up in writing code to write anything about
code.</p>
<h2 id="translateskyscraperim">translate.skyscraper.im</h2>
<div>
<img class="floatright" src="/images/skyscraper.png" alt="skyscraper" />
</div>
<p>The first part of this service is an xmpp <a href="http://xmpp.org/extensions/xep-0050.html">adhoc</a> interface to
google translate. It actually does support IM, but that’s incidental,
the real value is in the adhoc interface.</p>
<p>If you’re unfamiliar with xmpp adhoc, you can think of it much like
CGI, but using xmpp as a transport. You take a bunch of simple
key/multi-value pairs and send them to a resource somewhere, and it
sends you something back. The nice thing about xmpp, though, is that
the mechanism for determining what things exist and what parameters
they take are very programmatically accessible.</p>
<p>You can discover available commands through <code class="language-plaintext highlighter-rouge">translate.skyscraper.im</code>
as shown in <a href="http://www.vimeo.com/5558475">this video</a>, but I’ll just tell you what
it’ll tell you:</p>
<h3 id="the-input">The Input</h3>
<p>There is one field called <code class="language-plaintext highlighter-rouge">in</code> which is the input language in the form
of a two-character language code. You may have only one of these (it
is of type <code class="language-plaintext highlighter-rouge">list-single</code>).</p>
<p>There is one field called <code class="language-plaintext highlighter-rouge">out</code> which is the output language and is
also in the form of a two-character language code. You can have as
many of these as you like (it is of type <code class="language-plaintext highlighter-rouge">list-multi</code>).</p>
<p>Finally there’s a field called <code class="language-plaintext highlighter-rouge">text</code> which is the stuff you want to
translate. You may only have one of these.</p>
<h3 id="the-output">The Output</h3>
<p>The response is a form much like the one you sent it, the keys are
language codes and the values are the text translated in that
language.</p>
<p>Note that you will not receive more language translations than you
asked for, but you may receive fewer in the case where the upstream
translation service can’t perform such a translation.</p>
<p>The obvious benefit here over doing it yourself is that you get full
translations all at the same time without having to do any kind of
coordination as things are completing (i.e., I do that for you).</p>
<h2 id="conferenceskyscraperim">conference.skyscraper.im</h2>
<div>
<img class="floatright" src="/images/skyscraper-chat.png" alt="skyscraper chat" />
</div>
<p>A fun thing built atop the translate component is the skyscraper muc
– an xmpp multi-user chat with automatic translation.</p>
<p>What this means is that you can have several people enter a room with
no room in common, all speaking and reading their native language.</p>
<p>Of course, the dream is limited by the translation service, but it
<em>does</em> work within the reasonable limits.</p>
<p>If you’d like to try it out, find a friend who speaks another language
and both join a chat room at <code class="language-plaintext highlighter-rouge">conference.skyscraper.im</code>. Start by
each of you telling it your respective languages (e.g. <code class="language-plaintext highlighter-rouge">/lang en</code>) and
then talk.</p>
<p>I’ve spent very little time on this, so I imagine it falls apart in
all kinds of places, but it was <em>really</em> easy to get going with the
translate service from above, and as it’s an xmpp server component, it
does all this with just one file descriptor and the necessary state to
keep up with who’s in what room and what translations are outstanding.</p>
My Github Anniversary (Sort Of)2009-03-01T00:00:00+00:00http://dustin.github.com/2009/03/01/github-anniversary<h1 id="my-github-anniversary-sort-of">My Github Anniversary (Sort Of)</h1>
<div>
<img alt="[anniversary]" class="floatright" src="http://img.skitch.com/20090301-jqf9yrkfniqf2ysaa88ebshwqe.png" />
</div>
<p>I joined <a href="http://github.com/">github</a> about a year ago today. Kind of. It was
actually February 29th, but there isn’t one of those this year, so I’m
going to have to wait a few more years before I can properly have an
anniversary.</p>
<p>Between the time I joined and the time I typed this line, I’ve
generated 206 pages of activity (<a href="http://github.com/dustin?page=174">174 pages public</a>).</p>
<p>By the end of my first day, I had migrated two <a href="http://github.com/dustin/java-memcached-client">java</a>
<a href="http://github.com/dustin/photo">projects</a> over from mercurial, converted a <a href="http://github.com/dustin/ruby-freebase">ruby
project</a> from subversion, had that repo forked, watched
another <a href="http://github.com/mojombo/god">ruby project</a>, started a <a href="http://github.com/dustin/buildwatch">new objective c
project</a>, wrote some new code for some of my projects and
pushed it and invited a <a href="http://github.com/chriseppstein">couple</a> of <a href="http://github.com/verbal">friends</a> (both of
whom now share my leap-year-only start date) and added them to a
couple of my projects.</p>
<p><a href="http://github.com/chriseppstein">Chris</a> became somewhat a github evangelist and made some
really <a href="http://github.com/chriseppstein/compass">cool</a> <a href="http://github.com/chriseppstein/freebase">stuff</a> stuff there (some if it’s
cooler than most people can comprehend). <a href="http://github.com/verbal">Ian</a> throws awesome
parties (I’ll eventually make him give me code).</p>
<p>At the point where I started using github, I’d probably been a
(somewhat casual) git user for about two weeks. git is great, but the
documentation and tutorials were more about laying out an infinitely
complex decision tree – that is, git itself is easy to do anything
with, but you can do a lot with it, so it comes across as
unnecessarily complex.</p>
<p>Github has been really good about making really common paths really
easy so that you naturally fall into workflows that minimize the work
required to contribute to open source projects down to the point where
you can clone a repo, branch, edit some stuff, and notify the
maintainer of a project in just a few clicks on the web site.</p>
<p>Overall, it’s been a pretty <a href="http://calendaraboutnothing.com/~dustin">good year</a>. Just three more until
my real anniversary.</p>
Making Use of Caps Lock2009-02-09T00:00:00+00:00http://dustin.github.com/2009/02/09/caps-lock<h1 id="making-use-of-caps-lock">Making Use of Caps Lock</h1>
<div>
<img src="/images/capslock-pref.png" alt="caps lock" class="floatright" />
</div>
<p>If you’re like me (and who isn’t), the caps lock key is an annoying
waste of plastic. Its only value seems to be to type things to offend
people. Luckily, most operating systems allow you to map it to
control, or another useful key.</p>
<p>As a fairly new emacs user (and a long-term shell user), having a
control key near where my fingers already makes many things far more
accessible to me. Highly recommended.</p>
<p>But there’s another thing that the caps lock provides that quickly
moves from annoyance to useful feature:</p>
<div>
<img src="/images/capslock-key.jpg" alt="caps lock" class="floatleft" />
</div>
<p>Just about every keyboard ever made has a caps lock indicator. Such a
wonderful thing when used correctly.</p>
<p>Amit Singh over at google had a blog post about
<a href="http://googlemac.blogspot.com/2008/04/manipulating-keyboard-leds-through.html">manipulating keyboard LEDs</a> which inspired me to add this feature
to my <a href="http://code.google.com/p/buildwatch/">buildwatch</a> app pretty much immediately.</p>
<p>Due to a fairly dumb bug I fixed today, it hasn’t been working (and I
wasn’t paying attention to it anyway), but now, when anyone does a
build against my build farm and the build breaks, my keyboard light
will come on.</p>
<p>Sort of makes me want to write some bad code.</p>
Buildbot2009-01-30T00:00:00+00:00http://dustin.github.com/2009/01/30/buildbot<h1 id="buildbot">Buildbot</h1>
<p>I’ve used a few different continous integration systems, but
<a href="http://buildbot.net/">buildbot</a> has been my favorite for quite a while. It’s got a
really nice architecture, a great codebase, and all the tools I need.</p>
<div>
<img class="floatright" alt="buildwatch" src="/images/buildbot.png" />
</div>
<p>Most of these things seem to assume that if it builds anywhere, you’ve
done your job. buildbot assumes there may be a relatively large
number of build workers and an even larger number of configurations.</p>
<p>For example, I’m building out a buildbot configuration for a project
I’m working on now for some portable software that seems to be most
popular on Linux, possibly followed by OS X. However, depending on
the distribution, toolchain, architecture, and compile-time options,
some things just don’t work correctly. I also use it on FreeBSD, but
having added a slave for FreeBSD, I found a small compiler error.</p>
<h2 id="tbyb">TBYB</h2>
<p>Most CI systems are all about telling you when you’ve committed code
that breaks the build for other people. Isn’t it rather late by that
point?</p>
<p>buildbot has a <code class="language-plaintext highlighter-rouge">try</code> command that allows you run a complete build
across whichever nodes you want (or all) <em>before</em> making your code
available to anyone else.</p>
<p>One of the guys I’m working with on this project does most of his work
in Solaris. He wrote some code, tested his code, and sent me a
patch. I committed his patch to my local git repo, but before pushing
it, ran <code class="language-plaintext highlighter-rouge">buildbot try</code> to make sure nothing weird happened. There
were two different problems that caused build and/or test failures on
every OS that wasn’t Solaris.</p>
<p>I was able to fix up his changes so that they worked everywhere, and
they never actually made it into a public tree in their broken form.</p>
<h2 id="the-code">The Code</h2>
<p>buildbot’s <a href="http://github.com/djmitche/buildbot">codebase</a> has some very robust plumbing, and it seems
to support just about anything you might want to do (which other
systems allow you subscribe to the tail of the current step’s log in
realtime without having access to the slave?).</p>
<p>I’ve had to make some changes to get some features working as I
expect, or fixing bug in edge cases, though.</p>
<p>I’ve been doing some work lately with the <code class="language-plaintext highlighter-rouge">git</code> support in <code class="language-plaintext highlighter-rouge">try</code>.
Rather than repeating myself, you can see what I’ve done in my portion
of the changelog and imagine how much better your project would be if
every potential change could be tested across all your supported
platforms before you publish.</p>
<pre style="font-size: smaller">
commit dfb18e6c177d490da9dcab29e431eff22cfedfec
Author: Dustin Sallings <dustin@spy.net>
Date: Wed Jan 28 20:26:00 2009 -0800
Allow users to specify the remote git branch.
This allows for a case where someone has a repository that tracks
someone else's repository, has arbitrary local branches, but wants to
run tries with the delta from the reference repository (i.e. the one
the master knows about) to the local changes.
Without this, it's likely the reference repository will not have the
necessary objects to pull down a base revision to be able to apply
patches for the try to succeed.
This also ensures that the current client's view of the reference
repository is honored. That is, if the reference repository has moved
forward, the trier's current tip of the remote is used to compute the
delta, and that's sent along as the baserev.
commit 38a9c7fc719b44e2cdfa47884182da7128b369d2
Author: Dustin Sallings <dustin@spy.net>
Date: Wed Jan 28 16:30:08 2009 -0800
Added --dry-run (-n) support to buildbot try.
Need to be able to try try when I just want to know what it's even
going to consider doing.
commit f43143835cba3ca5963e07874da17c1416a031c2
Author: Dustin Sallings <dustin@spy.net>
Date: Wed Jan 28 08:34:04 2009 -0800
Refactored try buildName validation for reuse.
commit a88238cae5000c3481877aa354e3c76fc45770b8
Author: Dustin Sallings <dustin@spy.net>
Date: Wed Jan 28 08:25:52 2009 -0800
Don't require a list of builders for buildbot try.
This maintains the current restrictions around builder lists that
prevent one from trying a build that isn't in the list, but allows the
user to delegate the selection to the server by not listing the
builders at all.
I want my users to always try their builds on every build
configuration, but I don't want to be sending out buildbot options all
the time.
commit 99240ada38677a143971fe390beb714c3017c20b
Author: Dustin Sallings <dustin@spy.net>
Date: Sun Jan 25 10:47:57 2009 -0800
git_buildbot should show the author, not the committer
When I'm looking at my waterfall, I'd like to see the names of the
people who wrote code, not just mine because I happened to have
cherry-picked or am'd a bunch of changes.
commit 2c6865d83e967ca135acd3810e08af2dfab727b3
Author: Dustin Sallings <dustin@spy.net>
Date: Wed Jan 21 20:23:35 2009 -0800
Look at the remote tracking branch in git for buildbot try.
This allows us to try committed, but not pushed code.
commit a079d84d4056dbf5ab3489cb7f2f8f0e20d91b87
Author: Dustin Sallings <dustin@spy.net>
Date: Thu Jan 22 09:59:59 2009 -0800
Try to reclobber on retry.
On a failed git update in clobber mode, I was getting the following
error on the second try:
exceptions.OSError: [Errno 17] File exists: '/path/to/build'
It seems that the clobber only occurs once, and any error that happens
during the checkout should redo the clobber.
commit 6c36579a63b58bc986ec56e0272362038be08112
Author: Dustin Sallings <dustin@spy.net>
Date: Wed Nov 19 04:56:38 2008 +0800
Get rid of git- commands in git_buildbot.
Signed-off-by: Dustin J. Mitchell <dustin@zmanda.com>
commit ac70a83fa05c2b1b31dd9411ffc28876fb9e9f20
Author: Dustin Sallings <dustin@spy.net>
Date: Sat Apr 19 08:40:20 2008 +0800
Send merge changes from git.
Signed-off-by: Dustin J. Mitchell <dustin@zmanda.com>
</pre>
Publishing Changelogs2009-01-17T00:00:00+00:00http://dustin.github.com/2009/01/17/changelog<h1 id="publishing-changelogs">Publishing Changelogs</h1>
<p>A user filed a bug against my <a href="http://code.google.com/p/spymemcached/">memcached client</a> because he
couldn’t find the changelog and wanted to know what went into the new
version.</p>
<p>I have a decent structure around releases, especially with this
project. I tag it and write a good summary of changes in the tag
including an abbreviated shortlog output, then I send the same out to
the mailing list.</p>
<p>Somehow, I expected anyone not on the mailing list to just dig through
my tags to find out what’s changed. I suppose that’s asking quite a
bit.</p>
<p>Since I’ve been keeping good information in my tags since moving over
to git (which actually has proper tag objects), I’ve found it quite
easy to automate this process. My new <a href="http://github.com/dustin/bindir/blob/master/git-htmlchangelog">git htmlchangelog</a>
takes a list of tags and generates a reasonable changelog
automatically from this.</p>
<p>For example, the following command:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>git htmlchangelog `git tag | egrep -v pre\|rc` > changelog.html
</code></pre></div></div>
<p>created <a href="http://dustin.github.com/java-memcached-client/changelog.html">the changelog</a> for my memcached client.</p>
Visualizing Git Contributors2009-01-16T00:00:00+00:00http://dustin.github.com/2009/01/16/visualizing-contributors<p>I was looking for something quick to do today, so I started drawing
<a href="http://github.com/dustin/bindir/blob/master/git-contributors">pie charts</a> showing who commits to various projects.</p>
<p>Pie charts are particularly terrible for communicating something
useful to people, but they kind of look nice, so whatever.</p>
<p>Here are some examples:</p>
<h2 id="linux">Linux</h2>
<table>
<tbody>
<tr>
<td>![Linux](http://chart.apis.google.com/chart?cht=p&chs=600x300&chd=s:CBBBBB2&chl=Linus</td>
<td>Al</td>
<td>David</td>
<td>Adrian</td>
<td>Ralf</td>
<td>Jeff</td>
<td>Other)</td>
</tr>
</tbody>
</table>
<h2 id="git">Git</h2>
<table>
<tbody>
<tr>
<td>![Git](http://chart.apis.google.com/chart?cht=p&chs=600x300&chd=s:WFECBBa&chl=Junio</td>
<td>Shawn</td>
<td>Linus</td>
<td>Johannes</td>
<td>Eric</td>
<td>Jakub</td>
<td>Other)</td>
</tr>
</tbody>
</table>
<h2 id="memcached">Memcached</h2>
<table>
<tbody>
<tr>
<td>![Memcached](http://chart.apis.google.com/chart?cht=p&chs=600x300&chd=s:THGGEEP&chl=Brad</td>
<td>dormando</td>
<td>Paul</td>
<td>Dustin</td>
<td>Trond</td>
<td>Toru</td>
<td>Other)</td>
</tr>
</tbody>
</table>
<h2 id="emacs">Emacs</h2>
<table>
<tbody>
<tr>
<td>![Emacs](http://chart.apis.google.com/chart?cht=p&chs=600x300&chd=s:OEEDDDe&chl=Richard</td>
<td>Gerd</td>
<td>Eli</td>
<td>Stefan</td>
<td>Kenichi</td>
<td>Glenn</td>
<td>Other)</td>
</tr>
</tbody>
</table>
<h2 id="rails">Rails</h2>
<table>
<tbody>
<tr>
<td>![Rails](http://chart.apis.google.com/chart?cht=p&chs=600x300&chd=s:VPDDCCO&chl=David</td>
<td>Jeremy</td>
<td>Michael</td>
<td>Rick</td>
<td>Jamis</td>
<td>Joshua</td>
<td>Other)</td>
</tr>
</tbody>
</table>
Git Timecard2009-01-11T00:00:00+00:00http://dustin.github.com/2009/01/11/timecard<h1 id="git-timecard">Git Timecard</h1>
<p>I really like <a href="http://github.com/blog/159-one-more-thing">github’s punch card</a> feature. It’s a nice
way to quickly see when a project is worked on.</p>
<p>However, it’s very limited. You can only see everyone’s work on the
master branch. I have <em>lots</em> of other ways I want to look at my
repos (including some that aren’t on github).</p>
<p>So I wrote <a href="http://github.com/dustin/bindir/tree/master/git-timecard">my own</a>.</p>
<p>Here are the neat kinds of things I can do now:</p>
<h2 id="punch-card-of-my-work-repo">Punch Card of my Work Repo</h2>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>~/work-project/ % git timecard
</code></pre></div></div>
<table>
<tbody>
<tr>
<td>![work](http://chart.apis.google.com/chart?cht=s&chs=800x300&chd=e:CkFIHrKPMzPXR7UeXCZmcKeuhRj1mZo9rhuEwozM1w4U639bCkFIHrKPMzPXR7UeXCZmcKeuhRj1mZo9rhuEwozM1w4U639bCkFIHrKPMzPXR7UeXCZmcKeuhRj1mZo9rhuEwozM1w4U639bCkFIHrKPMzPXR7UeXCZmcKeuhRj1mZo9rhuEwozM1w4U639bCkFIHrKPMzPXR7UeXCZmcKeuhRj1mZo9rhuEwozM1w4U639bCkFIHrKPMzPXR7UeXCZmcKeuhRj1mZo9rhuEwozM1w4U639bCkFIHrKPMzPXR7UeXCZmcKeuhRj1mZo9rhuEwozM1w4U639bCkFIHrKPMzPXR7UeXCZmcKeuhRj1mZo9rhuEwozM1w4U639b,IAIAIAIAIAIAIAIAIAIAIAIAIAIAIAIAIAIAIAIAIAIAIAIAQAQAQAQAQAQAQAQAQAQAQAQAQAQAQAQAQAQAQAQAQAQAQAQAYAYAYAYAYAYAYAYAYAYAYAYAYAYAYAYAYAYAYAYAYAYAYAYAgAgAgAgAgAgAgAgAgAgAgAgAgAgAgAgAgAgAgAgAgAgAgAgAn.n.n.n.n.n.n.n.n.n.n.n.n.n.n.n.n.n.n.n.n.n.n.n.v.v.v.v.v.v.v.v.v.v.v.v.v.v.v.v.v.v.v.v.v.v.v.v.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3………………………………………….,BoAeAAAAAAAAAAAtDQDtGuE4IWMhIlJ-GfMECyGuH4L1L1KNFGGRAPAeAAAeBZCGLmeYgs1GjeiyijnavTbmMEHqIHG9IHGuELB3APAAAAAPAtCjGfcwdrpCZ-fTrXzP8SozSUFzGRCyI0H4CGB3AeAeAeB3AeCGEpYlkLjegdgsv.59..okTBJ-JvHMMhJvD8DfB3AAAAAAAADfHqabX4rIiFkLpRze28drVVQsLXI0J-JREpCUAAAAAPAAAACyGRVkchrXh2jPtc4kw7WCGfDBBoCGCjDtAPBKAAAAAAAAAABoBKD8GCHqGRIHGRGCIHDBGRELBoAeDQCjAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA&chxt=x,y&chxl=0:</td>
<td> </td>
<td>0</td>
<td>1</td>
<td>2</td>
<td>3</td>
<td>4</td>
<td>5</td>
<td>6</td>
<td>7</td>
<td>8</td>
<td>9</td>
<td>10</td>
<td>11</td>
<td>12</td>
<td>13</td>
<td>14</td>
<td>15</td>
<td>16</td>
<td>17</td>
<td>18</td>
<td>19</td>
<td>20</td>
<td>21</td>
<td>22</td>
<td>23</td>
<td> </td>
<td>1:</td>
<td> </td>
<td>Sun</td>
<td>Mon</td>
<td>Tue</td>
<td>Wed</td>
<td>Thu</td>
<td>Fri</td>
<td>Sat</td>
<td>&chm=o,333333,1,1.0,25,0&chds=-1,24,-1,7,0,20)</td>
</tr>
</tbody>
</table>
<h2 id="alternate-branch">Alternate Branch</h2>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>~/memcached % git timecard rewritten-bin
</code></pre></div></div>
<table>
<tbody>
<tr>
<td>![rewritten-bin](http://chart.apis.google.com/chart?cht=s&chs=800x300&chd=e:CkFIHrKPMzPXR7UeXCZmcKeuhRj1mZo9rhuEwozM1w4U639bCkFIHrKPMzPXR7UeXCZmcKeuhRj1mZo9rhuEwozM1w4U639bCkFIHrKPMzPXR7UeXCZmcKeuhRj1mZo9rhuEwozM1w4U639bCkFIHrKPMzPXR7UeXCZmcKeuhRj1mZo9rhuEwozM1w4U639bCkFIHrKPMzPXR7UeXCZmcKeuhRj1mZo9rhuEwozM1w4U639bCkFIHrKPMzPXR7UeXCZmcKeuhRj1mZo9rhuEwozM1w4U639bCkFIHrKPMzPXR7UeXCZmcKeuhRj1mZo9rhuEwozM1w4U639bCkFIHrKPMzPXR7UeXCZmcKeuhRj1mZo9rhuEwozM1w4U639b,IAIAIAIAIAIAIAIAIAIAIAIAIAIAIAIAIAIAIAIAIAIAIAIAQAQAQAQAQAQAQAQAQAQAQAQAQAQAQAQAQAQAQAQAQAQAQAQAYAYAYAYAYAYAYAYAYAYAYAYAYAYAYAYAYAYAYAYAYAYAYAYAgAgAgAgAgAgAgAgAgAgAgAgAgAgAgAgAgAgAgAgAgAgAgAgAn.n.n.n.n.n.n.n.n.n.n.n.n.n.n.n.n.n.n.n.n.n.n.n.v.v.v.v.v.v.v.v.v.v.v.v.v.v.v.v.v.v.v.v.v.v.v.v.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3………………………………………….,AAMzIiERAAMzERAAERAAAAAAIiIiREREu7MzERmZMz..iI..qqVVMzIiREERAAERMzZmREERMzZmd3VVREzMMzqqIiMziIu7VVVVMzIiIiERAAERERMzd3VVVVIiREREIiREVVVVREMziIMzMzREMzREERIiAAAAAAIiERmZVVmZAAREVVIiIiREIiIiIiIiVVd3ERERIiERAAERIiIiVVAAIiREIiMzMzERMzAAVVMzAAZmERMzIiIiERAAERAAAAREVVVVMzd3MzMzIiAAREERMzERERAAERAAMzAAERAAAAAAAAAAERAAMzIiERIiIiIiREERIiERMzIiAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA&chxt=x,y&chxl=0:</td>
<td> </td>
<td>0</td>
<td>1</td>
<td>2</td>
<td>3</td>
<td>4</td>
<td>5</td>
<td>6</td>
<td>7</td>
<td>8</td>
<td>9</td>
<td>10</td>
<td>11</td>
<td>12</td>
<td>13</td>
<td>14</td>
<td>15</td>
<td>16</td>
<td>17</td>
<td>18</td>
<td>19</td>
<td>20</td>
<td>21</td>
<td>22</td>
<td>23</td>
<td> </td>
<td>1:</td>
<td> </td>
<td>Sun</td>
<td>Mon</td>
<td>Tue</td>
<td>Wed</td>
<td>Thu</td>
<td>Fri</td>
<td>Sat</td>
<td>&chm=o,333333,1,1.0,25,0&chds=-1,24,-1,7,0,20)</td>
</tr>
</tbody>
</table>
<h2 id="one-users-work">One User’s Work</h2>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>~/memcached % git timecard --author=dustin
</code></pre></div></div>
<table>
<tbody>
<tr>
<td>![dustin’s timecard](http://chart.apis.google.com/chart?cht=s&chs=800x300&chd=e:CkFIHrKPMzPXR7UeXCZmcKeuhRj1mZo9rhuEwozM1w4U639bCkFIHrKPMzPXR7UeXCZmcKeuhRj1mZo9rhuEwozM1w4U639bCkFIHrKPMzPXR7UeXCZmcKeuhRj1mZo9rhuEwozM1w4U639bCkFIHrKPMzPXR7UeXCZmcKeuhRj1mZo9rhuEwozM1w4U639bCkFIHrKPMzPXR7UeXCZmcKeuhRj1mZo9rhuEwozM1w4U639bCkFIHrKPMzPXR7UeXCZmcKeuhRj1mZo9rhuEwozM1w4U639bCkFIHrKPMzPXR7UeXCZmcKeuhRj1mZo9rhuEwozM1w4U639bCkFIHrKPMzPXR7UeXCZmcKeuhRj1mZo9rhuEwozM1w4U639b,IAIAIAIAIAIAIAIAIAIAIAIAIAIAIAIAIAIAIAIAIAIAIAIAQAQAQAQAQAQAQAQAQAQAQAQAQAQAQAQAQAQAQAQAQAQAQAQAYAYAYAYAYAYAYAYAYAYAYAYAYAYAYAYAYAYAYAYAYAYAYAYAgAgAgAgAgAgAgAgAgAgAgAgAgAgAgAgAgAgAgAgAgAgAgAgAn.n.n.n.n.n.n.n.n.n.n.n.n.n.n.n.n.n.n.n.n.n.n.n.v.v.v.v.v.v.v.v.v.v.v.v.v.v.v.v.v.v.v.v.v.v.v.v.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3………………………………………….,AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAttAAAAAAAAAAAAbbJJAAAAAAAAAAAAAAAAJJJJAAAAAAAAAAAAAAAAkkAAAAAAAAAAAAAAAAAAAAAAAAAAAA..AAAAAAAAAAAAAAJJSSAAAAJJAAJJAAAAAAAAAAAAAAAAAAAAJJSSSSAAAAAAAAAAAAAAJJAAAAJJAAAAAAAAAAAAAAJJAAAAAAAAAAAAAAAAAAAAAAAAAAAASSAAAAAAAAAAAAAAAAAASSAAJJAAAAAAAAAAAAbbAAAAJJAAAAJJAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA&chxt=x,y&chxl=0:</td>
<td> </td>
<td>0</td>
<td>1</td>
<td>2</td>
<td>3</td>
<td>4</td>
<td>5</td>
<td>6</td>
<td>7</td>
<td>8</td>
<td>9</td>
<td>10</td>
<td>11</td>
<td>12</td>
<td>13</td>
<td>14</td>
<td>15</td>
<td>16</td>
<td>17</td>
<td>18</td>
<td>19</td>
<td>20</td>
<td>21</td>
<td>22</td>
<td>23</td>
<td> </td>
<td>1:</td>
<td> </td>
<td>Sun</td>
<td>Mon</td>
<td>Tue</td>
<td>Wed</td>
<td>Thu</td>
<td>Fri</td>
<td>Sat</td>
<td>&chm=o,333333,1,1.0,25,0&chds=-1,24,-1,7,0,20)</td>
</tr>
</tbody>
</table>
<h2 id="last-weeks-work">Last Week’s Work</h2>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>~/twitterspy % git timecard '@{1 week ago}'..
</code></pre></div></div>
<table>
<tbody>
<tr>
<td>![recent twitterspy work](http://chart.apis.google.com/chart?cht=s&chs=800x300&chd=e:CkFIHrKPMzPXR7UeXCZmcKeuhRj1mZo9rhuEwozM1w4U639bCkFIHrKPMzPXR7UeXCZmcKeuhRj1mZo9rhuEwozM1w4U639bCkFIHrKPMzPXR7UeXCZmcKeuhRj1mZo9rhuEwozM1w4U639bCkFIHrKPMzPXR7UeXCZmcKeuhRj1mZo9rhuEwozM1w4U639bCkFIHrKPMzPXR7UeXCZmcKeuhRj1mZo9rhuEwozM1w4U639bCkFIHrKPMzPXR7UeXCZmcKeuhRj1mZo9rhuEwozM1w4U639bCkFIHrKPMzPXR7UeXCZmcKeuhRj1mZo9rhuEwozM1w4U639bCkFIHrKPMzPXR7UeXCZmcKeuhRj1mZo9rhuEwozM1w4U639b,IAIAIAIAIAIAIAIAIAIAIAIAIAIAIAIAIAIAIAIAIAIAIAIAQAQAQAQAQAQAQAQAQAQAQAQAQAQAQAQAQAQAQAQAQAQAQAQAYAYAYAYAYAYAYAYAYAYAYAYAYAYAYAYAYAYAYAYAYAYAYAYAgAgAgAgAgAgAgAgAgAgAgAgAgAgAgAgAgAgAgAgAgAgAgAgAn.n.n.n.n.n.n.n.n.n.n.n.n.n.n.n.n.n.n.n.n.n.n.n.v.v.v.v.v.v.v.v.v.v.v.v.v.v.v.v.v.v.v.v.v.v.v.v.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3………………………………………….,AAAAAAAAAAAAAAAAAAAAQAAAAAAAv.AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA..AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAQAgAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAQAAAAAAAv.v.AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAQAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA&chxt=x,y&chxl=0:</td>
<td> </td>
<td>0</td>
<td>1</td>
<td>2</td>
<td>3</td>
<td>4</td>
<td>5</td>
<td>6</td>
<td>7</td>
<td>8</td>
<td>9</td>
<td>10</td>
<td>11</td>
<td>12</td>
<td>13</td>
<td>14</td>
<td>15</td>
<td>16</td>
<td>17</td>
<td>18</td>
<td>19</td>
<td>20</td>
<td>21</td>
<td>22</td>
<td>23</td>
<td> </td>
<td>1:</td>
<td> </td>
<td>Sun</td>
<td>Mon</td>
<td>Tue</td>
<td>Wed</td>
<td>Thu</td>
<td>Fri</td>
<td>Sat</td>
<td>&chm=o,333333,1,1.0,25,0&chds=-1,24,-1,7,0,20)</td>
</tr>
</tbody>
</table>
Git Reroot - When Rebase is Too Gentle2009-01-06T00:00:00+00:00http://dustin.github.com/2009/01/06/git-reroot<h1 id="git-reroot---when-rebase-is-too-gentle">Git Reroot - When Rebase is Too Gentle</h1>
<div>
<img src="/images/transplant.jpg" alt="transplant" class="floatright" />
</div>
<p>The fun thing about git is that it’ll do whatever you tell it.</p>
<p>Many newcomers look at is as this really complicated beast that is
impossible to understand, but the less resistant users find that it’s
very happy to just sit back and happily do whatever you ask of it
(even if you ask it to do something stupid).</p>
<p>Recently, I was working on a project, and wanted to rebase a branch
that had drifted quite a bit away from the master branch. <code class="language-plaintext highlighter-rouge">rebase</code>
itself wasn’t getting me anywhere due to various conflicts from some
partial merges and manual merges.</p>
<p>As an attempt towards a solution, I created <a href="http://gitorious.org/projects/bindir/repos/mainline/blobs/master/git-reroot">git reroot</a>.</p>
<h2 id="what-does-it-do">What Does it Do?</h2>
<p><code class="language-plaintext highlighter-rouge">git reroot</code> is very similar to <code class="language-plaintext highlighter-rouge">rebase</code> conceptually, with one subtle
detail – <code class="language-plaintext highlighter-rouge">rebase</code> works by rewinding to a merge point and replaying
deltas (while dropping duplicates). <code class="language-plaintext highlighter-rouge">reroot</code> works by taking a range
of commits and placing the commits at the end of the current <code class="language-plaintext highlighter-rouge">HEAD</code> by
exact tree state.</p>
<p>The distinction is subtle, but important. git does not record
changes, it snapshots tree states with some additional metadata.
Commit deltas may be computed between any arbitrary trees, so the
representations you often see are these deltas.</p>
<h2 id="when-should-i-use-it">When Should I Use It?</h2>
<p>Quite likely never. It was not appropriate for the project for which
I created it.</p>
<p>However, if</p>
<ol>
<li>you find yourself with a branch that has diverged too far,</li>
<li>you consider the result of this branch to be the desired state, and</li>
<li>it’s OK to think of the commits as snapshots of work instead of
changes to previous state,</li>
</ol>
<p>then you may find <code class="language-plaintext highlighter-rouge">reroot</code> helpful.</p>
<h2 id="how-do-i-use-it">How Do I Use It?</h2>
<p>The invocation recipes are different from that of <code class="language-plaintext highlighter-rouge">rebase</code> because
it’s more of a “do what I want” kind of tool.</p>
<p>In a really simple case, let’s say you have a branch <code class="language-plaintext highlighter-rouge">new-development</code>
that diverged from <code class="language-plaintext highlighter-rouge">master</code> a while back. Some work has been done on
master, but you really just want <code class="language-plaintext highlighter-rouge">new-development</code> to be master. For
whatever reason, you don’t want to do a merge to get it there, and
rebase fails you due to conflicts you really don’t care about.</p>
<p>You would invoke <code class="language-plaintext highlighter-rouge">reroot</code> as follows:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>git reroot master..new-development
</code></pre></div></div>
<p>You should see some output that’s showing you progress, and then a
line that looks like this:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>The newly created history is available as 2015200[...]
</code></pre></div></div>
<p>This command is <em>completely non-destrutive</em>, and will not affect <em>any</em>
ref, so it’s safe to do whenever and wherever you like.</p>
<p>This output is telling you that the new tree is available, but not
linked. You may use log (<code class="language-plaintext highlighter-rouge">git log 2015200</code>) to examine it, and when
you’re ready to overwrite the current ref:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>git reset --hard 2015200
</code></pre></div></div>
<p>If you look through the deltas (<code class="language-plaintext highlighter-rouge">git log -p</code>), you may see some
changes that are much larger than you’d expect (especially towards the
beginning, or any merge points), but at any given commit, the source
tree is guaranteed to be in the exact state it was in when the author
committed it.</p>
Git Archaeology2008-12-31T00:00:00+00:00http://dustin.github.com/2008/12/31/archaeology<h1 id="git-archaeology">Git Archaeology</h1>
<div>
<img class="floatright" src="/images/indiana_jones_small.jpg" alt="indy" />
</div>
<p>I just spent a while reconstructing the history of my code
<a href="http://github.com/dustin/snippets" title="snippets">junk drawer</a>. It’s on its fourth revision control system now
(<a href="http://www.nongnu.org/cvs/" title="concurrent version system">cvs</a> → <a href="http://www.gnu.org/software/gnu-arch/" title="gnu arch">tla</a> → <a href="http://www.selenic.com/mercurial/" title="mercurial">mercurial</a> →
<a href="http://git-scm.com/" title="git">git</a>) and has been through a lot of different tree states.</p>
<p>CVS really only versions files, but allows you to arrange things into
a hierarchy, so I had a natural hierarchy and reflected it in a
similar way in CVS.</p>
<p>Gnu arch favored smaller repositories, so when I did the conversion
from CVS, I broke the snippets down into several different “branches”
and versioned each language independently. I had one container branch
that had a build config that would recreate the tree. This codebase
lived through three different archives (repositories) and some of the
individual snippets had a couple versions within that.</p>
<p>Once I started using mercurial more, I needed my snippets with me, but
mercurial didn’t have a similar mechanism for managing a collection of
repositories (even today, the <a href="http://www.selenic.com/mercurial/wiki/index.cgi/ForestExtension" title="forest">forest extension</a> is not
distributed with mercurial). I had attempted to use <a href="http://darcs.net/" title="darcs">darcs</a> to
reconstruct a single tree with full history but the trees renamed, but
darcs wouldn’t ever complete with a subset of what needed to be
converted. I ended up just snapshotting what was in gnu arch and
dropping it into a single mercurial repository.</p>
<p>Having moved into git, I finally have the tools to actually put the
history back together correctly. By “correctly”, I mean I wanted a
single repository with all of the changes in it ordered
chronologically (the order in which junk was placed in the drawer)
without lots of weird merges that didn’t actually happen. I <em>also</em>
needed to dig up all of the history prior to the snapshot I took for
mercurial and get it all in place.</p>
<h2 id="bringing-up-snippets">Bringing up Snippets</h2>
<p>Just to add to the complexity story, keep the following in mind:</p>
<ol>
<li>After tla, code was committed into mercurial from a snapshot.</li>
<li>That snapshot was (cleanly) converted to git, and more code was
committed there.</li>
<li>One failed archaeological excursion had a few commits as well.</li>
</ol>
<p>I started by going to the latest gnu arch versions of each snippet set
and converting them to git repositories (by way of mercurial – but
that’s a different story).</p>
<h2 id="setting-up-the-repo">Setting up the Repo</h2>
<p>I created a repo with a single empty commit in it as the eventual root
of all of the other repos.</p>
<p>Once each repository was converted to individual git repositories, I
brought added them as remotes to the conversion repository. Each
branch needs to be considered related in order to facilitate the
eventual merge, so I created grafts that placed the root of each
branch atop my empty commit using the following script:</p>
<figure class="highlight"><pre><code class="language-sh" data-lang="sh"><span class="c">#!/bin/sh</span>
<span class="nv">empty</span><span class="o">=</span>6c417dd379ccdb46de57e7a3860379633c270c9e
<span class="k">for </span>b <span class="k">in</span> <span class="s2">"</span><span class="nv">$@</span><span class="s2">"</span>
<span class="k">do
</span><span class="nv">oldest</span><span class="o">=</span><span class="sb">`</span>git rev-list <span class="nt">--reverse</span> <span class="nv">$b</span> | <span class="nb">head</span> <span class="nt">-1</span><span class="sb">`</span>
<span class="nb">echo</span> <span class="s2">"Grafting </span><span class="nv">$b</span><span class="s2">"</span>
<span class="nb">echo</span> <span class="s2">"</span><span class="nv">$oldest</span><span class="s2"> </span><span class="nv">$empty</span><span class="s2">"</span> <span class="o">>></span> .git/info/grafts
<span class="k">done</span></code></pre></figure>
<p>This was run for every remote repo and then each branch was run
through <code class="language-plaintext highlighter-rouge">git filter-branch</code> to place the changes atop the empty branch
in a real history.</p>
<h3 id="rewriting-tree-structures">Rewriting Tree Structures</h3>
<p>These weren’t quite ready to merge just yet. Before I could even
consider an actual merge, I needed to modify the tree structures
(e.g. take all of the stuff at the toplevel of the <code class="language-plaintext highlighter-rouge">eiffel</code> directory
and move it under an <code class="language-plaintext highlighter-rouge">eiffel/</code> directory). The previous excursion had
done this using a recipe I’d found on the internet somewhere which
<em>worked</em>, but did the wrong thing with my version of sed. Using gsed
cleaned this up.</p>
<p>For each remote branch, I’d run the following filter:</p>
<figure class="highlight"><pre><code class="language-sh" data-lang="sh"><span class="c">#!/bin/sh</span>
git filter-branch <span class="nt">-f</span> <span class="nt">--index-filter</span> <span class="se">\</span>
<span class="s1">'git ls-files -s | gsed "s-\t-&eiffel/-" |
GIT_INDEX_FILE=$GIT_INDEX_FILE.new \
git update-index --index-info &&
mv $GIT_INDEX_FILE.new $GIT_INDEX_FILE'</span> <span class="nv">$1</span>/master</code></pre></figure>
<p><strong>Note</strong>: Out of pure laziness, I would edit this script for every
invocation and then run it for a single remote.</p>
<h3 id="doing-the-merge">Doing the Merge</h3>
<div>
<img alt="merge" class="floatleft" src="http://img.skitch.com/20090101-6s52spepjx7qgjaj3yuscuasa.png" />
</div>
<p>The merge was really rather exciting. The image to the left shows a
24-way octopus merge.</p>
<p>That is, after grafting the empty changeset to the bottom of every
branch, they now had common ancestry, making a merge possible. Since
each branch got its paths rewritten all throughout history, there was
no chance of conflict.</p>
<p>So enter the octopus.</p>
<p><br clear="both" /></p>
<h3 id="performing-a-linear-rewrite">Performing a Linear Rewrite</h3>
<p>As cool as it was to do a massive octopus merge, I wanted linear
history.</p>
<p>It would be possible to produce a graft file to place each change atop
a single parent, but that seemed quite hard.</p>
<p>The strategy I employed was to dump the entire history using
<code class="language-plaintext highlighter-rouge">git format-patch</code> and then write <a href="http://github.com/dustin/snippets/tree/master/python/misc/rewrite-patches.py">a script</a> to rename
all of the patches to be in chronological order so I could use
<code class="language-plaintext highlighter-rouge">git am</code> to reconstruct the tree.</p>
<p>So I created a new branch from “empty”, and ran <code class="language-plaintext highlighter-rouge">git am</code> for a while.
A nice bonus is that <code class="language-plaintext highlighter-rouge">git apply</code> strips off trailing whitespace for
me, so the changes were slightly cleaned on the way in (I could’ve
disabled that, but I rather liked it).</p>
<h3 id="removing-emptiness">Removing Emptiness</h3>
<p>I no longer needed the “empty” changeset after <code class="language-plaintext highlighter-rouge">git am</code> was complete,
so I had to get rid of that. The root node is generally a bit
difficult to touch, but I sort of guessed that I could add a graft of
a hash without a parent and it’d make that change the new root.</p>
<p>So another trip through <code class="language-plaintext highlighter-rouge">git filter-branch</code> and I’ve now got a pretty
decent set of history up throgh the snapshot that was taken for the
mercurial conversion.</p>
<h3 id="catching-up-to-the-present">Catching up to the Present</h3>
<p>So now that I’ve got everything up to the snapshot, what do I do?</p>
<p>I had a lot of options here – cherry-picking, grafting,
format-patch. I think I went with format-patch arbitrarily.
Basically, I did a <code class="language-plaintext highlighter-rouge">git format-patch</code> of the full history from the
latest git repo and applied those changes to the newly created one.</p>
<h3 id="verification">Verification</h3>
<p>So now that everything has been all hacked up and history is rewritten
and changests grafted, etc… how do I have any idea whether it’s
even close to where it was before?</p>
<p>This is where git’s content tracking stuff really saves the day. With
the git repo I’ve been using as a remote, I can do a simple diff
across the trees from the latest branches (and various other states).
The only differences I saw were some new scripts/etc… had been
added.</p>
<p>All’s well. I certainly learned a lot.</p>
Using Git Alternates2008-12-30T00:00:00+00:00http://dustin.github.com/2008/12/30/git-alternates<h1 id="using-git-alternates">Using Git Alternates</h1>
<p>Now that you’re happily using
<a href="/2008/12/29/github-sync.html">github sync</a> to pull down all your
repos into local bare trees, you may want to free up a bit of disk
space from duplicate objects (about 120MB for me).</p>
<p>git has a way for multiple repos to share object space by way of
alternates. You can read more about alternates in the
<a href="http://www.kernel.org/pub/software/scm/git/docs/gitrepository-layout.html">repository layout documentation</a>, but essentially it’s a text file
that contains the location of another <code class="language-plaintext highlighter-rouge">objects</code> directory from which
objects may be fetched when needed.</p>
<h2 id="example">Example:</h2>
<p>Let’s say you’re me and have checked out my photo album. You’d end up
with a .git directory that looks like this:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>dhcp-39:/tmp/photo 599% du -sh .git
18M .git
</code></pre></div></div>
<p>By setting up an alternate using my <a href="https://github.com/dustin/bindir/blob/master/git-alternate">git alternate</a> command:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>dhcp-39:/tmp/photo 600% git alternate ~/prog/github/photo.git
.git/objects -> /Users/dustin/prog/github/photo.git/objects
</code></pre></div></div>
<p>You can then gc and free up gangs of disk:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>dhcp-39:/tmp/photo 601% git gc
Nothing new to pack.
Removing duplicate objects: 100% (256/256), done.
dhcp-39:/tmp/photo 602% du -sh .git
144K .git
</code></pre></div></div>
<p>From 18MB to 144KB, and everything pretty much works as it did before.</p>
<p>You don’t need my <a href="https://github.com/dustin/bindir/blob/master/git-alternate">git alternate</a> command, for that, of course, but
it makes it a bit easier when you’ve got a lot of them to do.</p>
Using Github Sync to Track Your Projects2008-12-29T00:00:00+00:00http://dustin.github.com/2008/12/29/github-sync<h1 id="using-github-sync-to-track-your-projects">Using Github Sync to Track Your Projects</h1>
<div>
<img alt="octocat syncing" class="floatright" src="/images/octocat-sync-small.png" />
</div>
<p>When <a href="http://github.com/">github</a> announced their
<a href="http://github.com/guides/the-github-api">API</a>, I very quickly threw
together a <a href="http://github.com/dustin/py-github">python implementation</a>.</p>
<p>I didn’t end up doing very much with the project as a whole, but I did
write one tool in here that I end up using quite a bit:
<code class="language-plaintext highlighter-rouge">githubsync.py</code>.</p>
<p><code class="language-plaintext highlighter-rouge">githubsync.py</code> takes a github username and a directory and make sure
I’ve got a local copy of every public repo that user has on github.</p>
<p>Grab the repo and try it out:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>git clone git://github.com/dustin/py-github.git
cd py-github
./src/githubsync.py dustin /tmp/dustinatgithub
</code></pre></div></div>
<p>Once that finishes, you will have all of my current public repos in
<code class="language-plaintext highlighter-rouge">/tmp/dustinatgithub</code> and if you run it periodically, you’ll see new
repos I add appear while the existing ones are being updated.</p>
<p>But what about private repos, or even repos that aren’t on github?</p>
<p>The file <code class="language-plaintext highlighter-rouge">~/.github-private</code> is read as a tab-delimited list of repos
and their sources and those will also be synchronized. For example:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>cool-stuff git@github.com:dustin/cool-stuff.git
</code></pre></div></div>
<p>With that in place, the <code class="language-plaintext highlighter-rouge">cool-stuff</code> repo will be created and
synchronized along with all of the stuff found through the API.</p>
Wasted Time Developing for iPhone2008-12-26T00:00:00+00:00http://dustin.github.com/2008/12/26/wasted-time-on-iphone<div>
<img alt="wasted time" class="floatright" src="http://img.skitch.com/20081227-qukmwpnbu6u9qnruimsqnrdj2y.jpg" />
</div>
<p>OK, everybody’s written about this, but I just wasted a bunch of time making an
iPhone app.</p>
<p>I don’t actually feel too bad about it because it was a <a href="http://github.com/dustin/twister-iphone">pretty stupid</a>
iPhone app, anyway, but I’m not going to finish it because my first attempt to
run it outside of the simulator was going to cost me a hundred bucks.</p>
<p>The application is an iPhone port of my <a href="http://dustin.github.com/2008/12/25/twister.html">twister</a> app, but with worse
graphics and sound (though the sound is at least <em>potentially</em> better). It’s
functional enough to play a few games, but not fully polished.</p>
<p>I was hoping I could stick it on my daughter’s iPhone so she could play, but
doesn’t seem to be the case.</p>
<p>If anyone wants to do something with it, it’s over on <a href="http://github.com/dustin/twister-iphone">github</a>.</p>
<p>Of course, I’d love to find out I was wrong and I can actually run my own
program on my own phone without paying more…</p>
pfetch2008-12-26T00:00:00+00:00http://dustin.github.com/2008/12/26/pfetch<h1 id="about-pfetch">About pfetch</h1>
<div>
<img alt="octopus" src="/images/octopus.png" class="floatright" />
</div>
<p>For a long time now, I’ve had various cron jobs running to fetch various web
resources with which I’d build out parts of my own site, or supply myself with
custom RSS feeds after a pass through xsltproc.</p>
<p>This mostly worked OK, but there were a few things wrong with it:</p>
<ol>
<li>I had to be careful to avoid putting stuff in place upon fetch failure.</li>
<li>Fetch failures would send me email unless I put effort into avoiding that.</li>
<li>Network timeouts would cause cron jobs to start piling up.</li>
<li>I’ve actually had cron get sick of running my jobs and just stop altogether.</li>
<li>Various jobs that ran at various frequencies would be in various scripts and hard to keep up with.</li>
<li>Running through cron means all jobs start at the exact same moment in time, thus are more likely to cause strain on web servers (if everybody does it).</li>
<li>Conditional gets require cross-invocation state to be stored (though I wrote <a href="http://github.com/dustin/snippets/tree/master/python/net/http/fetch.py">a tool</a> for this).</li>
<li>Sequential processing meant the whole thing took longer.</li>
</ol>
<p>After a while, the problems added to enough of an annoyance that I decided to
do something about it, so a couple months ago I started
<a href="http://github.com/dustin/pfetch">pfetch</a>.</p>
<p>pfetch is a simple <a href="http://twistedmatrix.com/">twisted</a> app that does scheduled parallel http requests
and optionally runs scripts after successful execution.</p>
<p>Given a list of URLs each with a destination, frequency, and optional (with
arguments) to run after each successful (200) response, each URL will begin a
fetch cycle starting at a random offset from the start time and loop on the
defined interval.</p>
Twister2008-12-25T00:00:00+00:00http://dustin.github.com/2008/12/25/twister<h1 id="twister">Twister</h1>
<div>
<img alt="twister" class="floatright" src="http://upload.wikimedia.org/wikipedia/en/thumb/0/09/1966_Twister_Cover.jpg/275px-1966_Twister_Cover.jpg" />
</div>
<p>So, on Christmas, my kids decided they wanted to play twister. They wanted me
to spin the thingy and call out moves for them. That got <em>really</em> boring after
about five minutes.</p>
<p>I wrote a really simple python script to start calling the moves for me since
the spinny thing was getting annoying, and would sometimes end up pointing
between two colors or otherwise be too difficult to call.</p>
<p>The <a href="http://gist.github.com/40015">first version</a> of the script looked like
this:</p>
<figure class="highlight"><pre><code class="language-python" data-lang="python"><span class="c1">#!/usr/bin/env python
</span>
<span class="kn">import</span> <span class="nn">random</span>
<span class="k">if</span> <span class="n">__name__</span> <span class="o">==</span> <span class="s">'__main__'</span><span class="p">:</span>
<span class="n">colors</span><span class="o">=</span><span class="p">(</span><span class="s">'red'</span><span class="p">,</span> <span class="s">'green'</span><span class="p">,</span> <span class="s">'yellow'</span><span class="p">,</span> <span class="s">'blue'</span><span class="p">)</span>
<span class="n">limbs</span><span class="o">=</span><span class="p">(</span><span class="s">'left foot'</span><span class="p">,</span> <span class="s">'right foot'</span><span class="p">,</span> <span class="s">'left hand'</span><span class="p">,</span> <span class="s">'right hand'</span><span class="p">)</span>
<span class="k">print</span> <span class="n">random</span><span class="p">.</span><span class="n">choice</span><span class="p">(</span><span class="n">limbs</span><span class="p">),</span> <span class="n">random</span><span class="p">.</span><span class="n">choice</span><span class="p">(</span><span class="n">colors</span><span class="p">)</span></code></pre></figure>
<p>That was fine, but I still had to run it and call it out. Then I remembered
that someone made a <a href="http://www.gnufoo.org/macosx/">talking cat</a> for OS X. All
I needed to do was run the output of this thing through that, and there’d be
speech and then I could go about my business and let the computer call moves
for them.</p>
<p>I thought that was kind of cool, but wanted something a little…more. I ended
up writing a full <a href="http://github.com/dustin/twister">OS X desktop version</a>
complete with images, icons, a preference pane, etc…</p>
<p>The kids finished playing (using the prototype) long before I finished writing
the app. It was fun for all of us, though. :)</p>
<p>If anyone wants to play a two-player version of twister, though, you can grab
a copy.</p>
<h2 id="download">Download</h2>
<p><a href="http://public.west.spy.net/app/Twister_1.1.zip">Version 1.1</a></p>
Moody Bots2008-12-24T00:00:00+00:00http://dustin.github.com/2008/12/24/moody-bots<h1 id="moody-bots">Moody Bots</h1>
<p><a href="/twitterspy/">Twitterspy</a> is a rather brute-force way to achieve xmpp
functionality for twitter. It makes very heavy use of twitter search to
provide track-like functionality to end users.</p>
<p>I’ve noticed in watching the logs that I often will get more errors in
attempting searches than successes. This was at least the feel I got from
looking at the logs. I wanted a way to communicate this to look at this
information via xmpp.</p>
<p>Initially, it seemed like status would be a good way to do this. Currently,
the status is used to show stats on how many users and queries the bot knows
about. This is already a little weird, and I wouldn’t want to try to shove too
much stuff into it.</p>
<p>Next, I thought about using the vcard for it. The bio is a fine way to
describe such things. That wasn’t quite right, either. The bio is a better
general description of the bot, and not so much status.</p>
<p>Then I discovered <a href="http://xmpp.org/extensions/xep-0107.html">XEP-0107</a> – user
moods. User moods in a <a href="http://xmpp.org/extensions/xep-0163.html">PEP</a>
transport provides exactly the kind of thing I’m looking for.</p>
<div><img src="http://img.skitch.com/20081225-g8nbh7s3np2amubspgkas2ab1f.png" class="floatright" alt="twitterspy angry" /></div>
<p>Twitterspy keeps track of how many of its searches are successful, and how many
fail. When many searches are successful, it’s in a good mood, when few are,
it’s in a bad mood.</p>
<div><img src="http://ralphm.net/images/mood/knology/excited.gif" class="floatleft" alt="excited!" /></div>
<p>I had my kid look through the XEP to come up with some rules for how to select
a mood based on how successful recent searches are. I’ve applied many of her
changes, but some still require me to keep a bit more state than I do
currently. It’s kind of an exciting thing, though few people will ever
actually see it.</p>
<p>The pubsub mechanism will hopefully show itself to be useful, though. I’m
hoping to do something cool like have a web status showing moods and all.
<a href="http://ralphm.net/">Ralph Meijer’s</a> <a href="http://ralphm.net/moods">moods page</a> is
quite inspirational here – as long as I’m capturing the data.</p>
<p>For the rest of you out there: Bring your XMPP services to life. Show their
moods.</p>
Building Your Site Connectivity2008-12-24T00:00:00+00:00http://dustin.github.com/2008/12/24/building-site-connectivity<h1 id="building-your-site-connectivity">Building Your Site Connectivity</h1>
<p>When I started building this jekyll site, I thought it’d be nice to link to
all the other places I leave junk around the internet. Rather than manually
building a list, I took a bit of time to write something to do it for me using
the <a href="http://code.google.com/apis/socialgraph/">google social graph API</a>.</p>
<p>I made a simple <a href="http://public.west.spy.net/autolinks.html">web form</a> to do
this that generates HTML source so it can be further hand-edited if needed, but
more importantly, so that I can actually paste the results into my github page
and have it actually count for site connectivity.</p>
<p>If you don’t maintain a list of links on your own page, you may find it helpful
to link to your <a href="http://friendfeed.com/">friendfeed</a> account.</p>
<p>For example, you can see
<a href="http://public.west.spy.net/autolinks.html?u=http://friendfeed.com/dlsspy">how friendfeed links me</a>.
Change the username from dlsspy to yours for results that make more sense to
you.</p>
<p>As it’s just a simple chunk of HTML, I’ve created
<a href="http://gist.github.com/39613">a gist</a> to house it for now. If you’d like to
change this for the better, do it there and let me know about it.</p>
<p>I hope someone (else) finds this useful.</p>
Trying out Jekyll2008-12-23T00:00:00+00:00http://dustin.github.com/2008/12/23/trying-jekyll<h1 id="trying-out-jekyll">Trying out Jekyll</h1>
<p>Since <a href="http://github.com/mojombo/jekyll">Jekyll</a> seems to be all the craze, I
figured I’d give it a shot and see if it solved any problems for me.</p>
<p>So far, I like it.</p>