[video]
Guess who’s visiting Tumblr today?!
Alan will be visiting at 2:00. Come say hi.
We rely on memcache pretty heavily at Tumblr, with over 10TB of cache memory available across the stack. One of the things we’ve historically had a challenging time with at Tumblr is finding hot keys. A hot key is a memcache key getting dramatically more activity than other keys. This can have a significant performance impact on your cache backend.
We spent the past few days working on a C++ implementation of mctop*, which we’re happy to release today as memkeys. We do some pretty interesting stuff in memkeys to keep from dropping packets, some of which is documented in the wiki. I’m particularly proud of the striped lock-free queue implementation. In some basic benchmarks I found that memkeys dropped less than 2% of packets when seeing 1Gb/s of traffic. Additionally, the latency between a packet being picked up, parsed, processed, and reported on averages less than 1ms. Here is a screenshot of memkeys in action.
Interested in stuff like this? We’re hiring.
Footnote: Etsy created the excellent mctop tool which aims to be like unix top for memcache, showing you which keys are getting the most activity. Unfortunately (as noted in the known issues), mctop drops packets. It drops a lot of packets. This can be really problematic because depending on the packets being dropped, you’re getting a really incomplete view of your cache story.
Great cake or greatest cake?!?
Oh god… Its my face on a cake…
It’s also my ‘face’, I’m the man behind the mask.
This is why writing and testing code should always be done on a dev box. The dreaded accidental reboot.
Getting our donation on today!
No worries, no books or bikes are being given away. I am going to miss that red couch though.
Alan got a haircut today.
Love you Blake :)
It is our first anniversary. Love this woman.
[video]
by niraxoid
Sometimes.
