Too-biased - Home

Benjamin Zander

2008-12-20T18:07:00Z

Thanks to @igrigorik for the link to this wonderful Benjamin Zander talk

Facebook's memcached

2008-12-13T03:51:00Z

Facebook’s server installation has been a point of interest for a long time. When an engineer on the memcached mailing list casually mentioned that they are running 4TB worth of caches, a lot of people got their panties in a knot.

Since then Facebook has been more forthcoming with details about their exciting architecture. For example they explained how they created a custom Mysql to solve problems with cross-datacenter cache expiry and replication lag which is a great read for anyone looking into multi location hosting.

Today there was another memcached centric post that talks about their fork of the memcached codebase which they also made available on github.

We use more than 800 servers supplying over 28 terabytes of memory to our users. Over the past year as Facebook’s popularity has skyrocketed, we’ve run into a number of scaling issues. This ever increasing demand has required us to make modifications to both our operating system and memcached to achieve the performance that provides the best possible experience for our users. [...]

Fascinating read.

Passenger

2008-11-15T18:14:00Z

So there is a lot of talk about Phusion Passenger lately and I feel the need to chime in here. David pointed out that Shopify is running on passenger which is something I announced on Twitter a few months ago.

Some context on Shopify’s installation: We launched Shopify originally on Lighttpd with FastCGI and later migrated to nginx with mongrels. Obviously we had to use HAProxy between Nginx and mongrels to avoid the dreaded “queue behind long running process” problem. We also added Monit to the mix which observed all mongrels to make sure that everything is running according to plan. After a process reaches 260 mb of memory we signal it to shut down after the next request so that a new one can start out with less memory. For this we added runit to the mix which supervises the mongrels and starts them up quickly once they hit the ground.

It’s important to note that we are not talking about a memory leak here. The reason for the 260mb ceiling comes from two issues with Ruby’s garbage collector:

It allocates memory in very large chunks once the available memory gets low. This means a 140mb process increases to 260mb in a single go. It also never gives memory back to the operating system because Ruby’s GC is not able to move objects. Once it adds an object into the newly allocated space and that object remains alive, it cannot yield memory back to the OS.
Because Ruby’s garbage collector uses mark and sweep it has to traverse the entire memory space in search of pointers. There are no generations that help with that. It means that GC cycles become longer and longer the more memory is available. ~~-Rails mitigates these issues by moving a full GC run behind a HTTP response, into the time period when the process is waiting for a new request~~ (Update: Rails doesn’t do this anymore) but performance monitoring tools such as NewRelic clearly show that average response times is directly correlated with the amount of memory used across the server farm.

Now why did we switch to Passenger? Simple: the keyword is remove moving parts.

Every additional tool you add will come with it’s own bugs. Many people I talked to over the past years considered haproxy to be the most solid piece of infrastructure in their stack but even there was a really nasty bug recently (search for request queue handling).

We treat our server farm very similar to Shopify’s codebase. We are in this for the long haul and we cannot accept complex solutions when simple ones present themselves. Maintainability of our code and servers is paramount to the long term success of our product. Yes the Mongrel setup worked very well but Passenger allowed us to remove: Nginx, Haproxy, Runit and Monit. That’s a nice refactoring!

At the same time Passenger introduced some tangible improvements. We switched to enterprise ruby to get the full benefit of the COW memory characteristics and we can absolutely confirm the memory savings of 30% some others have reported. This is many thousand dollars of savings even at today’s hardware prices. We allow Passenger to adaptively spawn more processes with demand but most of the time our application servers are running about 40 processes to handle more than a million dynamic requests a day. However, because passenger constantly despawns and respawns rails processes they always stay fresh, run short GC cycles and are generally a lot more responsive. All this means that the total amount of memory that is used by Shopify during normal operations went from average of 9GB to an average of 5GB. We evenly distributed the savings amongst more Shopify processes and more memcached space which moved our average response time from 210ms to 130ms while traffic grew 30% in the last few months.

In conclusion: I cannot see any reason to choose a different deployment strategy at this point. Its simple, complete, fast and well documented.

Interview

2008-11-07T17:30:00Z

openvista posted their interview with me in which I talk about entrepreneurship and the early days of Shopify.

Rockstar Memcaching (video)

2008-10-28T18:02:00Z

InfoQ posted the video to my rockstar memcaching presentation from ruby fringe.

Rockstar Memcaching Video

Mobile Development

2008-10-23T22:08:00Z

I’ve been playing around with mobile development lately which is a nice change of pace. Before Shopify and Rails I used to count myself amongst the ranks of the C++ desktop developers ( go WTL!! ) so in many ways the concepts of mobile development feel like the good’ old times—without all the things that drive you up the walls.

What’s so fun about it is the innocence of it all. It’s the gold rush all over again. For example that bastard Hampton managed to sell his Wikipedia browser iPedia 50k times. Check out www.mobileorchard.com which just published an interview with him.

LHC Rap

2008-10-11T17:28:00Z

via youtube

Shopify Blogging

2008-10-06T18:41:00Z

Brand and community development where always the guiding principles behind Shopify and today we launched an important aspect of this: Your visitors can now comment on blog posts.

Liquid JS

2008-08-31T01:54:00Z

Color me impressed.

Matt Mccray ported Liquid to javascript. Go here to get the gist of it (yea, bad pun)

Pretty awesome work Matt :-)


<script>
 Liquid.readTemplateFile = function(path) {
  var elem = $(path);
  if(elem) {
    return elem.innerHTML;
  } else {
    path +" can't be found."; // Or throw and error, or whatever you want...
  }
 }

 var tmpl = Liquid.parse("{% include 'myOtherTemplate' with current_user %}"); 
 alert( tmpl.render({ current_user:'M@' }));
</script>

<script type="text/liquid" id="myOtherTemplate">
  Hello, {{ current_user }}!
</script>

Competitor Comedy

2008-08-22T19:14:00Z

Shopify got profiled last week in Practical E-Commerce as cart of the week . According to them they found 300 different Shopping cart packages, I know our market was big but that’s pretty insane.

Anyways, their Cart of the Week feature pits one Cart against another by asking a competitor to comment on the software, CNN style. In our case they asked Rick Wilson of Miva Merchant to comment on Shopify. Here is what he dislikes:

The obvious weaknesses I saw were the limits on what they called ‘enterprise’ level offering of only 10,000 SKUs. The other major weakness is the product is built using Ruby On Rails. Ruby On Rails seems to be a development environment with a lot of future possibilities, but as a general rule you don’t want your revenue-generating product to be based on cutting edge technology. There’s a lot of unforeseeable pitfalls in that area.

(emphasis mine).

So to deconstruct the first part of this argument let me just say that we have millions of products in Shopify right now and it’s a pure business decision to limit SKUs to 10k and has nothing to do with the software. There are millions of products in the database. In fact i’d buy Rick lunch if Shopify isn’t right now handling twice the traffic any given Miva Merchant store has ever sustained.

What’s left is the advice that you shouldn’t run a revenue-generating web site based on cutting edge technology. I don’t think I have to point out how ridiculous that statement is. First of all I presume that perl was pretty cutting edge ( definitely more than ruby is today ) in 1998 when Miva was written and also we are talking about bloody Miva Merchant here, the e-commerce system that runs on a proprietary closed source database that is known for corruption issues after several hundred products. Not only that, it seems to be one of their main sources of revenue: For 129$ setup + 20$ a month you can get some support and basic database recovery. Unfortunately:

** Due to the nature of database problems not all database corruption can be covered under this package

Isn’t it nice that you never have to worry about such tactics with hosted systems and SaaS packages like Shopify?

Rockstar Memcaching

2008-07-21T09:37:00Z

I’m back from Rubyfringe which was hands down the best conference i’ve been to.

Pete Forde asked me to present on memcached (mem-cache-dee) after my popular blog article Secret to memcached. The talk covers different use cases such as simple html snippet stores to advanced expiry systems such as generational cache keys.

Every talk at rubyfringe was taped so I’ll update this space with the video once its online. In the meantime enjoy the slides which probably make zero sense on their own.

Download the PDF or watch it on slideshare:

Rubyfringe<object height="355" width="425"><param /><param /><param /><embed src="http://static.slideshare.net/swf/ssplayer2.swf?doc=rubyfringe-1216650794678661-8" height="355" width="425"></embed></object>

view presentation (tags: memcached ruby code talk)

P.S: 30 minute is the ideal length for talks at a Tech Conference.

Update: The video was posted here

Shopify Sys Admin

2008-07-07T19:41:00Z

If you love servers you should consider applying for our system administrator position . Flexible work hours and you get to work with all the coolest and latest technologies and a fantastic team.

One million integers?!

2008-07-07T03:09:00Z

This is a great general purpose interview tip:

If you don’t know how to answer a question because it’s way outside of your expertise simply give it your best guess and negate the question.

via youtube:

This can sometimes have pretty impressive results.

Gecko/Webkit Screenshots

2008-05-17T16:07:00Z

For our Shopify Product Search we needed a good way to Screenshot web pages. There are some services on the web for this but we ended up building it but none of them fit our needs. They were either way to expensive, they didn’t produce nearly the quality we needed or they didn’t offer an API at all.

Our solution was to install a headless X server in our server farm which runs firefox 2.0. We used a python GTK automation script which navigates the Firefox instance to the page and then dumps the framebuffer into a png file when done. This works well enough but i’d like something more robust for a different project.

Ideally I’d like someone to build a screenshot tool based on Gecko or Webkit which can simply take an url and spit out an png. A dependency on an running X server is acceptable but I’d rather not have it running all the time because it complicates deployment a lot. It has to run on Linux and must not depend on a shared global resource, i.e. you should be able to take two screenshots at the same time.

If you know of a tool like the one I describe or if you think you could build something like this for me please contact me. This may be paid open source work.

Twitter

2008-05-09T01:13:00Z

I need more followers on twitter :-)