Article

Log files are one of the most important aspects of any web service. A webapp with a well designed logging strategy will allow you to essentially go back in time to track down even the most obscure bug. Unit tests have diminished the importance of log files somewhat but how do you write a unit test for a bug that only happens on server 2 around 2am in the morning when user fred is logged in?

Most of us run exception notification services such as exception_logger, hoptoad, exceptional or simply but log exceptions to a DB table ( MySQL protip: make that table MyISAM, otherwise exceptions that are added during a transaction will be removed when that transaction rolls back – duh ). Those exception notifications are great but they never provide a lot of context for how the user got to the point.

Lastly there is also the role that log files play in customer support. Have you ever gotten a complaint about data disappearing from your service? With good logging you can tell your customer in a matter of minutes that employee Bob went on data rampage friday evening before handing in his resignation.

At Shopify we use syslog-ng to have a centralized logging server which collects all the logs from the various machines in our cluster and combines the log files together. We used to give everyone access to this box for log analysis but as we grew this became a bit impractical. To solve this, we created Clarity, which provides a very nice web interface for the two staple tools of log analysis: grep and tail -f.

Clarity is very lightweight and only requires a few dependencies such as eventmachine and json. It’s completely evented which means that you can have many different greps and tails running at the same time on a single instance (as much as the server can handle). It even stops the grep utility on the server when you hit stop in your browser.

  ~  $ sudo gem install clarity 
  Password:
  Successfully installed clarity-0.9.8
  1 gem installed
  Installing ri documentation for clarity-0.9.8...
  Installing RDoc documentation for clarity-0.9.8...
  Could not find main page README.rdoc # anyone know how to get rid of this?!
  Could not find main page README.rdoc
  Could not find main page README.rdoc
  Could not find main page README.rdoc

  ~  $ clarity /var/log --include '*/**'
  Clarity 0.9.8 starting up.
   * listening on 0.0.0.0:8080
   * Log mask(s): */**

Additional command line parameters are:

~  $ clarity --help
Usage:  clarity [options] directory

Specific options:
    -p, --port=PORT                  Port to listen on
    -b, --address=ADDRESS            Address to bind to (default 0.0.0.0)
        --include=MASK               File mask of logs to add (default: **/*.log*)
        --user=USER                  User to run as

Password protection:
        --username=USER              Enable httpauth username
        --password=PASS              Enable httpauth password

Misc:
    -h, --help                       Show this message.

Clarity is now used by our support staff on a daily basis. It’s been so successful internally that we decided to release it as open source. You can read more about it on the github page. The URL structure of Clarity is pretty simple so it’s easy to add links to your internal admin area that directly open log files with the appropriate terms prefilled. You can for example add a link to a search that shows you all the DELETE requests of a certain store directly to your support system. This means that blaming Bob will only take one click in the future.

Enjoy the screencast:

Today marks two events.

First, Shopify was launched exactly 3 years ago. I started hacking on it more than 5 years ago, originally just for myself so that I could sell snowboards online, free from the tyranny of horrible online store software such as Yahoo Stores, but it quickly grew into something much larger. Now we are a profitable multi-million dollar company with one of the best teams in the industry. It’s been an amazing ride, at times a bumpy road but never less than exhilarating.

The other event is the launch of something I’m infinitely excited about: the Shopify Platform. Let me explain. E-commerce is one of those software areas where individualization matters. This has been clear from day one of running Snowdevil and selling our first snowboard. If you build a store on the Internet you are providing a customer experience that is not unlike walking into any physical store in the downtown area of the town you live in. If the floorboards squeak, the wall colors don’t match, if the service is slow or the lighting is off, you will not like the experience. You won’t be back. There will probably be no sale. This goes for online stores as well. An online store must look good because you are building a brand of trust with your client. Poor design begets poor customers and poor customers lead to unsustainable margins. This is the reason I wrote liquid, which allows you to build awesome looking stores on Shopify.

However, there are a lot of common elements to every e-commerce store. There is this small nucleus of core functionality that all software has to provide (and most do). These are things like inventory management, order processing, payment processing, shipping support and so on. Shopify excels at all these things—our customer satisfaction rate is north of 90%. However after you are done with all those features something funny happens. The next feature everyone wants is different for each store. Some people want live auctions, some people want a wholesale area, some people want community forums, license key generation, digital delivery, integrations with MS Commerce Server, Oracle Inventory, international tax form printers, etc.

None of these features is particularly hard to implement. The problem is that they fail our basic test which we use to determine whether we should implement a feature or not: Implement what most people need most of the time.

There is no cheating. Digital delivery is only needed by some people most of the time and international tax form printing is only needed by some people some of the time.

The trouble is, if we were to add all these features to Shopify, we would simply end up with software like the others on the market; filled to the brim with features that only some people need some of the time. I’m a firm believer that every time you add a feature to you are diminishing all other features. Adding features and especially adding elements to a user interface is not something to take lightly.

So that is the solution. Facebook and Salesforce showed us the way and this is what we are bringing to e-commerce: We are turning Shopify into a development platform and our merchants can supplement the pristine Shopify core with only the additional features they need. E-commerce à la carte.

The Shopify platform allows any programmer to create applications that integrate natively with the administration interface or storefront. These applications can be written in any language and communicate with Shopify using our handcrafted REST API. We even provide some amazing rails generators to get started quickly.

Obviously we need developers to make this happen. Reasons why you should develop for the Shopify Platform:

• Super fast start with the Shopify App rails generator
• Automatic marketing through the Shopify Application Store
• Soon we will launch the monetization system that allows you to bill merchants for using your applications directly through Shopify’s monthly billing system. We will deposit the money to you via Paypal.

The sum total is that Shopify is now as extensible as any self hosted Wordpress system but still hosted on a world-class server farm. It’s the best of both worlds and surely will be the way a lot of hosted apps will develop in the coming years.

Exciting times.

My friends from unspace are putting together FutureRuby, the spiritual successor to Rubyfringe which turned out to be one of the best conferences of last year.

The theme of the conference is obviously the future of ruby but one thing that is clear from the lineup is that the event doesn’t forget that ruby is part of a greater ecosystem and that ruby is as much a language as a mindset.

One great example of this is this workshop which Dan Grigsby is putting together which teaches iPhone development specifically to Ruby developers.

Shopify is shipping 8 people to the conference. Hope to see you there :-)

Ottawa friends of mine launched Gazaro, a really remarkable comparison shopping engine that transcribes a full history of price fluctuations and allows you to see how good of a deal a deal really is.

I’ve been using it for the last few weeks and it really works great, give it a try.

Now to get Shopify’s product base on there…

Thanks to @igrigorik for the link to this wonderful Benjamin Zander talk

Facebook’s server installation has been a point of interest for a long time. When an engineer on the memcached mailing list casually mentioned that they are running 4TB worth of caches, a lot of people got their panties in a knot.

Since then Facebook has been more forthcoming with details about their exciting architecture. For example they explained how they created a custom Mysql to solve problems with cross-datacenter cache expiry and replication lag which is a great read for anyone looking into multi location hosting.

Today there was another memcached centric post that talks about their fork of the memcached codebase which they also made available on github.

We use more than 800 servers supplying over 28 terabytes of memory to our users. Over the past year as Facebook’s popularity has skyrocketed, we’ve run into a number of scaling issues. This ever increasing demand has required us to make modifications to both our operating system and memcached to achieve the performance that provides the best possible experience for our users. [...]

Fascinating read.

So there is a lot of talk about Phusion Passenger lately and I feel the need to chime in here. David pointed out that Shopify is running on passenger which is something I announced on Twitter a few months ago.

Some context on Shopify’s installation: We launched Shopify originally on Lighttpd with FastCGI and later migrated to nginx with mongrels. Obviously we had to use HAProxy between Nginx and mongrels to avoid the dreaded “queue behind long running process” problem. We also added Monit to the mix which observed all mongrels to make sure that everything is running according to plan. After a process reaches 260 mb of memory we signal it to shut down after the next request so that a new one can start out with less memory. For this we added runit to the mix which supervises the mongrels and starts them up quickly once they hit the ground.

It’s important to note that we are not talking about a memory leak here. The reason for the 260mb ceiling comes from two issues with Ruby’s garbage collector:

1. It allocates memory in very large chunks once the available memory gets low. This means a 140mb process increases to 260mb in a single go. It also never gives memory back to the operating system because Ruby’s GC is not able to move objects. Once it adds an object into the newly allocated space and that object remains alive, it cannot yield memory back to the OS.
2. Because Ruby’s garbage collector uses mark and sweep it has to traverse the entire memory space in search of pointers. There are no generations that help with that. It means that GC cycles become longer and longer the more memory is available. -Rails mitigates these issues by moving a full GC run behind a HTTP response, into the time period when the process is waiting for a new request (Update: Rails doesn’t do this anymore) but performance monitoring tools such as NewRelic clearly show that average response times is directly correlated with the amount of memory used across the server farm.

Now why did we switch to Passenger? Simple: the keyword is remove moving parts.

Every additional tool you add will come with it’s own bugs. Many people I talked to over the past years considered haproxy to be the most solid piece of infrastructure in their stack but even there was a really nasty bug recently (search for request queue handling).

We treat our server farm very similar to Shopify’s codebase. We are in this for the long haul and we cannot accept complex solutions when simple ones present themselves. Maintainability of our code and servers is paramount to the long term success of our product. Yes the Mongrel setup worked very well but Passenger allowed us to remove: Nginx, Haproxy, Runit and Monit. That’s a nice refactoring!

At the same time Passenger introduced some tangible improvements. We switched to enterprise ruby to get the full benefit of the COW memory characteristics and we can absolutely confirm the memory savings of 30% some others have reported. This is many thousand dollars of savings even at today’s hardware prices. We allow Passenger to adaptively spawn more processes with demand but most of the time our application servers are running about 40 processes to handle more than a million dynamic requests a day. However, because passenger constantly despawns and respawns rails processes they always stay fresh, run short GC cycles and are generally a lot more responsive. All this means that the total amount of memory that is used by Shopify during normal operations went from average of 9GB to an average of 5GB. We evenly distributed the savings amongst more Shopify processes and more memcached space which moved our average response time from 210ms to 130ms while traffic grew 30% in the last few months.

In conclusion: I cannot see any reason to choose a different deployment strategy at this point. Its simple, complete, fast and well documented.

openvista posted their interview with me in which I talk about entrepreneurship and the early days of Shopify.

InfoQ posted the video to my rockstar memcaching presentation from ruby fringe.

Rockstar Memcaching Video

I’ve been playing around with mobile development lately which is a nice change of pace. Before Shopify and Rails I used to count myself amongst the ranks of the C++ desktop developers ( go WTL!! ) so in many ways the concepts of mobile development feel like the good’ old times—without all the things that drive you up the walls.

What’s so fun about it is the innocence of it all. It’s the gold rush all over again. For example that bastard Hampton managed to sell his Wikipedia browser iPedia 50k times. Check out www.mobileorchard.com which just published an interview with him.

via youtube

Brand and community development where always the guiding principles behind Shopify and today we launched an important aspect of this: Your visitors can now comment on blog posts.

Read more about it on the shopify blog

We will provide an import tool for wordpress and other blogging systems soon (implemented as an open source shopify api app. )

Color me impressed.

Matt Mccray ported Liquid to javascript. Go here to get the gist of it (yea, bad pun)

Pretty awesome work Matt :-)

<script>
 Liquid.readTemplateFile = function(path) {
  var elem = $(path);
  if(elem) {
    return elem.innerHTML;
  } else {
    path +" can't be found."; // Or throw and error, or whatever you want...
  }
 }

 var tmpl = Liquid.parse("{% include 'myOtherTemplate' with current_user %}"); 
 alert( tmpl.render({ current_user:'M@' }));
</script>

<script type="text/liquid" id="myOtherTemplate">
  Hello, {{ current_user }}!
</script>