Log files are one of the most important aspects of any web service. A webapp with a well designed logging strategy will allow you to essentially go back in time to track down even the most obscure bug. Unit tests have diminished the importance of log files somewhat but how do you write a unit test for a bug that only happens on server 2 around 2am in the morning when user fred is logged in?

Most of us run exception notification services such as exception_logger, hoptoad, exceptional or simply but log exceptions to a DB table ( MySQL protip: make that table MyISAM, otherwise exceptions that are added during a transaction will be removed when that transaction rolls back – duh ). Those exception notifications are great but they never provide a lot of context for how the user got to the point.

Lastly there is also the role that log files play in customer support. Have you ever gotten a complaint about data disappearing from your service? With good logging you can tell your customer in a matter of minutes that employee Bob went on data rampage friday evening before handing in his resignation.

At Shopify we use syslog-ng to have a centralized logging server which collects all the logs from the various machines in our cluster and combines the log files together. We used to give everyone access to this box for log analysis but as we grew this became a bit impractical. To solve this, we created Clarity, which provides a very nice web interface for the two staple tools of log analysis: grep and tail -f.

Clarity is very lightweight and only requires a few dependencies such as eventmachine and json. It’s completely evented which means that you can have many different greps and tails running at the same time on a single instance (as much as the server can handle). It even stops the grep utility on the server when you hit stop in your browser.

  ~  $ sudo gem install clarity 
  Password:
  Successfully installed clarity-0.9.8
  1 gem installed
  Installing ri documentation for clarity-0.9.8...
  Installing RDoc documentation for clarity-0.9.8...
  Could not find main page README.rdoc # anyone know how to get rid of this?!
  Could not find main page README.rdoc
  Could not find main page README.rdoc
  Could not find main page README.rdoc

  ~  $ clarity /var/log --include '*/**'
  Clarity 0.9.8 starting up.
   * listening on 0.0.0.0:8080
   * Log mask(s): */**

Additional command line parameters are:

~  $ clarity --help
Usage:  clarity [options] directory

Specific options:
    -p, --port=PORT                  Port to listen on
    -b, --address=ADDRESS            Address to bind to (default 0.0.0.0)
        --include=MASK               File mask of logs to add (default: **/*.log*)
        --user=USER                  User to run as

Password protection:
        --username=USER              Enable httpauth username
        --password=PASS              Enable httpauth password

Misc:
    -h, --help                       Show this message.

Clarity is now used by our support staff on a daily basis. It’s been so successful internally that we decided to release it as open source. You can read more about it on the github page. The URL structure of Clarity is pretty simple so it’s easy to add links to your internal admin area that directly open log files with the appropriate terms prefilled. You can for example add a link to a search that shows you all the DELETE requests of a certain store directly to your support system. This means that blaming Bob will only take one click in the future.