Posted by tobi — 01:14 PM Nov 15
So there is a lot of talk about Phusion Passenger lately and I feel the need to chime in here. David pointed out that Shopify is running on passenger which is something I announced on Twitter a few months ago.
Some context on Shopify’s installation: We launched Shopify originally on Lighttpd with FastCGI and later migrated to nginx with mongrels. Obviously we had to use HAProxy between Nginx and mongrels to avoid the dreaded “queue behind long running process” problem. We also added Monit to the mix which observed all mongrels to make sure that everything is running according to plan. After a process reaches 260 mb of memory we signal it to shut down after the next request so that a new one can start out with less memory. For this we added runit to the mix which supervises the mongrels and starts them up quickly once they hit the ground.
It’s important to note that we are not talking about a memory leak here. The reason for the 260mb ceiling comes from two issues with Ruby’s garbage collector:
- It allocates memory in very large chunks once the available memory gets low. This means a 140mb process increases to 260mb in a single go. It also never gives memory back to the operating system because Ruby’s GC is not able to move objects. Once it adds an object into the newly allocated space and that object remains alive, it cannot yield memory back to the OS.
- Because Ruby’s garbage collector uses mark and sweep it has to traverse the entire memory space in search of pointers. There are no generations that help with that. It means that GC cycles become longer and longer the more memory is available.
-Rails mitigates these issues by moving a full GC run behind a HTTP response, into the time period when the process is waiting for a new request(Update: Rails doesn’t do this anymore) but performance monitoring tools such as NewRelic clearly show that average response times is directly correlated with the amount of memory used across the server farm.
Now why did we switch to Passenger? Simple: the keyword is remove moving parts.
Every additional tool you add will come with it’s own bugs. Many people I talked to over the past years considered haproxy to be the most solid piece of infrastructure in their stack but even there was a really nasty bug recently (search for request queue handling).
We treat our server farm very similar to Shopify’s codebase. We are in this for the long haul and we cannot accept complex solutions when simple ones present themselves. Maintainability of our code and servers is paramount to the long term success of our product. Yes the Mongrel setup worked very well but Passenger allowed us to remove: Nginx, Haproxy, Runit and Monit. That’s a nice refactoring!
At the same time Passenger introduced some tangible improvements. We switched to enterprise ruby to get the full benefit of the COW memory characteristics and we can absolutely confirm the memory savings of 30% some others have reported. This is many thousand dollars of savings even at today’s hardware prices. We allow Passenger to adaptively spawn more processes with demand but most of the time our application servers are running about 40 processes to handle more than a million dynamic requests a day. However, because passenger constantly despawns and respawns rails processes they always stay fresh, run short GC cycles and are generally a lot more responsive. All this means that the total amount of memory that is used by Shopify during normal operations went from average of 9GB to an average of 5GB. We evenly distributed the savings amongst more Shopify processes and more memcached space which moved our average response time from 210ms to 130ms while traffic grew 30% in the last few months.
In conclusion: I cannot see any reason to choose a different deployment strategy at this point. Its simple, complete, fast and well documented.
Articles
Hongli Lai 15 Nov 13:50
Thanks for the kind words tobi. :)
However, I’m not sure I agree with the following statement: “Rails mitigates these issues by moving a full GC run behind a HTTP response, into the time period when the process is waiting for a new request”
As far as I know Rails does not do this.
tobi 15 Nov 14:12
I think you are right actually. I had a quick check and there is no code in rails that does this. I’m almost positive it used to do this, also the mongrel rails handler still seems to do it. Maybe Jeremy remembers what happened to the GC run.
Nikolay Kolev 15 Nov 15:46
Passenger has been working outstandingly for me, too (6+ months already). It gives me the peace of mind I needed so badly.
Grant 15 Nov 15:57
Thanks for this write-up. I was really curious about Passenger, but really comfortable with my current setup (similar to your old one). I’ll have to give it a try.
Paul 15 Nov 19:43
Great article Tobi, I definitely agree with “remove moving parts”. Going to have to try this out on my setup.
ActsAsFlinn 15 Nov 21:21
Agree 100%. Passenger+REE really does everything it claims. Have had great experiences with it on some very high profile sites. Another benefit, it really cuts down deployment to pull new code, symlink, touch tmp/restart.txt.
Scott 15 Nov 21:27
Why does it say ‘nginx/0.5.35’ in the server identifier at shopify.com if you’re running passenger?
hosiawak 15 Nov 21:34
Scott: they probably still use Nginx to proxy to Apache, it’s much faster serving static files.
Cody Fauser 15 Nov 22:09
Scott,
shopify.com is just our brochure page. Tobi is talking about the Shopify application that powers all of the Shopify stores. Shopify the application is running Nginx proxied to Apache. Try: curl -I www.snowdevil.ca.
We left Nginx in front of Apache because we have a very complex Nginx configuration that deals with SSL certs for several domains, rewrite rules, etc. We simply swapped the Haproxy + Mongrel portion of the stack with Passenger. This allowed us to make the switch to Passenger in minutes instead of spending a bunch of time porting our large and complex Nginx config to Apache. Also, since the switch to Passenger originated as an experiment this also allowed us the ability to quickly switch back to the Haproxy + Mongrel setup if we had to.
izidor 16 Nov 02:22
Tobi,
passenger benefits single server setups, but the clustered setups require balancer and in that case it is usual to just have mongrels running on servers and single haproxy doing the load-balancing among all of them, isn’t it?
There is no big benefit in such case for using passenger, or is it?
How do you load-balance your multi-server setup with passenger?
Michael Koziarski 16 Nov 04:45
@izidor: Passenger saves considerable effort with multi-server setups too, it mitigates the need to run a ‘special’ load balancer like haproxy as each http server can handle multiple simultaneous connections. With mongrel you can only realistically handle one at a time.
Before anyone jumps in and suggests that that’s rails’ fault for not being ‘threadsafe’, Ilya’s article should be required reading:
http://www.igvita.com/2008/11/13/concurrency-is-a-myth-in-ruby/
Michael Koziarski 16 Nov 04:48
Tobi,
Great write up that corresponds nicely with all my experiences so far. While none of my passenger projects are anything like as large as shopify, every single one of them shows significant memory savings and much simpler and less error prone deployments.
Jonas Galvez 16 Nov 10:05
So have you ditched Nginx altogether or is it still used in any way (eg., serving static files)?
tobi 16 Nov 10:59
Jonas: We still use nginx at the moment in front of apache because our (rather complicated) routing / proxy / ssl setup is handled nicely by it. I like nginx a good deal better than apache because it’s config file is very powerful and intuitive.
When we redo our setup in the future for a move we are planning, we will eliminate nginx from the dynamic request pipe simply to remove moving parts and flatten the pipe.
We also use nginx for serving static assets which is a problem domain where it excels at. However, static files will probably move to S3 and S3’s new CDN service very soon so this component may be eliminated from the stack.
Brian 16 Nov 13:26
We moved WellCare.com to use Passenger, and decreased our pain factor in just the same way. What was Mongrel + HAProxy behind hardware load balancers is now just Passenger behind hardware LBs. It’s taking millions of requests per week – many long running on the physician’s portal – and serving them all really well. We couldn’t be more pleased.
Greg Willits 20 Nov 04:18
I wish I could say the same about Passenger. I tried it with two apps, and it has compatibilty problems with both. Further, having posted pretty clear info about each problem on their forum site, neither one was apparantly interesting enough to look into. There was never any questions or suggestions.
Passenger would be a welcome change to Rails’ otherwise convoluted deployment options (Apache + proxy + mongrel_cluster + monit in my case), but the fact that it failed with two otherwise long-running apps leaves me to believe it’s a crapshoot, and I’ll never know when even an app that works might fail because of a Passenger problem.
vishi 22 Nov 09:21
I am another satisfied passenger user. Have been using it since 6 months on a few of my. Apps. I love the simplicity.
Charles Oliver Nutter 07 Dec 05:44
I’d be curious to see if Shopify could run on JRuby. With JRuby + Rails 2.2.2, you would have a single process for everything across cores. Could be another significant memory reduction, if performance was satisfactory. Feel free to contact me directly if you’d like to try it.
Philippe 14 Dec 21:21
Thanks for the article and comments. We’ll see how it goes, we just made the switch too and kept Nginx for now so it’s easier to go back to mongrel in case we needed to. We got used to nginx, it’s the single piece of software that never crashed on us for months.
Not only will we get rid of our seesaw + monit + mongrel setup, mongrel is not maintained anymore, I’d be pretty happy to not go back to using it and have to remember all the ports and instances started…