Solving the random Rails performance problem

Last week I uncovered a tricky performance problem on Scout I wanted to share.

In seemingly random fashion, when accessing my account home page, the render time was 5-6 seconds – way too slow. Other times, the render time would be far less than a second.

Here’s what I did to debug:

1. Tailed my production log
2. Opened up the mysql console
3. Generated a slow request
4. Ran the mysql SHOW PROCESSLIST command

Viola!

Plain as day, I saw a query taking 30+ seconds. A simple mysql EXPLAIN on the query revealed a missing index. A new index fixed the issue. This query was outside the Rails application – a background job – so I never saw the query when optimizing the application.

Checkout the impact on the DB server load:

It’s a great lesson that you really need the complete picture of your infrastructure when something goes wrong – just looking at the Rails application often doesn’t cut it.

P.S. You’ll love what’s coming up with Scout – finding these problems gets a lot easier.

Related:

• MySQL EXPLAIN
• MySQL SHOW PROCESSLIST
• What’s in store for Scout in 2009
• 4 Simple Steps to Detect & Fix Slow Rails Requests