Doug Breaker

Title:

Bio:

Recent posts

Monitor a Django app with Scout

BY Doug Breaker

In this post, I'll show how to setup Scout to monitor the performance of SQL queries, external HTTP calls, template rendering, and more in Wagtail , a Django CMS app. Wagtail is a fast, modern opensource content management system built on Django. Used at NASA, Google, MIT, and more, it's ...

Read More →

 

Monitoring Django apps on Heroku

BY Doug Breaker

I don't know of an easier way to deploy a Django app than letting Heroku do the work. That said, how do you stay on top of your app's performance, errors, and stability post-launch? Running an app on Heroku is a blissful experience, but it presents some monitoring challenges that ...

Read More →

 

Part I: How not to structure your database-backed web apps

BY Doug Breaker

Most scientific papers are unlikely to change your day-to-day approach as a Rails web developer. How not to structure your database-backed web applications: a study of performance bugs in the wild Yang et al., ICSE'18 is the exception to that rule. This study examined 12 popular, mature, opensource Rails apps ...

Read More →

 

Finding slow ActiveRecord queries with Scout

BY Doug Breaker

Once your Rails app begins seeing consistent traffic, slow SQL queries will likely rear their ugly head. Simple things like using find_by_sql can make a big improvement, but how can you easily tell where your app is slowing down? While PostgreSQL and MySQL can log slow queries, it's difficult to ...

Read More →

 

Finding and fixing N+1 queries in Django apps

BY Doug Breaker

The Django ORM makes it easy to fetch data, but there's a downside: it's easy to write inefficient queries as the number of records in your database grows. One area where the ease of writing queries can bite is you is with N+1 queries ( Rails N+1 queries can kill ...

Read More →

 

Why put Rust in our Python Monitoring agent?

BY Doug Breaker

Prior to adding Python performance monitoring , we'd written monitoring agents for Ruby and Elixir . Our Ruby and Elixir agents had duplicated much of their code between them, and we didn't want to add a third copy of the agent-plumbing code. The overlapping code included things like JSON payload ...

Read More →

 

Your Rails & Elixir performance metrics 📈 inside Chrome Dev Tools

BY Doug Breaker

Browser development tools - like Chrome Dev Tools - are vital for debugging client-side performance issues. However, server-side performance metrics have been outside the browser's reach. That changes with the Server Timing API . Supported by Chrome 65+, Firefox 59+, and more browsers, the Server Timing API defines a spec ...

Read More →

 

Scout's top-secret 4-point observability plan

BY Doug Breaker

Observability: the degree to which you can ask new questions of your system without having to ship new code or gather new data. Above is my slightly modified definition of observability, mostly stolen from Charity Majors in Observability: A Manifesto . Observability is increasingly important . Modern apps and services ...

Read More →

 

Django & Flask Performance Monitoring

BY Doug Breaker

7/31/18 Update: See how Scout monitors a real-world Django app , tracking SQL queries, deploys, slow requests, and more. GitHub's State of the Octoverse 2017 revealed that Python is now the second-most popular language on GitHub, with 40 percent more pull requests opened in 2017. We couldn't help but notice. ...

Read More →

 

Rollbar+Scout: a legit New Relic alternative

BY Doug Breaker

The New Relic price tag goes up dramatically as your server footprint grows. This might not be an issue if you are utilizing New Relic's full product suite, but what if you just care about error and performance monitoring? In that case, there's a solution that offers richer features as ...

Read More →

 

Setting up a Rails app for CodeBuild, CodeDeploy, and CodePipeline on AWS

BY Doug Breaker

If you've followed along with our previous episodes, we've covered many different aspects of setting up a production service. We've used many different products to simplify the day-to-day operations of running and maintaining an application. We've used Scout for monitoring our application, LogDNA for aggregating our logs, HoneyBadger for our ...

Read More →

 

5 traits of teams that make on-call less terrible for developers

BY Doug Breaker

Over the past two weeks, there's been considerable discussion on whether developers should be on-call. I understand the frustration of the anti on-call party. If you go to school to be a doctor, you know that being on-call is likely in your future. You didn't know that being on-call would ...

Read More →

 

Deploying a Faktory worker to AWS Fargate

BY Doug Breaker

Looking for a fresh, 2018 approach to deploying a Rails app to AWS? We've partnered with DailyDrip on a series of videos to guide you through the process. We're covering how to Dockerize a Rails app, AWS Fargate, logging, monitoring, setting up load balancing, SSL, CDN, and more. In the ...

Read More →

 

Deploying Faktory to AWS Fargate

BY Doug Breaker

Looking for a fresh, 2018 approach to deploying a Rails app to AWS? We've partnered with DailyDrip on a series of videos to guide you through the process. We're covering how to Dockerize a Rails app, AWS Fargate, logging, monitoring, setting up load balancing, SSL, CDN, and more. In today's ...

Read More →

 

Scout <> Rollbar Integration: unifying your stability metrics 🚀

BY Doug Breaker

When things are going wrong, the more signals you can view from one screen, the better. Today, we're excited to announce an integration with Rollbar . This brings Rollbar's best-of-breed error monitoring into Scout's performance-focused UI , creating a single source for your stability metrics: We display active Rollbar items ...

Read More →

 

AWS Disaster Recovery: Configuring a Read Replica and a Multi-AZ

BY Doug Breaker

We've partnered with DailyDrip on a two-part series guiding you through Automated Backups and Disaster Recovery on AWS. This is part 2. You can read part 1 here . This is the second portion of a two-part series on automated backups and disaster recovery for AWS. In part 1 , ...

Read More →

 

AWS Disaster Recovery: an overview and a backups tutorial

BY Doug Breaker

> We've partnered with DailyDrip on a two-part series guiding you through Automated Backups and Disaster Recovery on AWS. This is part 1. Read part 2 , which covers read replicas and multi-az. Have you ever lost data? It can make for a bad day. Now, imagine you lose your ...

Read More →

 

Deploying to AWS Part V: the final punch list (load balancer, CDN, SSL)

BY Doug Breaker

Looking for a fresh, 2018 approach to deploying a Rails app to AWS? We've partnered with DailyDrip on a series of videos to guide you through the process. We're covering how to Dockerize a Rails app, AWS Fargate, logging, monitoring, setting up load balancing, SSL, CDN, and more. In our ...

Read More →

 

Tutorial: Distributed Tracing in Ruby with OpenTracing

BY Doug Breaker

It's a lot harder connecting the dots of the request lifecycle when the final response is built from a number of separate microservices. However, distributed tracing - which connects a transaction trace across microservices - is getting a lot easier. In this short tutorial, I'll show how to add distributed ...

Read More →

 

A tour of Python monitoring tools

BY Doug Breaker

Prior to releasing our Python Performance Monitoring agent, we took a look at the Python ecosystem to see how Scout can compliment the existing landscape. What follows is a summary of our internal report. The Python ecosystem has a wealth of monitoring tools. That said, making sense of each tool's ...

Read More →

 

Deploying to AWS Part IV: Performance monitoring with Scout

BY Doug Breaker

Looking for a fresh, 2018 approach to deploying a Rails app to AWS? We've partnered with DailyDrip on a series of videos to guide you through the process. We'll be covering how to Dockerize a Rails app, AWS Fargate, logging, monitoring, and CDN support. Today, we're configuring application performance monitoring ...

Read More →

 

Tutorial: Tracing Python Flask requests with OpenTracing

BY Doug Breaker

A transaction trace is a GPS system for web performance: it paints a rich picture of the flow of a web request through your code. So, why doesn't everybody trace? I believe there are two reasons: Complex instrumentation : Adding in-app tracing instrumentation is more involved than calling logger.info() for ...

Read More →

 

Are You Monitoring Your Machine Learning Systems?

BY Doug Breaker

We're partnering with FusionCell to survey folks about how they monitor machine learning learning applications. FusionCell's Robert Dempsey joins us today to share some of the personal pain he's faced monitoring machine learning apps. Robert has been helping companies and engineers learn about, build and leverage machine learning systems. When ...

Read More →

 

Deploying to AWS Part III: Log aggregation

BY Doug Breaker

Looking for a fresh, 2018 approach to deploying a Rails app to AWS? We've partnered with DailyDrip on a series of videos to guide you through the process. We'll be covering how to Dockerize a Rails app, AWS Fargate, logging, monitoring, and CDN support. In our last video , we ...

Read More →

 

Deploying to AWS Part II: Running a Rails app on Fargate

BY Doug Breaker

What's the 2018 approach to deploying a Rails app to AWS? We've partnered with DailyDrip on a series of videos to guide you through the process. We'll be covering how to Dockerize a Rails app, AWS Fargate, logging, monitoring, and CDN support. Today's video is the next in the series ...

Read More →

 

Deploying to AWS Part I: Dockerizing a Rails app

BY Doug Breaker

What's the 2018 approach to deploying a Rails app to AWS? We've partnered with DailyDrip on a series of videos to guide you through the process. We'll be covering how to Dockerize a Rails app, AWS Fargate, logging, monitoring, and CDN support. Today, we will be working through a few ...

Read More →

 

Observability: the new wave or another buzzword?

BY Doug Breaker

OH - "Observability - because devs don't like to do "monitoring" we need to package it in new nomenclature to make it palatable and trendy." — Cindy Sridharan (@copyconstruct) July 28, 2017 I think you'll be hearing more about observability in 2018. In this post, I share why I believe ...

Read More →

 

What we shipped in 2017

BY Doug Breaker

In 2017, we focused our engineering time on reducing the painful, time-consuming investigation workflow that comes with fixing performance issues. We think that approach is working: we're now analyzing billions of web requests and background jobs every day. Here's some of the highlights from 2017: Database Monitoring The database is ...

Read More →

 

Real User Monitoring with Raygun

BY Doug Breaker

Most of a web page's load time is on the front-end. You can monitor the performance of the end-to-end page load time with a Real User Monitoring (RUM) service, and one of the more attractive ones is Raygun Pulse . Let's take a deep dive. Installing Raygun Pulse Installing Pulse ...

Read More →

 

Python Application Monitoring: comparing New Relic and Opbeat

BY Doug Breaker

App Overview Below is the initial page you see when checking on the health of your Python app: The contrast in breadth is clear when looking at the app overview page. New Relic has a denser display while Opbeat has a clean, polished look. Some thoughts: The overview chart - ...

Read More →

 

Introducing the Scout database monitoring addon

BY Doug Breaker

Perhaps the most significant performance problem spot in web apps is the database. In apps monitored by Scout, database queries account for nearly a third of the time spent in the average Rails web request. Just as important, the more time an app spends in the database, the more volatile ...

Read More →

 

Super-charge ActiveRecord#explain with pg-eyeballs 👀

BY Doug Breaker

When I have a slow Postgres ActiveRecord query, one of the first tools I reach for is EXPLAIN . pg-eyeballs is a Ruby gem that makes acting on this frequently noisy output easier. Those of you that know your way around ActiveRecord might be confused. ActiveRecord comes with an #explain ...

Read More →

 

Does team size impact app performance?

BY Doug Breaker

In 2014, CodeClimate published a blog post investigating the impact of team size on code quality . I was curious: are there correlations with app performance and team sizes as well? Determining if an app is well-performing is more subjective than the GPA-like results a code quality analysis provides. I'll ...

Read More →

 

A 5-point Rails app performance audit

BY Doug Breaker

Before we talk performance, lets talk entropy. Entropy usually refers to the idea that everything in the universe eventually moves from order to disorder, and entropy is the measurement of that change. Like entropy, the performance of a Rails app will trend toward disorder . An N+1 database query here, ...

Read More →

 

DataDog vs. Scout: which is right for you?

BY Doug Breaker

You're probably confused where DataDog and Scout overlap. You're not alone! They've each offered similar products since their inception, and come on, they both have dogs in their logos! Here's a timeline to clear things up: Both Scout (2009) and DataDog (2010) started around the same time and focused on ...

Read More →

 

New Relic vs. Scout: which is right for you?

BY Doug Breaker

There's no type of monitoring tool that can get to the heart of a performance problem faster than Application Performance Management (APM). If you're making this decision, you may be choosing between New Relic and Scout . There are cases where New Relic is a better fit, clear cases where ...

Read More →

 

Visualizing Rails logs With Jupyter Notebook

BY Doug Breaker

Your Rails app appears to be humming along just fine. But then, out of the blue, one of your most important customers emails your support team: "the site is loading slowly - is something up"? You frantically browse around the site, but everything seems fine. You double-check the charts: things ...

Read More →

 

ActiveRecord: biting the hand that feeds you

BY Doug Breaker

Richard Schneeman (better known as Schneems) recently wrote about how he reduced his database server load by 80% ...with one simple trick . In the Hacker News discussion that followed, much of the debate was on the merits of using an ORM like ActiveRecord...or not: In every case I can ...

Read More →

 

Finding fast (but expensive) database queries

BY Doug Breaker

A database is a shared resource for a Rails application. A single misbehaving query hitting your database can slow down unrelated queries throughout your system. When we began running our own database monitoring against our apps, we discovered several expensive queries that weren't apparent when viewing individual web endpoints and ...

Read More →

 

Autocorrect with Postgres Trigrams

BY Doug Breaker

Rails Postgres Guru Greg Navis shares another powerful tip. Greg is writing The Ultimate Guide to Deploying Rails on Heroku. Subscribe to his mailing list for performance tips & tricks. In my previous article on Postgres Trigrams , we used pg_trgm to speed up LIKE and other operators (even on ...

Read More →

 

We're sharing our roadmap

BY Doug Breaker

Years ago when we started Scout, we were vague on our roadmap. We recognized priorites and plans can shift and we didn't want that shifty-ness exposed. However, things have changed by 2017. All of us have a general understanding of this behavior: we see it everyday in the GitHub repos ...

Read More →

 

5 hard-earned lessons from a decade of Rails performance work

BY Doug Breaker

The last 3,650 days of my professional life have been focused on making Rails apps faster. Below are five lessons I've learned the hard way. 1. Facts alone won't convince the business folk to care about performance The typical pitch to prioritize making an app faster & more reliable goes ...

Read More →

 

Monitoring GenServer calls with Scout

BY Doug Breaker

The latest release (0.3.0) of Scout's Elixir APM package adds the ability to monitor GenServer calls. What is GenServer and how do you monitor it? A short intro to GenServer If you're coming from a Ruby background, you've probably used Sidekiq or similar to do background work outside of the ...

Read More →

 

Introducing free app monitoring for Elixir apps

BY Doug Breaker

A year ago, our team started playing with Elixir. We liked it - a lot. In fact, it's the first language we've used since Ruby that we'd unanimously love to build web apps with. With that, we're excited to announce BETA support for Elixir app monitoring @ Scout. Get started ...

Read More →

 

Alerting: get notified when performance goes bad

BY Doug Breaker

> We've added one of the top feature requets to app monitoring: alerting. Alerting keeps your team updated if your app's performance degrades. Alerts can be configured on the app as a whole and on individual endpoints. Metrics include: mean response time 95th percentile response time Apdex error rate throughput ...

Read More →

 

RailsConf 2017: 5 standout performance sessions

BY Doug Breaker

Along with a couple other members of the Scout team, I'm heading to sunny Phoenix, Arizona in mid-April for RailsConf . I'm a performance addict, so I've been scanning through the sessions and making notes of the ones that stand out. My performance short-list is below. 1. What's my app ...

Read More →

 

The danger of Rails.env.production?

BY Doug Breaker

When I do a code review, one of the scariest things I see is logic like this: if Rails.env.production? do_additional_work end Why? Your beautiful tests and tightly integrated CI system won't execute that code. You won't see that code execute as you refresh your browser in development. From syntax errors ...

Read More →

 

Five tools for profiling Rails apps

BY Doug Breaker

Starting with Fiveruns TuneUp in 2008, a number of in-browser Rails profilers have come, gone, or continue to exist but in a zombie-like half-alive state. Finding profilers that reliably work in 2017 with little effort was a bigger challenge than I thought. Here's a 2017 summary of the Rails profiler ...

Read More →

 

Rails::API vs. Sinatra vs. Grape: which Ruby microframework is right for you?

BY Doug Breaker

Rails dominates Ruby web frameworks: the next most popular framework, Sinatra, has 5% of the popularity of Rails . However, that doesn't mean non-Rails frameworks like Sinatra and Grape don't have their place. When does it make sense to step away from ActionController and use another framework? What are the ...

Read More →

 

Unicorn vs. Puma vs. Passenger: which app server is right for you?

BY Doug Breaker

The Ruby app server ecosystem has consolidated around three app servers in 2017: Unicorn , Puma , and Passenger 5 . What specific problems must an app server solve for Ruby? How do you pick the right app server? Is there a need for each of these app servers in ...

Read More →

 

Honeybadger's Ben Curtis on bootstrapping, competing aganst VC-funded companies, and life managing a large Rails app

BY Doug Breaker

Exceptions happen. To everyone. For half a decade, Honeybadger has given sanity to the art of bug hunting, monitoring exceptions for Heroku, eBay, DigitalOcean, and many more. I was able to steal some time from Ben Curtis, one of the Honeybadger co-founders, to talk about their origin story, life as ...

Read More →

 

Making sense of ActiveRecord joins, includes, preload, and eager_load

BY Doug Breaker

Like a pair of jumper cables, ActiveRecord's joins , includes , preload , and eager_load methods are incredibly useful, but also very dangerous when used incorrectly. Knowing when and where to use each approach - and even when to combine them - can save you considerable trouble as your app ...

Read More →

 

State of the 2017 Rails Stack

BY Doug Breaker

Are folks upgrading to Rails 5? Where is the Postgres vs. MySQL battle heading? Are devs embracing Puma and concurrency? If you're curious about the above, you've come to the right place. We collect gems used on the apps we monitor at Scout to assist with debugging issues and to ...

Read More →

 

Elixir foundations for Ruby Devs: Recursion

BY Doug Breaker

I can't recall writing a recursive method in Ruby. I think that's because I find recursion more confusing than iteration (ie Array#map ). Additionally, recursion isn't efficient in an imperative language like Ruby. While I find recursive functions a bit difficult to grok in any language, writing a recursive function ...

Read More →

 

2016 Year-in-Review

BY Doug Breaker

We've pushed a lot of features this year at Scout. Here's a look back at what we've added to app monitoring in 2016: February Completed a significant effort reducing our agent resource usage and open-sourced our agent overhead benchmarks . April Sidekiq monitoring Backtraces for N+1 queries ( background on ...

Read More →

 

There's a Slim Linter in your Atom

BY Doug Breaker

One of the technical bits I learned when joining Scout was Slim , a lightweight templating engine. I was a quick convert: Slim eliminates boilerplate code while making it easy to "break away" when full HTML markup is needed. I'm a big fan of Linters , especially when learning a ...

Read More →

 

Coming Soon: your Rails app performance trends & outliers, via email

BY Doug Breaker

I follow a simple rule before configuring a monitoring alert: if I receive this alert at 3am, will I act on it? If not, it shouldn't be an alert. Few performance-related alerts meet this criteria. For example, if our app is running 25% slower, it's not worth a hasty 3am ...

Read More →

 

Elixir foundations for Ruby Devs: transforming data

BY Doug Breaker

This is a guest post by Tomasz Kowal, a software developer currently working full time with Elixir at ClubCollect. He started with Erlang 6 years ago and is still amazed by the power functional languages provide. In his free time he likes tinkering with flying robots. Have you ever reached ...

Read More →

 

Introducing GitHub-enhanced Deploy Tracking

BY Doug Breaker

We're happy to introduce a lightweight addition to Scout: GitHub-enhanced deploy tracking . ​Deploy tracking makes it easy to correlate deploys to your app's performance. Scout's deploy tracking goes beyond chart markers: enable our GitHub integration to see which branch or tag was deployed, the team members that contributed, and ...

Read More →

 

DevTrace and the Art of Staying the F*** Out of the Way

BY Doug Breaker

DevTrace is a performance widget for your Rails applications in development. It sits unobtrusively in the corner of your page, just waiting to drop insight on your application: See stack traces, SQL timings, and more with just a click! This kind of insight is powerful. You can see how your ...

Read More →

 

How we're turbo-charging our traces with ScoutProf

BY Doug Breaker

The transaction trace is application monitoring . If you can't track a performance problem to a line-of-code, you aren't app monitoring. However, there's a gaping hole in the quality of transaction traces today. We've been instrumenting the quality of our traces since our launch, and noticed that 46% of the ...

Read More →

 

Know of an open-source Rails app in need of some performance love?

BY Doug Breaker

We've just released the BETA version of our new Ruby Profiler (Stackprof-inspired, but hopefully more readable). To really give it a go, we're looking for some open-source Rails apps that are need some performance love. If you know of - or our a contributor to - an open-source Rails app ...

Read More →

 

New Release: Memory Bloat Detection

BY Doug Breaker

When your app is experiencing memory bloat - a sharp increase in memory usage due to the allocation of many objects - it's a particularly stressful firefight. Slow performance is one thing. Exhausting all of the memory on your host? That can bring your application down. Memory bloat is a ...

Read More →

 

Faster PostgreSQL Searches with Trigrams

BY Doug Breaker

There's nothing quite like having a "tool-belt" full of tricks for getting the most performance out of your Rails app. This week, Rails Postgres Guru Greg Navis shares another powerful tip. A few months ago, I was working on a project that had about 100,000 users. Each user could have ...

Read More →

 

A Six-Pronged Rails Performance Philosophy

BY Doug Breaker

"An ounce of prevention is worth a pound of cure."• Benjamin Franklin Application performance problems can be annoying. With luck, you'll spend an hour or two resolving the problem and get back to your real job: building things. But what happens when the issues start piling up? What happens ...

Read More →

 

Product Update: enhanced zoom, 95th percentiles, trace diffs, and more.

BY Doug Breaker

Today we're excited to announce the release of our latest Ruby on Rails monitoring agent and a major UI update. This release continues our focus on quickly revealing your path to a faster, more reliable Rails app. Enhanced zoom Determining how to make your Rails app faster often involves investigating ...

Read More →

 

3 pleasantly surprising PostgreSQL Indexing tricks

BY Doug Breaker

Most Rails engineers know the basics of database performance. They know that if a query is slow, an index may be the solution. Some know the trade-offs between having and not having an index. Or why an index on a low-cardinality column might not help. But everyone is surprised when ...

Read More →

 

Rails Performance and the root of all evil

BY Doug Breaker

Donald Knuth wrote an often quoted paper in the 70s which is still referenced when talking about performance in web apps today. Premature optimization is the root of all evil. In my line of work, it is sometimes invoked as a sort of apology; an excuse for why more time ...

Read More →

 

Getting down with Stackprof: how we added N+1 query detection to Scout

BY Doug Breaker

Let's get the confusing part out of the way: we're going to get a little "meta". This post is about using one performance tool (Stackprof) to improve another performance tool (Scout's Rails Agent) so Scout can diagnose a common performance pitfall (N+1 queries). Did I just put "performance" in a ...

Read More →

 

"Scout Does Things Right": Gigmit on using Scout

BY Doug Breaker

> Benjamin Knofe, CTO of Gigmit , sat down with us to share how they use Scout to improve their performance. What is Gigmit? Gigmit is a platform to connect music professionals to one another. Artists get opportunities to play gigs, shows, and festivals all over the world by showcasing ...

Read More →

 

Monitor Sidekiq Jobs!

BY Doug Breaker

Teaser for a soon-to-be released capability: Overview metrics for all your background jobs Chart throughput, latency, error rate, and more. Detailed drill-down on slow jobs See what's making the job slow, identify N+1 queries, and more. Tech preview - want in? Background job monitoring currently supports sidekiq, and is tech ...

Read More →

 

3 Git Productivity Hacks

BY Doug Breaker

Most Ruby developers use Git for their version control system of choice. Git is a wonderful tool that can save you countless hours of lost productivity and makes collaborating with others a cinch. Git's distributed nature also allows devs to work anywhere with or without an internet connection without fear ...

Read More →

 

Now with Redis Instrumentation

BY Doug Breaker

Among our customers, Redis is the single most popular database . It's a terrific compliment to any of the relational database systems out there, so it finds a way into a lot of apps. We've added Redis instrumentation to our scout_apm gem - to upgrade: bundle update scout_apm PS - ...

Read More →

 

Java for Rubyists

BY Doug Breaker

The Scout Java Application Monitoring Agent is under active development and we have a few spots open in our alpha program. Email support@scoutapp.com for access. For many Rubyists, Ruby is the first language that they learn and perhaps the only programming language that they know. Ruby provides an excellent gateway ...

Read More →

 

The Dogs of Scout

BY Doug Breaker

If it isn't clear from our logo, or our offer to give you a puppy if you don't love Scout more than the competition , the Scout team loves dogs. Here are some of our favorite coding buddies: Dave's Dog: Bella Bella is my 10-year-old shepherd mix. I rescued her ...

Read More →

 

Which Ruby background job framework is right for you?

BY Doug Breaker

If you've been around the Ruby/Rails ecosystem for a bit you've likely heard the term 'background job' or 'offline processing'. But what does that actually mean? How do you know which tasks are suitable to be processed 'in the background'? Once you define those tasks, how do pick the right ...

Read More →

 

Overhead Benchmarks: New Relic vs. Scout

BY Doug Breaker

High monitoring overhead is a silent killer : your app's requests take longer, throughput capacity shrinks, end users requests start stacking up in a request queue, you react by provisioning more servers, and finally, more servers == more $$$. So how does Scout's overhead compare with the competition? To find ...

Read More →

 

State of the 2016 Rails Stack

BY Doug Breaker

What's a Rails stacks look like in the wild these days? Most popular Rails version? Most popular Ruby? Is Delayed Job still hanging on? If you're curious about the above, you've come to the right place. We collect gems used on the apps we monitor at Scout to assist with ...

Read More →

 

NEW! Request queuing and middleware instrumentation

BY Doug Breaker

Metrics for Christmas everyone! Get your metrics! We've added more metrics to Scout App Monitoring . These metrics will give you even greater visibility into the full request cycle of your Rails app: Request Queuing - judging if you have enough capacity to serve all your traffic is one of ...

Read More →

 

AcademicWorks' switch from New Relic to Scout

BY Doug Breaker

The engineering team at AcademicWorks , the leading provider of scholarship management solutions for public and private educational institutions, was frustrated. With over two million users and a datastore-heavy Rails app with hundreds of database instances, their ten-person development was feeling bogged down with the bloat of their existing application ...

Read More →

 

4 ways to get the most out of your Rails logs

BY Doug Breaker

Logging is the lowest common denominator of monitoring - it's low effort and low overhead. You can put ANYTHING into a log file (sans animated GIF memes - save that for Slack). The downside of being a logging addict: logs can quickly become a noisy mess. Digging through logs while ...

Read More →

 

Pillars of the Rails Monitoring Stack: 2018 Edition

BY Doug Breaker

Here's a behind-the-scenes rundown of how we ensure our apps are in peak condition in 2016* 2018. *Ed. Note- this is largely unchanged from 2016, so I've updated this post with our 2018 stack. Focus Service Approx. Price Uptime Pingdom $15/mo App Monitoring Scout $150/mo Log Monitoring ELK Stack $150/mo ...

Read More →

 

Thanksgiving Support

BY Doug Breaker

First, Happy Thanksgiving to all of our U.S. customers! We couldn't ask for a better group of folks to work with on an every-day basis. Second, our team will spending time with their family and friends tomorrow and Friday. We'll watch for urgent issues and will still be checking our ...

Read More →

 

App Monitoring for the Modern Dev Team

BY Doug Breaker

How we're building apps as developers is changing. Fast . The breadth of our responsibility is decreasing (yeah), yet the complexity of our code is increasing (meh). We've just launched our app monitoring service for the modern development era. We think it will dramatically change how you make your web ...

Read More →

 

Order your Scout cycling kit

BY Doug Breaker

We're big fans of the outdoors - and cycling - at Scout. I'm excited to announce Scout is the title sponsor for a development team in our home base of Fort Collins, Colorado . It's a super talented team you'll find all over North America in 2016. We'll have more ...

Read More →

 

App Monitoring: Stability, Stability, Stability ... and pricing.

BY Doug Breaker

We're officially in the Stability Phase™ of app monitoring: slamming our site with traffic, trying to break things, and adding resiliency for the spots that need it. Read on for an updated General Availability date, our planned stability improvements, and pricing information. General Availability GA is November 16th (GA will ...

Read More →

 

Learn about Go @ Scout: Denver Go Meetup Tomorrow

BY Doug Breaker

Go has become an increasing part of our stack at Scout. At tomorrow's Denver Gophers Meetup , we'll be delivering two short talks for the price of one: Dave will be talk about Native Ruby Extensions in Go . Lots of folks are using Ruby and Go together. We'll take ...

Read More →

 

Git-integrated app monitoring

BY Doug Breaker

Tracking down slow code is about to get a lot faster. We're excited to debut our Github Integration for Scout Application Monitoring: With the Github Integration enabled, instead of seeing backtraces for slow method calls (>= 500 ms), you'll see the relevant code in our UI. This goes beyond the ...

Read More →

 

From a spike in response time to a Git blame: our improved path to slow code

BY Doug Breaker

Before Scout, we used several app monitoring services. One of my biggest frustrations: I never found a way to select a spike in response time from an overview chart and view what's slow during that period. Those spikes are begging to be clicked. We wanted to make that happen. Introducing ...

Read More →

 

AWS re:Invent Day Two

BY Doug Breaker

The Scout conference team in full force! From left to right: Derek Haynes, Dave Anderson, me, and Megan Strahm. Our three big takeaways from conversations on the expo floor today: Developers are psyched about an alternative to New Relic for SaaS application performance monitoring. There are lots of polyglot applications ...

Read More →

 

AWS re:Invent Opening Day

BY Doug Breaker

It was a great opening day at AWS re:Invent in Las Vegas! Many, many developers and tech leaders coming by to see our vision of Next Generation Application Monitoring. If you're at the conference, come see us at booth 1130!

Read More →

 

App Monitoring Update

BY Doug Breaker

It's been just over a month since we opened our early access period for Scout App Monitoring. Whew! A lot has happened: What you've told us Over the last couple of weeks, our founders have talked with many of you via phone, Skype, or Hangout (if they haven't talked to ...

Read More →

 

Scout is going to AWS re:Invent!

BY Doug Breaker

Find us in Booth #1130 at AWS re:Invent. Be sure to stop in - meet the team, and rub Dave's head for good luck! Schedule a Meeting with a Scout Founder Derek (CEO) and Andre (CTO) would love to meet with you in Vegas. Contact me and I'll schedule a ...

Read More →

 

StackProf: The Holy Grail of Rails Profiling

BY Doug Breaker

Our Stackprof-inspired profiler, ScoutProf, is now in BETA. See our docs to get started. The holy grail of performance profiling is finding a tool that's safe to run in production. A tool that identifies slow code as it works "in the wild". Profiling code locally is never as good as ...

Read More →

 

Heading to the mountains

BY Doug Breaker

There's no better place for big thinking than the mountains. Just a short time ago, Scout was myself and Andre. Today, we're a fun, smart team of seven (and growing). We've got grand plans for the future of Scout (we'll be debuting those @ aws:reinvent ) and we're hunkering down ...

Read More →

 

StatsD. Build vs. Buy?

BY Doug Breaker

"Remember that time is money." A phrase used by Benjamin Franklin in Advice to a young Tradesman, written by an old One . Sage advice - and even after all these years, still relevant! It's no surprise that we all look for tools that make our jobs easier, deliver value ...

Read More →

 

Monitoring InfluxDB with Scout

BY Doug Breaker

We're using InfluxDB in our new app monitoring service . While InfluxDB hasn't reached 1.0 yet, it has loads of potential and has been holding up well during our BETA period. Don't worry, we'll talk more about InfluxDB in coming posts. So, how are monitoring InfluxDB performance? Here's how we ...

Read More →

 

Introducing easy StatsD with Scout

BY Doug Breaker

The easiest way to unleash StatsD. One agent. Minimal overhead. Robust language support. A unified monitoring solution for your servers and metrics. We've added StatsD support to our monitoring agent. With Scout, you are just minutes away from StatsD-backed charts and alerts. Use StatsD to report code execution times, user ...

Read More →

 

The making of app monitoring: the health dashboard

BY Doug Breaker

We're battle-scarred devs building the focused app monitoring service we've always wanted. We're blogging about the adventure below. Customers telling me our app is slow? I'm looking at a response time graph. On the front page of Hacker News? I'm looking at requests per-second and response time on a graph. ...

Read More →

 

Reversing the GoDaddy-ification of application monitoring

BY Doug Breaker

Scout is an "oops" company. We didn't build our product with the intention of turning it into a company, but we certainly can't imagine life without it today. Scout was started out of frustration . The prospect of setting up and using a Nagios-like server monitoring solution was so terrifying, ...

Read More →

 

The Curious Case of the StatsD Timer

BY Doug Breaker

Instrumenting our application with StatsD is easy, especially when we just stick to Counters and Gauges. These metrics return just a single value when implemented. When you get to Timers, however, StatsD steps up it's game and returns eight metrics. So let's explore the curious case of the timing metric. ...

Read More →

 

New Dog in the Scout Kennel

BY Doug Breaker

Please join us in welcoming Chris Schneider to the Scout team! We've had our eye on Chris for awhile now, and we couldn't be more excited about him joining the team. Nearly a Fort Collins native (rare around these parts), Chris joins us with over 15 years of development experience ...

Read More →

 

Hugging it Out @ Boulder DevOps w/StatsD

BY Doug Breaker

Last night, at DevOps Boulder, I presented the StatsD basics: key concepts, metric types & implementation strategies - and dug a little deeper. My mission was to talk about StatsD and how it made me a better developer by giving me a simple, repeatable tool that I could use to ...

Read More →

 

Kafka: "git branch" for production streams

BY Doug Breaker

I switched from SVN to Git for version control years back largely because it made experimenting a blast. Creating branches, viewing diffs, stashing - it was fun to veer off the highway and explore dirt roads. Making it easy to experiment on my dev box is great, but rolling those ...

Read More →

 

DevOpsDays Minneapolis

BY Doug Breaker

Some of the best things about summer are BBQ, baseball and roadtrips. As part of my summer celebration, I'm hitting the open road and heading up to Minneapolis, MN for DevOpsDays MSP. DevOpsDays MSP is a 2 day event on July 8th & 9th in downtown Minneapolis. This is their ...

Read More →

 

An intro to the Docker Stats API

BY Doug Breaker

If I was teaching Server Health 101, I'd start with four key metrics: CPU Usage Memory Usage Network I/O Disk Utilization The approach for fetching these metrics on Linux hosts is tried-and-true (hint: look in the /proc folder). However, I was curious about Docker Containers: where do I access these ...

Read More →

 

Docker ETP Program

BY Doug Breaker

Our friends at Docker today announced that we are part of their new ETP (Ecosystem Technology Partner) program. We couldn't be more excited! Docker has done a great job of creating an API that allows anybody to track performance metrics, container events and metadata about their containers. What we've done ...

Read More →

 

Implementing Docker event monitoring from scratch

BY Doug Breaker

Docker's API provides a ton of functionality around containers and images - but there is a hidden secret, one that is easy to miss in the documentation: Docker's API has the capability to report host wide events ! Container events like: die , restart & out of memory . With ...

Read More →

 

Django Application Monitoring (APM) with StatsD

BY Doug Breaker

Spring 2018 Update - This post covers instrumenting a Django app with StatsD. For a more full-featured APM solution, checkout Scout's Python Application Monitoring . One of the basic tenets in DevOps is that we measure EVERYTHING. StatsD is the open source darling that has quickly turned into our tool ...

Read More →

 

StatsD update to docker-scout

BY Doug Breaker

We've all been exploring Docker lately, and back in March, we published Monitoring Docker with docker-scout with how to get started with Scout and Docker. Today, we are excited to announce that we updated this container to include StatsD. Just like in our previous post, Rails App Monitoring (APM) with ...

Read More →

 

Rails App Monitoring (APM) with StatsD

BY Doug Breaker

Application monitoring doesn't have to be complicated. Introducing scout_statsd_rack , a drop-in Ruby gem for monitoring key performance metrics in your Rails app. scout_statsd_rack leverages StatsD for lightweight Rails app monitoring via Rack middleware. Lets see how fast we can go from no monitoring to a Rails performance dashboard plus ...

Read More →

 

DevOpsDaysRox

BY Doug Breaker

Last week, I spent Thursday (4/23/15) and Friday (4/24/15) at DevOpsDaysRox . If you've never attended a DevOpsDays events - find your closest one here and signup now. So what makes DevOps Day special? First, you'll find an active community that prides itself on inclusion and making everyone feel welcome. ...

Read More →

 

"Monitoring", "Sparkly DevOps Princess", "StatsD" ....BINGO!

BY Doug Breaker

The 2015 Conference Season is upon us and we couldn't be more excited. The first one of the year for us is right in our backyard - DevOpsDaysDenver . We're hoping to meet a lot of customers - and have a little fun. In talking with the event organizers, they ...

Read More →

 

Understanding page faults and memory swap-in/outs: when should you worry?

BY Doug Breaker

Imagine this: your library is trying to step up its game and compete in the Internet age. Rather than you browsing the shelfs, trying to remember how the Dewey Decimal works, you'll enter your book selections from your phone. A librarian will then bring your books to the front desk. ...

Read More →

 

Meet Mark Morris, Scout's New Team Member

BY Doug Breaker

Tap. Tap. Is this thing on? Hi. My name is Mark Morris, and I just joined the Scout team as the new Dev Evangelist. Whoa. I already like Scout, I don't need to be preached to. In my first two weeks, I've already had the chance to talk with some ...

Read More →

 

Monitoring Docker with docker-scout

BY Doug Breaker

When it comes to agent-based monitoring and Docker , you're typically choosing between two bad options: (1) install the agent and its dependencies directly on the host or (2) running an agent in every container. Installing the agent on the host is bad : Docker is best when everything is ...

Read More →

 

Twilio for SMS

BY Doug Breaker

Scout has long offered email-to-SMS alert delivery for free. Free is good, but there are a couple downsides: email-to-SMS doesn't work some places outside the US . Some foreign carriers don't enable it by default, and in some countries email-to-SMS just doesn't work well. with email-to-SMS, texts don't always come ...

Read More →

 

Understanding Linux CPU stats

BY Doug Breaker

Your Linux server is running slow, so you follow standard procedure and run top . You see the CPU metrics: But what do all of those 2-letter abbreviations mean? The 3 CPU states Let's take a step back. There are 3 general states your CPU can be in: Idle , ...

Read More →

 

The year at Scout - 2014 edition

BY Doug Breaker

2014 was a year of major updates to Scout. Some stats on what's been a fun year: Product Highlights Scout Realtime - In January , we released our open-source standalone realtime monitoring agent. Scout Realtime was the second most popular repository on Github during its release week (trailing Popcorn - ...

Read More →

 

From MySQL full-text search to Elasticsearch

BY Doug Breaker

Migrating backend search technologies on a high-throughput production site is no easy task, but Vector Media Group was recently faced with this decision. With a popular client site struggling under the load of complex MySQL full-text search queries, they recently switched to Elasticsearch . I spoke with Matt Weinberg to ...

Read More →

 

Restricting process CPU usage using nice, cpulimit, and cgroups

BY Doug Breaker

The Linux kernel is an incredible circus performer, carefully juggling many processes and their resource needs to keep your server humming along. The kernel is also all about equity: when there is competition for resources, the kernel tries to distribute those resources fairly. However, what if you've got an important ...

Read More →

 

Understanding disk inodes

BY Doug Breaker

You try creating a file on a server and see this error message: No space left on device ...but you've got plenty of space: df Filesystem 1K-blocks Used Available Use% Mounted on /dev/xvda1 10321208 3159012 6637908 33% / Who is the invisible monster chewing up all of your space? Why, ...

Read More →

 

From Ruby to Go: a rewrite for the future

BY Doug Breaker

During a team camp among the lofty peaks of Breckenridge, Colorado, we talked a lot about the future of Scout and monitoring in general. Big mountains and nature have a way of doing that. One thing that was getting our nerd juices flowing: Go . At Monitorima in May, it ...

Read More →

 

Slow Server? This is the Flow Chart You're Looking For

BY Doug Breaker

Your high-powered server is suddenly running dog slow, and you need to remember the troubleshooting steps again. Bookmark this page for a ready reminder the next time you need to diagnose a slow server. Get on "top" of it Linux's top command provides a wealth of troubleshooting information, but you ...

Read More →

 

Another round of updates for the new dashboard+charts UI

BY Doug Breaker

Dashboards have exited beta - see our launch post for the details. Thanks for your feedback on the first preview of our new charts UI. You spoke, we listened, we coded: Resize + drag-and-drop your charts. Total control over how you view your key metrics. Chart-specific settings to toggle either ...

Read More →

 

Scout is NOT vulnerable to CVE-2014-0160 AKA Heartbleed

BY Doug Breaker

All of Scout's infrastructure has been checked, and we have confirmed that Scout is not vulnerable to CVE-2014-0160.

Read More →

 

OpsGenie adds Scout Integration

BY Doug Breaker

Opsgenie , the IT alerts company, now integrates with Scout. Email, SMS, phone calls, and and mobile push - OpsGenie lets your team receive alerts exactly how and when they want. OpsGenie has a full integration guide on their website . Here's the gist: 1. In the OpsGenie UI, click ...

Read More →

 

Scout Realtime @ Boulder Ruby Group

BY Doug Breaker

Interested in the tech behind Scout Realtime , our open-source tool for realtime server metrics? Boy, are you in luck! I'll be diving into the guts of Scout Realtime @ the Boulder Ruby Group this evening. The fun starts at 6pm with a beginner's track.

Read More →

 

Scout is Hiring! Come Build the Future of Monitoring as Developer #2

BY Doug Breaker

We take pride in building a server monitoring product our customers love with a lean, flat team. We're looking to add the fourth human to our close-knit group. So, what's special about being a Ruby dev @ Scout? First, great people! Second, great tech: come build beautiful realtime monitoring visualizations ...

Read More →

 

Using Gmail and Scout emails marked as SPAM?

BY Doug Breaker

Last week, our background task that sends out emails went down. When it came back up, the task sent out a large amount of emails, and it looks like some folks marked the emails as SPAM to clear their inbox. This is causing emails from Scout for some customers to ...

Read More →

 

From MySQL+MMM to MariaDB+Galera Cluster: A High Availability Makeover

BY Doug Breaker

Kevin Lawver , President @ Rails Machine , is our guest author for this post. Few things feel worst than rolling out a High Availability (HA) system, then regularly seeing that system collapse. For our team at Rails Machine, that failing HA system was MySQL Multi-Master Replication Manager (MMM) . ...

Read More →

 

Docker: Git for deployment

BY Doug Breaker

I've been hearing how Docker is the new awesome, but it didn't click for me until I dug in with a practical question: if we deployed Scout via Docker, would deployment be a more pleasurable experience? My three takeaways are below. Finally, an efficient way to simulate our production environment ...

Read More →

 

Scout Puppet Module and PuppetConf

BY Doug Breaker

Just in time for PuppetConf 2013 , we've added an official Scout Puppet Module to Puppet Forge . Configuration instructions are right there in the Scout UI: We're also excited to be a sponsor for PuppetConf - if you'll be attending, drop us a note . We'd love to meetup ...

Read More →

 

Understanding CPU Steal Time - when should you be worried?

BY Doug Breaker

A big thanks to Eric Lindvall of Papertrail for adding steal time to Scout's CPU Usage Plugin and helping out on this blog post! Netflix tracks CPU Steal Time closely. In fact, if steal time exceeds their chosen threshold, they shut down the virtual machine and restart on a different ...

Read More →

 

Omnibus Tutorial: Package a standalone Ruby gem

BY Doug Breaker

A couple of years ago I visited Argentina. I have trouble enough pronouncing my limited English vocabulary and I don't speak Spanish, but after a bit of time, it was pretty easy to order food, buy groceries, and use a taxi. However, occasional hangups that happen during my regular life ...

Read More →

 

RVM, Bundler and Cron in Production: Round 2

BY Doug Breaker

Back in 2010 , we suggested using /bin/bash -l -c to run scout via Cron when using RVM . However, this was a brute approach: /bin/bash -l -c tells bash to behave as a login, interactive process. However, as Daniel Szmulewicz elequently stated in the comments for the original blog ...

Read More →

 

Debugging request bottlenecks with realtime charts

BY Doug Breaker

Last week, one of our application servers died. We have four app servers, so in theory, the death of one app server shouldn't bring the entire platoon down. However, real-life had other plans: 95% of requests were handled fine, but around 5% were being dropped. Here's the story of how ...

Read More →

 

RailsConf 2013!

BY Doug Breaker

I’m sitting in the Denver Airport – in a couple of minutes, I’ll board the plane to RailsConf in Portland, Oregon. I’m already getting amped for Voodoo Donuts, Stumpdown Coffee, well-trimmed beards, and of-course, lots of Rails-related chats. I’m bringing a fresh load of Scout T-Shirts. These aren’t your normal ...

Read More →

 

Three weeks with roles: by the numbers

BY Doug Breaker

It's been three weeks since the launch of the largest feature enhancement in Scout's existence: roles. Haven't heard of roles? Nutshell: roles let you monitor many serves with fewer clicks and more joy . Roles were driven by your feedback and it's showing in the fast adoption numbers below. Time ...

Read More →

 

Notification Schedules and PagerDuty

BY Doug Breaker

Whenever we’re asked how to make on-call notification schedules for Scout alerts, we recommend PagerDuty . PagerDuty has invested a ton of time in building a dedicated notification scheduling service, and it’s a great complement to Scout. With our recent release of notification groups , Scout’s integration with PagerDuty got ...

Read More →

 

The reward of redundancy: time

BY Doug Breaker

It’s been a month since I started attaching torture devices disguised as boots to my feet, long wooden sticks to each torture device, and tumbling down mountains. Skiing has changed my outlook on winter. It’s a season to enjoy, not a time where I gaze wistfully out the window, hoping ...

Read More →

 

Our DIY Homepage Process

BY Doug Breaker

We recently decided it was time for a major update to the public side of Scout . We’d start with a more polished homepage. Since we’re both developers, the obvious next step seemed like hiring a designer. However, working with an outside designer isn’t a hire-and-forget experience: Good designers are ...

Read More →

 

Team BFF

BY Doug Breaker

Startup Lessions from CCP - EVE Online Style : Another great example was the formation of "Team Best Friends Forever" ("BFF"is an inside EVE joke). This team is a group of CCP developers whose sole mission is not to work on major features and improvements, but rather to fix ...

Read More →

 

Notifcation groups coming Jan 15th

BY Doug Breaker

On Jan 15th, all Scout accounts will be switched over to notification groups. Notification groups are designed to make notifiation management easier and more flexible: instead of managing notifications per-plugin/per-user, you will assign users to notification groups, and apply notification groups to triggers. you can have multiple PagerDuty integrations and ...

Read More →

 

The year at Scout - 2012 edition

BY Doug Breaker

If 2011 was the year of fine-tuning, 2012 was the year of major feature enhancements at Scout. Some stats on what’s been a fun year: Feature Highlights Realtime Charts – We started the year off with a bang, adding realtime capabilities to Scout’s charts. Pagerduty Integration – We made it ...

Read More →

 

Self-funded = no compromises

BY Doug Breaker

Naomi Pomeroy, a Food & Wine Best New Chef, on starting an underground, self-funded restaurant: When major chefs hear about the way I run Beast, they say, ‘You’ve created a chef’s dream restaurant, because you don’t have to compromise,’ ” she told me. “What happens when you do a million-dollar ...

Read More →

 

Fort Collins Ruby Meetup - Getting real(time) with Ruby

BY Doug Breaker

In the Colorado Front Range area? I’ll be talking about the realtime web at the Fort Collins Ruby Meetup this Wednesday. I’ll show how adding realtime functionality is the easy : in less than 30 minutes, we’ll build a Sinatra app that uses Pusher for realtime functionality. UPDATED See the ...

Read More →

 

Heading to Rocky Mountain Ruby?

BY Doug Breaker

The Scout gang (including founders and co-dictators-for-life Derek and Andre) will be at Rocky Mountain Ruby Conference in beautiful Boulder, Colorado next week. If you’re going to be there, let us know . We’d love to take our relationship beyond our support email address (if you’re ready for that).

Read More →

 

Removing deadweight - cleaning up our Rails app

BY Doug Breaker

Every man has their breaking point when it comes to deadweight code. Andre and I hit ours recently and decided to spend all of last week focusing soley on cleaning up Scout (a Rails app). Our goals: Faster tests – our tests took 8 minutes to complete. While it’s the ...

Read More →

 

Kestrel in Production at Papertrail

BY Doug Breaker

We’ve added prolific plugin contributor Eric Lindvall ’s latest plugins to Scout: Kestrel Overall and Kestrel Queue Monitoring . Kestrel is a simple message queue built from production needs at Twitter. Being the gentleman he is, Eric shared his experiences with Kestrel at Papertrail , a hosted log aggregation service. ...

Read More →

 

Giving into reasonableness (acqui-hires)

BY Doug Breaker

Last week, Sparrow became the latest poster boy for talent acquisitions (Google gets the team, kills the product). Paying customers complain (I supported it!). Indie devs get depressed as one of their rank sells out. I disagree with Matt Gemmell that these are a good thing – this is not ...

Read More →

 

Tomorrow at Fort Collins New Tech: the story I wish I heard before we started Scout

BY Doug Breaker

There was a clear turning point for Andre and I with Scout – it happened in October 2009. It led to one of our most popular blog articles, “We Just Undid Three Months of Dev work. Here’s What We Learned.” I’ll be telling that story at the Fort Collins New ...

Read More →

 

Real-world Scout at Rails Machine

BY Doug Breaker

Josh Nichols of Rails Machine on their monitoring philosophy: Measure all the metrics and alert on metrics that are actionable. If you’ve got a bunch of servers, you’re going to want to read Josh Nichols’ How we roll with Scout article on the Rails Machine blog. Rails Machine is a ...

Read More →

 

Scout ♥ Pagerduty

BY Doug Breaker

The only thing worse than being woken up by a critical alert: not being woken up when you should be. That’s where Pagerduty comes in—Pagerduty uses a combination of email, SMS , and phone calls to you and your team to ensure you never miss an important notification. And now, ...

Read More →

 

The nuts and bolts of our Ruby-based realtime charts solution

BY Doug Breaker

Scout's realtime charts have been a big hit . Once you start using them for major deploys or performance incidents, going back to ten terminal windows running "top" feels like the dark ages. Realtime was a lot of fun to implement and it's been rock-solid so far. A big reason ...

Read More →

 

Easily compare the same metric across servers (the context menu)

BY Doug Breaker

When you’re building a chart in Scout, you select metrics from a tree. It branches like this: group => server => plugin => metrics. This makes it easy to drill-down to a given metric on a specific server. However, it’s very common to compare the same metric across servers. For ...

Read More →

 

Why doesn't anyone want to build companies anymore?

BY Doug Breaker

It seems that today, the light at the end of the startup tunnel is either (1) being acquired or (2) a billion-dollar IPO . Let’s just throw out the IPO - it’s incredibly rare . This means that many startups are created with an explicit goal to not exist in ...

Read More →

 

Hello Fort Collins!

BY Doug Breaker

A month ago my family and I moved from one special city to another: San Francisco to Fort Collins, Colorado . While we’ve only lived here a short time, we couldn’t be happier. You probably know many of the one-of-kind-things about San Francisco. Here’s some of the things that make ...

Read More →

 

On writing

BY Doug Breaker

Ernest Hemingway via Letters of Note I write one page of masterpiece to ninety one pages of shit. I try to put the shit in the wastebasket. Sounds a lot like writing code too, huh?

Read More →

 

Scout Alerts in Campfire via Hubot

BY Doug Breaker

Want to get Scout alerts piped into your Campfire room? You can, thanks to Scout user Gavin Stark’s Hubot script for Scout. Gavin describes the advantages for his team at Real Digital Media: Our support staff can now see the alerts from Scout as a team. We combine this with ...

Read More →

 

One Hundred Dollars

BY Doug Breaker

Recently I’ve been calling a couple of customers per-week to chat about their Scout experience. One of the questions I’ve been asking comes out of left field: ”What would you pay another $100/mo for?” I’ll ask the question first to see if they have any suggestions, then run a small ...

Read More →

 

Real Time Infographic

BY Doug Breaker

It’s been a little over a week since we rolled out real-time monitoring . Some people think it’s pretty cool . To commemorate one week of real time, our art department put together a basic infographic on its usage. You’re spending more time watching lines move on the screen than ...

Read More →

 

The Year At Scout: 2011 Edition

BY Doug Breaker

I’ll remember 2011 as the year of fine-tuning at Scout. From your experience with the end product to our experience delivering it, we invested a lot of our time sanding Scout. Feature Highlights Dashboards – Combine plugin displays and charts across servers for a complete view of your infrastructure’s health. ...

Read More →

 

Do we really need a phone number?

BY Doug Breaker

Like a bug zapper on a Midwestern summer evening, our company phone number attracted lots of annoying visitors: partnership offers from the Middle East, RFP requests for the wrong product, support inquires for a dating service, etc. In fact, I can only recall a handful of calls from real-life customers. ...

Read More →

 

Monitoring HAProxy

BY Doug Breaker

HAProxy is a rock-solid, high performance TCP /HTTP load balancer. It’s what we use at Scout to proxy traffic to our app servers. We’ve added a plugin to monitor HAProxy : just specify the URL to your HAProxy stats page and the name of the proxy. You’ll get the request ...

Read More →

 

Our poor choice of words: "Server Down" alerts are now "No Data" alerts

BY Doug Breaker

What we’ve been referring to as “Server Down Notifications” suffer from two problems: False Positives – The last thing you need is an alert from Scout at 3am telling you that your server is down when it isn’t. An overaggressive email subject – The subject line of these emails looks ...

Read More →

 

Making Scout feel young again: our 4-part tonic

BY Doug Breaker

Scout is no longer a puppy: in dog years, he’s old enough to drink, get drafted, and rent a car. During that time, cruft gathered around the edges of our server infrastructure. We’ve been using a hodgepodge of server hardware, some performing multiple roles, some not, all individually configured and ...

Read More →

 

Monitoring Apache ZooKeeper

BY Doug Breaker

With over 50 million plays, OMGPOP – the free multiplayer game site – is logging a lot of data. Tracking stats like app downloads and launches paint a picture of how their games are performing. This logging data is collected via Flume , a system for collecting streaming data, and ...

Read More →

 

Dashboards (for Ultimate Accounts)

BY Doug Breaker

On of the most common feature requests we get is the ability to place multiple charts on a dashboard page. We’re launching this feature today. It’s currently available for Ultimate accounts only. What are Dashboards? Multiple Charts : display any number of charts on one page. Arrange and resize the ...

Read More →

 

Where we're going, we don't need road(maps)

BY Doug Breaker

We don’t have a product roadmap. Why? We become that with which we busy our minds. I’m not preoccupied with hitting every deadline on a one-year pipeline of features. I’m focused on making it easier to monitor servers every week. Rather than thinking about deadlines, I wake up thinking: “How ...

Read More →

 

The bike break

BY Doug Breaker

A bike commute leaves me more refreshed than a run in the park, a round of video games, or reading a book. David Byrne wrote about this in Bicycle Diaries : It (biking) facilitates a state of mind that allows some but not much of the unconscious to bubble up. ...

Read More →

 

How Pixar and Toy Story almost didn't happen

BY Doug Breaker

When Pixar was working on their first film, Toy Story , the original draft closely followed a checklist of required elements provided by Disney. Disney has been producing animated films since 1923. Lion King , released a year before Toy Story , won two Academy Awards and grossed $783 million. ...

Read More →

 

Scaling Illustrated

BY Doug Breaker

Last week we added a third web server to one of our reporting applications. We’ve been growing at a steady rate and we wanted to reduce the load across our web tier (losing one of the web servers could put too much traffic on the remaining server). Before Will Farrington ...

Read More →

 

Don't want your server named after hostname?

BY Doug Breaker

If you’re using Scout’s cloud server functionality , you know that new server instances in Scout are named after the machine’s hostname. In some cases, the hostname isn’t that useful (especially if your servers are on EC2 ). Now, you can specify a name for the server from the Scout ...

Read More →

 

Behind the scenes of a self-funded business: a week on RescueTime

BY Doug Breaker

One of the things I was most curious about before starting Scout was what my day-to-day life would look like as a business owner versus life as a developer. How much time would I spend writing code? Answering emails? Handling the business bits? Would I work more? Three years after ...

Read More →

 

API Blank Slate

BY Doug Breaker

A couple of years ago I went to Argentina. I don’t speak Spanish. This meant a lot of the basic conversations I’d have with locals were frequently interrupted by me paging through a translation guide. I felt very disconnected: basic exchanges were cumbersome. When I work with an API for ...

Read More →

 

The Short and Sweet SSL How-to

BY Doug Breaker

Got a site you need to serve up via SSL ? Here are your Cliffs notes. This assumes 1) your site already runs without SSL ; 2) you’re using Apache and Ubuntu; 3) you don’t want any browser warnings, so no self-signed certificates. 1. Generate a Private Key $ openssl ...

Read More →

 

3 Developer-Centric Takeaways from Ben Franklin

BY Doug Breaker

Before a recent plane trip, I checked out the Autobiography of Benjamin Franklin from the library. Franklin might have the greatest list of accomplishments in American history. I was curious to learn how he did it. Three developer-centric takeaways: The Socratic Method Like developers, Franklin loved a spirited debate. In ...

Read More →

 

Three New Tools in my Mac Toolbox

BY Doug Breaker

It’s always satisfying to find tools that make your workflow smoother. Here are three that I started using recently: iTerm2 is a Terminal replacement with a ton of features. The three that make the most difference in my workflow are search , mouseless copy , and tabs which you can ...

Read More →

 

JMX Monitoring

BY Doug Breaker

One of the features Java Management Extensions provides is the ability to add instrumentation to an application. While this makes collecting metrics straightforward, it doesn’t address storage for these metrics. Enter David Dossot’s JMX Monitoring Scout Plugin . David has used the plugin to monitor an asynchronous data archiver designed ...

Read More →

 

The Only Two Business Metrics That Matter

BY Doug Breaker

How do you spot a successful business? It’s easiest to think in terms of size: Google, Starbucks, and Berkshire Hathaway are successful. If you’re a small business and fixate on size alone, you’ll drive yourself crazy. I will never run a Starbucks-scale company, nor do I want to. But do ...

Read More →

 

Detect and Fix HAProxy+Apache+Passenger Queue Backlogs

BY Doug Breaker

To inspire hard work, some young men hang a poster on their wall that includes: (1) an exotic sports car (2) a scantly clad lady and (3) a beach house. My inspirational poster would be much less attractive: a friendly butler who offers time-honored wisdom (with an accent because people ...

Read More →

 

You Are Making Scout Even Better

BY Doug Breaker

A huge thank you to Scout users who have sent us plugin fixes and enhancements! It’s really a pleasure to see the plugin library grow and improve. Here are the latest updates: Memory Profiler Solaris Compatibility added by Ankur Bulsara URL Monitoring Better exception handling for bad hostnames and request ...

Read More →

 

Tina Fey: go with the bran muffin on the white plate

BY Doug Breaker

First, Tina Fey needs to write more. Both of her New Yorker articles have been terrific. Second, her latest New Yorker piece had a great nugget on how producing is about discouraging creativity: You may have an occasion where the script calls for a bran muffin on a white plate, ...

Read More →

 

Quick & dirty log monitoring

BY Doug Breaker

Sometimes we have an immediate need to watch for a term in a log file. For example, if we’re doing a major deploy, we might watch for the term error in a log file. We want to make sure the rate of errors doesn’t increase. To do this, we’ll use ...

Read More →

 

Behind our financial dashboard

BY Doug Breaker

We started Scout to make monitoring server infrastructure easier. With our heads deep in code, it took us a while to realize that monitoring our business was just as important. Here’s a look at the financial dashboard we created (as you’ll guess, the data isn’t real): Key Components Activity by ...

Read More →

 

When you are not the smartest programmer in the room

BY Doug Breaker

Most of the freshman engineering students at my college took two computer science courses together. These were the hardest classes I would take during college ( my colleague was a TA for these courses so he deserves partial blame). The experience for Sam Epstein, a friend of mine, was decidedly ...

Read More →

 

Understanding Disk I/O - when should you be worried?

BY Doug Breaker

Our co-author today is Christian Paredes , Senior System Administrator at Blue Box Group , a Ruby on Rails-focused web host that specializes in providing the operations expertise required to keep powerful apps running at peak performance. Christian keeps Blue Box Group’s internal infrastructure in top-shape and provides tier 3 ...

Read More →

 

How much slower is Disk vs. RAM latency?

BY Doug Breaker

It takes longer to access data stored on your hard disk vs. RAM . But how big is the difference? It’s really big. How can you tell if you your web application is being impacted by slow disk access? What impacts I/O performance? What’s the best path to fixing the ...

Read More →

 

Sleep Better with a Proper Staging Environment

BY Doug Breaker

Nothing helps you sleep better at night like a staging environment that’s faithful to your production setup. That means your staging environment has the same Linux distro, same version of Ruby and gems, the same Apache and Passenger configuration, etc. VPS not cloud We’ve found that an inexpensive “always-on” VPS ...

Read More →

 

The 11" MacBook Air: like a good Linux tool

BY Doug Breaker

I’ve been tuning Scout’s Apache setup lately. To start, I looked at the output of: apachectl status ...but this only provides an instant snapshot. I wanted to watch the results over a longer period of time: watch --interval=1 apachectl status ...but this generates a lot of output and I was ...

Read More →

 

Relentlessly Shortcut: .bashrc & Thor

BY Doug Breaker

Check out the incredible shortcut Lance Armstrong takes in the above clip. As developers, we should try to shortcut as smoothly as Lance does. You might not get cheered on quite as much—but then again, you have a lot more shortcut opportunities! Shortcuts and Development Workflow The quicker I can ...

Read More →

 

CouchDB in production

BY Doug Breaker

John P. Wood of Signal , which offers a mobile customer engagement platform used by many top brands, recently created a couple of Scout Plugins for monitoring CouchDB . I’ve always been impressed by the team at Signal, so I was curious how they were using CouchDB in production. It ...

Read More →

 

Developer Happiness (2 of 5): Speed Up Your Tests

BY Doug Breaker

Seeing your tests pass is a great feeling. Waiting a long time for it—not so much. Faster tests mean a happier developer! Use Parallel Tests If you’re not using parallel tests by Michael Grosser, you’re wasting time. Setup is just a few simple steps . You’ll be glad you did. ...

Read More →

 

Developer Happiness: a Five-Part Series

BY Doug Breaker

The New York Times ran a popular Op-Ed on Happiness last year. The Cliff Notes version? The daily activities most associated with happiness are sex, socializing after work and having dinner with others. The daily activity most injurious to happiness is commuting. Many software developers are fortunate to have flexible ...

Read More →

 

The best $650 I spent last year

BY Doug Breaker

Last year I ponied up for a 256GB SSD for my Macbook Pro. Six months later, I can confidently give the verdict that it’s the best investment I made in my work setup. Just a few datapoints: Scout’s unit test suite runs in half the time : 1m34s vs. 2m58s ...

Read More →

 

Optimizing Elevator pitch visits

BY Doug Breaker

A quarter of returning visits to Scout are less than five minutes in length . It’s a tad longer than an elevator pitch on the high end. Like a real-life elevator pitch, you need to make a good impression quickly for these short visits. A user that spends twenty minutes ...

Read More →

 

2010 Scout Highlights

BY Doug Breaker

2010 was a busy year at Scout. The highlights: Top Features An updated server view A redesigned dashboard & server groups Charts: Now much faster Updating monitoring on 100 servers with 1 mouse click Scout for the iPhone: BirdDog Rails + MySQL Monitoring Enhancements Top Plugins MySQL Stats w/MySQLTuner Plugin ...

Read More →

 

Forget The Technology - Just Do The Business

BY Doug Breaker

A friend of mine is brainstorming a business in the dating space. It’s an interesting idea—a spin on the traditional dating service in potentially lucrative niche. My friend asked me about technology—which technology, who to partner with, etc. My advice for the dating idea was this: forget the technology. Just ...

Read More →

 

Don't build to get acquired. Build to say "No".

BY Doug Breaker

Groupon rejected a $6 billion acquisition offer from Google. They don’t need to cash out . Our goal isn’t to get acquired. It’s to say “No” as often as possible: No, we’re not going to support Windows. No, that partnership doesn’t feel right. No, we’re not working late. Saying “No” ...

Read More →

 

Modifying browser history with Javascript

BY Doug Breaker

We lost the browser state when we replaced our Charts UI w/AJAX manipulation . While it’s possible to maintain state using an anchor ala Gmail (ie – gmail.com/mail?#inbox/THREAD), it breaks the separation between business and view logic. The server cannot access the state information contained in the anchor – it’s ...

Read More →

 

Data processing slowing you down? Try a teaser check

BY Doug Breaker

Your cluster of servers ping Scout every minute, but you may only login to our website once a day. There’s a large gap between the amount of work on the data-processing side and the number of times it results in a visit to Scout. It’s a typical profile of a ...

Read More →

 

Announcing Redwood: Spotlight-like search for your web apps

BY Doug Breaker

Our knowledge at Scout is spread across web apps: Gmail, Basecamp, Delicious, Dropbox, and more. It’s become more cumbersome to find information as our pile of content has grown. We saw two options: devote more time to organizing our information or make it easier to search through it. Spending more ...

Read More →

 

Monitoring mod_pagespeed

BY Doug Breaker

Josh Nichols of Rails Machine has developed a Scout plugin for monitoring mod_pagespeed , Google's Apache 2.x module for performing on-the-fly optimization in the Apache 2 HTTP Server. mod_pagespeed has several filters that optimize a web page’s resources. These filters combine CSS files, optimize images, and more. mod_pagespeed is still ...

Read More →

 

"I expected it to ..." feature requests

BY Doug Breaker

When I bike into work, I coast down a decent hill. With gentle curves and smooth pavement for 99% of the decent, it’s a refreshing way to start the day. My one complaint: there’s a 2×2 ft. section in the final turn with bad pavement. This tiny imperfection breaks up ...

Read More →

 

One change that lowered our DB I/O by 80%

BY Doug Breaker

When a server reports data to Scout , a small bit of data is updated in a MySQL MyISAM table. Each of these updates doesn’t change much data: UPDATE `plugins` SET `last_reported_at` = '2010-11-17 19:53:00' WHERE `id` = 999999 I assumed that only the updated data was written to disk. ...

Read More →

 

A redesigned dashboard & server groups

BY Doug Breaker

Answering “how is my cluster performing?” just got a lot easier. We’ve rolled out a redesigned account dashboard that makes filtering the noise in your server cluster a lot more efficient. The dashboard The updated account homepage lists each server and 3 key metrics: Server load over the last minute ...

Read More →

 

A lesson from my favorite TV movie

BY Doug Breaker

Duel , Steven Spielberg first film, had an estimated budget of $450K. For comparison, the opening scene in Spielberg’s Saving Private Ryan had a budget of $12 million. However, I’m just as happy watching Duel . It’s the only television movie I’m proud to talk about in public. Spielberg on ...

Read More →

 

Where are the Rails infrastructure support firms?

BY Doug Breaker

There are no formal documents to sign if you and your spouse decide to have children. You don’t have to sit through an accreditation class. There is no credit check. You don’t need a high school diploma. Procreation can even happen accidentally. A baby is a lot like a Rails ...

Read More →

 

Apple's Hidden Genius

BY Doug Breaker

James Duncan Davidson on support at the Apple Store : "one of the secrets of Apple's product design in the last decade has been the use of the data that these stores generate…it's the data that's generated at the Genius bar that fascinates me. This data, in aggregate, can tell ...

Read More →

 

Your Launch is Not an Event

BY Doug Breaker

photo credit: Steven Depolo Most entrepreneurial articles portray the product launch as an event. Sometimes the launch “event” goes well, sometimes it doesn’t. I think talking about the launch “event” misrepresents how most successful product launches actually work. Your launch isn’t an event. Launching is a series of small validations, ...

Read More →

 

Here's to the unsung developer heros

BY Doug Breaker

We’re fascinated with complexity. Write an article about the enormously high throughput/disk space/concurrency your application pushes and people will read it. You’ll speak at conferences. People will follow you. It’s fun talking about a complex system, but it can be tiring caring for one. I love Woz’s focus on efficiency ...

Read More →

 

Why we don't schedule deployments during off-hours

BY Doug Breaker

Years back, before Scout , I used to schedule deployments outside the regular workday. I didn’t give it much thought – it was what my consulting clients were used to. However, we changed that practice with Scout. Many web applications, including Scout, have customers around the world. There isn’t a ...

Read More →

 

5 Pillars of Rails Cluster Monitoring

BY Doug Breaker

Photo by mollypop My how you’ve grown! A couple of years ago your little Rails app was on a single server. Now you’re on a whole cluster – you’ve got web servers, database servers, HAProxy servers, and more. I’m so proud of you! Monitoring your Rails cluster has gotten more ...

Read More →

 

Determining free memory on Linux

BY Doug Breaker

When checking the amount of free memory on a Linux server, it’s easy to think you’re running out of memory when you’re not. For example, here’s the output of free -m on a server with 4GB of RAM : With a quick glance, you might start sweating. Only 39 MB ...

Read More →

 

Our failed experiment: great on the rack, bad in the mirror

BY Doug Breaker

My left arm is one inch longer than my right. My right foot is a half inch longer than the left. For me, trying on clothes is an adventure: what looks good on the rack often doesn’t on me. I thought of this recently when we decided to end development ...

Read More →

 

Scout for the iPhone: BirdDog

BY Doug Breaker

Shawn Veader of V8 Logic has released BirdDog , an iPhone application for Scout. The app is on sale for $1.99 through Thursday . With BirdDog, you can track the health of servers you are monitoring with Scout . BirdDog provides an alert history for each server, highlighting any active ...

Read More →

 

Embedded Customer Support With Rapportive

BY Doug Breaker

A big part of providing good support is making it painless. At Scout , Andre and I handle all of the support requests. Once we’ve gathered the account information, it usually doesn’t take much time to help. The problem is quickly putting the account information together. We don’t want to ...

Read More →

 

RVM and cron in production

BY Doug Breaker

UPDATED 6/7/2013 – We no longer recommend using bash -l -c to run Cron jobs with RVM . See our updated approach . This info is for Scout users, but it’s also applicable to anyone who’s wondered how to use cron together with RVM . This assumes 1) you are ...

Read More →

 

A Holodeck for deployments?

BY Doug Breaker

Recently I watched the documentary miniseries When We Left Earth . A constant theme from early space flight was the huge number of unknowns. For example, before the first astronaut went into orbit, NASA had concerns humans would lose their vision and wouldn’t be able to swallow in zero gravity. ...

Read More →

 

Scout is Getting a New Load Balancer

BY Doug Breaker

We are in the process of replacing our hardware load balancer with an HAProxy setup. We plan on making the switch tomorrow evening (Tuesday, 8/24) at 4:30pm PST . We are not expecting any downtime, and you shouldn’t experience any service interruption. As a result of the switchover, the scoutapp.com ...

Read More →

 

So, what do you do?

BY Doug Breaker

I was back in Michigan a few weeks ago visiting family. These trips always involve a “what do you do?” question at some point. I’m never sure how to answer. “I have a startup.” Honestly, I don’t think of Scout as a startup. Scout is a business built the old-fashion ...

Read More →

 

MySQL Tuning Tips with Scout

BY Doug Breaker

If you’ve used Major Hayden’s MySQLTuner before, you know it’s a great source of MySQL optimization tips. Now you can get MySQLTuner reports automatically delivered through Scout. All you need to do is install the MySQL Stats w/MySQLTuner plugin , or update the plugin if you already have it installed. ...

Read More →

 

Monitoring MongoDB

BY Doug Breaker

Updated 3/21/2013 – The MongoDB Overview plugin discussed below has been split into 2 separate plugins: MongoDB Server Status and MongoDB Database Stats . The server status plugin reports global MongoDB metrics and the database stats plugin reports metrics specific to a particular database. John Nunemaker of Ordered List knows ...

Read More →

 

Oink + Request Log Analyzer = Rails Monitoring in one report

BY Doug Breaker

If you’ve ever had to track down a memory leak in a Rails application (and who hasn’t forgotten to use will_paginate occasionally), you’re probably already familiar with the excellent Oink plugin by Noah Davis . Oink spits out the actions that are leaking the most. Oink is a huge help ...

Read More →

 

The Page Load Paradox

BY Doug Breaker

The irony is that, with broadband nowadays more or less everywhere, overall connection speeds have gone up by leaps and bounds, yet the time taken to load web pages seems only to have got longer. - World Wide Wait , The Economist, Feb. 12 2010 For more than 40 years, ...

Read More →

 

There's Scout in my Safari

BY Doug Breaker

The smart folks over at Rails Machine are always quick on the draw. Still, we were impressed when developers Will Farrington and Mike Skalnik had a working Safari 5 extension for Scout within 24 hours after Safari 5 extensions were announced. I pinged Will for the scoop: What does the ...

Read More →

 

Small Teams Need Redundant Skills

BY Doug Breaker

If you’re running a small company, it’s absolutely key that you have redundant skills on your team. Does that seem counter-intuitive? Isn’t the typical pairing a business guy and a technical guy ? In our experience, you’re far better off with two technical guys . The redundancy is key if ...

Read More →

 

Win a free Scout account at the Red Dirt Ruby Conference website

BY Doug Breaker

UPDATED 4/29 – We’ve got 5 winners! Answers inline… The Red Dirt Ruby Conference is hitting all of the things we like about regional Ruby conferences – a capped attendance, emerging technologies important to Ruby developers, great speakers, and plenty of opportunities to hack with other Ruby developers. We’re sponsoring ...

Read More →

 

Is your Rails app under-provisioned?

BY Doug Breaker

You maintain a growing Rails application and you’re seeing something peculiar. Sometimes when you use the application, it feels like the performance deteriorates significantly. However, all of your performance data shows no issues – requests in the Rails log file look speedy, CPU utilization is fine, database performance is solid, ...

Read More →

 

Monitoring Apache Request Processing Time

BY Doug Breaker

With a little help from the excellent Request Log Analyzer gem , created by those smart folks at RailsDoctors , the Apache Log Analyzer plugin now tracks the duration of Apache requests. Upgrading & Install If you already have the Apache Log Analyzer plugin installed, you’ll need to upgrade to ...

Read More →

 

Monitoring a Log File

BY Doug Breaker

A log file is a lot like a neglected board game covered in dust. I don’t play board games that often, but when I do, I always say “we should do this more often”. The log files we have for Scout collect lots of great information, but we only reference ...

Read More →

 

Memcached Monitoring

BY Doug Breaker

Jesse Newland of Rails Machine has created a Memcached monitor plugin to track all the key stats from your Memcached instance, including gets/ sets/ hits/ misses/ evictions/ per second, uptime, memory used, KB read per second, KB written per second, and more. Install the Memcached plugin from our directory , ...

Read More →

 

Why web developers have more to learn from Wall Street than Steve Jobs

BY Doug Breaker

In 2006, the biggest risk-taker on Wall Street looked like John Paulson . This certainly wasn’t based on Paulson’s past behavior. Paulson managed a middle-of-the pack hedge fund. He made careful, boring deals. He rode the bus and liked to ride his bicycle. In other words, he was the anti- ...

Read More →

 

MongoDB Slow Queries Monitoring

BY Doug Breaker

Thanks to Jacob Harris , we have a MongoDB Slow Queries plugin for Scout. The MongoDB plugin captures the number of slow queries per minute according to the threshold you specify. It also reports details of any slow queries that take place. Of course, like any other Scout plugin, you ...

Read More →

 

Scout makes the big screen at Runa

BY Doug Breaker

There are lots of things that look great on big, wall-mounted flat screen displays: Halo, Dark Knight, the Super Bowl. How about Scout-generated graphs? Robert Berger , CTO of Runa sent us a picture of their Scout setup. Runa has an LCD screen of Scout graphs positioned above the most-visited ...

Read More →

 

When tech startups == closed restaurants

BY Doug Breaker

Bootstrapped tech startups are a lot like restaurants started by quality chefs. It’s easy to find restaurants serving award-winning food that fail . There are lots of self-funded startups started by smart developers that either never launch or die a painful death. Writing quality software – just like serving good ...

Read More →

 

InfoQ Scout Interview

BY Doug Breaker

Robert Bazinet , Community Editor for InfoQ and owner of expens’d recently interviewed my co-founder Andre Lewis . You can find the Q&A below: Scout – Extensible Server and Application Monitoring Andre talks about data security with hosted monitoring, remodeling his bathroom, how Scout fits in with existing tools like ...

Read More →

 

Instrument Your Business

BY Doug Breaker

Instrument your business. It's like instrumenting your code, but profitable. If you're a developer or a sysadmin, periodic optimization is a fact of life. As more people use your code, you'll see performance issues. Unoptimized use cases bubble up. Systems get strained by continued or heavy use. On the technical ...

Read More →

 

Production Server Sysadmin Essentials

BY Doug Breaker

~ or ~ Sysadmin Eye for the Dev Guy Developers! You can churn out a Rails or Sinatra app in no time. What about putting it out there in production? Occasionally forget the syntax for crontab or logrotate? Yeah, me too. That's why I wrote up a few essential notes ...

Read More →

 

3 competitive advantages of developer-run businesses

BY Doug Breaker

Older baseball players typically have plenty of power and discipline but are slow runners with lower batting averages. It’s kind of obvious: as we get older, we get bigger and more patient, but we also get slower. This characteristic even has a name, described as (you guessed it): old player ...

Read More →

 

What makes a great developer?

BY Doug Breaker

James Gray, lead developer of the Scout Agent, takes a stab at the eternal question “What makes a great developer?” in the latest RailsCoach Podcast Episode . Besides talking about Scout’s blended open-source approach to monitoring, James also talks about The Pomodoro Technique (a time management process), his time at ...

Read More →

 

Rackspace & Scout Webinar - Monitoring in the Cloud

BY Doug Breaker

A couple of months back when we announced our partnership with the Rackspace Cloud I mentioned that 1 of 5 hosts monitored by Scout is in the cloud. Monitoring in a cloud environment brings some unique challenges. On Thursday, January 21st at 1pm Central I’ll be leading a webinar with ...

Read More →

 

Free memory on Linux: free -m vs /proc/meminfo

BY Doug Breaker

How much memory is really available on your Linux box? Don't use /proc/meminfo to find out, use free -m instead. You may have more memory available than you thought. Here's an example. /proc/meminfo says about 330MB is free: ~ $cat /proc/meminfo MemFree: 340996 kB .. free -m gives the following: ...

Read More →

 

Production Rails Tuning with Passenger: PassengerMaxProcesses

BY Doug Breaker

Our co-author today is Jesse Newland , CTO of RailsMachine . Jesse keeps RailsMachine customers up and running and troubleshoots their toughest problems. We’re pleased to have him share some of his expertise on Phusion Passenger tuning. Say your Rails application is running in production and it’s getting good traffic. ...

Read More →

 

Partnership with Rackspace Cloud Computing

BY Doug Breaker

In less than two months, we’ve seen strong adoption of Scout’s simple cloud monitoring functionality . 1 of every 5 hosts monitored by Scout is in the cloud . With more movement to the cloud, we’re happy to announce a new partnership with Rackspace Cloud Computing . Rackspace has an ...

Read More →

 

Process Memory Usage: Troubleshooting with Scout

BY Doug Breaker

Scout customer Eric Lindvall wrote up a nice piece on finding and fixing memory leaks in god -- specifically, when issuing "god load" on long-running god processes. Give it a read, it provides good insight into the troubleshooting process and the tools he used. Eric points to the Scout graphs ...

Read More →

 

Running a freemium web app? Here's a big reason we're growing.

BY Doug Breaker

Before Scout , my experience developing software was primarily consulting. Success was measured by delivering software on time and on budget. With Scout, a subscription-based service, my focus isn’t on scheduling. We are self-funded and we didn’t have the luxury of a venture-backed startup. We’re focused on figuring out which ...

Read More →

 

Rails + Google Analytics = easy goal tracking

BY Doug Breaker

Google Analytics is an indispensable tool as you optimize the business side of your operation. If you haven't already set up goals in Analytics for viewing your pricing information, accessing the sign-up form, and signing up for an account -- go do it! It's vital information. However, Google Analytics' goals ...

Read More →

 

Part II: We Just Undid Three Months of Dev work. Here's What We Learned.

BY Doug Breaker

Two weeks ago I covered some of the business lessons learned from a large (~3 months) investment in new features, and the hard decision to roll them back. I discussed how you will underestimate the ongoing cost of complexity in your product , and how cool new capabilities don’t sell ...

Read More →

 

The MySQL MyISAM and InnoDB engines and a grocery checkout

BY Doug Breaker

There’s no shortage of resources comparing the MyISAM and InnoDB storage engines. You’ll quickly see it isn’t a black-and-white decision after reading through various discussions debating MyISAM and InnoDB. Why is the decision so hard? Setting up your database is one of the first steps when building a web application. ...

Read More →

 

First Impressions Count

BY Doug Breaker

Scout is making a better first impression than ever starting today. When you start monitoring a new server, you'll immediately get a high-level summary of the vital stats: Scout reports this for you automatically. From there, you choose the deeper metrics you need, like Ruby on Rails monitoring , MySQL ...

Read More →

 

We Just Undid Three Months of Dev work. Here's What We Learned.

BY Doug Breaker

We’ve been deleting a lot of code from Scout. We’re ripping out major infrastructure, and in doing so, pulling the plug on functionality which, just six months ago, we believed would be crucial to our business. Most importantly, we’re simplifying the most complex, error-prone, and poorly-performing parts of the application. ...

Read More →

 

Simplify. Get an order of magnitude speedup.

BY Doug Breaker

Have you noticed Scout feels snappier lately? We made some major simplifications that sped things up a lot. Here’s the CPU load on one of our DB servers: ( and yes, we use Scout to monitor itself! ) Even better, the response time for users improved dramatically. Scout’s longest actions ...

Read More →

 

In-depth Rails Monitoring using only a production log file

BY Doug Breaker

No Rails plugins to install. No performance hit during the request cycle. Nothing to break your application code. Nothing to restart. With just the path to your production Rails log file, Scout's new Rails monitoring plugin alerts you when your Ruby on Rails application is slowing down and provides detailed ...

Read More →

 

A weather dashboard

BY Doug Breaker

Server Monitoring and weather forecasts have a lot in common – (1) I want to know what’s happening now, (2) what the forecast looks like for the immediate future, and (3) how the long-term is shaping up. I’m really impressed with Matthew Ericson’s Weather Page . It condenses the 3 ...

Read More →

 

Using Scout in the cloud?

BY Doug Breaker

We’re developing features that make using Scout in a cloud hosting environment super simple ( Amazon ec2 , Rackspace Cloud , GoGrid , etc). We’re looking for BETA testers, so shoot us an email at support@scoutapp.com with your account name if you are interested in giving Scout’s cloud support a ...

Read More →

 

EC2 CloudWatch graphs, trends, and alerts

BY Doug Breaker

If you're using Amazon EC2, you may be familiar with CloudWatch, Amazon's analytic system that provides metrics on CPU usage, Network I/O, and Disk I/O of your instances. While CloudWatch collects metrics, it doesn't provide a web interface for viewing the metrics, graphs, trending, or alerting. Enter our Scout EC2 ...

Read More →

 

RubyKaigi 2009 wrap up

BY Doug Breaker

James Gray's July 19th talk at RubyKaigi 2009 focused on best practices for long-running Ruby daemon processes. What types of questions did the audience ask? What did they seem most interested in? In general, users always want to know about our RRD usage, extracting the daemon functionality from Scout's agent, ...

Read More →

 

Moving from FiveRuns to Scout

BY Doug Breaker

As FiveRuns posted on their blog they have announced End-of-Life for FiveRuns Manage. We have made arrangements with FiveRuns to ease the transition for customers who still need a robust, easy-to-use monitoring solution. For current Fiveruns customers, we are offering 50% off your first paid month here with Scout . ...

Read More →

 

Understanding Linux CPU Load - when should you be worried?

BY Doug Breaker

You might be familiar with Linux load averages already. Load averages are the three numbers shown with the uptime and top commands - they look like this: load average: 0.09, 0.05, 0.01 Most people have an inkling of what the load averages mean: the three numbers represent averages over progressively ...

Read More →

 

Presenting the Scout Agent at Ruby Kaigi - Tokyo, Japan July 19

BY Doug Breaker

Scout takes a trek to Ruby’s birthplace – Japan – as James Gray presents How Lazy Americans Monitor Servers at the sold out Ruby Kaigi . James’ July 19th talk focuses on the architecture of the Scout agent , the Ruby gem that is installed on a server you wish ...

Read More →

 

Q&A with the Scout Agent - An overview

BY Doug Breaker

Our recent update to Scout featured a revised UI, more functionality, and a new Scout Agent. While it’s easy to see the changes in the UI, a lot of the work conducted by the agent happens beneath the surface. The Scout Agent, which is installed on a server you wish ...

Read More →

 

Scout Agent Updated - and do your Net::HTTP calls ever hang?

BY Doug Breaker

UPDATED 6/30 – The fix for the old scout client (run via cron) is now available in version 2.0.7 (sudo gem install scout). In rare (and difficult to reproduce) cases we’ve seen the Scout Agent not observe a Timeout during a checkin error with the Scout server. Scout uses Ruby’s ...

Read More →

 

On Sinatra

BY Doug Breaker

Sinatra , a Ruby DSL for quickly creating web applications with minimal effort, forms a key part of the Scout infrastructure. James Gray talks about how we use Sinatra at Scout via RubyLearning.org – 20+ Rubyists are using Sinatra ...“ Do you? For more on how Scout works: A tour ...

Read More →

 

Rails Machine selects Scout for Performance Monitoring

BY Doug Breaker

Rails Machine , one of the first specialized Rails hosting providers, has selected Scout for performance monitoring. Why Scout? “Scout’s versatility was a key benefit: “the open-source plugins are the killer app of Monitoring,” says Rails Machine CEO Bradly Taylor . “We have so much flexibilty—and we’ll be contributing some ...

Read More →

 

Learn about the Scout Agent, API, and more at OK.rb

BY Doug Breaker

James will be speaking at Thursday’s (June 11th) Ok.rb meetup in Edmond, Oklahoma, giving a tour of the Scout Agent and the API . See the Ok.rb website for more details.

Read More →

 

Scout at RailConf

BY Doug Breaker

It’s been a big year for Scout at conferences. We sponsored GoGaRuCo here in San Francisco last month, and we had a booth on the floor at RailsConf last week. A few highlights from the conference: We came to the conference armed with all kinds of “selling points” for Scout ...

Read More →

 

Interview on akitaonrails.com

BY Doug Breaker

Fabio Akita tracked down our own Matt Todd to talk about Scout. Listen to the interview: Download (13:43)

Read More →

 

I'm not a ShamWow salesman and other lessons from our first conference

BY Doug Breaker

I despise sales pitches, but you wouldn’t know it if you were the first person to visit our Scout exhibit booth. After introducing myself, I delved into a 5 minute whirl-wind tour of Scout 2.0, trying to show every possible feature that might help him monitor his Rails stack. He ...

Read More →

 

Scout 2.0 Behind the Scenes on the Ruby on Rails Podcast

BY Doug Breaker

We sat down with Geoffrey Grosenbach of the Ruby on Rails Podcast to chat about the heavy lifting we’re doing behind the scenes for the big ‘ol Scout refresh. What plan of attack did we choose for building a daemon? What are some of our new approaches for data visualization? ...

Read More →

 

Updates to memcached, nginx monitoring

BY Doug Breaker

Colin Surprenant did some quality GitHub forking today, enhancing the monitoring for memcached and nginx . The memcached stats plugin reports the following data: Current connections Current data MB Evictions Get count Get hits Get misses Max data MB Set count Total items ...and alerts you if the plugin can’t ...

Read More →

 

See Scout's big changes at Golden Gate Ruby Conference and RailsConf

BY Doug Breaker

It’s been almost a year since we launched Scout , and over the past couple of months, we’ve been working on a big new release. We’ll be showing off Scout’s new features at the Golden Gate Ruby Conference on April 17th and 18th and at RailsConf on May 4th-7th. Hope ...

Read More →

 

Hardware Upgrade - Downtime 3pm-3:30pm

BY Doug Breaker

A quick note that we are continuing our hardware upgrades, and we’ll have downtime from 3pm-3:30pm PST today, February 12.

Read More →

 

Scout reviewed in French

BY Doug Breaker

If French suites you better than English (or you just like imagining yourself coding in a Paris coffee shop), Raphaël Emourgeon has penned a  review of Scout in French. Raphaël sums up Scout nicely: Après avoir installé un gem et ajouté une tâche cron sur le serveur à ...

Read More →

 

Bigger Hardware = Downtime today, 1PM PST

BY Doug Breaker

We’re upgrading our infrastructure and Scout will be down from approximately 1PM – 1:30 PM PST . Sorry for any inconvenience this may cause.

Read More →

 

Solving the random Rails performance problem

BY Doug Breaker

Last week I uncovered a tricky performance problem on Scout I wanted to share. In seemingly random fashion, when accessing my account home page, the render time was 5-6 seconds – way too slow. Other times, the render time would be far less than a second. Here’s what I did ...

Read More →

 

What's in store for 2009 - beyond the numbers

BY Doug Breaker

2008 was a year full of numbers – 34% ( fall of the the Dow Jones Industrial Average ), $1.84B ( Apple iPhone revenue ), and $100M ( cost of the Beijing Olympics Opening Ceremony ). But raw numbers are just the start – in 2009, our goal is to ...

Read More →

 

Say hey at Merb Day

BY Doug Breaker

Merb Day Atlanta – a “one-day, one-track conference for Ruby and Rails developers looking to learn Merb” – is this Saturday and Scout’s extremely proud to be one of its sponsors. Our own Charles Brian Quinn and Matt Todd will be there, taking in the sights, and Matt will also ...

Read More →

 

A simple Rails performance tuning workflow

BY Doug Breaker

I really like Dan Mange’s workflow for Rails performance tuning : Use curl for benchmarking. It’s good enough (and damn simple). Use the ruby-prof gem along with an optional request parameter so you can profile any action, any time. Dan also does a solid job explaining CPU time and what ...

Read More →

 

The 3 pillars of our Rails Monitoring Stack

BY Doug Breaker

Life as a cross-dresser must be unnerving. I could buy a wig, put some makeup on my adam’s apple. Buy a skirt. Does a sock-filled bra work for my bosom? With a bit of mood lighting, I might actually look like a woman, but it would be difficult to relax. ...

Read More →

 

Scout Checks in on Passenger

BY Doug Breaker

The number one requested plugin for Scout has been a Phusion Passenger monitor. I’m happy to report that is now available. Using the new Passenger Monitor plugin , Scout can keep track of Passenger’s VM size, process count, and the amount of private memory it has squirreled away. Aside from ...

Read More →

 

Rails Consultancy Elevated Rails: "using Scout has been a real joy"

BY Doug Breaker

Ensuring 15 Rails apps are running smoothly can be a real chore, especially when they have core pieces that fall outside the typical monitoring stack. Mike Mangino of Chicago-based Elevated Rails recently wrote about consolidating his Rails monitoring efforts using Scout : This simple interface to reporting and alerting is ...

Read More →

 

4 Simple Steps to Detect & Fix Slow Rails Requests

BY Doug Breaker

2016 UPDATE : We’ve released Scout App Monitoring , which automates these steps for you. In Blink , Malcom Gladwell’s book on split-second decisions, Gladwell tells the story of how the Emergency Department at Chicago’s Cook County Hospital changed their process for diagnosing chest pain. Dr. Brendan Reilly instituted a ...

Read More →

 

Monitoring Amazon EC2 Instances

BY Doug Breaker

UPDATED – We’ve made it even easier to monitor ec2 instances. Check out our post on cloud monitoring .

Read More →

 

Monitor MySQL queries with Scout

BY Doug Breaker

Robin Ward of Forumwarz has released a MySQL slow queries plugin for Scout that does exactly what you’d expect: sends an alert when a slow query is run. Derek Laptop"/> The alert contains the full query along with the query time: Alert"/> Note that you’ll need to enable slow query ...

Read More →

 

Meet our team at RailsConf

BY Doug Breaker

Charles, myself (Derek), and Andre are boarding a plan soon for RailsConf in Portland. If you want to do a little coding, chat about monitoring, etc. look for us there!

Read More →

 

Giving your web host access to your Scout account

BY Doug Breaker

A couple of times already, we’ve seen users give their web host access to their Scout account. It’s a big help when debugging problems – especially things that a basic monitoring tool might not pick up (like slow web requests ). Click the ‘People’ link in the header: Select the ...

Read More →

 

How we handle background jobs

BY Doug Breaker

We run background jobs on our Scout servers. Lots of them. As we’ve grown, they’ve used dramatically more resources. We needed a way to simmer them down. Most of these jobs load the entire Rails environment – that’s a hefty overhead. We’ve modified how we run background jobs and we’re ...

Read More →

 

Recording Historical Data in Scout

BY Doug Breaker

By default, when you create a report in Scout , we assume that the data corresponds to the time the report was submitted. Sometimes this isn’t the case – for example, you can retrieve the number of subscribers to your RSS feeds in Feedburner , but not from the current ...

Read More →

 

Taking the guesswork out of scaling

BY Doug Breaker

Determining a web application’s hardware resources isn’t easy (or cheap). Frankly, it’s often just guesswork. Even when you build benchmarking scripts, they can miss key behaviors and ignore important metrics. Scaling becomes a lot less stressful when you can quickly compare a history of your application data with server performance. ...

Read More →

 

Track Active Logins & more

BY Doug Breaker

Ever wanted to track how many people are logged into your server? Mark Hasse built an Active Logins plugin that does just that, returning the total number of people logged in via console, ssh, telnet, etc. Mark has built several other plugins as well: Monitor /proc/mdstat Monitor time drift from ...

Read More →

 

Monitoring Your Rails Sessions With Scout

BY Doug Breaker

An oft-forgotten maintenance issue on Ruby on Rails applications is forgetting to expire old sessions. Lots of sessions in your database can dramatically slow performance. You can use a cron job to expire old sessions, but it’s not always foolproof. We’ve seen timeouts when attempting to delete lots of session ...

Read More →

 

Scout Opens to Public

BY Doug Breaker

The floor is waxed, the windows are washed, and the paint is fresh. Scout, our server monitoring and reporting application , is at your service. As a web developer, there are few things I value more than a solid block of focused development time. Because Scout has largely removed the ...

Read More →

 

Hands on with Scout at Atlanta Ruby User Group

BY Doug Breaker

Last night, I demoed Scout to a room-full of Rubyists at the Atlanta Ruby User Group Meeting. I would love to share all the wonderful feedback, but instead, I’ll share some of the excellent questions (and more elaborate answers) that were asked of Scout: What are the security pitfalls, i.e. ...

Read More →

 

Graphing Rails Performance With Scout

BY Doug Breaker

We’re using Scout, our monitoring and reporting application , to graph the performance of our Rails applications and servers. I’ve uploaded a video that looks at how one of our applications, PlaceShout , impacts the server load and Mongrel memory usage. I also compare PlaceShout’s footprint to another server. Watch ...

Read More →

 

How popular are you? Find out with Scout

BY Doug Breaker

Tracking the results of your blatant self-promotion campaign can be a time-consuming effort. You might be using Google Analytics for web traffic and FeedBurner for blog subscribers. You’re probably checking link referrals. You’re querying the database for usage statistics (user signups, logins, etc.), etc. Scout is an honest friend that ...

Read More →

 

Get notified of slow Rails web requests with Scout

BY Doug Breaker

When something bad happens you want to find out about it as quickly as possible. You’re probably notified of uncaught exceptions. What about slow web requests, which can be just as annoying to a user? Find about about slow web requests (and what might be causing them) in near real-time ...

Read More →

 

See Scout Run - 2 screencasts added

BY Doug Breaker

I’ve added 2 videos to the Scout homepage – they demonstrate 3 things: Installing the Scout client on a remote server is almost too fast Watch the video (1 min 39 sec) One-click plugin configuration (I’m installing the Ruby on Rails Request Monitoring Plugin , so if you’re a Rails ...

Read More →