Nine Fives

When this happened it caused a great deal of churn within the network as all of our aggregated links had to be re-established, leader election for spanning-tree had to take place, and all of the links in the network had to go through a spanning-tree reconvergence. This effectively caused all traffic between access switches to be blocked for roughly a minute and a half.

[…] When the network froze, many of our fileservers which are intentionally located in different racks for redundancy, exceeded their heartbeat timeouts and decided that they needed to take control of the fileserver resources. They issued STONITH commands to their partner nodes and attempted to take control of resources, however some of those commands were not delivered due to the compromised network. When the network recovered and the cluster messaging between nodes came back, a number of pairs were in a state where both nodes expected to be active for the same resource. This resulted in a race where the nodes terminated one another and we wound up with both nodes stopped for a number of our fileserver pairs.

In the course of our troubleshooting we discovered that our aggregation switches were missing a number of MAC addresses from their tables, and thus were flooding any traffic that was sent to those devices across all of their ports. Because of these missing addresses, a large percentage of our traffic was being sent to every access switch and not just the switch that the destination devices was connected to. During normal operation, the switch should “learn” which port each MAC address is connected through as it processes traffic. For some reason, our switches were unable to learn a significant percentage of our MAC addresses and this aggregate traffic was enough to saturate all of the links between the access and aggregation switches, causing the poor performance we saw throughout the day.

[…]

We need to be more mindful of tunnel-vision during incident response. We fixated for a very long time on the idea of a bridge loop and it blinded us to other possible causes. We hope to begin doing more scheduled incident response exercises in the coming months and will build scenarios that reinforce this.

Though we cannot be absolutely certain, we believe that the combination of our growing number of users and the loss of the database cache that resulted when we resized the database caused a sudden, sharp increase in lock contention within mysql. The issues were compounded by the fact that Amazon uses proprietary technology to power the file system that RDS runs on, but doesn’t offer documentation about how this technology works during a resize operation. Further, the lack of root access to our database made the cause of the problems more difficult for us to understand.

Monday’s migration caused higher load on the database than our operations team has previously seen during these sorts of migrations. So high, in fact, that they caused Percona Replication Manager’s health checks to fail on the master. In response to the failed master health check, Percona Replication manager moved the ‘active’ role and the master database to another server in the cluster and stopped MySQL on the node it perceived as failed.

At the time of this failover, the new database selected for the ‘active’ role had a cold InnoDB buffer pool and performed rather poorly. The system load generated by the site’s query load on a cold cache soon caused Percona Replication Manager’s health checks to fail again, and the ‘active’ role failed back to the server it was on originally.

[…]

Upon attempting to disable maintenance-mode, a Pacemaker segfault occurred that resulted in a cluster state partition. … In the second, single-node cluster, node ‘c’ was elected at 8:19 AM, and any subsequent messages from the other two-node cluster were discarded. As luck would have it, the ‘c’ node was the node that our operations team previously determined to be out of date. … As a result of this data drift, inconsistencies between MySQL and other data stores in our infrastructure were possible. … Consequentially, some events created during this window appeared on the wrong users’ dashboards. Also, some repositories created during this window were incorrectly routed.

[…]

In summary, three primary events contributed to the downtime of the past few days. First, several failovers of the ‘active’ database role happened when they shouldn’t have. Second, a cluster partition occurred that resulted in incorrect actions being performed by our cluster management software. Finally, the failovers triggered by these first two events impacted performance and availability more than they should have.

Shortly after this, our DNS provider, GoDaddy, experienced a totally unrelated outage.

The disruption was precipitated by an Amazon Web Services outage which affected the US East region, beginning at 8:04PM PDT. … Approximately 30% of our EC2 instances, which were responsible for running applications, databases and supporting infrastructure (including some components specific to the Bamboo stack), went offline. …

• The management API for the AWS US East region became unavailable
  • In order to restore sufficient capacity quickly, we needed to bring additional instances online in the same region. Without the API, we couldn’t start any new instances.
  • We regularly rely on this API in order to collect information about the state of our infrastructure, and to diagnose problems. This lack of visibility slowed the recovery process.
• Elastic Load Balancer instances and Elastic IP addresses failed to respond promptly to configuration changes
  • We use ELBs and EIPs to redirect traffic away from failed instances to redundant and secondary systems, so when these mechanisms responded slowly or malfunctioned, traffic continued to be routed to systems which were down.

…A large number of EBS volumes, which stored data for Heroku Postgres services, went offline and their data was potentially corrupted. As a result, customer databases remained down until they could be recovered…

In past EBS incidents, it was very rare for volumes to be damaged even if they sustained downtime, so they were very fast to recover. We were therefore not fully prepared for a recovery effort of this magnitude, and it took several hours to automatically restore the affected databases.

2012-06-30 05:25 UTC: I finish configuring the replacement Tarsnap server and start the process of regenerating its local state from S3. The first phase of this process involves reading millions of stored S3 objects; unfortunately, these reads were performed in sequential order, triggering a worst-case performance behaviour in S3. As a result, this phase of recovery took much longer than I had anticipated; unfortunately, the design of the code meant that changing the order in which objects were read was not something I could do “on the fly”.

In the single datacenter that did not successfully transfer to the generator backup, all servers continued to operate normally on Uninterruptable Power Supply (“UPS”) power. As onsite personnel worked to stabilize the primary and backup power generators, the UPS systems were depleting and servers began losing power at 8:04pm PDT. Ten minutes later, the backup generator power was stabilized, the UPSs were restarted, and power started to be restored by 8:14pm PDT. At 8:24pm PDT, the full facility had power to all racks.

… Time for the completion of this recovery was extended by a bottleneck in our server booting process. Removing this bottleneck is one of the actions we’ll take to improve recovery times in the face of power failure.

… As the power and systems returned, a large number of ELBs came up in a state which triggered a bug we hadn’t seen before. The bug caused the ELB control plane to attempt to scale these ELBs to larger ELB instance sizes. This resulted in a sudden flood of requests which began to backlog the control plane. At the same time, customers began launching new EC2 instances to replace capacity lost in the impacted Availability Zone, requesting the instances be added to existing load balancers in the other zones. These requests further increased the ELB control plane backlog. Because the ELB control plane currently manages requests for the US East-1 Region through a shared queue, it fell increasingly behind in processing these requests; and pretty soon, these requests started taking a very long time to complete.

The routing outage was the result of three root causes.

The first root cause is related to the streaming data API which connects the dyno manifold to the routing mesh. On the dyno management side, an engineer was performing a manual garbage collection process which created an unusual record in the data stream. On the routing side, the subprocess of the router which handles the incoming stream could not parse this record.

On Tuesday (Feb 21), we deployed a code change to the dyno management agent which runs on the machines which comprise our dyno manifold. This change introduced a latent problem with creating extra (“phantom”) dynos, which only became visible under unusual load conditions that would occur that weekend.

When the GA creates the transfer certificate, it gives it a one year validity range. It uses midnight UST of the current day as the valid-from date and one year from that date as the valid-to date. The leap day bug is that the GA calculated the valid-to date by simply taking the current date and adding one to its year. That meant that any GA that tried to create a transfer certificate on leap day set a valid-to date of February 29, 2013, an invalid date that caused the certificate creation to fail.

As mentioned, transfer certificate creation is the first step of the GA initialization and is required before it will connect to the HA. When a GA fails to create its certificates, it terminates. The HA has a 25-minute timeout for hearing from the GA. When a GA doesn’t connect within that timeout, the HA reinitializes the VM’s OS and restarts it.

If a clean VM (one in which no customer code has executed) times out its GA connection three times in a row, the HA decides that a hardware problem must be the cause since the GA would otherwise have reported an error. The HA then reports to the FC that the server is faulty and the FC moves it to a state called Human Investigate (HI).

On Saturday we had a rough night. There were problems routing HTTP traffic to customer applications, to varying degrees, for more than four hours. … Upon further investigation, we discovered that we were being attacked by a very large scale distributed denial of service (DDoS) attack. This attack was sending multiples of our normal production traffic levels to our HTTP routing system which was unable to handle the load.

The hot spare for the database was not known to be in a usable state due to missing information in the management UI. We are improving our monitoring to monitor the status of the hot spare to ensure that it is in a usable state. We are also improving the documentation around failing over to the hot spare since it was also not immediately obvious to the responding engineers what the procedure was.

First, our EC2 management service (which handles API requests to RunInstance, CreateVolume, etc.), has servers in each Availability Zone. The management servers which receive requests continued to route requests to management servers in the affected Availability Zone. Because the management servers in the affected Availability Zone were inaccessible, requests routed to those servers failed. Second, the EC2 management servers receiving requests were continuing to accept RunInstances requests targeted at the impacted Availability Zone. Rather than failing these requests immediately, they were queued and our management servers attempted to process them. Fairly quickly, a large number of these requests began to queue up and we overloaded the management servers receiving requests, which were waiting for these queued requests to complete. The combination of these two factors caused long delays in launching instances and higher error rates for the EU West EC2 APIs.

At approximately 03:30 Pacific time, it was determined that Heroku was the target of a distributed denial of service (DDoS) attack. The method of the DDoS was a SYN flood [1]. Our network engineers applied a variety of host-based techniques to reduce the effect of the attacks. A combination of tweaked networking configuration and some quickly built tools for maintaining system firewall rules helped us get the affected servers to the point where they were no longer crashing, but we were still seeing very high levels of network packet loss to the nodes — users often had to reload multiple times in order to get a web page to load.