Summary of events (times in PDT):

• 20:37 - A database indexing operation was started to add functionality for an upcoming release for contactless signatures
• 20:45 - As a result of the indexing operation, disk IO times began to spike causing database operations to be queued and slow down
• 21:00 - The indexing operation was cancelled on the primary DB node but follow-on effects had already begun to put increased IO pressure on the primary DB instance
• 21:20 - In an effort to restore normal functionality in the most expedient manner, we failed over to our secondary DB node
• 21:55 - The new primary node warmed up sufficiently to handle production traffic at moderately degraded levels
• 22:00 - We began to terminate long-running database queries to take pressure off the DB
• 22:20 - Metrics returned to nominal levels

Rationale for the database operation:

The motivation for this database operation was to add support for contactless deliveries. We'll be shipping this feature soon once we have a solid plan in place for how to perform the DB operation.

Understanding the follow-on effects:

What hindered recovery in database performance was a series of suboptimal query patterns which poisoned the cache and put increased load on the disk for the primary. These queries were primarily due to the use of the GET /tasks endpoint by API clients. They were able to time out externally while remaining long-running on the database.

What could have been improved:

• The performance characteristics of the indexing operation should have been better understood ahead of the operation
• We should have waited until closer to the bottom of our peak load to perform this DB operation (~+120 mins)
• Long-running database operations should never have continued beyond the request lifecycle
• We should have begun to terminate these long-running queries earlier

What we'll be changing in the future:

• We will do better planning for indexing updates like this in the future
• These production events will be planned further in advance and publicly visible on our status page
• We will ensure database operations cannot continue beyond the request lifecycle
• We will put additional monitoring and logging in place to ensure these kinds of queries are detected in advance

Conclusion:

The database operations we performed should have been planned and modeled better to reduce risk. In response to the COVID-19 crisis, it has been a priority of ours to support our customers and ongoing community efforts. We shipped this change too casually and will ensure this operation can be done in a safe manner. We will be deprecating the GET /tasks endpoint for the API as its characteristics do not allow for it to be efficient as it is not paginated. The GET /tasks/all endpoint is the preferred choice here as it is more efficient, is paginated, and has a broader feature set. We will coordinate with customers to ensure a smooth transition for integrators. We will be shipping code to ensure that long-running database operations cannot extend beyond the request lifecycle (i.e. they time out on the same timescale as the request at the load balancer). We apologize for this blip in performance and want you to know that we can handle increased load; this event was not related to limitations in infrastructure.