Amazon Web Services (AWS) is having issues. Amazon said that it has identified the issue as high error rates with AWS's scalable cloud storage service S3 in US-EAST-1, which is also impacting applications and services dependent on S3.
Because the service is used by pretty much of the web, many websites are affected. They include Slack, Trello, Quora, Business Insider, Coursera, and Time. What's more, some users have reported that they were unable to control Internet of Things devices such as lightbulbs because IFTTT is also down.
After a while offline, AWS updated its status page, saying that "As of 1:49 PM PST, we are fully recovered for operations for adding new objects in S3, which was our last operation showing a high error rate. The Amazon S3 service is operating normally."
Most websites that were offline because of the outage, are back online.
What caused AWS S3 to has issue was a typo made by an employee.
The employee entered what was thought to be a routine command to remove servers from an S3 subsystem. But a mistake was made when the employee entered a larger number than intended. The command issued a take down to a much larger set of servers instead of the usual small set of servers. Because the servers are supporting two other S3 subsystems which manage the storage and metadata, troubles started to happen.
The command removed a significant amount of storage capacity and this caused the systems to automatically restart. As this happened, S3 couldn't service requests, which caused even AWS's dashboard to go offline.
In order to revert back to the default, Amazon needed to reboot the subsystems. However, AWS admitted that the system has never been restarted in years. And because the S3 has grown a lot larger since its last reboot, it took a considerable amount of time to make it back online.
What's more, running all of the safety checks and validating the integrity of the underlying metadata also took a bit longer than expectation.
