Impact: Users using app passwords to connect third party IMAP/POP3 clients were unable to view emails during the affected period.
- The problem began at 2:00 AM PST, when a new change was deployed to production.
- At about 2:10 PM PST, staff monitoring error logs noticed a large number of errors fetching emails.
- By 2:30 PM PST, the problem was identified and a fix began deployment.
- By around 2:50 PM PST, the problem was resolved.
Technical details: Passwords are used to unlock private keys which can in turn decode messages. App passwords thus maintain their own encrypted version of user private keys, which can be unlocked when the app password is supplied.
While refactoring code, a change was made that inadvertently used the encrypted private key of the regular password and tried to decode it with the app password.
Remediation: We will add more tests to our automatic test suites to make sure app password logins work correctly.
- Remember that code that needs refactoring probably has other latent issues too. Hiding in confusion and redundancy is often a small bit of actually necessary business logic.
- Still need faster deploy cycles.