Firebase Postmortem Blames a Two-Line Cleanup for iOS Crashes

Google has published the postmortem for the Sept. 28 Firebase Analytics outage, and the cause is smaller than the damage: a 2-line configuration cleanup that left the iOS SDK reading a flag that no longer existed (Firebase Blog, Oct. 2, 2026).

What the postmortem says

Engineering manager Jerome Mourits and product manager Don Kahn wrote that the team removed a stale configuration flag at 17:38 PDT but “failed to remove a pointer to that flag in the SDK.” The malformed payload began rolling out globally at 17:41. Crash alerts spiked and the first GitHub bug reports arrived at 17:59. The team identified the cleanup as the cause at 19:16 and finished the rollback at 19:52. The backend served the invalid configuration for 2 hours and 11 minutes.

The SDK did not check that a flag’s name was non-nil, so affected iOS and other Apple-platform apps hit a fatal error on launch. Google said active users saw a one-time crash before the SDK fell back to its previous config. It saw no devices enter a permanent crash loop. Crash reports queue on the device and upload at the next launch. Developers kept seeing elevated numbers after the fix. The post does not name affected SDK versions.

Existing tests passed on the change that produced the bad configuration, the authors said. The configuration also shipped everywhere at once. The kill-switch mechanism is released globally.

Where the response fell short

Google’s Firebase and Google Ads status dashboards stayed green throughout. The team used GitHub and a console banner to reach developers instead. Gergely Orosz, writing in The Pragmatic Engineer’s newsletter, put detection and acknowledgment at about 70 minutes and criticized the lack of status updates. Google’s postmortem dates the first crash alerts and GitHub reports to 18 minutes after rollout.

What Google promised

• A patch “in the next week” that strictly validates server payloads and ignores corrupt flags, falling back to cached defaults. Google is also auditing its other iOS and Android SDKs for similar parsing weaknesses.
• Status dashboard changes that fold SDK-related outages into the dashboards and give Analytics for Firebase better representation.
• Pipeline checks that block flag cleanups leaving orphaned identifiers.
• Phased configuration rollouts with better monitoring.

Our take

Our Sept. 29 article argued that “no SDK update required” dodged the cause. The postmortem’s own fixes include a client-side SDK change. Hardening the SDK is the right promise, but it is still a promise. The crash path remains in iOS apps that bundle Analytics until the patch ships.

The dashboard failure matters as much as the crash. A status page that measures server health cannot see an outage that happens inside a client library. Google says it will change that. Teams that depend on Firebase should keep their own crash alerting on the Analytics path rather than trust the green light.

What to do

• Watch for the Firebase iOS SDK release carrying the validation patch and plan to update.
• Expect delayed crash reports tied to Sept. 28 to keep appearing in Crashlytics.
• Track firebase-ios-sdk issue #16728, where Google communicated during the incident.