Bean counters took over, now it's maximum profit extraction until slow decline takes over. Decline in profit mind you, their software and service quality took a dive a while ago.
Resolved, but can still affect customers for up to 4 hours...
So many problems here, and no signs of an updated SDK that behaves properly on malformed input.
alex_suzuki 12 hours ago [-]
I remember first learning about Firebase when working on Android push notifications, a loooooong time ago (~2011 i think). Over time it grew into this full-featured app development platform, after an aquisition (don’t remember the name).
These days however it feels a bit neglected, and somehow poorly bolted on to GCP. I still have a few production apps running on it and news like this reinforces my belief that it’s in decline.
What are folks using these days that (ideally) is open source and self-hostable? I don’t want to lock in with another platform. Some of these apps use the offline sync feature of Firestore (apps used in basements and other low-connectivity areas).
wilsynet 6 hours ago [-]
I was a new employee at Google when we started talking to Firebase. The founders are super smart and get stuff done.
However, I remember saying to the VP who was leading the acquisition, “shouldn’t we decide what market we want to be in, then evaluate if we should build, buy, or partner?”
He said “no, we should buy it first then figure out what to do”.
I’m not surprised Firebase feels like it’s bolted on. They never really knew what they wanted to do with it. That was true then and it’s true now.
pranshuchittora 11 hours ago [-]
Yes that's the sad part, if you want to integrate Push Notification in your app, and you want to use a 3rd party vendor like OneSignal etc, you still need a fuckin FIREBASE account & FIREBASE dependencies in your app. So every app on the Play Store comes with firebase dependency.
mcsniff 3 hours ago [-]
Er no?
Be hyperbolic if you want, but there are LOTS of apps on the Play Store that do not have internet access, let alone a dependency on Firebase.
hackernud3s 11 hours ago [-]
Calling them a 3rd party push notification vendor is being generous. Google is still doing the push notification and I guess OneSignal offers a wrapper around the libs.
pranshuchittora 6 hours ago [-]
Yes, so basically all Android devices comes with Play Services that's why we need firebase. But in regions like China they have their own Play Services alternative and inorder to send notifications to those devices OneSignal or others provides a unified wrapper.
Leonard_of_Q 9 hours ago [-]
Using UnifiedPush with ntfy [1] as push distributor, used Prosody as well with Conversations as distributor which did work but I prefer ntfy because it does its one thing well.
I guess Supabase is the obvious contender. It doesn't provide the actual frontend hosting though, which is a bit of a pain because now you have two problems.
alex_suzuki 11 hours ago [-]
I wasn't aware that Supabase is self-hostable - thanks for pointing that out, will have a look. Customer is in Europe, if they decide to move away from Firebase, they don't want to migrate to another managed platform run by a US company (digital sovereignty etc.)
ryan_beethe 9 hours ago [-]
I'm working on an alternative (phaselock.io) which is still in alpha but it is open-source and self-hostable.
Offline mode and cross-platform support are some of the features most important to me.
bcye 11 hours ago [-]
Especially for offline sync Firebase is still much stronger than other all-in-one solutions.
The real alternative there would be to use some SQLite syncing solution and a different solution for auth and cloud compute.
matharmin 11 hours ago [-]
While Firebase has support for queries with realtime updates and offline caching, I wouldn't quite call it offline sync.
ChrisMarshallNY 12 hours ago [-]
Ugh.
That's the dark side of these types of dependencies.
But they provide a great deal of utility, so I can understand the attraction.
amelius 12 hours ago [-]
I guess it's time Apple sherlocked Firebase.
oefrha 11 hours ago [-]
Apple's web services are shit, so I hope not. When it works it's great/okay, but when it breaks 5-10% of the time (that's about the rate AirDrop doesn't work for me) there's absolutely no indication what the hell is going on. Apple's allergic to progress indicators and useful error messages.
plorkyeran 7 hours ago [-]
Apple has had various forms of iCloud sync that theoretically compete with firebase but are really awful for the entire life of firebase.
saagarjha 11 hours ago [-]
You can write your own crash reporter now.
hnisjafx40 11 hours ago [-]
Yeah, and pinning versions doesn't save you when the config comes from their server.
frizlab 10 hours ago [-]
All iOS apps that are using firebase…
It’s an important distinction.
saghm 7 hours ago [-]
Important but also pretty obvious. If literally every iOS app were crashing, I would expect a more generic headline from a major news outlet, not a tweet discussing the cause with ALLLLLL CAPS in the middle of it.
jaen 10 hours ago [-]
Exactly the same thing also happened in 2020 with the Facebook SDK:
Ironic. They could save others from crashes, but not themselves.
rvz 12 hours ago [-]
Keep vibe coding and breaking everything.
This will be the new normal as all human “engineers” from staff to seniors are now down levelled to interns and junior engineers when using AI; unable to understand what they are doing and pushing broken updates like this into production with “agents”.
So we have not learned anything since then, on both sides from the SDK builder side and the client side.
Why is it that iOS is crashing but not Android?
Google should know better to not have such an incident like this as the Firebase SDK runs on millions of apps.
If you did not implement a fall back mechanism, perhaps you based your testing assumptions on that “Firebase could never crash my app” which is now false.
pranshuchittora 11 hours ago [-]
No matter who writes the code, the name in the commit is responsible as well as the person who approved / reviewed the PR.
swiftcoder 11 hours ago [-]
The coding side of this isn't even the problem - that a broken change like this made it to production is an operational failure.
Why didn't testing catch this? Why was there no canary? How come alarms didn't wake up an on call as soon as the call-volume on the backend dropped? Why was this update rolled out to every customer at once, instead of gradually?
gib444 11 hours ago [-]
Why pay for and bother with all of that when you can outsource it to your users?
Users who will forget about it, and due to inertia stay on your platform.
pranshuchittora 11 hours ago [-]
I think verification is still a grey area in the agentic software dev.
QA and ppl testing it often resort to coding agents for the QA work, and we all know how good agents are at convincing themselves.
pranshuchittora 11 hours ago [-]
-
2muchcoffeeman 11 hours ago [-]
The solution is testing it yourself, not more AI
Macha 10 hours ago [-]
And in many companies, if they move at a responsible speed (rather than the speed the AI vendors promise, if you let their tools work in yolo mode), that person will get replaced by someone who will just yolo it
lelanthran 11 hours ago [-]
No matter how many times this is said, the reality is that it is not true: no one will get blamed if the AI can be blamed.
darkwater 10 hours ago [-]
Wrong.
They will be totally blamed. And the pushback will be that management forced me to use AI and go fast. Then, management will be the one NOT being blamed for this.
fernandotakai 10 hours ago [-]
there's a LOT of push from vibecoders to stop with PR reviews.
i've also heard "the prompt is the only thing that should be reviewed"
PunchyHamster 9 hours ago [-]
Just blaming AI seems to be the norm now, we have companies literally hacking other companies by "rogue LLM" and not getting any responsibility for it
illegalbyte2 12 hours ago [-]
I was wondering why random apps kept crashing today. Good to know.
pranshuchittora 12 hours ago [-]
But this is completely not acceptable from the Google
solarkraft 11 hours ago [-]
They’ve been accepting this for years and so have users
whstl 6 hours ago [-]
It has been completely normalized and rationalized by the tech community.
Small company is down? Time to move to something else. Pitchforks.
AWS goes down, or Google brings down half the internet? Well lots of other apps are down, so it's not a problem.
yreg 12 hours ago [-]
This is not the first time this has happened.
redwood 10 hours ago [-]
A friendly piece of vocabulary assistance: it's "this morning" ("today morning" is not correct English)
deergomoo 10 hours ago [-]
It’s a common phrase in Indian English
redwood 7 hours ago [-]
That's true. It's one of those phrases that, for example in an interview, an Indian might not be aware holds them back because of how jarring it sounds to the native speaker. My feeling is that while it's true it's effectively colloquial English in the subcontinent it's also appreciated if people understand rather than everyone being too polite to say anything and never coach the person. While I'm here I will admit I love you the words thrice and the needful
Skwid 12 hours ago [-]
I'm hearing an awful lot of "How could they do this to us?" and not a lot of "Wow, maybe we should have read some of this 3rd party code we bundled into 'ALLLL' of our apps"
Just an ignorant and naive outsider's take, but to me it paints a pretty damning picture of the state of mobile development.
skeledrew 11 hours ago [-]
> maybe we should have read some of this 3rd party code we bundled
Not really practical though, if you push the thought to its limit. That 3rd party code is literally everything that's not written by you, from firmware to the launch UI. And if you don't push the thought to the limit then there's always that risk leading to the same "maybe we should've read X" if something happens in X. Trusting that others will be good stewards of all that 3rd party stuff is a hard requirement for making progress.
Skwid 9 hours ago [-]
I appreciate what you're saying. Commercial realities are very different to theoretical ideals, and of course you have to draw the boundary of trust somewhere to get anything done.
I don't think anyone is saying it's an app developers responsibility to ensure the user's bootloader is securely implemented.
There is a lot of space between verify everything and trust nothing, and I don't think it's unreasonable to question whether that trust boundary is in the right place.
I also think that the code compiled to produce the binary you ship is a perfectly reasonable place to put that boundary, would you disagree?
I'm sure that within the mobile dev world it is normal, accepted practice to include lots of unseen code. I don't begrudge anyone involve for taking the money and doing what's expected.
But I still think it's a mad way to run a business or community project, and it's worth considering how we got here and whether it really has to be this way.
alt227 11 hours ago [-]
I agree its not practical, but including any 3rd party libraries in your project puts it at real risk of upstream bugs. There needs to be acceptance of this rather than blame culture.
freakynit 12 hours ago [-]
Mobile and frontend development, both, now-a-days contain way way way more dependencies than they should.
pranshuchittora 11 hours ago [-]
Yes, but the SaaS adoption is also the reason. Many analytics SDKs, observability tooling etc.
bcye 11 hours ago [-]
You're already trusting them with their proprietary cloud products, whose code you can't read. Why wouldn't you trust them with their client SDK code.
jeroenhd 11 hours ago [-]
Because their client SDK might crash the entire app if the server does something funky. We saw it before with Facebook, and now it's Firebase's turn.
A dependency on code you can't read reaching out to servers you don't control is a recipe for disaster. If you can control the server, you can try to add fixes, redirect DNS to a backup cluster, you name it. If you implement the client, you can write the code in a way that doesn't cause full crashes when the remote server does something weird. If you can do neither, you're handing over your business flow and uptime to a third party that doesn't care about you in the slightest.
bcye 4 hours ago [-]
Well your database crashing isn't much better for your uptime really
whstl 6 hours ago [-]
> state of mobile development
If only it was just mobile development.
12 hours ago [-]
nasretdinov 12 hours ago [-]
If only there was any way to avoid this, right..? (I don't mean from Google's side, that as well, but that's not the point)
pranshuchittora 11 hours ago [-]
No, not under your control. It is like "trust me bro" thing from your dependency maintainers.
nasretdinov 10 hours ago [-]
Well, you don't have to use the proprietary third-party SDKs in your apps... If you want analytics you can either implement your own, or use some open-source ones that aren't dependent on some third party backends switching some features on for everyone all at once...
elzbardico 10 hours ago [-]
The more we insist on the idiotic idea that we can delegate all coding to agents and don't even need to read it anymore, the more such things will happen. And they will accelerate, as the hidden errors will compound.
See this 2016 article for the cultural background: https://cloud.google.com/blog/products/gcp/adventures-in-sre...
1. https://support.google.com/googleplay/android-developer/answ...
So many problems here, and no signs of an updated SDK that behaves properly on malformed input.
These days however it feels a bit neglected, and somehow poorly bolted on to GCP. I still have a few production apps running on it and news like this reinforces my belief that it’s in decline.
What are folks using these days that (ideally) is open source and self-hostable? I don’t want to lock in with another platform. Some of these apps use the offline sync feature of Firestore (apps used in basements and other low-connectivity areas).
However, I remember saying to the VP who was leading the acquisition, “shouldn’t we decide what market we want to be in, then evaluate if we should build, buy, or partner?”
He said “no, we should buy it first then figure out what to do”.
I’m not surprised Firebase feels like it’s bolted on. They never really knew what they wanted to do with it. That was true then and it’s true now.
Be hyperbolic if you want, but there are LOTS of apps on the Play Store that do not have internet access, let alone a dependency on Firebase.
[1] https://github.com/binwiederhier/ntfy
Offline mode and cross-platform support are some of the features most important to me.
The real alternative there would be to use some SQLite syncing solution and a different solution for auth and cloud compute.
That's the dark side of these types of dependencies.
But they provide a great deal of utility, so I can understand the attraction.
https://news.ycombinator.com/item?id=23097459
This will be the new normal as all human “engineers” from staff to seniors are now down levelled to interns and junior engineers when using AI; unable to understand what they are doing and pushing broken updates like this into production with “agents”.
Better not blame Claude on this outage.
Why is it that iOS is crashing but not Android?
Google should know better to not have such an incident like this as the Firebase SDK runs on millions of apps.
If you did not implement a fall back mechanism, perhaps you based your testing assumptions on that “Firebase could never crash my app” which is now false.
Why didn't testing catch this? Why was there no canary? How come alarms didn't wake up an on call as soon as the call-volume on the backend dropped? Why was this update rolled out to every customer at once, instead of gradually?
Users who will forget about it, and due to inertia stay on your platform.
i've also heard "the prompt is the only thing that should be reviewed"
Small company is down? Time to move to something else. Pitchforks.
AWS goes down, or Google brings down half the internet? Well lots of other apps are down, so it's not a problem.
Just an ignorant and naive outsider's take, but to me it paints a pretty damning picture of the state of mobile development.
Not really practical though, if you push the thought to its limit. That 3rd party code is literally everything that's not written by you, from firmware to the launch UI. And if you don't push the thought to the limit then there's always that risk leading to the same "maybe we should've read X" if something happens in X. Trusting that others will be good stewards of all that 3rd party stuff is a hard requirement for making progress.
I don't think anyone is saying it's an app developers responsibility to ensure the user's bootloader is securely implemented.
There is a lot of space between verify everything and trust nothing, and I don't think it's unreasonable to question whether that trust boundary is in the right place.
I also think that the code compiled to produce the binary you ship is a perfectly reasonable place to put that boundary, would you disagree?
I'm sure that within the mobile dev world it is normal, accepted practice to include lots of unseen code. I don't begrudge anyone involve for taking the money and doing what's expected.
But I still think it's a mad way to run a business or community project, and it's worth considering how we got here and whether it really has to be this way.
A dependency on code you can't read reaching out to servers you don't control is a recipe for disaster. If you can control the server, you can try to add fixes, redirect DNS to a backup cluster, you name it. If you implement the client, you can write the code in a way that doesn't cause full crashes when the remote server does something weird. If you can do neither, you're handing over your business flow and uptime to a third party that doesn't care about you in the slightest.
If only it was just mobile development.