The Algorithm Rewrite That Almost Broke Everything

I spent three days last week convinced I had broken the app beyond repair. Here's the full story.

The social feed algorithm I built in streams 4–7 worked fine in testing. Posts showed up in roughly the right order, engagement signals were being captured, and the ranking felt reasonable. Then I added 500 synthetic test users and everything fell apart.

The feed started serving the same 12 posts to everyone. New content wasn't surfacing. Users with zero followers were seeing posts from accounts with 10K followers, while accounts they actually followed were buried. It was a mess.

Day 1 was pure panic. I streamed the whole thing live and honestly considered rolling back to a simple chronological feed. The chat was helpful — several people pointed out that my ranking score calculation was overflowing a 32-bit integer when follower counts got large. That wasn't the root cause, but fixing it helped.

Day 2 I found the real issue. My "recency decay" function — which reduces a post's score over time — was using the wrong timezone. The server was running in UTC, but I was calculating decay based on local time. Posts published at 11pm EST were being treated as 4am UTC the next day, which made them look 5 hours older than they were. At scale, this completely distorted the feed.

Day 3 was the fix and the rewrite. I pulled the decay calculation out of the ranking function entirely and moved it to a pre-computed field that gets updated by a background job every 15 minutes. This is actually a better architecture — the ranking query is now much simpler and faster.

The thing that unlocked the fix was realizing I was trying to do too much in a single database query. I had one massive SQL query that was joining 6 tables, calculating scores, applying decay, filtering blocked users, and paginating — all at once.

Breaking it into stages (pre-compute → filter → rank → paginate) made each step debuggable in isolation. I could test the decay calculation independently, verify the filter logic separately, and confirm the ranking was working before adding pagination.

This is obvious in retrospect. But when you're building fast and streaming, you tend to reach for the "one query to rule them all" approach because it feels elegant. It's not.

The feed is working well. It's not perfect — I still need to tune the weights for different engagement signals — but it's serving diverse content, surfacing new posts appropriately, and handling the test load without issues.

I'm planning to stream the weight-tuning process next week. It's a good example of a problem that's more product than engineering: what does "good" actually mean for a social feed?

The full debugging session is on YouTube if you want to watch me suffer in real time. Stream #8 in the archive.