mark: A photo of Mark kneeling on top of the Taal Volcano in the Philippines. It was a long hike. (Default)
Mark Smith ([staff profile] mark) wrote in [site community profile] dw_maintenance2026-04-22 09:19 am

Search maintenance

Happy Wednesday!

I'm taking search offline sometime today to upgrade the server to a new instance type. It should be down for a day or so -- sorry for the inconvenience. If you're curious, the existing search machine is over 10 years old and was starting to accumulate a decade of cruft...!

Also, apparently these older machines cost more than twice what the newer ones cost, on top of being slower. Trying to save a bit of maintenance and cost, and hopefully a Wednesday is okay!

Edited: The other cool thing is that this also means that the search index will be effectively realtime afterwards... no more waiting a few minutes for the indexer to catch new content.

denise: Image: Me, facing away from camera, on top of the Castel Sant'Angelo in Rome (Default)
Denise ([staff profile] denise) wrote in [site community profile] dw_maintenance2026-04-11 11:58 pm

The case of the missing notifications

I keep forgetting to post about this: we've been troubleshooting the "missing notifications" problem for the past few days. (Well, I say "we", really I mean Mark and Robby; I'm just the amanuensis.) It's been one of those annoying loops of "find a logical explanation for what could be causing the problem, fix that thing, observe that the problem gets better for some people but doesn't go away completely, go back to step one and start again", sigh.

Mark is hauling out the heavy debugging ordinance to try to find the root cause. Once he's done building all the extra logging tools he needs, he'll comment to this entry. After he does, if you find a comment that should have gone to your inbox and sent an email notification but didn't, leave him a link to the comment that should have sent the notification, as long as the comment itself was made after Mark says he's collecting them. (I'd wait and post this after he gets the debug code in but I need to go to sleep and he's not sure how long it will take!)

We're sorry about the hassle! Irregular/sporadic issues like this are really hard to troubleshoot because it's impossible to know if they're fixed or if they're just not happening while you're looking. With luck, this will give us enough information to figure out the root cause for real this time.