It may be the 'new normal', but it sure is a pain

I've said before that "downtime happens", "outages are a fact of networked life", and "we should all have a Plan B". I think it's clear that having sporadic problems with hosted apps is The New Normal.

But it's hard to be sanguine when PayPal stops telling me about purchases my customers have made and email sent to my Google Apps accounts starts bouncing. Both of these things happened last week.

In addition to the widely-reported problems with international subscriptions, PayPal has been having a hard time keeping its traditional Instant Payment Notification system working reliably. In the past, IPN outages were simply delays, and notifications of payments would eventually be sent. Starting two weeks ago, IPN's have simply been dropped, meaning that PayPal never sent any automated notifications (either via API calls or email) of some payments.

We were able to manually update affected customers' accounts to reflect their payments, but not without doing a lot of manual searching through our records on the (insanely slow) PayPal site. The whole process was time-consuming and frustrating and frankly made us look bad. Google Checkout is looking more attractive every day. But Google's not immune to outages either.

Last week, following reports that a vulnerability had been discovered in Gmail allowing hackers to use it as an open spam relay, emails to our Google Apps account started bouncing. The problem was soon fixed, or so I thought, but then recurred daily for several days after the end of the business day in the US. Hey Google, my business runs 24/7. There is no "good time" to screw around with my email service.

Around the same time, a new option appeared in our Google Apps admin console:

I suspect the email outages are related to Google's fix for the routing problem. I'm glad they're working to address the open relay issue, but bouncing emails sent to my domain isn't acceptable.

So two of my mission-critical services—PayPal and Google Apps Email—both went flakey in the same week. Bummer. I think it's time to take my own advice and have a Plan B.

Update: Five hours after I posted this, PayPal reported yet another problem with IPN's. This is getting ridiculous.