Salesforce Outages — Its not how well you play when you’re playing well
May 10th/11th 2o16 is a 24 hour period that will live long in the minds of Salesforce customers who were unlucky enough to reside on the NA14 instance, as they endured a 20 hour outage culminating in the loss of 5 hours of data. The fact that this is such big news is a testament to the usual reliability of the platform — this is the largest outage that I can remember in nearly 8 years (there have been some that were wider and impacted many instances, but nothing like this duration) and the first time that I can remember a loss of data.
While outages are frustrating, like death and taxes they are very much a certainty in the modern connected world. What was surprising to me was how poorly the communications around this outage were handled.
If you can keep your head while all around are losing theirs …
As Martin Navratilova once said:
What matters isn’t how well you play when you’re playing well. What matters is how well you play when you’re playing badly.
Much the same applies to any kind of service offering — when everything is working well its easy to be in control, responding quickly to queries and issues and publicising compliments from satisfied customers. It’s how you handle the situation when things are not working well, or at all, that really defines you for your customers.
We’ve all heard many times from Salesforce that we need to become a customer company, respond instantly to any concerns, and delight our customers 24/7. During the NA14 outage it seemed that Salesforce were more about talking the talk than walking the walk.
Trust Salesforce?
A fairly common accusation levelled against the Salesforce Trust site is that it doesn’t reflect the current situation. It often feels like issues surface several hours after they occur, typically after they are rectified — personally I find Twitter to be a much better place to find if others are experiencing issues and see if Salesforce support have responded with any confirmation.
During the NA14 outage customers were directed to the Salesforce Trust site which was being regularly updated — every 30 minutes or so when I was monitoring it. In this situation, the problem wasn’t with the timeliness of the updates, it was the lack of information that was present in the updates. For many hours, the updates were identical with only the timestamps changing. In fact, I’m not sure that they could even be called updates, clones might have been a better term. No further information about what the underlying issue was, no ETA for resolution, just there is a problem and we continue to work on it. However, customers were continually directed to the site as though that would aid their decision making. Do these look like the tweets of delighted customers?
Things even got so bad that Marc Benioff took to twitter to try to calm things down :
However, this still doesn’t chime with the customer company mantra. Why are specific customers being offered calls rather than putting the details out on Trust? Should we be aspiring to delight all of our customers, or just the chosen few? Clearly Salesforce felt it was much easier to control the situation by disseminating the bare minimum of information, but that backfired fairly spectacularly.
Idle hands are the devil’s tools
What do people do when they are unable to access their system of record, are frustrated with the lack of information and have time on their hands? Why they make Twitter memes of course:
Now I don’t know what the culture is like in your organisation, but I can tell you that my CEO would be absolutely furious if this was our company being publicly mocked in this way. Obviously he wouldn’t be pleased at the outage, but things do go awry and as long as everybody is working towards a resolution there wouldn’t be too much trouble in that regard, but the loss of credibility would have severe repercussions. Would the memes have still surfaced if there had been more transparency around this incident? Maybe, but I suspect there would have been members of the community defending Salesforce if that were the case, whereas by putting up the barricades and hoping to ride the storm out they were left to fight their own battles.
The first cut is the deepest
Am I being harsh on Salesforce here, given that these kind of outages are exceptionally rare so they won’t be that experienced at handling the communications? I don’t think so. Leaving aside that they are a multi-billion dollar company with communications experts all over the show, those of us on the EU2 server experienced a 9 hour outage earlier this year on March 3rd. Here was the reaction on twitter — look familiar:
Now I get that Salesforce are a public company and they have to be careful about information that they put out lest if affect confidence and their share price, but this doesn’t seem to worry Marc Benioff when he’s tackling unfairness outside of the tech industry, so it seems odd that he’s more muted about something under, I would imagine, his direct control.
But you’re an MVP — does this make you a traitor?
I’d imagine that some people will be a little surprised to read a post from me of this nature, given that I’m usually such a fanboy in this area. However, if you look at my Twitter feed you’ll see that I don’t just blindly praise Salesforce at all times — especially around support, which I’ve found to be variable, to put it charitably.
So while I’m an MVP and one of Salesforce’s biggest supporters, I also consider myself one of their fiercest critics, mainly when I see areas that I feel could be so much better. The outages on EU2 and NA14 certainly fall into that category.
I still think Salesforce is an amazing and transformational platform and I have no doubt that I’ll be building applications on it for many years to come. I just want it to be the best Salesforce that it can be.
I’m better known in the Salesforce community as Bob Buzzard —10 x Certified, including Technical Architect, 5 x MVP and CTO of BrightGen, a Platinum Cloud Alliance Partner in the United Kingdom.
You can find my (usually) more technical thoughts at the Bob Buzzard Blog
