Reflections on what happened today when Clickfunnels when down for a little bit.
On this episode Russell talks about a situation that happened with Clickfunnels shutting down for a few hours. He talks about how he and his team handled the problem and were able to get things up and running in a short amount of time.
Here are some of the cool things you will hear in this episode:
So listen below to find out why Clickfunnels went down, but was able to recover in a relatively short amount of time.
---Transcript---
Hey everyone, this is Russell, I hope you guys are doing awesome today. Today’s Marketing In Your Car is actually happening in my office for a couple of reasons.
First off, I was getting ready this morning. I was planning on waking up at 5 to start working on the book. I got the last edits on the book going, I have to get it done by the end of the month if we’re going to get it published in time for our launch date of April 18th. But then I had only slept a couple hours that night, so I just every 9 minutes for 2 hours kept pushing snooze. Then finally it was 7:00 I’m like, “Oh crap, the kids gotta get up.” So my wife and I got up and started working on the kids, get everything ready and then check my phone to find out that Clickfunnels was down.
Some of you guys probably had the same sinking feeling in your chest this morning when you saw that, as I did. So I just wanted to kind of walk through what’s going on in my head. Because one, so you guys know, but more importantly because there are times when there’s going to be disasters and problems and issues with you business. And I’m just going to go through what I did and what our team did. Hopefully it’ll help some of you guys when disaster hits for you.
One good thing I have on my side, is I’ve got the most amazing development team on planet Earth. So by the time I’d noticed they’d already known and they were on it and working on it. What’s funny is the last time we had an issue, I’m pretty sure, I’m positive I did a podcast about this when I was in London. It was over a year and like 3 or 4 months ago. Almost a year and a half was the last time we had any issues.
It’s funny, because at the time Clickfunnels had been built on a database company called Clear DB was the name of it. And we were kind of outgrown them and we knew that and we were trying to move over to the other servers and we were trying to move and they wouldn’t let us, they basically had handcuffed us saying, “No, we don’t want you to move your data and if you do it’s going to have to go this other way and you’re going to have down time for 3 or 4 days.” Whatever.
So as we were getting closer and closer to what became the first day our servers crashed, things started working less and less and more issues and more hiccups. It was just their company couldn’t handle what we were putting through to them. And at that time we had about 10 thousand members so we wanted to shift over but couldn’t and couldn’t.
I was actually speaking at an event in London with my family, so I was in a plane flying to London, when I landed in London I figured out how to get international phone, which is not as easy as you’d think. Look at my phone and basically get there and realize Clickfunnels is down. Messaging my team and they’re freaking out, everyone’s freaking out and it was just like, “huh.”
We didn’t know the issue, but basically, luckily on my team Ryan Montgomery and Todd Dickerson kind of called an audible, “Hey look, we can either wait for Clear DB to rebuild or we’ve got to rebuild this whole structure over on Amazon, where we want to move things.” So they made the decision to start rebuilding. So they jumped into it and rebuilt, basically the whole database structure of Clickfunnels over here in Amazon. And then started restoring entire Clickfunnels database from an old backup. Not old backup, backup that had been, I think we back up every 20 minutes or 30 minutes or 10 minutes, whatever it is, and restored it from backup.
Ended up, I think we were down for 12 hours, which was insane. People were ready to kill me. Luckily I was in London, no one could find me. But it was tough. Luckily because our developers had the foresight to rebuild instead of waiting, it ended up taking 3 weeks to restore our servers. So we would have been down for 3 weeks if it hadn’t been for amazing people. I remember when I got there, Facebook was blowing up, my messages. Everything was blowing up and people were angry and I was like, “What do I do?”
I didn’t know what to do. I’d never been in this situation before when literally 10’s of thousands of people’s lives and livelihood depended on us and what we were doing. It was a scary, scary situation. At first I was like, well maybe I’ll make a video and kind of try to calm the storm. Make them feel good. But I was like, you know what? No. I’m pissed, they’re pissed. This is not a good situation, this is not acceptable for us, it’s not acceptable for our upstream providers. So instead in the hotel room I had my kids go in the bathroom and try to be quiet and I recorded, this is pre Facebook Live, so I recorded a video with my phone and I didn’t clean it up or polish it. I said, “Look, this is what’s happening.” And I was pissed and people felt it. And I posted that in our Facebook group.
And what came from that was amazing. We had hundreds of people come and comment on our site. Basically like, “Look, we’re here for you.” And they fought hard for us and our guys luckily got things up within 12 hours. We were back stable. And what was interesting was because of that we learned a bunch of things. And because of that we were able to move to a new server which was awesome. Things became more stable and then we spent the next 5 or 6 months just working on stability, which if you remember there wasn’t a lot of new features that came out in that time because we were focusing on building this back end structure making sure that never happened again.
And it didn’t for a year and 4 months. And then today it happened again. So the first thing is like, what in the world? Was it our infrastructure? We thought we had built things in a way, looking at server recourses, we had the ability to probably give out ten times where we’re at right now and still be fine. As of today, so this is a year and a half later, we’re at 30 thousand members right now. So we have 3 times as many people using the system, 3 times as many people’s lives are affected when we go down. So we go down and we’re looking on, from a server standpoint we were fine. We weren’t taxing the servers more than we should be, so we tried to figure that out and dig deeper.
Finally we figured out it was Amazon’s….something to do with them. They were the issue. So it was like this really weird thing where we were helpless. We can’t do anything, they have to fix it on their side. And their service isn’t typically the best, we actually just upgraded from the, I don’t know the $1000 a month service plan to $15000 a month service plan. So now we’ve got direct lines to all the backend developers. So we were able to upgrade that for the future. But we were kind of stuck. We don’t know what to do. What do you do?
So while we’re waiting for them to fix it, our guys were like, “Okay well let’s rebuild.” And they start going to work and it was just so cool to watch them in crisis mode. They didn’t sit there and do nothing. They said, “Okay, alright. What are our options? How many contingency plans do we have?” And they went and executed the first one and I watched as they tried and failed. Try the next one, failed. Try the next one, failed. And it kept going and going and finally they figured out something to get it live again. And this time we were down for 4 hours, which is the longest we’ve been down since the other one.
But they got it up and working again before the Amazon team was able to figure out the issues on their side. Now we’re communicating with them and they’re fixing the stuff on their side. And this time is kindo f the same thing. What do I do? I have all this fear. We’ve got 40 thousand people in our Facebook group, its our only means of communication I have with our audience. People in the Facebook group were like, “Why didn’t you email us.” I’m like, “We use Clickfunnels as well. We can’t log into our email. We’re in the same problem you are. We have status.clickfunnels.com page, we’ve got the Facebook group and most people on the Facebook group are like, that’s the only channel we have to communicate.”
So we decided to quickly do a Facebook live and let people know this was happening and be very aware. Again not trying to sugar coat things. In fact people messaged me, “Are you pissed or are you sick? I can’t tell.” I’m like, “A little bit of both right now, to be completely honest.” But we did that and then an hour later when we had more updates, we did another Facebook Live for updates. Then an hour later we were back live and kind of shared that as well. What’s interesting as you watch people that are, the message, the overwhelming thought was the fact that people appreciated our communications.
It was interesting, I remember about 6 months ago Lead Pages went down for about 18 or 19 hours if I remember right. We were watching their server logs and they were down for almost a full day. And they did the opposite. They went radio silent and didn’t say anything to anybody. Didn’t message, didn’t talk about it. It was the opposite and I know a lot of people left Lead Pages that day because of that. And I’m sure, unfortunately we’re going to lose people from Clickfunnels. But I had other people message me in the forum that said, “Hey I haven’t joined Clickfunnels yet, but after seeing your communication throughout the day, I’m now joining. “ So I think we’ll gain a lot of people as well.
It’s just one of those things. It’s kind of, for a lot of us, for me, it’s my first time being a real CEO and trying to figure out these things and I think what I’ve found from this experience is a couple of things. First one is over communication is key. People are okay if there’s problems happening as long as they know about them and you validate it. Number two I think that my initial response when people would complain on Facebook is I want to be very defensive.
I had people who were like, “Hey, Clickfunnels is down. I lost hundreds of dollars. Do I get a free month. I want you to pay that money back.” The defensive Russell, wrestler Russell wants to fight everybody he sees. You know what? Dude, the second you start paying me, right now you’re paying $97 a month to use our entire platform. The same thing that in the past was costing me 20 to 30 thousand dollars a month to run on our own. For a $100 you’re paying for that. As soon as you’re willing to give me 25% of all the money you make from us, then I’m willing to give you back money when we’re down for an hour or two.”
I just, in fact, I even started commenting multiple times that. And then I’m like no, I can’t. That’s not the right response. I think us as a team we all wanted to get defensive because we’re doing our best. It’s not even our fault. It’s Amazon’s fault. Amazon’s not refunding my money, I guarantee that. In fact, we’re spending more to be able to actually get answers from them. It’s so frustrating and multiple times I wanted to do that, so I messaged my team, everyone on support. “People are upset, and they have every right to be. I’m upset, they’re upset and we cannot be defensive. That’s going to cause, that’s just the wrong thing. It doesn’t help anybody. We have to have empathy. Empathy’s the only thing that’s right in this situation.”
So when people on Facebook complain and send me, mouthing off, by default I want to say, I tried to go to everybody and say, “Look, that sucks. I’m so sorry. It’s not okay. We’re not okay with it. You shouldn’t be okay with it. We apologize. We’re sorry.” And try to bring empathy to the situation as opposed to where we naturally go. I think that was a big part of it. Communication, I think empathy is big.
And I think the third thing is understanding. A lot of the things I talk to you guys about in the last year or so and what the new book is about is really when you build a culture of people where it’s more than just…..the best way to explain it in one of our masterminds is a dude who’s super cool named Noah. Noah in one of the things said, “Because I’ve bought into you, Russell and I’ve bought into Clickfunnels, I see all your competitors pop out. I see them in my Facebook feed, I see the emails. I look at them for less than a second, just long enough to laugh, then I close it down. Because I bought into you.”
And I think that reason why the community is doing okay through something like this and why we’re able to recover from something like this is because we’re not just a product, we try to become more than that. We’ve tried to become a community, tried to build a family, tried to not just give you guys the tool, but the training and the platform and everything else that ties into that. It’s because of that, again on the same side I’ve talked about before, when we were building Clickfunnels, I didn’t want everyone to say, “That’s Russell’s company.” I want them to say, “This is our company.” It works both ways. When we have down time I don’t want people, I want people saying, “We have down time, this is something we’re all struggling with.” And we look at it more as a community as opposed to us, not us versus them, but you know, here’s the service provider and here’s us.
I think for all of us, the more you can focus on that. You’ve heard me talk about that a lot in this podcast, but you’ll hear me talk about it more when the book comes out. Because that’s what the whole first section is all about. How are we building a following of people that are part of something bigger that you’re creating as opposed to someone who’s buying your product. And I think that those are the 3 things I kind of learned and more reaffirmed of today as we went through this mini disaster.
So hopefully, I hope to see you guys because it may not be today, might not be tomorrow or a year from now, or maybe it will never happen. But if and when it does happen, those are some of the tools that have helped us to survive now 2 disasters in the 2 and a half years we’ve been live. Hopefully if any more come in the future it will help us to wave the, survive the storm as well.
But I just wanted to thank you guys all for being listeners of the podcast and I want to thank you guys for being members of Clickfunnels. If you’re not using Clickfunnels yet, hopefully this is just one more reason for you guys to come try it out. Because we really do care. We’re here for you and we’re not leaving. We’re not going anywhere. So if you’re not a member yet, go to Clickfunnels.com, get your free 2 week trial, test us out and see what everyone is talking about.
This week Eli said we passed 30 thousand active members, which is a huge milestone for us and our goal is to get 100 thousand by the end of this year. And obviously the only way we’re going to do that on our side is to become better. Become a better infrastructure, become better people, become better teachers, become better marketers, become better at the fulfillment side, become better at on-boarding and I think that this was a really good chance for me and for my team to look at that. Look at what happened and make some really good choices based on that and some decisions and build it stronger and better so that it keeps getting better for you guys every single day.
The one big difference I would say between our platform and anything else that I’ve ever seen in our industry is just because of the success we’ve had so far, we’re able to put a lot of time, energy, money back into the infrastructure and make it better and better and better. So that’s why hopefully you feel safe and comfortable building your business and your platform here with Clickfunnels.
So with that said, I hope that helps you guys. I hope that in your time of trial, as the leader, the entrepreneur, the CEO or whatever it is your role is, that some of these lessons will help you and give you that foundation you need so that you can wave the storms. And that’s about it. With that said, someone’s ringing my doorbell, I’m going to check out who it is. Appreciate you all for listening. Talk to you all soon.