How we lost our official YouTube channel

This is the story of how we lost our official YouTube channel and the related Google+ page for TalentLMS. It is also the story of how big companies lose sight of the customer or, similarly, why small companies will always have a place in our world.

We run an online business for over a decade now. Online businesses rely heavily on other online businesses to build solid products. We are no exception. TalentLMS relies on over a dozen of services for credit card handling, translations, video encoding, transactional emails etc. Selecting wisely your technology partners is important. A failed partnership can bring down hell to your business and reputation. Consider waking-up one day to find your e-commerce processor gone or your hosting provider hacked. No one wants to find themselves in such a mess.

Selecting BIG

Selecting wisely usually means selecting BIG. The bigger your partner is the less likely to go bankrupt or to not have proper security or pivot or break overnight its API. The bigger the partner, the better the support. At least this is what conventional wisdom dictates.

Take for example our case. When we started TalentLMS we decided in favor of Rackspace as our hosting provider. Rackspace is notoriously expensive. But it is BIG, so we thought it would be a better infrastructure on top of which we could build our service without headaches. For the most part, Rackspace has been proven to be exactly that.

Selecting BIG also means you have to deal with Google, one way or another. Internally, we use Google for a dozen of things; from Adwords and Analytics to internal mail handling and file-sharing. We also use it for sharing videos through YouTube. Our videos demonstrate how our product works and are also an effective marketing tool. We considered YouTube to be “indestructible”, so we made those videos part of the product. For example, whenever someone creates a new TalentLMS account, he or she can check out a small course that comprises YouTube videos from our channel. We never really thought that those videos would be lost one way or another. But we were wrong.

The chain of events

When we started TalentLMS, we selected Google Apps as our email infrastructure. Google Apps offered a free tier for small businesses which was more than enough at that time. As we experienced exponential growth we decided to merge this free account with our paid Google Apps account established for eFront, our enterprise LMS product. This would reduce the complexity of managing two Google Apps accounts and would allow our employees to have one mail box with two aliases for both our domains – @efrontlearning.net and @talentlms.com.

The merging process is a bit technical but it boils down to this: export your data, delete the previous Google App account, transfer the domain ownership and re-create users inside the target Google App account. The destructive nature of merge seemed like a small problem as we could simply re-create the lost entities.

Indeed, the merge completed without a problem and everything looked normal. Then things turned interesting.

A couple of days after the merge, our YouTube channel suddenly disappeared. Whenever someone tried to access it he got a “Channel permanently deleted” message. Without a YouTube channel, TalentLMS users could not view the product’s tour or take the introductory course. Recall that video assets were part of the product. We really made a “good” first impression for the unlucky fresh users of those days.

It was not immediately clear what caused this deletion. We initially thought that we somehow broke Youtube’s rules or that our channel was hacked. Only a posteriori it became obvious that the merge triggered a chain of undesired automated events that led to the channel deletion.

And here is what happened.

The nature of the merge caused the deletion of entities within the original Google Apps account. This included all talentlms.com email addresses as well. The re-created users on the merged Google-Apps account had the same email addresses but they were not the same entities.

One of the deleted email entities was the administrator of TalentLMS’ Google+ page. And this Google+ page, on its turn, controlled TalentLMS’ YouTube channel. The deleted administrator turned the Google+ page an orphan. And this orphan was deleted by an automated clean-up process. The same way, without a page to control it, the YouTube channel became an orphan and was deleted via a second automated clean-up process.

Game over.

The curious case of Google’s support

It is obvious that we could transfer ownership of Google+ and YouTube channel before the migration and save ourselves from all the trouble. The thing is that it did not cross our minds that something like that could occur. You can get prepared as much as you want but reality is a strange beast.

We turned to Google for support. Google offers different levels of support for different products. Our problem spanned over a number of products; from Google Apps to Google+ and YouTube. The only one offering a chat/phone support option was the (paid) Google Apps account. However, they could not really help, as Google+ and YouTube are not part of Google Apps offering.

Google+ on the other hand offers only forum-based support. Just a bunch of nice people that suggested talking with Google Apps reps.

YouTube offers a “channel undelete” option. Phew! That seemed to be the solution to our problem. We contacted YouTube and our account was restored within a few days. We had access to our videos and the hole within the product was fixed.

BUT this was not the end of our troubles as we could not administer the restored channel. The technician on YouTube undeleted the channel and (somehow) restored a “ghost” of our original Google+ page. However, trying to administer that page returned a 500 page error as its original Administrator was deleted.

This 500 Error was alarming. It also offered a strong hint on what was really happening. Our hypothesis that there is an underlying process that deletes ghost Google+ and YouTube accounts was confirmed a few days later as our YouTube channel was deleted for a second time.

Back to support. We contacted YouTube and Google Apps once more, explaining the situation. We tried to be as detailed as possible, pinpointing that simply restoring the YouTube channel won’t work. The YouTube representative helped us restore the account and escalated the problem to “someone” within the organization who could help us “more”.

A few days after our communication, the YouTube channel was deleted for a third time.

Current situation

As I write these lines, our account looks deleted. We still expect some “elaborate” help from Google on this matter.

The funny thing is our channel is there (somewhere) and its data is intact. The obvious solution was to have a Google technician edit our Google+ account and manually modify its administrator. However, when you deal with BIG corporations with highly specialized roles and your problem spans across responsibilities and divisions, solving simple problems becomes a highly non-trivial task. A smaller, more agile, company would have been able to tackle the issue with a personalized and holistic approach. We regularly do so for our own customers.

Finally, we decided to solve our problem the hard way; we reconstructed our Youtube and Google+ accounts and built a script to update the sample courses for our users. We still miss our followers from the old YouTube channel and Google+ page. But, at the end of the day, the real impact on our business was negligible.

The Bottom Line

Size is a vanity metric. A big company can rarely offer the personalized support required to tackle complex problems in a timely manner. Fixing such problems fast makes all the difference between life and death for your online business. With the right culture, a smaller company can overcome its “size” and be an equally good or even better partner altogether.

Choose wisely 🙂