DevOps and cloud set up Amaysim for growth

Source – computerworld.com.au

Since its launch in 2010, Amaysim has shaken-up the Australian mobile market, pioneering a bring-your-own handset model and undercutting its more established “dinosaur” rivals with cheap and low-commitment monthly plans.

Yet despite its digital-native disruptor status, for the first four years of its existence the company ran on-premise servers and outsourced all of its IT and operations.

Over the last three years that situation has been turned on its head. The company’s 20 servers running 100 virtual machines located in three data centres have all been switched off and shifted to the cloud. As well as the cloud move, the company has taken on developers and engineers and established a DevOps culture.

The approach means the company is all set to scale – it surpassed a million customers in August – and ready to take on the behemoths in new markets like energy and the NBN.

“I really do feel for them,” says Amaysim’s head of IT operations, Peter James.

Cloud shift

Amaysim’s on premise servers were used to support its entire operation from back office systems to front end applications. But in 2014 disaster struck: a 10-hour outage of the systems that managed call data records used to generate invoices and advise customers of their mobile phone usage.

There was barely enough capacity to catch up.

“That was one of the things that actually triggered our decision, because effectively with the on premise equipment we had we couldn’t actually recover from that outage in time because of the volume of data that was getting ingested. So we realised at that point we were never going to catch up,” says James who joined the company shortly after from telco software provider ECConnect.

Amaysim did make it through, just, thanks to four people working flat-out for eight days.

“We recovered but it was along the lines of if it was any longer it would not have been able to catch up with the current hardware capacity that we had,” James adds.

A lift and shift to Amazon Web Services took place in 2014 within about seven months, assisted by partners Polar Seven and Bulletproof.

As Amaysim continues to grow its mobile user base, making acquisitions like Click Energy in April and expands into home broadband, scaling up is now, James says “almost a non-issue”.

Even when there are problems, cloud can pick up the slack. Speaking on the sidelines of the Amazon Web Services ReInvent conference in Las Vegas, James recounts a defect that made it into production when the company launched 4G in 2015 which sent CPU “through the roof”.

“Instead of actually having to fix that code in production – and I have a couple of opinions on hot fixes, I don’t believe they should be done wherever possible – but we didn’t have to hotfix we just added [capacity] and we did that with within half an hour,” he says. “These days that’s hideous us having to wait that long to scale that failure.”

The potentially major incident cost just $64.

The core data-processing application has been scaled up to be able to process about 500 million records a day – 10 times more than usually required.

The business uses Amazon EC2 instances for compute resources, while its main customer database runs in Aurora (since earlier this year, “a massive weight off my mind,” James says).

DevOps delights

As a lean start-up, Amaysim started life without an IT or operations team instead choosing to outsource. The company publicly listed in 2015 and James was tasked with bringing that capability in-house, give structure to product releases, and foster a DevOps culture.

“I’m a really big believer that DevOps is a culture, it’s something that the entire organisation has to get involved with. It’s bringing the engineers on a journey to say you can be more than just going input, output and actually see it holistically,” he says.

“It’s engaging with them to say okay this is how you can write your own infrastructure or for example one of our partners may not always return a successful request so how do you code for failure? How do you code for absolute failure? Really just trying to break down that barrier of ‘it’s someone else’s problem’ when you’re on call or ‘Ops will fix that’. We’re a big believer in our version of the shared responsibility model in DevOps.”

The technology team is made up of a platform and operations team, a DevOps team and an internal IT support team. The engineering function has around 60 engineers.

Developers now replicate an entire EC2 stack and can have it operational in less than 10 minutes, while databases are replicated in an hour with configuration management tools. Increasingly Amaysim has a “serverless front end” with AWS Lambda being the second most common API call after CloudFront.

Releases have gone from a respectable seven per month in October last year, to 90 in October this year.

“The speed to market and being able to build amazing next generation products I just can’t imagine doing it another way anymore,” James adds.

DevOps continues to evolve within Amaysim, as needs change.

“It’s really important to just keep challenging yourself, because what was awesome about the way we did DevOps a year ago may not be working now,” James explains.

“We want to make sure we’re constantly evaluating and that comes back to the agile mindset of iteration, of self-analysis, working out exactly what you want to do and then working out when you’ve done well, what you haven’t done so well and moving on from that.”

Customer experience focus

Amaysim are making some strong plays into other markets – “quite aggressive but that’s sort of how we how we do things,” James says. As well as energy and the NBN it has also set-up a mobile device store.

With on premise “there’s no way we would have been able to do that” James explains.

The team is now considering how the services might join up. At the suggestion that unused mobile data could be put towards paying off an electricity bill for example, James says “definitely things have been thrown around about what we can do”.

Consistent across all its offerings will be a focus on customer experience, James says.

“We’re really focused on giving customers an amazingly simple customer experience because the idea behind our proposition is really, if you don’t like us, because we’ve got no contracts, if you don’t like us you can just leave,” James says.

“For a lot of our customers it’s more than just price. It’s about experience. You don’t want to be sitting there on hold for too long or you want be able to interact with your telco in your way. So there are areas where we think we want to continue and improve but we really want to be an experienced brand more so than just a budget telco.”