AWS billing is broken and Kubernetes won’t last, says irreverent economist Corey Quinn

Source:- techrepublic.com

The Duckbill Group’s outspoken co-founder Corey Quinn shares his thoughts on the cloud industry, Walmart’s paranoia about AWS, why Docker is nearly out of money, and why Larry Ellison “is not people.”

There are fairly few people watching the clouds, so to speak, that set off fire alarms whenever they take to Twitter. Corey Quinn, self-styled cloud economist at The Duckbill Group, is perhaps the most visible. Quinn’s Twitter antics pull no punches. He’s very upfront about his views and presents them in oneliners tailor-made for the Twitter age, like the comedy stylings of Stephen Lynch or Mitch Hedberg, but with significantly more edge:

TechRepublic interviewed Quinn on the labyrinthine billing practices of Amazon Web Services, why transit pricing on AWS is a silent budget-killer, and the adoption of popular technology trends such as Kubernetes and Docker, as well as Walmart’s noted distrust of AWS, and Larry Ellison’s boasting about Amazon’s protracted migration away from Oracle.

TechRepublic: What’s the most consistently wrong thing you see AWS users do?

Quinn: The most consistent mistake that everyone makes when using AWS—this extends to life as well—is once people learn something, they stop keeping current on that thing. There is an entire ecosystem of people who know something about AWS, with a certainty. That is simply no longer true, because capabilities change. Restrictions get relaxed. Constraints stop applying. If you learned a few years ago that there are only 10 tags permitted per resource, you aren’t necessarily keeping current to understand that that limit is now 50.

TechRepublic: If I gave you a magic wand, and you had the power to kill three AWS products, what would they be?

Quinn: Three products that I’d love to see killed. That’s going to win me some enemies like you wouldn’t believe. Okay, I’ll give it a whack. I would say Amazon CodeCommit because it’s a distraction, it’s their GitHub equivalent.

I would kill their data transfer pricing twice. I could take that a step further, because the reasons that I kill it each time are different. The first is that it is completely impossible for people to figure out in advance what it’s going to cost, it’s inscrutable. Secondly, it’s never seen a price drop, we’re still paying 1998 prices for data transfer. It’s really the cloud’s Achilles heel.The most consistent mistake that everyone makes when using AWS—this extends to life as well—is once people learn something, they stop keeping current on that thing.Corey Quinn

TechRepublic: That’s a lot more detailed than what I expected. I thought you’d kill it once for ingress and once for egress.

Quinn: No, no. If I kill either ingress or egress, it’ll just retry.

TechRepublic: So, why is Amazon still charging 1998 prices for data transfer? 

Quinn: I wouldn’t want to speak out of turn, but I think that because it is so inscrutable, you’re not seeing significant push back. It’s one of those things that disappears into the background noise of the bill. 

Yes, you have customers who are sending millions of dollars on data transfer, but they’re spending tens of millions of dollars on EC2 instances or storage. You don’t have a big data transfer problem for most workloads until you have a much bigger problem with other services. 

I’d also suspect it exists between a bunch of different service teams. It feels like there may not be an explicit single point of ownership, or “throat to choke,” depending upon what term you prefer.

SEE: Amazon Web Services: An insider’s guide (free PDF) (TechRepublic)

TechRepublic: Is it because programmers are relying on outdated understandings of AWS, is this what results in outsized transfer bills?

Quinn: I think so. You’re also seeing people deploy software that wasn’t written with zone affinity in mind: You’re following best practices, deploying things into multiple regions or availability zones. [Perhaps] things like Cassandra or MongoDB want to do a replication between them. 

Those aren’t really built with an eye towards, “Huh. Maybe I’m paying per gigabyte on that replication factor,” and “Oh, it turns out I’m processing petabytes a month,” and that’s not an exaggeration, that means that I’m spending tens of thousands of dollars.

At small scale, a lot of what AWS does is awesome and makes stupendous sense. As you scale it out, there are certain inflection points where using services that come nicely gift wrapped for you no longer make sense.

Managed NAT Gateway, for example, charges a $0.045 per GB processing fee, on top of any transfer that passes through it. That isn’t a big deal until you’re have massive amounts of traffic to the Internet or S3 passing through that, and then you’re spending—in some cases—millions of dollars on that. It disappears into the background noise of the EC2 section of the AWS bill. It doesn’t show up in data transfer.

TechRepublic: How large are the companies finding their pockets lighter by a million dollars per month in data transfer fees?

Quinn: Here’s the fun part. It’s easy to think of this as, “These are only multinational companies that have this hit.” But we’re seeing relatively small-scale startups impacted by this. What people lose sight of is that infrastructure, in almost every case, costs less than payroll. The people who run these things are inherently more expensive than the infrastructure itself, and losing sight of that leads to ridiculous things. 

At some point, stop cutting costs. You’re not going to cost optimize your way to a better business model. Take a look at Lyft, for example. They very publicly said they’re committed to spend a minimum of $100 million a year to AWS for the next three years and the internet exploded with fits of, “That’s stupid. You could save so much money by running your own data centers.” Sure. At the time, they were also losing $900 million a year, so doubling their AWS bill or cutting it to zero, does not meaningfully alter the economics of their service. Cutting the Amazon bill, surprisingly, should not necessarily be on their Top 10 list of items that they’re focusing on, as a company. What people lose sight of is that infrastructure, in almost every case, costs less than payroll. Corey Quinn

Why Kubernetes is better for building a resume than a platform

TechRepublic: From an engineering standpoint, the solution to everything is just add another layer of abstraction. That said, what’s your complaint against Kubernetes?

Quinn: My problem with Kubernetes, as it currently stands, is two fold. First, it is tremendously complex to understand, and not just get up and running, but to maintain it. In six months, if you’re running this and suddenly things get broken—or worse, intermittently slow—how do you begin to diagnose it? There’s an awful lot of hand waving and abstractions built on top of it.

Secondly, people are building their entire career around Kubernetes. In a few years, no one is going to care about it at all. It’s going to have simplified and slipped below the surface of awareness. We’ve seen this in tech time and again and again. Originally, it was web servers. You had to have an intimate knowledge of GCC compiler flags, then RPM, then YUM, and… now it’s a checkbox on an S3 bucket. Things don’t get more complex over time.

Right now, the burden for maintaining Kubernetes is…at least a million dollars, [either] in managed consultancies or engineering time. It’s valuable, but it’s not an end state. I think you can safely skip to what comes beyond without too much trouble from an engineer perspective.

It also…feels like exactly what you would expect to come out of Google. It’s relatively opinionated in a super condescending way. It has an awful lot of moving parts. It is stupendous overkill for most workloads that people are putting on it, but it is “the new hotness,” to quote an old movie, and people are embracing it regardless of what I say. I tend to be somewhat down on a lot of technologies that are over hyped. On the other side of the coin, I’m usually right.

TechRepublic: Well, what, in your opinion, did Docker do wrong?

Quinn: A lot. I think the number one mistake that they made is they didn’t make friends. I think that they weren’t sure what their business model was going to be. I think, as a result, they were very cautious about partnering closely with anyone. They kept everyone at arm’s length.

So that when it turned out that—like Kubernetes eventually will—suddenly nobody cared about Docker anymore, because the container runtime is so far away from the interesting part of whatever it is that they’re building, that it just doesn’t matter anymore. There’s no story—I might use Docker, if you’re using Kubernetes, you almost certainly are—but there’s no part of that story where you’re cutting a check to Docker.

TechRepublic: How similar is that to developers of open source software who are saying Amazon is taking their project, offering it in the cloud, and monetizing it.

Quinn: You had to expect that would happen. I think there’s sour grapes. People expected, when they were building things with those licenses, that everyone else who tried to sell a product would be crappy at it. It turns out, Amazon kind of isn’t. People mistook open source for a business model, and I think that that’s having reverberations throughout the ecosystem.

Amazon would point out that they have never cloned a service or open source project that then drove them out of business. Take a look across the board companies like Mongo, Elastic, RedisLabs, are all doing very well. If anything, Amazon claims this is a validation of the product that they’ve built.

Whether or not that’s true, I don’t know. I don’t tend to spend enough time navel gazing in the open source world. There are a lot of very angry people with spittle flowing out of their mouth as they rant incoherently about the validity of these models. None of these companies seem to be doing mass layoffs.

Why “more or less everything” is wrong with multicloud

TechRepublic: So, what’s wrong with multicloud?

Quinn: As expressed, as a best practice, more or less everything. There are a few problems with it. For starters, it forces everyone to cater to the common denominators that exist between clouds, and those are far smaller than most people think they are.

If you’re just using cloud as a place to run a bunch of your VM’s, it works. It’s annoying, but it works. It also removes a whole bunch of higher level offerings. Now, there are valid use cases to do it. When regulators require it, sure. That makes sense. When your customers demand it, there are times where you have to bow to the inevitable. 

But we’re seeing people push this as a greenfield best practice which is, to be polite, nuts. Who’s pushing this? It’s a whole bunch of second- and third-tier operators who know that if you’re going to go all-in on one provider, it’s not theirs. 

Despite the fact that I focus on AWS’s ecosystem, I’m not a partner. I don’t care which cloud provider any company takes. I suggest that they pick the one that works for their business model and in many cases, that is very boasted vocally not AWS, but pick whatever it is and go all in.

I think that there are a lot of companies who pound their chests and claim “we do nothing with AWS because we consider them a competitor.” If you go on LinkedIn, they employ an awful lot of people whose primary area of focus is AWS. So, who’s someone to believe?

TechRepublic: What do you think of Walmart’s alleged paranoia about Amazon and AWS?

Quinn: I think there is validity to not wanting to finance your competitors operations. I think that going to your vendors and your suppliers, effectively being a bully, demanding that they operate their business differently, is shitty. I think that that speaks of tremendous insecurity. I think it is putting an awful lot of your competitive disadvantage onto the backs of smaller companies that may very well not have the bandwidth to approach these things, and I find it to be perfectly honest, reprehensible. 

If you want to compete, compete on your merits. Not out of what would be anti-competitive behavior, in a functioning antitrust regulatory environment.

TechRepublic: What would your advice be for migrating away from Oracle?

Quinn: I think it’s a good idea. Take a look at Amazon, who is incentivized more than most to move off of Oracle, and they’re saying it’s still in process. It’s still going to take years, and you have Larry Ellison on stage crowing about this. I don’t think that sounds as good as Oracle thinks it does. These people freaking hate us and it’s still almost impossible for them to migrate off of our nonsense. 

It doesn’t give new prospective customers the warm fuzzies because you know that whenever you make a choice of database provider, there’s going to be lock-in. But saying that some of the most technically adept people on the planet, at Amazon, who are incredibly motivated to do this way faster than almost any other company on the planet, still hasn’t gotten there? That says a lot. Not much of it good.

Becoming the face of billing on Amazon Web Services

TechRepublic: How did you wind up in this position? How did you become Corey Quinn?

Quinn: My previous job was running operations at a giant asset manager. I went there by way of acquisition and as it turns out, my personality works exactly as well as you think it would in a large compound regulated financial institution.

I dabbled with consulting on and off for years and I figured, If I were going to…run a consulting company the way that I wanted to, what would I care about? I want it to be built around an expensive problem the people would be willing to make go away, in a sense, and because over the course of my career, I’ve dealt with some pretty abusive on-call environments, I wanted it to be a problem that was strictly business hours, so no one is waking me up screaming at 2 am.

At first, I was afraid that Amazon was going to fix the billing system and put me out of business in six months. Now, I’m scared that they never will. 

My prospects, when I talk to them, they don’t say, “Not interested. No.” They say, “Not this quarter. Can we talk again soon?” This problem doesn’t go away on its own. My marketing is inbound, I don’t have a sales office dialing for dollars on the consulting side.

[Becoming] the name brand for AWS billing is kind of heady. The way I did it was by finally embracing the thing that always got me in serious trouble with previous jobs: my personality. 

In most companies, you can’t have a representative go on stage and, more or less, rip Amazon a new one over some decision they’ve made and not have fall out for that. When you own the company, and the company is built around your brand, what is Amazon going to do? Take away my birthday? I don’t see this as a significantly large threat on them.