Identify all the ingredients in your AWS salad

Now that we’ve laid the foundation for the budgeting process in Chapter 2: Beginning budgeting for CTOs, not dummies, and we’ve identified what qualities define a truly excellent AWS budget in Chapter 1: The four traits of a phenomenal AWS budget, it’s time to give you the tools to create your own AWS budget for your coming fiscal year.

We’re going to step back from the unfamiliar territory of financial accounting and re-enter your area of expertise: cloud infrastructure. Here, we’ll give you the tools and techniques for translating what you know about the historical usage of your cloud resources into a real budget. Come along!

Starting with the bill

Let’s talk about that darned AWS bill. While it provides a nice monthly summary of your spending associated with a variety of branded services, these carry no financial meaning when it comes to budgeting.

For budgeting purposes, you need to know what values may change and what values are going to stay the same as the company evolves. From your AWS billing history, and your own personal cloud resource expertise, you’ll be able to divide spending into four separate categories, as they carry very different implications for your financial planning. These categories are actual cloud resource usage, contracts for expected cloud resource usage, ancillary services purchased via Amazon, and taxes.

You might notice a key drawback to the AWS bill now that you’ve divided it up: we need to analyze your actual historical usage to predict next year’s usage, but it’s not detailed enough to do this! While the remaining total from your last year’s worth of AWS bills gives you a total of your usage-based spending for the past year, that total cannot help you predict next year’s spending. That’s because there is no way to determine from these bills what resources, infrastructure, or sub-components built that spend, nor which will increase meaningfully in the coming year.

To get to the bottom of usage planning, prepare to roll up your sleeves. It’s about to get complex. In this chapter, we’re going to talk about the theory behind finding detailed usage data and applying it to future predictions, but it’s important to note that in order to execute this yourself, it’s going to require some detailed spreadsheet analysis on your part.

Get the real source of usage truth

Your AWS bill looks pretty simple. For each of the member accounts in your organization, it shows individual branded services (think: Amazon Redshift) and identifies the total due for that service and the subset in tax.

While this is certainly a clean and simple summary of the technical systems used to build your infrastructure, this will simply not help you to analyze your actual historical usage, nor will it help you to predict your future usage. Here we’ll explain how you can source and understand your own historical usage data.

Retrieve the raw data

There are a number of tools you can use to pull accurate resource-level usage, including Aimably Insight, but without purchasing any third-party tools, your best resource is to use your most recent complete month’s Cost and Usage Report (CUR). We’ll assume for the purposes of this article that you’re using the CUR.

Isolate usage spend

Your CUR shows line items for every element of AWS billing imaginable, sometimes even divided up into sub-segments of every day. Since you’ve already isolated other forms of spending by reviewing your annual bills, it’s critically important to isolate everything that’s not actual usage data before you do any further data analysis.

Now, you might be concerned about removing line items associated with credits, reserved instances or savings plans, as the act of doing this could hide usage that was offset or paid for using such contract tools. Never fear! The CUR is such a comprehensive document, that there are always separate lines for tracking the credit side of a contract and the usage side of it as well.

Divide by business purpose

At the very least, you must separate what resources and usage represents a production environment. This is because these estimates have to be reported in your final bill as a separate Cost of Goods Sold line item.

Often, companies will separate these resources by AWS account, which can make this process very easy. If you're one of the folks to draw the short stray of intermixed production resources across acounts, you'll need to review your resources line by line to identify and isolate production resources from others.

For bonus points, divide out the remaining resources by purpose, such as staging, development, QA, data science, and sales engineering. This added context will come in handy later on down the road!

Decide whether to carry forward subscription agreements

By and large, once a company has subscribed to a program or service, it’s pretty unlikely that they’ll cancel. That inertia against evaluating subscription spending is even more profound in businesses, as companies rarely take the time to consider the impact of the subscription on the actual business and may worry about unintended consequences if the subscription were to end.

In the case of AWS spending, as you prepare your AWS budget, we do strongly recommend performing an analysis of these subscriptions.

First, your team is the only team in the company with the AWS expertise to properly do this. If it’s not going to be you to do the job, the job simply won’t be done.

Second, many of the contractual subscriptions offered by AWS may deliver a significant return on investment but it’s not a guarantee that this return on investment is being delivered to your business. You should make a determination on the effectiveness of each.

Finally, your business might benefit from an expansion of a subscription or the addition of further subscriptions. It’s incredibly important to plan for these now, during the budgeting process. Otherwise, if an unbudgeted large item gets tacked on to your AWS bill halfway through the year, it could set you up for needing to reduce spending elsewhere.

When it comes to doing this work, the actual job depends on whether we’re talking about Ancillary Purchases or Contracts for Usage. We’ll explain.

Ancillary Purchases

These are typically fairly straightforward recurring monthly or annual expenses. You’ll want to review each of the Ancillary Purchases made in the previous year and craft an opinion as to whether these purchases should be carried forward into the new year. You should also consider at this point whether there are AWS programs or marketplace purchases that you want to bring to the company for added benefit.

Making a final determination on each of these suggestions will require a specific conversation with the purchasing decision-maker from the affected departments to determine whether the investment will be carried forward to the next year. We recommend you get these conversations started now, as these decisions often aren’t made easily. By the time you’re done with your usage projects for the new year, you should have a very good understanding of which Ancillary Purchases you’ll bring with you into the new fiscal year.

Contracts for Usage

As you plan your next year’s budget, you need to understand which contracts will last throughout the year’s term, and for any that may end during the year, you’ll want to determine whether you’ll want to enter similar contracts for a further period of time.

When it comes to reserved instances, savings plans, and even enterprise discount programs or private pricing term sheets, it’s entirely possible that your company’s actual usage did not meet the targets set in your selected contracts, resulting in spending more on your commitments than you would have spent if you had paid list price for your actual usage.

Luckily, AWS provides helpful resource utilization tools for analysis of the contractual plans that depend on specific resource utilization like reserved instances and savings plans. For the high-end enterprise discount programs and private pricing term sheets, each is utilized on a dollar basis, therefore complex analysis is not required.

In this stage, your goal is to ensure that none of these programs that are up for renewal are being underutilized. If so, you’ll want to project a smaller contract for the renewal date. In the event that you find reservations that are not in use at all, you can even consider relinquishing them via the reservation marketplace.

Once you’ve determined which contracts you will carry forward into the upcoming year, you’ll want to track when any upfront payments are due as well as record any custom pricing that should be applied to any of your future usage, and any restrictions on that pricing that may exist.

Identify which usage is likely to change, and what’s likely to stay the same

This section is where we start to build the bridges between your technical expertise and your company’s financial planning. In order to properly estimate how much your company will spend on AWS resources, you need to predict which costs will scale as more customers are added to your production systems and more complexity is added to the operations of your business.

At Aimably, we are big fans of using a three-tiered framework for categorizing cloud infrastructure usage: fixed systems, expandable systems, and scaling usage. By assigning all of the components of your AWS organization to each of these categories, you can begin to see how your budget planning becomes clearer.

Gather intelligence on business growth plans

Barring the development of new products and features, or a major initiative for re-architecture, your company’s AWS usage increases are going to be driven by forces external to the infrastructure itself. (Don’t worry, we have your new product and feature budget needs covered in Chapter 4: Make future plans a reality with well-modeled budget additions)

For most companies, the primary force for AWS usage growth will be increased customer counts, but it’s also important not to discount other elements, such as increased engineering team members, increased frequency of sales demonstrations, and others.

In order for your AWS budget to be closely aligned with the full company’s growth plans, your projections must depend on data inputs from other departments. You’re going to need to go talk to people.

Projected customer growth

Customers’ use of your SaaS product produces usage of a wide-ranging number of AWS services, so gathering insight into planned customer growth is critical for modeling out future AWS spending.

Meet with your finance team, with the goal to retrieve expected customer counts for each month of the upcoming year, agreed upon by the sales and marketing teams during their budgeting process (you might need to work directly with your sales team, depending on your company’s budgeting process). It’s highly likely these numbers are not yet final, so make sure to keep the lines of communication open, ensuring you can properly update your budget projections when and if the customer count projections change.

Just as important to the future plans, you’ll need a good estimate of the number of the company’s previous twelve months of customers have looked like. With these data, you’ll be able to build models for projecting usage.

While you may be tempted to look up how many individual customers were in your product’s database at any given time, this is not the correct data source. This is because the company’s projections throughout the upcoming year will be sourced by the company’s finance team from their own current recording of the total clients you have. By using a different data source you would skew your results. So, in your conversations with finance, make sure to get their end-of-month reporting on your company’s total client number.

Planned operations expansion

A number of your resources in AWS are not customer-facing. Whether they’re development, quality testing, data science, or sales demonstration environments, their usage is driven by other business growth variables than simply the count of expected customers. Remember, you’re not just budgeting for your production environment, you’re setting a budget for your company’s entire AWS spending, which can extend far beyond the responsibility of the employees who report directly to you.

In order to effectively model expected usage growth for your cloud resources, your job is to understand your current resource capacity, the variables upon which that capacity depends, the intended growth trajectory of those variables, and the rate at which each resource’s capacity will scale alongside that variable growth.

From an interdepartmental coordination perspective, this will be the most complex part of the budgeting process. Examine the usage details provided to you by the CUR, and associate these with specific departmental programs, then work with the departmental leaders to identify what single variable will most likely drive increased usage of the related resources. Then, request monthly projections for the growth of this variable, as well as the previous twelve months, just as you did for customers.

Project your future raw spending, using business growth metrics

Up until this point, we’ve divided out and categorized usage rates, but we have yet to associate these with actual projected spending amounts. At Aimably, when we perform budget analyses for our clients, we always create a first version of the budget based on our concept of ‘raw spending’ which refers to the cost of your projected usage, if it were to be billed at the list price.

When you analyze your company’s raw usage and, simultaneously, project raw spending, you’re able to develop recommendations for the application of contractual agreements that can optimize that usage for the best cash management strategies for the company. If you were to analyze the projected usage assuming a carry-forward of any pre-existing contracts, you may miss opportunities for the best cost optimization available.

This section will show you how to follow the Aimably method of first calculating raw spending, before applying existing contracts and following up with new contract recommendations.

Let’s dive in!

The raw spend formula

Now that you have all your historical spending from the most recently completed month identified by category, each of these categories will feed into this simple raw spend-projection formula, where raw spend refers to the cost to run your AWS infrastructure before applying contractual agreements (such as credits, reserved instances or savings plans) to reduce overall cost. The formula looks like this:

Monthly Projected Raw Spend = Base Cost + Scaling Spend + Compounded Step Changes

Let’s first discuss the formula, then we’ll walk you through creating each of the critical values to plug into the formula, followed by a plan to produce your monthly raw spend projections. Keep in mind: this formula is limited to projecting AWS spending so long as your company operates its cloud infrastructure in a business-as-usual mindset and does not take into consideration any major new initiatives or launches (don’t worry, we’ll get into that in Chapter 4: Make future plans a reality with well-modeled budget additions).

Going back to our categorization of usage types, we can begin to see how the formula gets built. Fixed systems will remain constant and therefore represent the Base Costs of your AWS environment. Expandable systems may expand (or contract) as a result of customer or operational growth demands. We can forecast these potential expansions into compounding monthly step changes (and we’ll explain that in a little bit).

Finally, we derive your Scaling Spend by distributing the value of your total current Unit Costs for your current customer count, then re-apply it to each coming month based on customer count predictions. Unit Costs are derived from the cost to run systems categorized as scaling usage.

We’ll explain how to derive the values for the formula in the sections below.

Calculate your base costs

As we mentioned above, fixed systems drive your base cost. Luckily, the impact is easy to analyze since it remains flat throughout the year. The job simply requires totaling up all of the previous month’s spending on resources you’ve already designated as fixed systems.

That being said, In the event that your company currently takes advantage of corporate discounts, reserved instances, or savings plans, the challenge in this process is to associate each fixed usage example in your last month’s data with the actual list price without these contractual agreements.

To come up with a good base cost estimate for these cases, first, you’ll isolate out current spending that is incurred at list price from usage that is covered by contractual plans, then, you’ll carry forward list price spend, and finally, you’ll apply list pricing to calculate the raw value of any usage covered by current contracts.

Calculate your scaling spend

As we mentioned above, the calculation of Scaling Spend is derived from a Unit Cost calculation. The purpose of the Unit Cost is to model the direct impact of each additional customer on the cost to run your infrastructure. As you know, much of the AWS billing model is based on the real traffic running through your systems, which can be rather surprising for CFOs whose previous experience was not in Software as a Service.

Luckily, the process to get to your Unit Costs is nearly identical to the process to get to Initial Base Costs, just with some additional division. You’ll want to go through the same process of adding forward the list price cost of each resource designated as Scaling Usage to reach a total cost of systems designated as Scaling Usage for the most recent month.

Once you have a total raw cost for Scaling Usage systems, you’ll want to divide that value by your company’s current customer count. The result is your unit cost.

Now, to project your future Scaling Spend, it’s just a matter of simple multiplication of your unit cost by your projected customer count for each month.

(It’s important to note that we have assumed all of your customers use your system in largely the same way. If you have distinct populations of customers that use your system in distinctly different ways, you’ll need to perform multiple unit cost calculations and scaling spend projections.)

Calculate your compunded step changes

In a previous section of this chapter, we talked about collecting all the projected monthly operational growth statistics for any variable you know is affecting one or more cloud resources in your AWS organization. We’re going to take these into consideration when considering compounded step changes.

In order to effectively model expected step changes to cloud resources, your job is to understand your current resource capacity, the variables upon which that capacity depends, the intended growth trajectory of those variables, and the rate at which each resource’s capacity will scale alongside that variable growth.

It’s not a quick process, but ultimately you need to review the usage history of each of your resources (we recommend using CloudWatch). If you pull the maximum usage values for these resources for the previous twelve months, you’ll be able to build a model to project future usage against the previous twelve months of operational growth variables.

Now that you have an expected utilization growth prediction assigned for each of your resources, you need to make a decision as to your level of comfort with resource utilization. For example, is it your expectation that no resource should ever hit 100% usage at any time? If so, what is the maximum utilization you will accept for any of these resources? Once you make a determination, this becomes your Resource Health Threshold.

Now that your threshold is set, in all likelihood, you’re going to notice that some of your predicted usage rates will exceed that resource health threshold in the upcoming year.

Congratulations! You’ve found the months where investments will need to be made. You now get to plan for these in advance.

How you upgrade your resources is a matter of both opinion and architecture. In some cases, you’ll upgrade the size of the resource and in some cases, you’ll add an additional resource. You probably won’t know this in advance.

It can help to streamline the budgeting process by accounting for the larger cost of adding an additional resource in all cases, so as to give you the flexibility in the future to make an appropriate determination without going over budget. Following this logic, you can assume that the cost of every resource should double when its usage threshold hits (and triple in the event that the usage threshold doubles).

Building the monthly raw spend calculation

To finish this project, we’re going to revisit our original raw monthly spend formula and plug in the values generated in previous steps to build a raw spend AWS budget. To recap, the formula is below.

Monthly Projected Raw Spend = Base Cost + Scaling Spend + Compounded Step Changes

Simply reference the values calculated, insert them into the formula and… Voila! You now have a monthly projection of raw spending, based on actual business growth plans, presuming all your usage would be billed at the list price.

Transform projected raw spending into a real AWS budget

Now we now find ourselves armed with a realistic model for your future cloud resource usage (assuming business-as-usual and now new products, features, or re-architecture adventures) and you have a monthly sticker price available to you in the event that you were to pay for all these resources at their list price.

There are three major hurdles in place before we reach a true AWS budget: you need to factor back in the effects of your existing contracts for usage, plans for future ancillary purchasing, and taxes. Just as we separated out these effects at the very beginning of this chapter, we must add them back in before we are finished.

Now is the time when we’ll revisit the subscription analysis you performed early on and add the final touches.

Contracts for Usage

To add back in contracts for usage, you’ll need to consider both upfront charges and adjustments to pricing that are included in the contracts. It’s a rather straightforward process to insert expected upfront charges into the months where they are due, yet it’s a bit more complex to handle pricing adjustments.

Let’s assume you make zero changes to your contractual commitments, their terms are renewed for the full upcoming year, and you have already resolved situations in which contractual plans are not being underutilized. In this case, for Base Costs, we simply alter our raw spend budget base costs to pull from actual pricing recorded in your historical data in lieu of list prices.

When it comes to Compounded Step Changes and Scaling Spend, however, in the case where you haven’t expanded your contractual commitments, you can only assume that the discounted pricing will affect spending that matches historical rates and not any further increases. In these cases, you’ll want to use the actual pricing for usage that matches the historical data, and apply the list price to the projected usage that is in excess of historical data.

You might be tempted at this point to consider adding additional contracts into your plan to expand your cost optimization footprint. If you are, we recommend jumping to Chapter 5: Select the cost optimization techniques that best suit your unique infrastructure and business goals so you have all the information at your fingertips to make good choices.

Ancillary Purchases

Now, in comparison to Contracts for Usage, this process is pretty basic. You’ll simply grab the agreed-upon purchases and plug them into their monthly or annual costs on the dates on which they are due.

Taxes

For tax calculations, apply the applicable sales or VAT tax percentage for your jurisdiction to the current monthly totals, resulting in a final AWS budget per month.

Divide by business purpose

In your final draft, separate the subtotals of your production resources from all others previously-identified business purposes.