AWS Spend Anomaly Detection

SoftwareONE's Spend Anomaly Detection (SAD) is a comprehensive solution designed to help customers monitor and control their AWS costs effectively. With a primary focus on detecting and reacting to dangerous spending trends and anomalies, SAD ensures that SoftwareONE and its customers remain informed about substantial spending spikes and budget breaches. The key value of SAD lies in safeguarding SoftwareONE customer's best interests by proactively identifying and addressing cost-related issues.

Objectives

The main objectives of the Spend Anomaly Detection (SAD) solution are as follows:

1. Set Dynamic and Recurring Monthly Cost Budget:
   SAD establishes a dynamic and recurring monthly cost budget for all customer AWS Master Payer accounts (and optionally for each member account). The budget is set based on the following principles:

   • If the previous month's spend data is available, the budget is calculated as three times the previous month's spend. If the calculated budget is less than $10,000, it is set as the default budget value of $10,000.

   • For new customers without any previous month's spend data, the default budget is set to $10,000.

2. Dynamically Adjust Budget Threshold After Every Month:
   SAD continuously monitors the actual spend and forecasted spend against the monthly budget. After every month, it automatically adjusts the budget threshold based on the actual spend, ensuring proactive budget management.

3. Integrate with Event Management Shared Service (Splunk+ServiceNOW):
   SAD is tightly integrated with the Event Management service, leveraging Splunk and ServiceNow. It generates individual anomaly alerts whenever customers breach the actual spend and forecasted spend thresholds. This integration allows SoftwareONE to take timely actions and raise incidents to address any potential issues.

Diagram

Solution Architecture

The Spend Anomaly Detection (SAD) solution is built on AWS native services for cost monitoring and event management, providing a comprehensive approach to detect and react to dangerous spending trends and anomalies. The core components of the solution include:

1. Budget Configuration and Monitoring:

   • SAD utilizes AWS Budgets, a native AWS service, to track and manage cost-related data. For each customer's AWS Master Payer account (and optionally for each member account), a budget named "SWOSpendAnomalyBudget" is created with a cost type of "COST." The budget is designed to auto-adjust its data based on historical cost data for the last three months.

   • The "SWOSpendAnomalyBudget" budget is configured with monthly granularity, providing a recurring and dynamic cost budget. It leverages historical data to forecast the budget for upcoming months, enabling proactive cost management based on historical trends.

   • Budget notifications are configured to trigger alerts to the "SWOSNSTopicSAD" SNS topic when specific thresholds are breached. The SNS topic acts as a central hub for receiving budget-related notifications and distributing them to relevant subscribers.

   • The "SWOSpendAnomalyBudget" budget is configured with 2 separate alarms - one Actual cost and one Forecast alert. They both could trigger events for Spend Anomaly Detection.

2. Lambda Function (SWOBudgetSADLambda):

   • The SWOBudgetSADLambda function is a custom AWS Lambda function written in Python (runtime: python3.9). It handles events related to the SAD budget and thresholds.

   • Trigger Mechanism:

     • The Lambda function is automatically triggered in two scenarios:

       • During the CloudFormation stack creation or update process, the Lambda function is triggered through the custom resource "SWOBudgetSADLambdaRunner." This enables the function to configure and set up the necessary budget notifications.

       • Additionally, the Lambda function is triggered when there is a change in its configuration by CloudFormation, utilizing the EventBridge rule "SWOSADLambdaTriggerEventRule."

   • The Lambda function is responsible for updating the budget's notification thresholds based on specific conditions. It dynamically adjusts the threshold values after every month to reflect the customer's cost profile accurately.

   • When budget alarms are triggered due to threshold breaches, the Lambda function sends messages to the SQS queue specified by "SQSSADQueueURL." These messages contain relevant information about the customer's spending, which can be processed further by external systems or applications.

3. SNS Topic (SWOSNSTopicSAD) and EventBridge Rule:

   • The "SWOSNSTopicSAD" is an AWS SNS topic that acts as a central hub for receiving and distributing budget-related notifications.

   • AWS Budgets sends notifications to the SNS topic when budget alarms are triggered or when there is an automatic update to the budget threshold. Subscribers, such as the Lambda function "SWOBudgetSADLambda," are subscribed to this topic to receive and process the notifications.

   • The EventBridge rule "SWOSADLambdaTriggerEventRule" monitors changes to the Lambda function's configuration made by CloudFormation. When such changes occur, the rule automatically triggers the Lambda function to update the budget's notification thresholds, ensuring accurate budget management.

4. IAM Role and Policies:

   • The IAM role "SWOBudgetSADFnExecutionRole" is created to grant the necessary permissions to the Lambda function "SWOBudgetSADLambda" for executing actions and accessing other AWS services.

   • Two IAM policies, "SWOBudgetSADPolicy" and "SWOCrossAccountSQSWriteSADPolicy," are associated with the role "SWOBudgetSADFnExecutionRole" to provide specific permissions.

   • The "SWOBudgetSADPolicy" allows the Lambda function to view and modify the "SWOSpendAnomalyBudget" budget, enabling it to update the budget's notification thresholds accurately.

   • The "SWOCrossAccountSQSWriteSADPolicy" enables the Lambda function to send messages to the specified SQS queue ("SQSSADQueueARN"), facilitating the delivery of budget-related data for further processing.

Default Configuration

The Spend Anomaly Detection (SAD) solution comes with a default configuration for budget thresholds and other parameters to provide initial settings for cost monitoring and anomaly detection. These defaults are applied during the initial deployment of the SAD stack. The default configuration ensures that the solution is functional out of the box, but it is essential to review and adjust these settings according to each customer's specific requirements.

Default Budget Thresholds

1. SADMinimumNotificationThreshold:

   • Default Value: $10,000

   • Description: This parameter sets the minimum absolute amount for a threshold breach that would trigger a notification to Support Services for Spend Anomaly. If the actual or forecasted cost exceeds this value, the Lambda function will take action and send notifications.

2. SADPercentageThreshold:

   • Default Value: 300

   • Description: This parameter sets the default alarm threshold given as a percentage of the budget. It determines the percentage above which an alert will be triggered when actual or forecasted costs exceed the budget.

Deployment Flexibility for Service Provider Account Model (SPAM)

By default, the Spend Anomaly Detection (SAD) solution is deployed on dedicated payer accounts. However, for customers in the Service Provider Account Model (SPAM), it can be deployed to linked accounts using AWS StackSets. This deployment flexibility ensures that the SAD solution can adapt to different account structures and meet the needs of various customer scenarios.