My Hidden Infrastructure Costs Were Killing My Budget

Hey everyone, Jules Martin here, back at it from agntmax.com. Hope you’re all crushing it out there. Today, I want to talk about something that’s been nagging at me lately, something I’ve seen pop up in more conversations and project post-mortems than I care to admit: the invisible drag of unoptimized infrastructure costs. We all know we need to build fast, scale quick, and deliver features yesterday. But often, in that mad dash, we leave behind a trail of forgotten resources, oversized instances, and services running on autopilot, racking up bills we barely glance at until the quarterly budget review hits like a ton of bricks.

So, for this piece, I’m diving headfirst into cost optimization, but with a very specific, timely angle: how to stop bleeding money on “always-on” resources that should be “on-demand” or “event-driven.” It’s 2026, people. The days of set-and-forget server provisioning are long gone. If your cloud bill still looks like a phone book, it’s time for an intervention.

The Silent Killer: Always-On When It Should Be On-Demand

Let’s be real. When we’re under pressure to get a new agent-facing tool or a customer service enhancement out the door, cost usually takes a backseat to functionality and speed. We provision an EC2 instance that’s “big enough,” maybe even “a bit bigger just in case.” We spin up a database with provisioned IOPS that could handle the entire internet, only for it to sit mostly idle during off-peak hours. We forget to set up proper scaling policies, or we just leave things running 24/7 because, well, it’s easier than thinking about it.

I saw this firsthand a few months ago with a client’s new internal analytics dashboard. The team, bless their hearts, had built a fantastic system that gave agents real-time insights into customer interactions. It was a huge win for performance. But when the first full cloud bill came in, the CFO nearly had a heart attack. They had provisioned a beefy EKS cluster, a couple of high-end RDS instances, and a whole slew of Lambda functions with generous memory allocations, all running non-stop. The kicker? The dashboard was primarily used by agents during business hours, 9 AM to 5 PM, Monday to Friday. Outside of that, it was a ghost town.

They were paying for enterprise-grade capacity for a system that was effectively idle for 70% of the week. That’s like buying a Formula 1 car to drive to the grocery store once a week.

Identify the Culprits: Where Your Money Is Really Going

Before you can fix anything, you need to know what’s broken. Most cloud providers offer tools to help you visualize your spending, and you absolutely need to use them. AWS Cost Explorer, Azure Cost Management, Google Cloud Billing reports – these aren’t just for finance. They’re your first line of defense.

The Usual Suspects

• Compute Instances (EC2, VMs): These are often the biggest offenders. Are they oversized? Are they running when they don’t need to be? Are you using the right instance family for your workload?
• Databases (RDS, Azure SQL, Cloud SQL): Similar to compute, databases can be over-provisioned for IOPS, CPU, or memory. Many offer serverless options now that scale down to zero or near-zero cost when idle.
• Storage (EBS volumes, unattached disks): Ever launched an instance, terminated it, but left the associated storage volume hanging around? It happens more than you think.
• Networking (Data transfer, NAT Gateways): Data transfer costs can sneak up on you, especially cross-region. NAT Gateways also have an hourly charge, even if they’re doing nothing.
• Underutilized Services: Are you paying for a dedicated Redis cache that only gets a few hits a day? A managed Kafka cluster for a trickle of messages?

My client from the analytics dashboard story started by looking at their AWS Cost Explorer. The biggest line items were, predictably, EC2 and RDS. They also found a couple of EBS volumes attached to terminated instances and a NAT Gateway in a VPC that was no longer actively used for production traffic. Small things, but they add up.

Strategies for Turning Always-On into On-Demand (or Off-Peak)

Okay, so you’ve identified the areas where you’re overspending. Now for the fun part: fixing it. The goal isn’t just to save money, but to build a more resilient, efficient system that only consumes resources when it genuinely needs them.

1. Schedule Instance Start/Stop

This is probably the lowest-hanging fruit for many applications. If your internal tools or staging environments are only used during business hours, there’s no reason for them to be running 24/7. Most cloud providers offer native ways to schedule instance power cycles, or you can roll your own with serverless functions.

Practical Example: AWS EC2 Scheduler with Lambda

You can create a simple Lambda function triggered by CloudWatch Events (CRON expressions) to stop and start EC2 instances based on tags. Here’s a simplified version of the Lambda function code (Python):

import boto3

def lambda_handler(event, context):
 ec2 = boto3.client('ec2')
 
 # Define tags to identify instances for stopping/starting
 # For example, 'Schedule': 'business-hours'
 
 # Get all running instances with the 'Schedule' tag set to 'business-hours'
 running_instances = ec2.describe_instances(
 Filters=[
 {'Name': 'instance-state-name', 'Values': ['running']},
 {'Name': 'tag:Schedule', 'Values': ['business-hours']}
 ]
 )
 
 stop_instance_ids = []
 for reservation in running_instances['Reservations']:
 for instance in reservation['Instances']:
 stop_instance_ids.append(instance['InstanceId'])
 
 if stop_instance_ids:
 print(f"Stopping instances: {stop_instance_ids}")
 ec2.stop_instances(InstanceIds=stop_instance_ids)
 else:
 print("No instances to stop.")
 
 # --- Similar logic for starting instances at a different time ---
 # You'd have another Lambda/CloudWatch Event for starting,
 # or combine logic with a 'start' tag.
 
 return {
 'statusCode': 200,
 'body': 'EC2 instance scheduling complete.'
 }

You’d set up two CloudWatch Event rules: one to trigger this Lambda at, say, 6 PM UTC to stop instances, and another at 7 AM UTC to start them. This alone can cut compute costs by over 70% for those specific resources.

2. Embrace Serverless and Container Orchestration

If your workload is truly sporadic or event-driven, serverless is your best friend. AWS Lambda, Azure Functions, Google Cloud Functions – they scale down to zero when not in use, meaning you only pay for compute when your code is actually running. This is a massive shift from the “always-on” paradigm.

For more complex applications that still need persistent services but have fluctuating demand, container orchestration platforms like Kubernetes (EKS, AKS, GKE) combined with intelligent autoscaling are powerful. Horizontal Pod Autoscalers (HPA) can scale your application pods up and down based on CPU utilization or custom metrics. Cluster Autoscalers can even add or remove nodes from your cluster as demand changes.

My client refactored parts of their analytics dashboard to use Lambda for generating certain reports that were only requested a few times a day. Instead of a dedicated EC2 instance running a cron job, a Lambda function was triggered by an S3 event (new data uploaded) or an API Gateway request. The cost savings were immediate and significant.

3. Right-Size Your Databases with Serverless or Auto-Scaling

Databases are often a tricky one because data persistence is critical. However, many modern databases offer serverless or auto-scaling options that weren’t widely available a few years ago.

• AWS Aurora Serverless v2: This is a significant shift. It scales capacity based on actual usage, from fractions of an ACU (Aurora Capacity Unit) up to hundreds, and you only pay for what you use. No more provisioning for peak capacity when most of the time you’re operating at base load.
• Azure SQL Database Serverless: Similar to Aurora Serverless, it auto-scales compute and pauses when inactive, saving significant costs for intermittent workloads.
• DynamoDB On-Demand: For NoSQL workloads, DynamoDB’s on-demand capacity mode means you pay per request, without having to provision read/write capacity units. Perfect for unpredictable traffic patterns.

The analytics dashboard originally used a large RDS PostgreSQL instance with provisioned IOPS. After migration to Aurora Serverless v2, their database costs dropped by almost 60%, simply because it was no longer running at full tilt during off-hours.

4. Clean Up Unattached Storage and Snapshots

This sounds basic, but it’s a constant source of wasted money. When you terminate an EC2 instance, its associated EBS volume isn’t always deleted by default, especially if it was a non-root volume. Same goes for snapshots – they accumulate quickly and can become expensive.

Practical Example: Finding and Deleting Unattached EBS Volumes (AWS CLI)

You can use the AWS CLI to find unattached volumes and delete them. This is a common cleanup task.

# List all unattached volumes
aws ec2 describe-volumes --filters Name=status,Values=available --query 'Volumes[*].[VolumeId,Size,CreateTime]' --output table

# To delete a specific volume (BE CAREFUL, THIS IS IRREVERSIBLE)
# Replace 'vol-xxxxxxxxxxxxxxxxx' with the actual volume ID
# aws ec2 delete-volume --volume-id vol-xxxxxxxxxxxxxxxxx

Automate this with a scheduled Lambda function if you frequently spin up and tear down environments. The client found several terabytes of old, unattached EBS volumes and hundreds of outdated snapshots. Deleting them shaved a few hundred dollars off their monthly bill – not massive, but every little bit counts.

5. Optimize Network Costs

NAT Gateways are fantastic for allowing instances in private subnets to access the internet, but they incur an hourly charge and a data processing charge. If you have multiple NAT Gateways in different availability zones, but only one is actively used, you’re paying for redundant ones.

• Consolidate NAT Gateways: If your architecture allows, consolidate to fewer NAT Gateways.
• VPC Endpoints: For accessing AWS services like S3 or DynamoDB from within your VPC, use VPC Endpoints. Traffic flows privately within the AWS network, avoiding NAT Gateway costs and offering better security.

We found the client had a NAT Gateway in every AZ, even though their primary application only ran in two. They were able to consolidate and save a bit there, and then later implemented VPC Endpoints for S3 access, which cut down on data processing costs through the NAT Gateway.

Actionable Takeaways for Your Next Sprint

This isn’t just about cutting costs; it’s about building smarter, more efficient systems that are inherently cost-aware. Here’s what you can start doing today:

1. Audit Your Cloud Bill Regularly: Make it a habit. Use your cloud provider’s cost management tools. Don’t just hand it to finance. Understand where every dollar is going.
2. Tag Everything: This is non-negotiable. Tag resources by project, owner, environment (dev, staging, prod), and whether they can be scheduled for shutdown. This makes identification and automation infinitely easier.
3. Prioritize Scheduling for Non-Prod Environments: Staging, dev, QA environments are prime candidates for scheduled shutdowns outside business hours. This is usually the easiest and fastest win.
4. Evaluate Serverless for New Workloads: If you’re building something new, especially event-driven microservices or background tasks, always consider serverless first.
5. Revisit Database Choices: If you have databases running 24/7 with highly variable loads, investigate serverless or auto-scaling options for your specific database technology.
6. Automate Cleanup: Implement automated scripts or serverless functions to identify and delete unattached storage volumes, old snapshots, and other orphaned resources.
7. Educate Your Team: Foster a culture of cost awareness. Make sure developers understand the cost implications of their provisioning choices. It’s not just an ops problem anymore.

Stopping the bleed from “always-on” resources isn’t a one-time fix; it’s an ongoing discipline. But by making these shifts, you’ll not only save your company a significant amount of money but also build a more agile, resilient, and future-proof infrastructure. And frankly, that just makes you a better agent in the tech game.

That’s it for me this time. Keep building smart, and I’ll catch you in the next one!

🕒 Last updated: March 16, 2026  ·  Originally published: March 12, 2026