AWS clustering platform with elastic warehousing
Multi-tenant clustering compute on ECS Fargate, paired with Redshift Serverless for downstream analytics.
The problem
An enterprise needed to run large-scale clustering jobs on customer data, then expose the results to analyst teams via SQL. They had outgrown a single Redshift cluster but didn't have the headcount to run a Kubernetes platform.
Why standard tooling didn't fit
Kubernetes is overkill when the workload is task-shaped, not service-shaped. A provisioned Redshift cluster sits idle 80% of the time. Lambda can't run multi-hour clustering jobs. We needed managed infrastructure that scales with workload, not constant capacity.
What we built
A platform where the workload decides how much compute gets provisioned, not a capacity plan. Clustering work is task-shaped, so it runs as isolated, short-lived units that start when there is work and stop when there isn't. The analytics layer behaves the same way and scales to the queries analysts actually run, instead of holding a cluster open in case someone runs one. Shared storage sits underneath both, and older data ages into cheaper tiers on its own. The whole environment is described as code, so standing up a new analytics workspace is a configuration change. That is what lets a small data team operate the platform without a dedicated infrastructure function behind it.
Outcomes
- Compute cost down ~60% vs. always-on cluster baseline.
- Zero infrastructure operations burden on the data team.
- New analytics workspaces stood up in minutes, not days.