Software Development Engineer II, Amazon EMR Serverless (Amazon Web Services)
Apr, 2021 - Present
Led the design, implementation, and release of custom EMR Serverless images, enabling customers to install/configure packages optimized for their workloads and integrate with current CI/CD best practices. +100 customers successfully submitted +5000 jobs within the first two months of launch.
Contributed towards launching AWS EMR Serverless, a new AWS EMR offering which enables clients to setup an EMR cluster 75% faster. This is achieved by avoiding the need to setup, maintain, or configure EC2 hosts or EKS clusters.
Developed one of EMR Serverless control plane components with >99% availability and >15 TPS at launch that provides nodes to client's distributed application (i.e. Spark and Hive)
Improved clients' applications' start-time by >85% by 1/ adding an application configuration to clients that proactively created nodes before application submission and 2/ evaluated various optimization strategies to reduce Docker images pull times by 90%.
Designed a multi-tenant service that enables multiple clients to run applications while ensuring network and compute isolation between clients.
Led the implementation of metrics and creation of monitoring dashboards in parts of EMR Serverless to improve debugging and proactively identify issues.