Principal Software Engineer, Maxar
Jan, 2015 - Present
Currently the tech lead of a big data platform consisting of Hadoop/Yarn/Spark, Elasticsearch and Openstack using ansible and terraform. I'm involved in hardware configuration/purchasing decisions, devops and sre roles, sharing my knowledge with team members, data pipeline ETL, app development, and ci/cd deployments. I deploy, configure and maintain several Openstack environments with openstack-ansible and Ceph, monitor everything with Prometheus and Grafana, and configured alerts across infrastructure, data ETL and applications. I build data pipelines using Apache NiFi, HDFS, Apache Spark and Elasticsearch to process raw data to DeltaLake format and then indexed to Elasticsearch. I develop, deploy and maintain apps to support analysts requirements which are dockerized and deployed to Nomad clusters
Designed and developed a clientside federated search application with Django, React, Redux and Typescript. The app uses client PKI certs for authentication and aggregates results from multiple sources into single view. Setup AWS/OpenStack hosted environment to integrate our search capability with other applications for a unified user experience.
Created a data ingest pipeline and visualization that performed object detection in video. Achieved a goal of processing a ~5 min video in under a minute using multiple GPUs and parallel processing using YOLO, python and C. Visualized hundreds of thousands of detects and tracks over multiple processed videos using cesiumjs and ElasticSearch.
Worked on a project that rendered geo points as heatmaps in a 3D desktop visualization application. One of the goals was to apply data blending techniques using multiple sources of data, which was achieved by ingesting and enriching the original dataset and adding a generic geohash filtering capability that allowed for multiple external datasets to be used to dynamically alter the visualization. Enhanced the heatmap display to render by aggregated fields and metrics of the data.