About

Work Experience

• Senior Data Engineer, SecureKloud Technlogies

  Aug, 2023 - Present

  Client: Sorcero, Designing and refining data system architectures to meet organizational needs for scalability, performance, and reliability. Building and optimizing ETL pipelines to process and transform large volumes of data efficiently. Integrating diverse data sources into unified systems to ensure data consistency and accuracy across platforms. Monitoring and tuning system performance to enhance efficiency and manage increased data loads. Maintaining high data quality through rigorous validation, cleansing, and auditing practices.

  • Developed and implemented a high-performance data pipeline processing over TB's of daily data, resulting in cost effective de-duplicated data-warehouse

  • Developed and implemented the Great Expectations Data Quality Management tool across all data sources, resulting in highly efficient and reliable data.

  • Developed and implemented a range of test cases for the PySpark framework, leading to highly efficient and bug-free logic.

  • Optimized other internal applications, increasing efficiency and reliability by 50%.

• Staff Software Engineer, moEVing Urban Technlogies

  Apr, 2022 - Aug, 20231 year 4 months

  Collaborating with stakeholders to gather and understand software requirements, translating business needs into technical specifications and functional requirements. Design software architecture and system components, creating detailed specifications and technical designs for new features or systems. Write, test, and maintain code for applications, implementing features, fixing bugs, and optimizing performance. Review peer code to ensure it meets quality standards, adheres to best practices, and integrates effectively with existing code.

  • Designed and implemented Mysql Database for Charging Hub Application.

  • Devoloped and implemented 40+ Api's for Charging Hub Application.

  • Devoloped test cases for internal inhouse application, resulting in reliable code base.

  • Developed and integrated Api's with multiple third party Api's such as Firebase for OAUTH2 and Razorpay for wallets.

• Senior Data Engineer (Senior Manager, Engineering), Reliance Jio Infocomm

  Sep, 2021 - Apr, 20227 months

  HCMP proactively discovers public, private, and hybrid cloud infrastructures, and maps interdependencies between tiers. It predicts incidents, before they occur, self-heals outliers using intelligent algorithms, and recommends resolution actions using machine intelligence.

  • Developed and implemented a high-performance streaming data pipeline transforming,processing,enriching and aggregation over 1 billion events daily.

  • Developed and implemented Log Analytics As Service (LAAS) inhouse replacement for Splunk migrated it as an SAAS application, resulting in plug and play service for any SAAS application via configuration.

  • Developed and implemented 20+ API's and test cases for in house SAAS application Hyper Cloud Management Service (HCMP).

  • Developed and implemented Activity logger framework for HCMP.

  • Managed a team of 7 engineers.

• Lead, Data Engineer, CACTUS Communications

  Nov, 2019 - Sep, 20211 year 10 months

  R Discovery is the best AI tool for academic research, Effortless academic research reading with personalized recommendations

  • Developed and implemented high-performance batch data-pipeline to transform, process and de-duplicate data from multiple data-sources, such as Pubmed, Openalex, etc

  • Developed and implemented Big-data-lake (Dyson), Currently powers R-Discovery Application.

  • Developed and Solved Industry level issue for Data-Depulication on fuzzy data without unique key from multiple data-sources, resulting highly reliable data

  • Managed a team of 3 data engineers

• Senior Data Engineer (Manager, Engineering), Reliance Jio Infocomm

  Apr, 2018 - Nov, 20191 year 7 months

  A unified portal that brings all operations related to the production application lifecycle into a single, cost-effective, and smart self-healing approach. This portal will not only handle operations but also unify log management and provide various other views.

  • Developed and implemented various real time dataframe jobs using pyspark and scala.

  • Developed and Optimized data-pipeline jobs to minimize data lags while processing streaming data from Kafka

  • Developed and implemented Log Analytics As Service (LAAS) inhouse replacement for Splunk.

  • Developed various ETL Jobs for Next Generation Operations (NGOps) to handle very large amount of daily streaming data.

  • Build Next Gen OPs(NGO) GATEWAY(Real Time DataTransformer) using kafka stream and java

  • Build Url Monitoring Application (Async), Monitor thousands of endpoint and various parameters within a second for High Availabilty and for Operations.

  • Build NGO Monitoring DashBoard, Monitors streaming data on kafka and processed data on various persistant storage for Delays/Lags if any.

  • Developed and implemented logstash configurations to ingest very high volume bof processed data in to Elasticsearch Cluster

  • Developed and implemented many complex Elasticsearch queries

• Senior Software Engineer, Mindtech iSolutions

  Apr, 2013 - Jul, 20185 years 3 months

  A unified portal that brings all operations related to the production application lifecycle into a single, cost-effective, and smart self-healing approach. This portal will not only handle operations but also unify log management and provide various other views.

  • Developed and implemented Search Engine using Elasticsearch and Logstash

  • Developed and implemented Product Categorisation on top of search engine, resulting in high relevance score within right category.

  • Elasticsearch Performance Tuning & Migration.

  • Developed and implemented various web crawling scripts to scrape multiple ecommerce websites.

  • Developed and implemented 30+ Api's for Full Fledged ecommerce Comparison Site

Projects Experience

• SSCR, Sorcero

  Aug, 2023 - Present

  To enhance medical customer insights, we developed a data lake and data warehouse solution designed for comprehensive cross-data analysis. We continuously ingest data from various sources, including PubMed, OpenAlex, and MedArchive, to construct a Scalable Source and Consumer Repository (SSCR). This system employs a Data Vault architecture where Raw Data Vault serves as the data lake, aggregating and storing raw data from diverse sources. Business Data Vault functions as the data warehouse, built on top of the Raw Data Vault, allowing the creation of multiple business-specific data vaults as needed. This design provides flexibility and scalability, enabling tailored data storage and analysis solutions based on evolving requirements.

  • Developed and implemented a high-performance data pipeline processing over TB's of daily data, resulting in cost effective de-duplicated data-warehouse

  • Developed and implemented the Great Expectations Data Quality Management tool across all data sources, resulting in highly efficient and reliable data.

  • Developed and implemented a range of test cases for the PySpark framework, leading to highly efficient and bug-free logic.

  • Implemented the Raw Data Vault as the foundational data lake, which aggregates and stores raw, unprocessed data from various sources.

  • Built the Business Data Vault on top of the Raw Data Vault, functioning as the data warehouse. This design allows for the creation of multiple business data vaults tailored to specific analytical and reporting needs.

  • Designed the architecture to support the dynamic creation of additional business data vaults as required, ensuring scalability and adaptability to changing business needs.

• Sorcero MIM, Sorcero

  Aug, 2023 - Present

  Cobbling together fragmented data takes significant time, let alone analyzing information to identify trends and insights crucial for patient care, provider education, and clinical impact monitoring. To keep pace with the explosion in medical data and deliver the best outcomes for patients, Medical Affairs teams need a modern approach for managing, collaborating on, and harnessing all of their medical data, Hence Sorcero MIM unifies and organizes your structured and unstructured medical data in a single, collaborative platform. Through the Instant Insights Engine MIMs AI-powered conversational interface teams can easily query their data using simple language, mirroring the experience of chatting with an expert colleague. Now, Medical Affairs teams can ask complex medical questions and instantly receive high-quality answers rooted in real data to guide strategy and respond to KOL needs faster than ever before.

  • Designed and implemented Data Quality Management framework using GreatExpectations

  • Implemented multiple test cases for code reliability

  • Developed and implemented transformation hub which can transform and ingest any format of customer data within data-warehouse

  • Performed Data Cleaning and Mirgrations to move application from Postgresql to alloyDB

  • Developed and implemented Spark job for syncing data from Postgresql/AlloyDB to Big-Query for Analytics and Customer Dashboards.

  • Developed and integrated multiple API's.

• moEVing Charge, moEVing Urban Technologies

  Apr, 2022 - Aug, 20231 year 4 months

  MoEVing Charge is a service that manages charging hubs and infrastructure for electric vehicles. It provides an automated application for users to access charging stations, with features like login options, wallets, payment integrations, and real-time data. Users can start and stop charging and perform other activities through the app.

  • Designed and implemented Mysql Database for Charging Hub Application.

  • Devoloped and implemented 40+ Api's for Charging Hub Application.

  • Devoloped test cases for internal inhouse application, resulting in reliable code base.

  • Developed and integrated Api's with multiple third party Api's such as Firebase for OAUTH2 and Razorpay for wallets.

• Log Analytics As Service (LAAS), Reliance Jio

  Sep, 2021 - Apr, 20227 months

  Log Analytics as a Service (LAAS) seamlessly integrates with existing products and offers features such as On Demand Log Analytics, Log Search, Log Tail, a Rule Engine for alerts, and metrics tracking. On Demand Log Analytics allows for quick access to log data without manual collection and analysis. Log Search enables easy querying and filtering, while Log Tail monitors real-time logs. The Rule Engine generates alerts based on specific log events, and metrics tracking allows KPI monitoring. LAAS provides a comprehensive log management solution that integrates into workflows, providing insights and alerts for system and application optimization.

  • Developed and implemented a robust solution designed to handle billions of daily streaming events, efficiently managing tasks such as cleaning, enriching, processing, and ingesting the data.

  • Developed and implemented a solution that integrates seamlessly with existing products, enhancing overall log management. This integration enables rapid access to log data without manual collection or analysis and supports easy querying and filtering.

  • Developed and implemented real-time log monitoring to provide immediate insights and established an alert system that triggers notifications based on specific log events, facilitating proactive monitoring.

  • Developed and implemented KPI tracking capabilities to evaluate system and application performance.

  • Delivered a comprehensive log management framework that integrates seamlessly into existing workflows, providing actionable insights and alerts to enhance the optimization of systems and applications.

• User Activity Tracking System, Reliance Jio

  Sep, 2021 - Apr, 20227 months

  Activity Logging feature tracks and records all user-based activities across the product, providing valuable insights into user behavior, usage patterns, and optimization opportunities. Activity logs can also be used for security and compliance purposes, helping to track user behavior and investigate any suspicious activity. Overall, our feature provides a comprehensive record of user activity for insights, optimization, and security.

  • Developed and built a scalable system capable of recording and analyzing large volumes of user activity data

  • Enhanced product performance and security through actionable insights and proactive monitoring.

  • Streamlined integration processes with existing systems, improving data accessibility and usability.

  • Created analytical tools to visualize and interpret user activity data, identifying trends and performance opportunities.

• R-Discovery Dyson, CACTUS Communications

  Nov, 2019 - Sep, 20211 year 10 months

  We have created a singular corpus of academic information that spans across various data sources, including open access publications, licensed content from publishers, and publicly available information. Our method of collecting data varies from source to source, and we are committed to continuously expanding and improving the corpus to ensure it remains accurate and up-to-date. By providing a comprehensive source of data, we aim to facilitate research and knowledge dissemination across the academic community. R-Discovery(Available on all platforms).

  • Developed and implemented methods for collecting and aggregating data from various sources, ensuring a unified and accurate corpus.

  • Established processes for the ongoing expansion and refinement of the corpus, maintaining its accuracy and relevance in the rapidly evolving academic landscape.

  • Ensured that R-Discovery is accessible across all major platforms, facilitating broad access and use by the academic community.

  • Provided a valuable resource that supports research and knowledge dissemination, enhancing the accessibility of academic content.

  • Designed and executed data collection strategies tailored to different types of sources, including automated data scraping and API integration.

  • Implemented data validation and cleaning processes to maintain the quality and reliability of the corpus.

  • Successfully developed a singular, comprehensive corpus of academic information, enhancing research capabilities and knowledge sharing within the academic community.

  • Developed and solved industry level issue for Data-Depulication on fuzzy data without unique key from multiple data-sources, resulting highly reliable data

  • Improved data accuracy and accessibility through continuous updates and expansion of the corpus.

• Next Generation Operations, Reliance Jio

  Apr, 2018 - Nov, 20191 year 7 months

  The goal of this project is to create a unified portal that brings all operations related to the production application lifecycle into a single, cost-effective, and smart self-healing approach. This portal will not only handle operations but also unify log management and provide various other views. By consolidating all operations into a single portal, we can streamline and simplify the application lifecycle, reducing the time and effort required for operational tasks. The self-healing approach will enable the portal to automatically detect and resolve issues, minimizing the need for manual intervention and reducing the risk of downtime. In addition to operational tasks, the portal will also provide log management capabilities, allowing users to easily search, view, and analyze logs from a centralized location. Other views, such as performance metrics and security logs, will also be available, providing users with a comprehensive view of the application and its environment. Overall, this project will significantly enhance the operational efficiency of the production application lifecycle, providing a centralized, cost-effective, and smart self-healing approach that simplifies operations, reduces downtime, and improves overall application performance.

  • Developed and implemented various real time dataframe jobs using pyspark and scala.

  • Developed and Optimized data-pipeline jobs to minimize data lags while processing streaming data from Kafka

  • Developed and Integrated 50+ Api's

  • Developed and Integrated comprehensive log management features into the portal, enabling users to search, view, and analyze logs from a centralized location.

  • Streamlined application lifecycle operations, significantly reducing the time and effort required for operational tasks through centralized management.

  • Conducted thorough testing and validation to ensure the portal met operational and performance standards.

  • Enhanced user experience with centralized log management, performance metrics, and security views, providing a comprehensive overview of application health and performance.

• API Monitoring and SSL Certificate Management, Reliance Jio

  Apr, 2018 - Nov, 20191 year 7 months

  To ensure that all APIs were functioning properly, both internally and externally, we developed a monitoring system that could check whether they were UP or Down. This system was designed to run periodic checks on each API and generate alerts in the event of a failure or downtime. In addition to monitoring API availability, we also developed a system to check for expiring SSL certificates. This system would periodically scan all certificates associated with the APIs and generate alerts when a certificate was approaching its expiration date. This allowed us to proactively renew certificates before they expired and avoid any potential disruptions to API services. Overall, the monitoring system has been effective in ensuring that all APIs are functioning properly and that any potential issues are detected and addressed in a timely manner. By proactively monitoring and managing the APIs and associated SSL certificates, we have been able to maintain high levels of reliability and availability for the company's services.

  • Developed a system to periodically check the status of APIs, ensuring they are UP or DOWN and generating alerts in case of failures or downtime.

  • Implemented a system to scan and monitor SSL certificates for expiration, generating alerts as certificates approach their expiration dates.

  • Enabled timely renewal of SSL certificates to prevent disruptions in API services.

  • Enhanced the reliability and availability of the company’s services by addressing issues promptly through proactive monitoring.

  • Ensured all APIs functioned properly with timely detection and resolution of potential issues.

• Next Gen OPs(NGO) GATEWAY(Real Time DataTransformer), Reliance Jio

  Apr, 2018 - Nov, 20191 year 7 months

  In this project, various kinds of data such as application logs, system logs, and script logs are collected and sent to Kafka in an unstructured format along with metadata for further processing. However, in order to effectively use this data for analysis and other processes, it was necessary to transform it into a generic universal format or map it to a universal schema that could be used by various processes. To achieve this, we developed a data transformation process that would convert the unstructured data into a structured format that could be used by different processes. This involved mapping the data to a universal schema that defined the structure and format of the data. Once the data was transformed into a universal format, it could be easily processed and analyzed by various applications and systems, making it more accessible and useful for business intelligence and other purposes. Overall, this project has been critical in enabling the efficient processing and analysis of large volumes of data from various sources by converting it into a universal format that can be easily used by various processes.

  • Collected various types of data, including application logs, system logs, and script logs, and sent them to Kafka in an unstructured format with metadata.

  • Developed a data transformation process to convert unstructured data into a structured format using a universal schema.

  • Enabled effective use of transformed data across different applications and systems, facilitating easier processing and analysis.

  • Improved accessibility and utility of data for business intelligence and other purposes by standardizing the data format.

  • Critical for the efficient processing and analysis of large volumes of data from multiple sources by converting it into a universally usable format.

• LogAnalyzer, Reliance Jio

  Apr, 2018 - Nov, 20191 year 7 months

  The company was facing the need for cost-cutting measures and sought to replace its current use of Splunk with an in-house technology. In response, we developed LogAnalyzer, a tool designed to provide the same functionality as Splunk, but with additional features to meet the company's specific needs. LogAnalyzer includes advanced searching, aggregating, bifurcation, and search capabilities with selections on date and time, among other features. It also includes a Live Log Tail feature, allowing users to view logs in real-time without the need to log into the server, saving time and improving efficiency. Overall, LogAnalyzer has proven to be a valuable replacement for Splunk, providing the necessary functionality while also being cost-effective for the company. It has streamlined the logging process and improved the ability to monitor and analyze system logs, contributing to the overall success of the company.

  • Developed LogAnalyzer to replace Splunk, addressing the company's need for cost-cutting while providing equivalent and enhanced functionality.

  • Includes sophisticated searching, aggregating, bifurcation, and date/time-based search capabilities.

  • Features real-time log viewing without server login, saving time and boosting efficiency.

  • Simplified and improved the monitoring and analysis of system logs.

  • Proven to be an effective and cost-efficient alternative to Splunk, contributing to the company’s overall success.

• Migration and Performance Optimization of Search Platform, Mindtech iSolutions

  Oct, 2017 - Apr, 20186 months

  As part of our efforts to improve performance and support, we recently migrated our older search platform from Elasticsearch v2 to v7. I played a key role in designing the algorithm and building the pipeline for this migration. Following the migration, we observed significant improvements in performance, with the new platform delivering faster and more accurate search results. In particular, our older search CTR (unique click to unique query by user) increased from around 32 to 35% to a consistent 42 to 44%. This improvement was achieved through end-to-end tuning of the search platform, including search algorithms, relevance scoring, and user interface design. As a result of these improvements, we have seen a boost in our GMV/V by 2 to 3%. This increase in revenue has been a significant achievement for the company. Overall, tuning and migration to Elasticsearch v7 has been a success, enabling us to provide faster, more accurate search results and drive greater revenue for our business.

  • Designed and implemented the algorithm, complex queries and migration pipeline for transitioning to Elasticsearch v7, ensuring a smooth and efficient upgrade process.

  • Increased the click-through rate (CTR) from approximately 32-35% to a consistent 42-44%, through comprehensive tuning of search algorithms, relevance scoring, and user interface design.

  • Contributed to a 2-3% boost in Gross Merchandise Volume (GMV) by enhancing search effectiveness and driving greater user engagement and conversion.

  • Conducted end-to-end tuning of the search platform, including fine-tuning search algorithms and relevance scoring to align with user needs and improve search accuracy.

  • Monitored post-migration performance and made iterative improvements based on user feedback and performance metrics.

• Enhanced Search Experience Module (Product Categorisation), Mindtech iSolutions

  Sep, 2017 - Apr, 20187 months

  The primary objective of this module was to enhance the user search experience by ensuring that search queries landed in the relevant category and enabling auto category tagging for new products. This module facilitated the manipulation of search results based on business logic while still maintaining search relevance. Since the implementation of this module, we have observed significant improvements in several key metrics. For instance, it has led to an increase in the GMV/V and user time on the app for search queries. Additionally, the number of user scrolls has increased from 5 to 9 per user, indicating that more products are being viewed by users. To track these improvements, we have set up monitoring dashboards that provide real-time updates on key metrics. Overall, this module has been instrumental in improving the user search experience and driving business growth by increasing user engagement and revenue. We continue to work on optimizing the module to further enhance its performance and contribute to the success of our platform

  • Designed and implemented a system to ensure search queries were accurately categorized and new products were automatically tagged with relevant categories, enhancing the search experience and relevance.

  • Incorporated business logic into search result manipulation to align search outcomes with business objectives while maintaining search relevance.

  • Established monitoring dashboards to track real-time updates on key metrics such as GMV/V, user engagement, and search behavior.

  • Boosted user time on the app and scroll depth, demonstrating improved product discovery and interaction.

  • Created and maintained real-time monitoring dashboards to track performance metrics and guide ongoing optimizations.

  • Observed a rise in the number of user scrolls from 5 to 9 per user, indicating greater product visibility and interaction.

• TheDealapp, Mindtech iSolutions

  Jul, 2016 - Apr, 20181 year 9 months

  The project was focused on building a comprehensive price comparison website for customers to find the best deals across various e-commerce sites. The website was built using a technology stack that included Django, HTML, and PHP, with databases created using MongoDB and MySQL. To gather data from different e-commerce sites, web crawling techniques were used, and search functionality was implemented using Elasticsearch. The ultimate goal was to provide users with a comprehensive comparison of similar products on different websites, along with price graphs and other relevant information to help them make informed purchasing decisions. (thedealapp.in)

  • Utilized Django for backend development, HTML for frontend design, and PHP for server-side scripting. Managed databases using MongoDB and MySQL to handle diverse data storage needs.

  • Implemented web crawling techniques to gather product data from multiple e-commerce sites, ensuring a broad and accurate dataset for comparison.

  • Integrated Elasticsearch to enable efficient and scalable search functionality, allowing users to quickly find and compare products across different sites.

  • Developed features such as price graphs and detailed product comparisons to provide users with relevant information for making informed purchasing decisions.

Skills

Education