About

Work Experience

• Big Data Engineer, 

  January 6, 2020 - March 6, 2023

  As a Big Data Engineer at eBay, I was responsible for data generation and production pipelines in the selling domain, including sellers' listed products, product information, stores, and performance metrics for listing tools.

  • Developed ETL pipelines using Spark SQL and KSH, and optimized performance.

  • Increased the stability of the feed pipeline sent to SellerHub, reducing failures by 33% and eliminating the need for manual intervention for common issues, while supporting more seller IDs without altering field sizes.

  • Enhanced the coverage of structured products among listed items, using a pipeline combined with NLP/TF-IDF to match sellers' listings with structured product information, raising sneaker category coverage from 17% to 64%.

  • Developed an integrated data product for sellers during a company hackathon, leveraging maintained data to provide a product selection tool addressing the 'what to sell' need.

  • Created internal tools to boost data engineering efficiency, including AUT, a UC4 enhancement tool.

  • Collaborated with the web scraping team to deliver pricing guidance data services for sellers.

  • Maintained ETL processes for Spark, Flink, and Cassandra.

• Java Engineer, 

  Sep, 2018 - Jul, 201910 months

  We made a social app called onecan, which provides chat services to ones have social anxiety. I was the server side engineer which provides the solid apis to clients. And did the technical research on the data migration tools and distributed databases.

  • Refactored and maintained the whole auditing report management system. Features in the management system are authentication, access control, user content filter, user report, topic operation, high quality content recommendation, user feedback, etc

  • Migrated the big table from Mysql to HBase, about 0.6 billion records. Planed the whole data migration process, include rowkey design, performance testing, edge case testing, new data double writing, old data migration

  • Setup HBase cluster. Customized the Spring Data Hadoop framework, which provides some common logic, for example HBase client reconnection and CRUD apis which is easy to use. Designed rowkey to fit the query requirements. Created table with region pre split, to prevent the region split which may bring performance issues.

  • Maintened Instant Messaging system based on Protocol Buffer

  • Installed the CI/CD deployment chain based on Jenkins, Gitlab and Supervisor

  • Researched on pt-online-schema-change, and use it as our online schema change tool. The native Mysql DDl might produce the lagging replication when the load is high. And pt-osc brought less load and traffic.

  • Wrote groovy scripts to generate POJOs from Mysql table, automate repetitive work.

  • Wrote bash scripts to simplify the deploy and server maintenance. Such as scheduling backup database, clean up old logs, scheduling update HTTPS certification files, monitor errors in server logs.

• Software Engineer, 

  Jul, 2016 - Sep, 20182 years 2 months

  Zeonic was a startup based on public traffic, such as bike, bus, metro info service. Recently we were focused on mobile payment. If you use our app and show the qr code, you could get on the taizhou's bus with no cash nor transportation card.

  • Designed the qr code payment system, which can work offline because the bus may has no internet connection. To support the offline job some encryptions was used, for example, TDES, RSA.

  • Developed the algorithm for locating running bus, and calculate the time to arrive the stop

  • Provided the Oauth2 service to the partners. And use the RFC7662 Token Introspection to fit the business logic

  • Developed the NFC adapter which can operate the transportation card(PBOC), include determine which transportation card, check and loading card balance

  • Optimized the android application with smooth operation and nature animation

  • Built Java project from zero, it's SpringBoot based, date access use the JPA, and Gradle to build the project. Try to introduce the ProtoBuf to reduce the data exchanged

  • Wrote the unit tests, installed the ci/cd based on Jenkins, installed Bitbucket, wrote some backup scripts

• Android Developer and PHP Developer, 

  Jul, 2015 - Mar, 20168 months

  I joined Clink Finance at very early stage. We helped our customers which are also startupers to coding their dreams. I played the role as an android developer, a php developer and a general technical support in our team.

  • Developed hybird app. Union the native app login and webapp login

  • Developed js bridge to send message between android and js

  • Developed the order system by finite state machine

  • Developed distribution cluster to split the platform-server and client-server which use the same code base. Use http-digest and md5 to validate certification to each other in the cluster.

• Full Stack Engineer, 

  Sep, 2014 - Jul, 201510 months

  My second job was working for a startup called Geex Finance. It was an Internet Finance company with series A term raised by Sequoia Capital. We helped merchant to build their website and apps, sold our loan product by the way.

  • Built loan app in android pad

  • Built loan app server side

  • Built wechat based web app

  • Fixed IE compatibility with web engineer when it is at top priority.

  • Built non-block order process system, use Crontab and Redis.

• Java engineer, 

  Jul, 2012 - Sep, 20142 years 2 months

  My first job was coding for enterprise software. One project was for NanJing Airport which using algorithm to deal daily shift schedule and alarm delay flights.

  • Built Spring based JavaEE program and Java pc client

  • Wrote Oracle procedure and view to modeling the data, and export the report as pdf

  • Migrated Groovy and Java code at one project

  • Debugged in ios code

Skills

Education

Interests