ETL Pipeline for India’s Voter List

Our client had a request to compile a unified voter list encompassing every state and administrative division in India. The goal was to create a centralized, easily updatable database that could be refreshed annually.

Challenges Intsurfing Had to Deal With

• Handling over one billion records required substantial computational power and storage capacity.
• The data came in various formats: PDFs and images, often containing handwritten entries.
• Voter information was available in 22 official languages.
• Ensuring data accuracy involved cross-referencing with India Post records, which presented challenges due to differing formats and structures. 

Solutions Implemented

• Data Collection. We developed custom modules to gather voter lists from the National Voter Service Portal (NVSP) and various electoral authorities across India, handling both PDFs and images. 
• Machine Learning Integration. Collaborating with linguistic experts, we created algorithms enabling our systems to process and understand 22 different Indian languages.
• Data Extraction. We utilized Optical Character Recognition (OCR) technology to extract information from PDFs and images to handle handwritten forms. 
• Data Standardization and Transliteration. Post-extraction, we cleaned, standardized, and transliterated the data into a unified format.
• Data Validation. To ensure accuracy, we cross-referenced the standardized data with India Post records, verifying names and addresses. 
• Annual Updates. We established a system to monitor and incorporate updates from electoral authorities and India Post. 

Technologies Used

• AWS
• Nannostomus
• .NET
• Tesseract OCR

Results

• Centralized Database. Delivered a digital voter ID database containing over one billion records from 36 sources, accessible in native languages and transliterated into English.
• Comprehensive Data Fields. Provided 63 fully verified and normalized data fields: voter name, relation’s name, EPIC number, address, age, sex, year of birth, year of electoral roll revision, and polling station name. 
• Scalability and Efficiency. Established a system capable of annual updates. 

Learn more here: https://www.intsurfing.com/big-data-projects/etl-india-voter-list/