Big Data Pipeline for Data Marketplace

About the customer: DDMR, an American data marketplace for clickstream data.

Business challenge: The company needed to manage and process large-scale data effectively and required high-quality engineering support to bring its product to life. Intelliarts was engaged to redesign, develop, and implement robust systems for data transformation and augmentation.

Solution: Our team developed a platform-agnostic data collection tool with browsers as a data source.  We recommended the customer split data into cold and hot storage, which helped to access the necessary data easily and without delays and saved costs on data storage. Also, we redesigned the data pipeline and transferred data processing services to more advanced technology. Finally, our engineers improved the infrastructure by increasing reliability and reducing human factors; adding automatic notifications and introducing trend monitoring; and moving to multi-region infrastructure to speed up the services.

Business outcome: Intelliarts helped the company build an end-to-end data pipeline, helped to process very large data sets, and transformed them into valuable insights.

Results:

• our customer multiplied its annual revenue a few times and won a base of loyal customers
• also, the customer can process hundreds of terabytes of data per day
• our team secured the business in terms of data leakage.