They had started developing their SaaS with an ad-hoc data lake system. It was difficult to extend, scale and integrate with new API providers. They were a data-intensive application leveraging databases that were not optimized at all for data consumption by the SaaS, taking seconds for simple requests, impacting user experience.

Re-design and implement a robust data system that can ingest, transform and store massive amounts of data from different sources like Reddit, Twitter and Instagram. The system should support batch processing to efficiently compute and load data into curated layer for ML processing and into the databases the processed data for consumption by the SaaS backend.

How we helped:

We helped them build a solid data system that was scalable by designing and creating a server-less ingestion system that could scale on-demand. Our core contribution was around data consumption and ML batch pipeline processes.