The situation
This case study is about an innovative, young startup that provides insights and collaboration tools to life sciences organizations, to solve the life science insights gap. Their insights management platform provides their customers a better way to understand the diseases and conditions they study, and data-driven insights that can change the course of clinical research. They empower their clients to make smarter decisions faster by enabling them to more efficiently gather, integrate, and analyze insights from data across multiple sources.
Complication
The company decided to explore the use of Large Language Models (LLMs) to enhance the analytics capabilities for its clients. Addressing this required the development of complex data pipelines. Troubleshooting these data pipelines was a manual, error-prone, resource-intensive process. As data volumes increased by 5-20x with new customers, the time to onboard them increased by 100% due to limitations of the data infrastructure. This also impacted business agility with dashboards being refreshed only once every quarter.
The Solution
After a comprehensive evaluation, the company decided to use Dataworkz to make its data ready for LLMs and drive unification of its AI and BI initiatives for improved analytics capabilities. The solution automates data preparation and transformation to support complex data pipelines. This involves taking raw prescription and medical claims data stored in Databricks Delta Lake tables, running it through a 280-step transformation workflow to combine and aggregate data from multiple external sources including geospatial parameters, and creating downstream tables in Snowflake that are connected to Tableau dashboards.
The Impact
$75k savings per quarter
300+ steps of Transformation automated
2 days vs 6 weeks to process new data source