CASE STUDY

AUDITING A DATA ECOSYSTEM AND RECOMMENDING BEST PRACTICES

A UK charity that focuses on research and support for Parkinson’s disease. Funded by different constituents including individuals and organisations, their primary purpose is to support those affected by Parkinson’s. Their main goal is to find a cure for Parkinson’s, and they intend to do this by supporting research.

FIND OUT MORE

The Challenge

The core aim was to align a data strategy and recommend best practices around improvements in tool usage while upskilling the in house team in these technologies. 

To combat rising costs with their current data solution, the client needed an audit of their data ecosystem. The critical area of focus was Snowflake, Matillion, Raiser’s Edge and Tableau. We also recommended the creation of a single version of truth for constituents who have multiple points of interaction with the organisation.

The technologies being evaluated did not have defined success criteria or a holistic view, and there was a need to define how the technology would be measured as a true fit for the business. There was also a lack of expertise in these technologies within the business, as well as a lack of governance around the usage of tools.  Particular areas of inefficiency were highlighted resulting in rising costs for the data estate.

 

The Solution

Over 14 days, BI:PROCSI undertook a rapid, complete data discovery and engineering engagement to provide the client with a set of recommendations to enable them to alleviate the issues by fully understanding them and effectively decrease the data usage costs.

The first step in the solution process was to identify the main offending queries and use these as benchmarks in a BI:PROCSI led project with a focus on tooling evaluation and data migration.

Parallel to this BI:PROCSI ran a full and detailed Tableau audit at every level from users to content and security with the intention of capturing the inefficiencies in platform usage. 

The BI & reporting estate was scrutinised and evaluated to identify offending design flaws and improvement opportunities.

Additionally, the BI:PROCSI developed a core data migration plan to move data from Raiser’s Edge, a legacy platform into the clients newly purchased Snowflake instance. This approach was to remove duplication of efforts, ensure a single version of the truth and increase ROI on the technology stack. 

Post-migration of the Raiser’s Edge data feeds, BI:PROCSI developed a single version of truth for constituents across multiple touchpoints.  

The final element for the BI:PROCSI engagement was a full GDPR and PII audit to identify gaps, ensure compliance and if necessary, suggest the best way forward. 

The Result

BI:PROCSI demonstrated that Matillion usage was not optimised in the most efficient way and the Matillion database had the highest unnecessary cost by a large margin. 

Matillion was generating a high volume of queries in Snowflake hitting an average of 100k queries per month and costing an unnecessarily high portion of Snowflake credits, Snowflakes cost metric. Certain jobs within the Matillion platform had been set up incorrectly and performing menial tasks with no business benefit while impacting resources and cost. 

BI:PROCSI highlighted that an API component was deprecated and recommended audit, documentation and recreation of relevant components. Additionally, BI:PROCSI also discovered that status log components were being used inefficiently creating a large volume of interactions and again unnecessarily using resources and dramatically increasing technical costs. 

Raiser’s Edge, a legacy system, was being used for storing constituent details and their interactions with the organisation and the BI:PROCSI-led Data migration plan was implemented removing the need for legacy systems. 

Matillion orchestration and transformation jobs were created and the core datasets were migrated from Raiser’s Edge to Snowflake. This was a huge win in terms of showcasing best practice when using Matillion and also the migration of key data sources which enabled movement from a legacy platform, reducing technical debt and cognitive overhead.

The primary value-add in this engagement was the creation of a single view of truth for constituents. Prior to its implementation, the organisation had to look up various sources of data in order to identify whether a constituent was active in the last 24 months. BI:PROCSI developed and implemented a transformation job in Snowflake which combined and aggregated data from multiple sources – to create a single view. This highlighted the active constituents across various touchpoints and helped understand the first point of initiation of interaction of a customer with the organisation. This was a primary goal for the client in house teams for a substantial amount of time, BI:PROCSI was able to plan, develop and deliver this solution alongside the other workstreams in a matter of weeks. 

Alongside this, it was highlighted that internal teams required training and enablement in the new technologies to understand best practices and how to create valuable solutions from these newly purchased technologies. BI:PROCSI planned and led several hands-on training workshops with the client in Tableau, Matillion and Snowflake to upskill the existing team with a focus on using their data in a practical setting. Q and A sessions were held to allow the team to ask questions and highlight problems they had been trying to solve in their day to day.  

BI:PROCSI has always had a strong focus on training and enablement of our clients teams and believe this is the foundation for success for any business. By helping the clients team understand the tips, tricks and common pitfalls when using these technologies and processes, we can ensure continued success for future in house projects. s

Linkedin

Get in Touch

Third Floor, 1 London Square,
Guildford, GU1 1UN

Quick Links

Quick Links

Get in Touch

Third Floor, 1 London Square,
Guildford, GU1 1UN

Linkedin

© 2024 BIPROCSI Ltd. All Rights Reserved