 Bad decisions from bad data. Business intelligence is a ubiquitous term in the technology world of today. Companies large and small are dependent on data to make both tactical and strategic decisions. Reports, dashboards and key performance indicators are some of the tools that can be used to make sense of data. While BI tends to be the star of the show, it's just the last stage of an overall data management plan. If the data being reported against is bad, the decisions based on it will be inherently flawed. Data management comprises many activities across an organization including governance, gathering, sanitizing, ETL, and BI. Each stage is as important than the last. If one step is overlooked, the quality of data being reported on may be compromised. This is why having a well thought-out and documented data management plan is critical. How we develop a successful data management plan. An organization may need assistance in the process of developing a data management plan. Like many cases, data is managed with varying degrees of precision, whether transcribed from written to digital, or a lack of consistency around data types, specificity, and homogeneity. For example, a data field that was expected to contain a yes or no response may have been gathered as y, Y, yes, Yes, YES or even X. Another aspect of data management requirements may need to publish parts of data without including personally identifiable information. In this kind of engagement, Pivotal has helped clients understand the many methods of cleaning data and de-identifying data for publication. As an example from a previous engagement, we evaluated the risks and virtues of various techniques and technologies, and created an easy to use tool with Excel, VBA, and Power Query that could perform the required tasks. This approach was beneficial because of the low cost of implementation, the ability to tailor it to industry specific requirements, and the shallow learning curve required for a typical user to employ the tools. The outcome is a solution primarily automated with limited user input which would prepare the data both for reporting and analysis as well as publication.  Concerned about data privacy? Another benefit of de-identification.
Like sanitizing data during the data management life cycle is critical to ensuring the final outcome, BI is able to produce meaningful and useful analysis. De-identification adds a layer of protection when data privacy is of concern. By removing or replacing pieces of data like names and complete addresses, in-depth analysis is still possible without the risk of privacy loss. With proper security, it is even possible to maintain the ability to re-identify data but limit that capability to a few key users. Data cleaning can greatly improve the efficacy of BI. Things like standardizing date formats, range checking on numeric values, and consistency in how data is recorded takes the burden of handling those exceptions off the BI implementation. Sanitized data not only benefits your BI, but your bottom line. From a high level, data management plans are invaluable at every stage of data processing. Understanding where the data is coming from, defining a data dictionary and analyzing the raw data all contribute to the sanitization process. Sanitized data can then aid in the creation of better data warehouses which in turn can be used for improved business intelligence. While it may be a time intensive activity to define a data management plan initially, the return on investment can be quickly realized. Having sanitized data to report on could very well mean the difference in millions of dollars of revenue |
Categories
All
Archives
June 2023
Subscribe |
RSS Feed
