“I cannot imagine any condition which would cause a ship to founder. Modern shipbuilding has gone beyond that.” — E.J. Smith, Captain of the Titanic, in 1910
“BP Exploration & Production Inc. does not propose to utilize new techniques or unusual technologies for these operations; however, the best available and safest technologies …will be incorporated as standard operational procedures,” — excerpted from Section 2.4, BP’s Initial Exploration Plan for the Deepwater Horizon oil rig, May 2009
I have a friend who recently discussed taking a family cruise, and his wife said that she would prefer not to, given what had happened on the Titanic in 1912. This comment somewhat surprised me being that the famous Titanic incident–in which an ocean liner, disregarding warnings, sped through icy waters and hit an an iceberg with fatal consequences—happened a century ago. After all, technological improvements have been substantial since then.
Despite my logic regarding telegraph versus internet communications, the bridge’s use of a real-time GPS with radar over paper foldout maps, and, most importantly, that this was a Carribean cruise where the ocean was too warm to have icebergs, my friend’s wife remained adamant. To her, just because she couldn’t see the iceberg didn’t mean they weren’t there, at least in her mind.
And of course, I took the big leap into thinking about data quality, project management, and the obvious similarities with the Titanic’s fateful voyage.
In a recent article, I proposed that Mistrust of Technology and Fear of the Unknown are two of the culprits that cause others to blame technology for data issues, thereby pitting the business team against the IT department. I can extend this thinking with my Titanic metaphor: Business often blames technology for being the iceberg that may not really exist.
Of course, this conceptual imagery does have its advantages, especially in terms of risk management. If we can see the cause of the problem before it occurs, then we can easily avoid it. Just like charting a ship’s path through treacherous waters, we can stay away from the big icebergs before they hit.
But this is easier said than done. Icebergs can and do exist and sometimes they occur, despite the warnings and signs that were apparent in hindsight. This happens many times due to the phenomenon of technological hubris. This form of pride occurs when project leaders choose to march forward with disregard towards safety, planning, or quality.
Technological hubris is the faith that experts know what they are doing although the facts may not suggest as much. It translates into the presumption that data wracked with data quality issues plugged into a trendy new data warehouse tool will invariably become good data. And when it is discovered that no matter the price or performance of the technological efficiency, bad data continues to be bad data no matter what, the team blames the technology. This is akin to blaming the Titanic, and not the crew, for sailing into the iceberg.
My example above in which “bad data + technology = good data” is difficult to avoid in practice, especially because human beings are highly susceptible to the modern-day hubris of deadlines, milestones, costs, customer satisfaction, political influences, and so on. Sometimes the best we can do is to know the icebergs exist, to keep monitoring the waters ahead with risk assessments, to pay close attention to the signs of trouble ahead, and then to follow up with good disaster recovery practices such as code repository storage and database backup.
Data quality issues, like icebergs, can be hidden deep below the surface and very big indeed. When the data quality icebergs are encountered, simply attacking the data itself is only a short-term fix. The long-term solution requires time and effort to perform a deep analysis and even deeper remediation that identifies the source systems and the people involved with that data.
Taking our time to see the true icebergs we face, then to plan, evaluate and analyze them will help minimize potential damages and lead the way towards a successful project. With this in place, both the IT and business team members have the security and confidence in the project that will keep them pushing forward, rather than crowding the lifeboats and blaming the technology for the disaster.