The term “big data” is hot these days, driven by increases in real-time computing power while the costs of software, computing, and data storage continue to rapidly fall. Big data is seen by many as a promising pathway to improve corporate efficiency, lower costs, and understand wants and needs to better satisfy customers. However, big data has long been around, whether in analog form, digital form, or both. The twin challenges have always been in how to present and correctly interpret data, and reduce delays in decision-making.

The existence of big data does not in itself mean that information will flow without interruption, that correct decisions will be made, or that decisions will be made in a timely manner. The digital world must interact with the analog world called human beings – leaders and managers – who must make judgments based upon the information they receive. They can easily block information by exhibiting behavioral waste: blame, micromanaging, bullying, condescension, disrespect, etc.

big_data1Batch-and-queue information processing will still exist in a big data world where humans mediate information processing and make judgments. While the cost of big data may be low to acquire and the data vast and important, poor judgment, slow decision-making, and faulty decision-making will remain. Information can be blocked merely by virtue of being human.

Calls for a new generation of leaders capable of comprehending the data and making good decisions based on the data are illusory, absent of training in what constitutes good decision-making. That skill is best learning by studying how other leaders have failed in their decision-making as a result of poor human information processing.

I teach a graduate course called “Decision Failure Analysis in Technology Management” where we carefully analyze ten real-world cases each semester (click here and here for more information). We study leaders who made bad decisions that resulted in terrible outcomes for people. We find that bad decisions impact all stakeholders: employees, suppliers, customers, investors, and communities. Competitors are the one stakeholder that is positively impacted – plus the people who replace the failed leaders and those who bought stock in the company for the first time when the price was at a multi-year low.

In my course, we use a structured problem-solving process that I developed a decade ago called the “A4 Failure Analysis Method.” Part of the failure analysis involves identifying factors that afflict all humans in their ability to process information and make decisions:

  • Favoritism (inclusive of subconscious biases and stereotypes); i.e. favoring one stakeholder over another, such as investors over employees, or one department over another, such as finance over operations.
  • Inconsistencies between what leaders say versus what they actually do.
  • Untested beliefs and assumptions.
  • Decision-making traps (anchoring, status quo, sunk cost, confirming evidence, framing, forecasting, etc.)
  • Illogical thinking (false assumptions, using and abusing tradition, avoiding the force of reason, abuse of expertise, false dilemma, special pleading, expedience, etc.)

Improving one’s information processing capabilities and decision-making skills requires careful study of other’s failures in relation to the above 5 items. Simply relying on generational differences that favor computer literate leaders over those who are not is an unrealistic answer. The better way is to train leaders in the era of big data is the A4 Failure Analysis Method. This will result in two practical outcomes:

  • Learn how to avoid the types of information processing errors commonly made by managers that contribute to flawed decision-making.
  • Develop an ability to anticipate future failures.

While big data holds much promise, nothing is all upside. A likely major downside is big data fooling leaders into thinking they can manage and lead through data. This will create even greater separation between management and the people who actually do the work. The leaders who have appeared in the television show Undercover Boss may end up looking good compared to the big data bosses. There will be other downsides as well, such as depersonalizing the customer or employee experience. Will socially challenged new generation big data CEOs be able to overcome such personal limitations?

Big data has as much an opportunity to favorably impact “Respect for People” as it does to unfavorably impact it.

Big data without applying the scientific method to management processes will simply be bad data. The data collected and how it is presented are merely artifacts of how software was written. Data integrity and sound analysis are not assured simply because society holds in high esteem entrepreneurs, programmers, and data scientists. They are human and subject to factors that afflict all humans in their ability to process information and made decisions.

Let us not forget that, far more often than not, senior managers reduce big data data to little data in order to facilitate decision-making – which most often is zero-sum, itself a handy reduction to simplify decision-making. Humans were wired that way to assure survival, not for any other purpose. So, the likelihood that leaders will make many bad decisions that negatively impact stakeholders remains very high, big data or not.

Lean principles and practices will be needed more than ever as big data and artificial intelligence expand their reach into business processes and organizations. For example, top managers will still need training and development, both to improve interpersonal skills, leadership skills, and their ability to process information with few errors and make good decisions. Leaders will still need to “go see” to comprehend actual workplace, customer, or supplier conditions and improve their ability to make fact-based decisions.

There will be a continuing role for kaizen because it puts leaders into contact with workers, customers, suppliers, and others. In addition, the programmers who write code to capture data, as well as the data scientists who work to display and analyze big data, conduct their work according to various processes that can no doubt be improved through kaizen.

One Response to Big Data, Leaders, and Lean

  1. Robert Drescher says:

    Hi Bob

    The biggest problem with relevance on Big Data is that like accounting information it is all historical information. It has zero ability to react or predict what disruptions can occur in the future. Any decision that is made needs to be tempered by real time human understanding about what is happening in the present real world, and what may likely happen in the future.

    Unfortunately there is no way for a data base to gather future information, nor can programmers or data scientists design software or equations to predict it (you may as well go to a fortune teller). It takes people capable of understanding what has happened, yet also capable of seeing what could happen in the future. I doubt if you took all of the data on computer technology and software of the 1970s and ran it through today’s software with today’s computers that it would predict how computers have evolved, the internet, tablets, smart phones, tvs, or anything else we accept today. Hell most people that admired Apple in the early 80s and their cutting edge Mac of the day, would not have disagreed with what the data would say that Apple should have died and disappeared from the computer industry just like Commodore and Nintendo did, yet today, it is the world’s highest valued company, and the highest ranking tech company in the Fortune 500.

    It takes visionary humans to lead an organization to a successful future, big data like a history book can only tell you the past and help a leader learn from the past. Neither though will ever successfully predict the future, there are just to many unknown variables that cannot be accounted for.

Leave a Reply