Learning and Life: Big Data, Management Transformation and Voyager of Star Trek Movie

V'jer of Star Trek

The growth of digitization and computerization and its assimilation of everything and everyone that it touches reminds me of the first Start Trek - Movie, where the enterprise encounters a huge energy cloud that is devouring extra-terrestrial objects and even planets on its path as it moves towards the solar system and then eventually towards earth. There seems to be nothing that can stop this energy cloud/mass and there is fear that even the entire earth may be swallowed by it. With each
star swallowed and with each planet gobbled, its size increases and so does its power and energy. What eventually happens to this energy cloud is part of the story that needs to be seen in this movie. What is of relevance for us is that the core of this ever increasing energy cloud eventually turns out to be none other than our old NASA spaceship Voyager that was released from the earth in 1977 and which only recently i.e. in 2012 teared out from the solar system and became the first man made craft to enter the inter-stellar space. In this movie, the Voyager during its two hundred or so years of its journey in the interstellar space has mysterious encounters. Through one such mysterious encounter with an energy source, it transforms and acquires the capacity to start attracting objects, gobbling them and becoming bigger and bigger and amassing more and more energy. Anything and everything that comes in its path is not only swallowed by it, but is also transformed into the energy, which accumulated on top of its existing energy, not only increases its power but also the speed with which it gobbles up things with greater and greater ferocity and an ever increasing appetite.

When I see the relentless growth of computerization and its insatiable appetite for everything that it touches, I am reminded of this Voyager turning into that massive energy cloud in Star Trek Movie. Starting from Babbage's mechanical computing machine in 19th century, computer had its encounter with Turing Machine in the 20th century followed by its transformation into an electronic form with computer tubes, which transformed into transistors, which transformed into IC chips and microprocessors and then suddenly Moore's law took over; heralding an age of the stupendous exponential growth of computerization with no end in sight:

Computerization first digitizes everything that it comes in contact with and then devours it by integrating with it. Its size continues to grow and the digitization of the information gets bigger and bigger and gets transformed into this cloud of data that becomes something similar to the energy cloud of Start Trek Movie that feeds upon the planets and extraterrestrial objects that come in its way. There seems to be no end to this insatiable appetite of the data cloud to integrate, adapt and internalize everything that it touches, and get bigger and bigger; starting from organizational books of accounts, to their transactions, to their operations, it has extended its reach to their tools, their devices, their workplaces, and has further entered the personal lives of the employees and into their homes, their lives, their biology, their thoughts and even everything they stand for.

Let's start with "convergence", a buzz word of the 1990s. This represented how at that time computer industry was merging with the networking industry. Next to merge with the computer industry was the telecommunication industry with electro-mechanical switching equipment getting transformed into computerized switching and so on. The advent of mobile phones indicated the merging of the computer industry with the telephone industry. Movies like Toy Story heralded an era where the computer industry started gobbling up the movie industry. Then computer industry started swallowing up the camera industry and now we have smart phones and while looking at them, we can not even imagine that once these industries had ever been separate. The multimedia projectors industry is also in the process of being gobbled up by the mobile devices, reference to which used to be the fun part of my lectures 18 years ago. Individual elements of the computer hardware that used to be separate industries such as microprocessors, memory chips, ASICs, database servers, etc had all converged sometime back and are now mostly are up there in the cloud. We are just now left with the thinnest of the clients; the i-phones and the i-tabs that are going to may be even disappear in few years in that cloud.

A similar thing had been happening on the software front. It all started with computerization of the books of accounts in the early 1960s. Soon there was a need to integrate the accounting system with payables and receivables. Then of course integration of purchasing and sales were close behind which could not be done before inventory systems could be integrated. This led to the material requirements planning (MRP) systems. This led to the integration with automatic generation of work orders and purchase orders and eventually MRP-2 systems got all the aligned systems integrated. Computerized systems have eventually transformed themselves into ERPs with 40+ modules capturing end to end all the aspects of structured transactions in an enterprise.

Computerized systems could not be contented with the integration of systems within the organization. Soon EDI technologies had led to the inter connectivity of the logistics systems and eventually to E-Commerce. On one hand, this drive led to integration across the supply chains; producer linking to the supplier, and the supplier linking to its supplier, and so on till the supply chains extended to the mining/farming industries. On the other hand, the drive for integration led to the connectivity with the whole-sellers, to the distributors, to the retailers and to the consumers. The intra-organization and inter-organizational integration during this spread of computerization primarily targeted structured information. Given this wealth of information and their linkages gave rise to the data warehouses, to data farms and from data-mining we were soon using business intelligence software for harvesting this data. This gave rise to the technologies that we study here.

The hardware integration, convergence of technologies and miniaturization and the security concerns following 9-11 led to the data sensors and data collection equipment getting installed on every gate, every machine, every hallway, every road, and every nook and corner. Each product could now be individually tracked from the time it leaves the factory to where ever it went or wherever it was used till its consumption. On the other hand movement of people, products, vehicles and services all started producing data that could be collected, tracked, monitored and processed.

The greatest explosion of data started taking place when even the unstructured transactions also started getting voluntarily or inadvertently recorded whether the subject subscribed or not. Each click on Google started getting recorded on the servers and helping Google to grow bigger and bigger, each Like on Facebook started enriching their graph database enabling Facebook to harvest that data and sell it for marketing purposes, each email started getting processed and producing wealth of information, each email account and each cell phone yielded a list of contacts, and for each of these contact that contact's cell phone/email account yielded another list of contacts. Linking of such list of contacts provided interesting graphs of who-knows-who, and who-talks-to-who and who-messages-who with what frequency and what type of information. Coupled with profile of information stored in various social networks and email systems suddenly all the people and their connections became visible, started getting recorded, and linked.

The stage has now come when we can track when a particular faucet of which bathroom gets operated when, which light of which room gets on or off, when the garage door opens, when you visit which place and meet with whom with what frequency, which pages you visit for how long and which links are more stimulated than the others and categories of information that get more hits from you and what times. All unstructured communications are now within the reach of this mass of computer system, this energy cloud that has already gobbled the structured information and most devices and is now readily gobbling up all the hitherto unstructured information and the entities that generate it and converting them into mine-able information.

Semantic technologies have already converted the Wikipedia into query-able semantic database. These technologies are now converting even the analog content of voice and picture data into a query-able semantic database. Emotions, personalities, ideas, and thoughts are become transparently visible, record-able and amenable to semantic processing. Our biology is now more than a series of bits and bytes known as genomes and issues related to health and medicine are getting transformed into cutting, copying and pasting of some gene information. Our mind and our thinking is already getting represented as bits that are getting picked, tracked and processed by neuroscience. The day is not far when there would be Wifi hot spots all over the place picking up our thoughts and using them for processing.

We are now sitting on this wealth of information. This being the information age it is all about transforming data into information. It is no longer the industrial age, where raw material could only be converted with the help of specialized machines into specific products. The data can now be converted by the generalized computing machines into any useful form that we can now think.

Accounting industry is already fighting a loosing battle with the computerized ERP systems that have all but replaced the need for professional accountants. Encyclopedia industry is already gone, gobbled up by wikipedia and its editors and contributors who are all up there in that cloud of information. Not only the robots are taking over the machine operators, but the mundane jobs such as those of drivers of cars and others automobiles. With the advent of the Singularity in about 20 years, robots would have overtaken the man in intelligence and nimbleness of action.

The growth of big data is predicted to be annually 4300% till 2020. We are talking now in terms of billions of Terabytes. Forget about a few Giga bytes offered for free by Google Drive, QQ of china is now offering 35TB of space free of cost. Just estimate the information explosion that we are anticipating. This does not include the organizational data.

How can workplace and organizations not get transformed with such massive transformation of everything around us. It is already difficult for us to understand the data contained in our ERPs and CRMs. What kind of management would be required for managing organizations with big data.

Abstraction Skills

Power of abstraction is to be able to reduce huge amount of data into chunks that are manageable.
Today the efficiency, decision making and presentations of those managers is preferred who are power users of Excel and can develop visual representations of data in tabular or graphical form that are easy to understand. Even today we find many managers having difficulty in handling such data and consequently are unable to use the data for decision making.

Considering the avalanche of data coming from multiple sources there would be a huge need to abstract it without compromising on the loss of value. As one abstracts to the higher levels, important details are lost. Ability to move up and down the abstraction level of data, and the ability to track and drill down to the information that is valuable is now of paramount importance.

Mental Models

Graphical representations of the relationships and conceptual frameworks that exist between various categories and abstractions of data. Given the large number of sources, and their categorization, there is a need to develop mental models that capture relationships and provide the ability to plan and predict. The relationships need themselves to be first hypothesized and then validated by the data. Typically a human mind can not keep and process more than five to seven different concepts simultaneously in the mind. Therefore, the ability to encapsulate the large number of concepts into smaller models that can be processed by human mind is now the requirement. Creating such models and verifying them using formal methods needs to be an ongoing activity in organizations dealing with Big Data.

Mental Models and
Representations [1]

Understanding the capabilities of tools and their limitations

Not every tool is specialized for doing all the tasks. For example, Excel is appropriate for certain types of tasks, while Word is better for other types of tasks. The two can not do the task of a database. Each tool has its limitations and its areas of expertise. Learning the strengths of each tool is necessary. Certain visualizations are easier to do on PowerPoint. Others require Excel. Yet others require a database. How these are linked together.

With Big Data, now we are getting into tools with their specialized query, data representation and data transformation capabilities. But, we must understand the limitations of what a human mind can perceive. How to reduce this humongous information to a level where it could be used for decision making is essential. The expertise required will involve presenting the information to a level of abstraction that is understandable for a decision maker.

The insight about the capabilities of tools, their limitations and their power only comes through experiential learning. This can not be learned through lectures or seminars. Managers would have to dive in, start using the tools, understand their capabilities, and only then would be able to guide the techies to generate the information that they require.

My experience of dealing with managers who has this experiential learning and those who do not makes them clearly stand out and distinguish in their productivity. Power users who acquire this depth of understanding through experiential learning and their ability to quickly respond to challenges and generate ideas is far superior than others.

Organizational Design

Design of organization would have to reflect these changes. Our organizations are still largely living in the industrial age with functional silos as indicated by the names of their departments and their designations. The era when finance and marketing were two different functions is largely gone. Both marketing and finance are simply categorization of data that are related to one another and to operations, purchasing and customer relationships. These can no longer be looked at separately. They as well as the customer sentiments as represented in the likes and dislikes of the Facebook pages of the current and potential customers are all related together. We need to make sense of all of this to determine whether the organizational mission and vision objectives are obtained.

Integrators and Consolidators

Integration of supply chains across organizations and integration of demand chains across the customers to consumers simply means that the survival of an organization will depend upon its ability to integrate and consolidate. Organizations focusing on a single step of these huge chains would only exist for a small period of time before they are gobbled by a neighboring more powerful organization. Organizations that think that they can survive on the basis of creating value in a tiny domain are just a small pawn in the great game. Nokia could not survive because it forgot to integrate with the web. Samsung not only overtook Nokia but is challenging Apple because of the crucial android system which is an integrator. Before long this would be integrated by a neighboring organization which Google is and is the huge challenge.

Organizations have to redesign and redefine their managerial transforming in terms of how they can become integrators and consolidators of information with management capable of abstraction of huge amount of data, capable of developing mental models and verifying them against this mass of data using sophisticated tools to derive strategic advantage.

Most of the ideas in this post were delivered extemporaneously as keynote speech at Open BDA Hadoop Summit 2014 held at Mariott Hotel, Karachi on November 18, 2014.

See Also:

References: