The differences between Operational Intelligence (OI) and Business Intelligence (BI) can be confusing. Just the name, Business Intelligence sounds like Nirvana. Show of hands, who doesn’t want their business to be intelligent? No, the names are fairly ambiguous so let’s turn to Google define to shed some light on their meaning;

Business intelligence, or BI, is an umbrella term that refers to a variety of software applications used to analyze an organization’s raw data. BI as a discipline is made up of several related activities, including data mining, online analytical processing, querying and reporting.

Operational intelligence (OI) is a category of real-time dynamic, business analytics that delivers visibility and insight into data, streaming events and business operations.

These definitions are helpful, but I think the picture above really illustrates the differences quite clearly.  Business Intelligence comes after the fact which is illustrated by looking in the rear view mirror of a car. Therefore, it’s helpful to think about BI as a reference to where you’ve been or what’s happened in the past.  Yes, you can store information in a data mart or data warehouse, and you can “mine that data”, but that doesn’t fundamentally change the fact that the information you are looking at or analyzing occurred sometime in the past.

On the other hand, operational intelligence is represented in the above photograph as the front windshield of a car depicting what’s happening right now in real-time. If you spot a large pothole in the distance, OI will alert you to that fact, and enable you to make a course correction to avoid ruining your alignment;  whereas, BI will only let you know that you had driven through a pothole as your car is wobbling down the road from all the damage.

Most businesses have the potential to leverage Operational Intelligence for competitive gain, but many are still stuck in the past with traditional BI tools. If you want to really crank up your business, I say it’s time to get real-time and discover what a paradigm shift of moving to OI can do for your business.

Let me begin this post by first describing what machine data is.  Machine data is essentially log file information.  When engineers build systems (hardware or software), they usually incorporate some element of log file capture into the design of those systems for several reasons: first for troubleshooting purposes and second as a backup, in case something unintended happens with the primary system.

As a result, almost every electronic device and software program generate this “machine data”.   It’s fairly safe to say, that most things we interact with on a day-to-day basis captures this machine data.  For example, our automobile, cell phone, ATM, EZ-Pass, Electric Meters, laptops, TV, Online Activity, Servers, Storage Devices, pacemakers, elevators, etc. all generate and locally store this machine data in one form or another.  When we call the mechanic about a “check engine” light warning on our automobile, they ask us to bring the car in to the shop so that they can hook it up to the computer to diagnose the problem, we are leveraging machine data.  What the mechanic is really doing is accessing the machine data stored on our automobile to identify error codes or anomalies that would help them to pinpoint a mechanical problem.  And, the proverbial “Black Box” that is so crucial to explaining why an airplane may have crashed also leverages this machine data.

So, if machine data is everywhere, how come we never heard much about it?

In a word, it’s difficult.  Since machine data comes in lots of different shapes and sizes, it is a difficult proposition to collect and analyze this information against lots of different sources.  Going back to the car example, information collected from different sensors are all fed into one collection point.  The engineers building the automobile are able to dictate requirements to component manufacturers about the shape, format, and frequency of data collection, etc. of all this machine data.  Since they design and build the entire process, they are able to correlate and present this information in a way that useful for mechanics troubleshooting a car problem.

However, if we look at an enterprise IT infrastructure, this same collaboration & integration doesn’t exist.  A typical enterprise will have lots of unrelated components.  From, load balancers, web servers, application server, operating systems, pc’s, storage devices, multiple sites (on premise and in the cloud), to virtual environments, mobile devices, card readers, etc.  So, depending upon the size and scale of the business, they could have lots and lots of machines generating this data.  I personally work with some customers whose server counts are measured in the tens of thousands.

Within the enterprise, no universal format for machine data exists.  This fact creates an enormous challenge for any enterprise looking to unlock the value of machine data. That, combined with the variety, volume, and variability of this machine data can be downright overwhelming.  As a result, enterprises collect the information in silos, and resort to an old school, brute force approach to analyzing this data only when it’s necessary.  If a system or process is failing, a team is assembled from the various IT departments to engage in what can best be compared to an IT scavenger hunt, manually pouring through log files, comparing those files to the cause and effect across other log files throughout the network.  This whole process is so labor intensive and time consuming that if the problem is only intermittent a decision may be made to abandon even identifying the root cause.

Let’s go back to the car example.  Imagine that we bring our car to the mechanic, but instead of simply hooking a computer up to a command and control sensor, the mechanic instead had to connect to and analyze hundreds of different data points on the automobile, and compare all the available data against other data, with the hope of finding the problem.  To further build on this point, let’s suppose that our automobile emits an annoying screech at 35 mph.  We’ve had the car in the shop three times already for the same problem and have spent hundreds of dollars all to no avail.  Eventually, we come to accept the fact the screech as the new normal, and turn the radio up when approaching 35 mph.

There has to be a better way!

Let’s think about this for a minute, what would be needed to get the most value out of this machine data?  Well, if we tried to structure the information by storing it in a database using a schema, we wouldn’t be able to account for the variety of the data.  No, instead we’ll need a way to store information in an unstructured format.  Next, we’ll need a way to get the data from all the different devices to send the information to our unstructured storage in real-time.  Building connectors will be too expensive and difficult to maintain, so what we’ll need is a way to simply forward this machine data in any format to our unstructured storage.  Next, we’ll need be able to search the data, but how can we do that if it’s totally unstructured?  Well, to do that we’’ll need some way to catalog all the data.  Since the value of the data raises exponentially in relation to corresponding information, we’ll also need some way to correlate information across different data types, but how?  So we start to think, what’s common across all these different data types? Eureka! We discover the date that something happened, and the time that it occurred is present within all this data.  We’ll also need a way to extract information from all this data, otherwise, what’s the point of doing all this in the first place?  Hmm, since the data has no structure, creating reports with a traditional BI tool won’t work, besides reports are too rigid to ask the complex questions we will likely be looking for in your data.  Lastly, we’ll need to address the issue of scale and performance.  Whatever we design has to able to bring in massive amounts of data, in real-time, because knowing what’s happening in the present across everything we are running in our enterprise is way more interesting and valuable than what happened last week.

Well, we can continue to ponder ways to solve all these technical challenges, or we can just opt to use Splunk, whose brilliant engineers seemed to have totally nailed it.

So much has been written about the US Government’s Health Insurance Exchange that I’m almost afraid to mention it.  For this posting, I’m going to stay out of the political fray and avoid rendering any opinion about whether we should, or should not have the Affordable Care Act, aka Obamacare.  Instead, I would like to discuss the Challenge of the Health Insurance Exchange strictly from an IT perspective.

The US Government has spent approximately 400 million and counting on the current system.  So far, the system has been down more often than it’s been operational.  Secretary Kathleen Sebelius is on the defensive, and she’s been called before congress to testify about how she spent the money,  what went wrong?, and how she plans to fix it?.  On top of that, her boss, the President of the United States has been forced to acknowledge the problems with the American Public.  You get the idea – the site is a train wreck.  What we discovered is that the project was rushed, the supporting technology was dated, the systems are vastly more complex than originally thought and nothing works as advertised.

Hypothetically speaking, how would solve these technical problem if you were Sebelius?   Bear in mind, you have to change the proverbial tires on the bus while its driving down the road.  Well, I’ve actually given this some thought.  Throwing the whole thing out and starting from scratch isn’t an option – it would take too long, and you have the President of the United States, Congress and US Public breathing down your neck.  No, about the only thing you could do in the short-term is identify, isolate and repair the glitches.  The trouble is, a single transaction spans multiple systems and technologies.  What’s needed is the ability to trace a transaction end-to-end in order to ferret out and address the problems.   Stabilize and fix what you can, and replace what you must.  Once stabilized, you can test and upgrade fragile components.  All this sounds great, but without End-to-End visibility and a single pane of glass to identify problems you wouldn’t know where to start.

I’m quite proud of the fact that I work for a software company that has actually solved this problem.  In fact, my employer (Splunk) is the only machine data platform that I’m aware of that can provide this level of visibility and insight across heterogeneous environments in real-time.  If you simply Splunk it, find it, and fix it, you’ll quickly get a handle on what you need to fix and your priorities.

Are you able to quickly identify and isolate technology problems across all your environments?

Imagine this scenario – you just discovered that your IT systems have been hacked into.  Even worse, after pouring over logs and conducting exhaustive analysis you discover that the data breach had been going on for weeks.  Since that time, the perpetrator or perpetrators  have been systematically siphoning sensitive data from your network.   Now imagine that it’s your job to report this Data Breach to the CEO.   Would your heart just skip a beat? I know mine would.

This  awful scenario, while hypothetical has no doubt been played out both within businesses and government agencies throughout the world.      In addition to the lost trust, embarrassment, operational disruption, and the financial impact on your brand, data privacy laws and regulations  further raise the stakes through imposed fines.  Moreover, Wall Street definitely doesn’t take kindly to Data Breaches.

Here is a not so typical example of how a data breach can play out in the real world, but one you need to be aware of.  This is an account of happened to Heartland Payment System, a Payment Process Provider (NYSE: HPY).  In 2008, Heartland Payment Systems was alerted  to suspicious account  activity related to their customers from their partners Visa and MasterCard.   Heartland conducted an all out investigation and were horrified to discover that they were unaware that a packet sniffer had been surreptitiously installed on their network.

At the time of this discovery, Heartland’s stock was trading at $25. per share, with 36.83m outstanding shares.  As the full extent of the data breach came to light, Wall St. punished the stock.  Over the ensuing months, the stock hit a low of just under $5 per share.  That’s over and 80% drop in value or approximately $736 million. In addition to the massive drop in market cap, Heartland was forced to pay substantial fines, and spent vast sums on consultants and both software and hardware to harden their network.  Overall,according to news articles, the Data Breach cost Heartland $140 million in fines and other hard dollar costs.

What truly shocked everyone was the sheer magnitude of the breach – some 130 million credit card accounts were stolen.   This single data breach could have put Heartland out of business.  It’s taken years, but thanks to some shrewd crisis management, Heartland has regained customer trust, and thankfully, the stock is doing better than ever – trading at $44.47 per share as of this writing.

Heartland’s troubles could have been greatly minimized if they had an early warning system alerting them to network anomalies.   Much like a smoke alarm alerts us to fire, monitoring inbound requests and outbound responses with Splunk would have alerted and directed management to suspicious network activity which could quickly have been rooted out quickly.

To use an analogy – A tiny fire quickly extinguished causes little damage, but if you aren’t aware, that same small fire can become a raging inferno and take down your business.

Public service announcement – Remember to check the batteries in your smoke detectors on your birthday.