Big Data?

July 24, 2013

 “Big data” seems to be the latest flavor of the month concept. We hear it used in context of the NSA, Obama Care, and cloud computing (which is not really done in the cloud, but in places like Utah and Norway).

The question is “does Big Data have anything to do with competitive intelligence?” Yes, but probably not in the way that you think. CI is certainly the beneficiary of the improving ability to collect and process more raw data. But, unless you are an arm of government, the data your firm usually processes is your own. That means, for most businesses it is inward, not outward looking.

CI is outward looking. That is not to say that the data that you need is not within your company – quite often it is. But is unlikely to be a part of its Big Data operations. Those typically deal with finance and production issues.

But Big Data has an impact on CI in indirect ways.

First, as companies can find out more and more about their internal activities at a greater level of detail, they have a tendency to expect that the competitive intelligence operations will be able to produce such detail. That is not likely since the number of people who have access to that detail at a competitor is limited, assuming the competitor has even bothered to generate that kind of data.

A second impact is that people increasingly believe that, somehow, CI can be conducted by computers, just by applying greater and greater computing power. Until computers are able to draw inferences from incomplete data, which they cannot do now, they will be of little impact to CI, other than to speed up and expand the amount of data that is collected. This is not always a good thing. More data means more time and effort spent processing it and a greater likelihood that the data contains within it increasing amounts of noise. That means time and effort wasted.

Competitive intelligence’s strength has always been in the analysis, not in the data collection. The world of Big Data does not provide CI with any help there; if anything, it hinders CI.

It ain’t over till it’s over

July 15, 2013

The conclusion of the Zimmerman – Martin criminal trial illustrates an important lesson for CI analysts.

Regardless of what level of interest you had in the matter, you have to admit that what came out of the trial had slight resemblance to what you heard or read in the early stages of the case. I’m not going to dwell on the details, because they are not important here.

This case can be used as example of dealing with limited information and limited data. What finally was presented at trial by both the prosecution and the defense seemed to be a mere skeleton of what the media reports and commentators seem to believe were the “facts” of the case over the past year and a half. But the jury had to make a decision on the set of facts that were given to it, and made it, unanimously.

What is important is that when analyzing any kind of a problem, you must withhold judgment until you have all the facts. Coming to a conclusion quickly is almost never the same as coming to a conclusion correctly. You also have to be careful to avoid being influenced by “advocates” of any point of view as to what the facts, as they emerge, mean. Of course, listen, but withhold your decision. In this case, the facts introduced at trial may or may not have come as a surprise to anyone. But the prosecution and the defense interpreted them very differently to the very end.

When analyzing your CI problem, don’t be either the prosecution or the defense. Be the jury and wait until all of the facts are in, and the rumors and other “non-facts” are sifted out.


July 9, 2013

The recent weeks are filled with discussions of whether or not residents of the United States, or for that matter of any Western country, have any privacy left. The answer is yes, but not as much as we did.

The underlying issue is whether and to what degree we’ve surrendered our privacy voluntarily or involuntarily. Involuntary loss of privacy is a political issue of major consequence. The voluntary surrendering of privacy is not.

Think about this in the context of competitive intelligence. Much of what you can learn through competitive intelligence is because we do not live in a closed society. In a free society, people communicate, they make public statements, they post on the Internet, they advertise, they meet, they speak freely, etc.

In that context competitive intelligence, which to remind anyone who is reading this for the first time, is perfectly legal and ethical, can be conducted and can be very useful. From an economic point of view, the more information one has about the marketplace and on competitors, the more perfect, from an economic sense, competition can be. So competitive intelligence, in a free society, contributes to, rather than impedes competition.

In a society where there are significant involuntary losses of privacy, competitive intelligence cannot function. That is because people and institutions there closely guard information, or affirmatively distort information (disinformation), or just withhold information from others. Without free access to public information, competitive intelligence cannot function.

The best estimates are that from 80 to 90% of all the information that a business needs to know about its competition can be collected through legal and ethical means, that is, by competitive intelligence. The last 10 to 20% requires unethical, or even illegal means. But that is another story.

One might even say that the inability to practice competitive intelligence effectively is one measure of the degree to which any losses of privacy are involuntary rather than voluntary.

There is no denying that we’ve suffered involuntary losses of our privacy. Actually some may argue that we have had no such involuntary losses, because our freely elected representatives have voted for and supervise those “invasions of privacy”. But the existence of and current practice of CI may tell us how much.

Going to the source

July 3, 2013


One way you look for data for competitive intelligence, when you’re doing your own research, is to visualize the data as something that is flowing, like a stream. You want to figure out where it originated, and where it moves. Then you try to figure out where is a good place to intercept some or all of that data.

One thing that is often overlooked is going directly to the source of the data. In most cases, that is your competitor, which of course, can cause problems. Before you even think of doing this, check to see if your company has a policy against contacting competitors. You’d be surprised how many corporations, particularly those that it been in business for 75 or more years, have policies dealing with this. They tend to come out of misadventures with antitrust authorities in the past.

Assuming that your company has no policy against you contacting a competitor, think out whether or not you should do so. Who would you call there? What would you ask for? What would you say when someone asks who are you and why are you calling? The answer the latter one is — the truth. You are not a “student calling to research a paper”, you are not a hassled customer calling about a bill. Now you don’t have to volunteer information, but do not lie.

Now you probably can’t pick up the phone and call your opposite number, say a product manager. We can, and do, but that is another story. What you can do is look at your competitor and find out where your competitor faces the public. More specifically, does it have a consumer information line, or a place on its website allowing people to ask questions? If so, exploit these. These people are there to provide information to the public, usually customers or shareholders or potential customers, and there is no reason not to at least ask. The worst they can say is “no”.

Let me give you one quick example. Several years ago, we were developing intelligence for client on the possible rollout of a new product. It happened to be food, but that is not really that important. We found out that the product was being offered in another city and wanted to figure out if the company was positioning itself for a national or regional roll-out. Our background research indicated it was likely that the company was positioning for a national rollout, but to confirm that we called the consumer help line. We simply said we had heard about this new product, that we lived in Pennsylvania, which is true, and that we wanted to know when it might be available here. The individual in the help center, being helpful, asked that we wait for a moment while she checked. She came back in a moment and said “I have answer for you. It appears that we are rolling the product out on a national basis and we expect it will be available in Pennsylvania in about 6 to 8 weeks.” I said, “Thank you very much.” By the way, I actually went out and bought the product when it showed up here. It was pretty good.

My point is never ignore the obvious – the source.