What we're now getting closer to is to be able to pair up the speed and processing power of computers with human intelligence - letting computers analyze potentially thousands of rules per second and when new opportunities emerge or problems occur, let humans into the loop presented with pre-defined applications, pre-loaded with data and analytic workflows, that allows those humans to rapidly assess problems and opportunities - and either update rules or take actual actions in operational systems. We're making some great headway in this direction and will look forward to speaking more to you about that. There is a unique opportunity in the history of computing here, finally pairing up the very best of human intelligence and computer smarts, speed, and stamina!

As we've seen financial markets develop over the last 25 years with a wealth of data surrounding them something very interesting has happened.

Knowledge of the future has always carried an enormous premium - think back at Julius Reuters in 1850 using a fleet of 200 pigeons to transfer information between Brussels and Aachen within two hours, beating the railroad by six hours. or think about the billions and billions being spent by governments every year in intelligence - just to be able to get a glimpse into the future.

Being able to know something about the future of a financial instrument has arguably become more valuable than owning it itself (you can clearly argue that you'll have better leverage on capital in such future knowledge than having your money locked up in an actual underlying instrument).

What the financial markets have taught us here is that information that can tell us something about the future is a currency in itself, and we're quickly entering a world where every (!) industry will come to realize that such insights will carry greater value than the underlying assets or processes from which they were derived. Information that can tell us something about the future is becoming a currency itself - and I will argue it's by far the most important currency of the future.

Below I will lay out some examples. Obviously you can shoot down each and one of them based on our current (in)ability to actually say something about the future - as well as based on the stronger argument that some things may never be very predictable - which are fine points, but my over reaching premise is simple: the pendulum is swinging towards an era where we can enable man and machine combinations to say something about the future better than ever (if you don't believe that, stop reading).

- Would you rather be a producer in an oil field or be able to predict where oil can be found?
- And accordingly be able to either sell such information
or be able to buy/sell oil fields/patches without bothering with dirty/complicated oil drilling...

- Would you rather hold lots of inventory of Christmas toys locked up in a global supply chain or be able to predict what ports and shipping terminals are going to go into gridlock before Christmas?
- And accordingly be able to buy up port/shipping capacity elsewhere...

- Would you rather have 10,000 customers that you know will be loyal over
5 years or 100,000 customers and being really unsure about who will stay and who will leave?
- And accordingly be able to invest in improved service for those 10,000 loyalists.

- Would you rather own a factory and its ability to turn out goods or the ability to predict when it will come to a standstill?
- And accordingly be able to tell the factory manager what he needs
to do right, or if you don't like consulting businesses, be able to
sell him a product that predicts when problems will arise, or even
better be able to provide back up manufacturing capacity when needed
(at a nice premium price point).

- Would you rather run/own a pharmaceutical company and its ability to produce cancer drugs that might in best cases work in 30% of the cases or have the ability to predict what drug will work on what cancer patient?
- And accordingly be able to actually improve that chance of success
dramatically for your own drug, or even be an information arbitrator
that doesn't even need to develop the drug.

- Would you rather insure 10M people at the same rate or segment them very carefully to a point where you are matching risk and premium exceedingly well?
- Yes, of course this can be morally complicated - but stop crying, it
will happen, and if the insurance companies aren't using it towards
consumers, consumers will be ganging up against insurance companies
to gain an information advantage against them. The insurance business
will be very different in the future.

- Would you rather have an army of 1M men with old rifles and equipment or an ability to predict where these 1M men, if they were your enemy, will show up and have an army with 50,000 men ready to meet them at the exact right time and place?

Now, of course you may say "is this just some dumb old artificial intelligence overconfidence in what computers are/will be able to do?!". I think that is a good argument actually, but the interesting point is that there is a counter effect built into all of this. The better we get at nailing one of these prediction problems you can be very certain that other people will be able to replicate it, and accordingly markets will neutralize any gain you can get to faster than may feel comfortable. And rest assured that this argument doesn't hold true only for financial markets (we all know the old argument that if you had a great stock picking algorithm the knowledge would soon slip out and the algorithm would be worth nothing) - this will hold true in every market and domain. If you devise a magnificent way of predicting supply chain flow and problems, somebody else will counter, and fast.

So, given that you might say "why even waste your time on this?" Here the point is I think that you can and will be able to gain some truly majestic information/competitive advantages - but you have to expect them to be short lived in themselves, so you need to equip yourself to invent over and over again. You will be in a constant information and analytics warfare. A brutal contest. And btw, if you don't you'll be outplayed. You will not even have a ticket to the game.

What does it mean to be in information and analytics warfare? Step #1 I can't really help you with. You need to hire and retain remarkable people - who understand both your business and understand what it means to compete on information/analytics. But you've always needed to hire and retain remarkable people to win, so perhaps no difference there.

The other end of the coin, Step #2, is that you need an infrastructure that is built to be remarkably adaptable to new business problems. Remember, you will be in a constant innovation game, and you will win based on your ability to innovate.

This means you need an infrastructure that is equally adaptable for both man and machine, as fast as business owners (yourself perhaps!) come up with clever ideas, and you need to be able to convert these ideas into action. This means deploying new user experiences, new algorithms, new applications, new data feeds, etc. - basically at the pace of business (think carefully about what that means, pace of business - not pace of software development, not pace of traditional systems roll-out, etc. - pace of business!). Building such an infrastructure should not be a pipe dream by any means. Perhaps you will not get there through some magic overnight transformation, but the bottom line that I want you to carry away from reading this is that these architectures exist, and you can put yourself in the pole position today.

Have fun doing so!

Today we announced that Spotfire is being acquired by TIBCO (NASDAQ: TIBX), a leading supplier of real time infrastructure software.

Over the past 10 years, we've built Spotfire to be the leading and most innovative (not so modest!) designer, developer, and supplier of software for finding interesting insights in data - based on an awesome user experience and application infrastructure. We've had the opportunity to work with fantastic customers (you!) across the world to make this happen.

Spotfire is at a very exciting point in time. I've talked in this blog about business intelligence 2.0 and what is driving the change in the market - user experience, speed, interactivity, relevance to business problems, etc. We've seen an incredible momentum in the market for our offering across nearly every type of business.

At the same time we've also seen how customers are looking for robust infrastructure as it relates to driving timely data to decisions. The traditional approach which results in a 2 year data warehouse project followed by mountains of static reports emailed out every week just doesn't work anymore. So - it makes a whole lot of sense for us to combine with the team at TIBCO - allowing us integrate our user experience and next gen BI platform with absolutely the industry’s leading SOA infrastructure.

Together TIBCO and Spotfire will be able to provide an unprecedented user experience coupled with an infrastructure that can drive data in real time to that user experience. I believe this will disrupt the BI market as it stands. Think of portfolio managers, sales managers, or manufacturing managers – in fact, anyone! – who need to make critical decisions in a world drowning in data. Now they'll be able to interact with that data in a real time interactive experience configured for their type of work (their "business process")! Who else can do that for you?! (not so modest again!)

I am very excited as you might read into the above text. We will have a lot of fun continuing to build Spotfire as part of TIBCO. We will be the Spotfire business unit of TIBCO, preserving the Spotfire name - but will have great opportunities to increase our presence in the market, increase investments in R&D and product development, and of course, leverage the TIBCO product infrastructure to help you create an information advantage for your business.

Looking forward to seeing you in the market place!

So what a pleasure to see NYTimes being gutsy enough to not only try but also keep up an interactive display of the stock market. If you haven't seen already - definitly do so!

You will find it in the business section of the NYTimes

Recently the book Long Tail has gotten a lot of attention. I can clearly recommend it, great book and also short and elegant(!). The basic premise of the book is that in the past limited resources like shelf space, movie screens and tv channels, expensive resources like shipping and storage, and difficult tasks like finding willing buyers, and sellers, etc. have forced businesses to focus only on the top sellers in all categories or the vital 20% of the 80/20 rule.. The rise of cable television, the internet, , the browser, Fedex new technology for data storage, user access, networking, etc. - allowed innovators like Amazon, Netflix, and Ebay, to open up new markets to the 80% of customers and niches not served or the “long tail”. For example, an obscure movie from 1950 suddenly is available and can be a money maker - this is part of the attraction of market places such as Amazon and Netflix. This ability to not only serve the "long tail" of customer demand but also make it a very good business proposition is the premise of the Long tail.

When I read the book I clearly thought of how it related to analytics but really only thought about what kind of analytics we at Spotfire could do on the long tail itself. Then last week a long standing colleague of mine, Tim Loser, emailed me and suggested something quite clever. Basically he made the point that the actual long tail of analysis is quite interesting in itself. And he is right on!

Think about it - as I mentioned before certain kinds of analytics has gotten its own software - analytic application and that's great. I think IDC calls out some 600 companies doing analytic applications - but I also think it's fair to say that there are tens of thousands if not hundreds of thousands of variations of analytic applications. Just think of all the different Excel spreadsheets out there capturing various kinds of analytics - in your company alone I'm sure there are hundreds or thousands.

The obvious question is how has this [very] long tail of analysis been served to date?

Interesting observations can be done here: some times it's served by IT departments building applications serving business users. But only a few can obviously be served that way by traditional means. It's expensive, it's heavy - both in development and maintenance. I meet plenty of CIOs telling me “I’d love to provide better analytics to my business users but I’m drowning in maintenance of what I’ve already got”.

Another approach is where the tail is served by in-house analytics groups - they exist in many large companies. A similar approach is external consulting firms - everything from small one-man-bands to McKinsey helping out companies with analytically oriented work - delivering everything from glitzy Powerpoint presentations to highly configured Excel models. In fact this business is a very large business sector - and can be very profitable for the consultants, even though it's man power intensive, and not exactly scalable (or economic?) for the customer.

Now - if we assume that the long tail of books, films, music, etc. became available to customers with the arrival of new technologies - what happens with analytics with the arrival of new technologies? Shouldn’t we be able to fundamentally change the world of analytics here?

The first step is what we’d like to think we’re doing at Spotfire, by providing a platform that is extremely easy to use but also very to configure to a particular task, we can have one platform serving many many users across many many business problems. Good for end users, good for the CIO and the IT staff. Business analysis groups like [should like] it as it allows them to spread not just conclusions but also share methodology and approaches.

The next step is the really interesting. What are the economics of analytics in a digital world? One obvious one would be the “iTunes of analytics”. We have seen ManyEyes and Swivel (I vote for ManyEyes) as interesting examples of visualization of data on the web. They certainly have a lot of content, but the question is if they really help an analyst doing analysis? I think they do allow somebody to share an analysis and point to it – but they are still a ways to go to provide this in a form where you can find an analysis that can help you solve a particular problem, and also allows you to apply it to your own data, etc. A really attractive version of this certainly could be helpful to a lot of people – and truly be the iTunes of analytics.

Another obvious one in a world of digital analytics is how the role of the consultant changes. Instead of giving glitzy PowerPoint presentations with canned conclutions, taking questions, billing more hours till the next presentation, repeat, etc. Innovative consultants should provide analytic applications where customers can ask and answer their own questions and that can be reapplied time and time again. You may say that that would be too hard for them (programming, etc.) – but I don’t think that’s a valid excuse anymore – easy to use platform now exist. You may also say that the economics works against analytic applications – consulting firms make their money by doing the same PowerPoint presentations over and over again. I say “welcome to an outsourced and down staffed world”! The consultant who not only wants to survive but also move up-market will move from PowerPoint to providing reusable analytics. A consultant who wants to open up the long tail analytics market will provide reusable analytics.

What else changes in a world of digital analytics? What does IT need to do differently, even with a great platform, to serve the tail? Should a company trying to gain a competitive advantage (Competing on analytics) focus on the head or the tail – easy to say the head, but is it in the tail the difference really is made? Is there a new class of knowledge worker who’s job it is to orchestrate how a company manages its tail of analytics? Is there a role for academia in this? What’s the supply chain of analytics that allows analytics and smarts, not just data, to be shared across companies – from Walmart to P&G, from P & G to the suppliers and back?

He calls out three key themes "Simple, Free Platforms for Self-Expression", "Emergent Structures, Rather than Imposed Ones", and "Order from Chaos" - which very much are similar to the themes I think are behind Business Intelligence 2.0. Classic IT has been all about imposing Command and Control structures through good governance, and now we can build upon that good governance to enable similar freedom of expression (within boundaries of good governance!) of YouTubeand Facebook in Business Intelligence 2.0.

Read Andrew's article, very good!

Not everybody was thrilled - "BI1.0 fan" wrote 'I think the last thing the world needs is something else slugged with "2.0" at the end of it.' Fair comment at some level - but you know what, I think the world of business intelligence is ready for something new [in an upcoming post I'll compare notes with some other definitions of what's to come]. Reporting and data warehousing is great, serves many purposes, and most importantly, has put in place the platform for us all to go to the next level. But we can do more than distribute reports.

Now, BI2.0 - people centric Business Intelligence - is only really possible once the infrastructure of warehouses, user modelling, and security is put in place. This also means that those who have built out that infrastructure - in many cases great IT departments out there - will be continue to be incredibly important going forward.

The great thing about BI2.0 is that it gives people the power to create and be creative while at the same time reduces the repetitive and time consuming constant changes to reports and OLAP cubes, that IT doesn't want to spend its scarce time and resources doing anyway. IT and the professionals they serve both win! IT information management resources can be more concentrated on ensuring information is correct, scalable, secure, and well-organized; and on enabling information workflows that are configured for specific people or business challenges; all of which are far more important than formatting reports and running them again when someone thinks of a question they didn't ask!

Think about it this way - other social/people oriented platforms - like YouTube, MySpace, Facebook, and SecondLife - which are all about user created content and experiences, *only* work because they run on great platforms that are carefully constructed and maintained.

Technorati Profile

James Taylor talks about some nice angles on next generation BI in his enterprise decision management blog, similarly Charles Nichols has laid out some good ideas in
a recent article in DM Review.

Both James and Charles talks about the importance of events, real time, automation, and action without intervention. I think these points are all very important - and it signifies an interesting fork in the road for thinking about business intelligence and analytics.

Certainly it will be very important and compelling to build systems that takes action on streams of events and information in real time - no question.

Now - I [and for that sake Spotfire!] wanted to align next generation thinking in BI [BI2.0!] with the great ideas coming out of the whole Web2.0 wave

"Web 2.0, a phrase coined by O'Reilly Media in 2004[1], refers to a supposed second generation of Internet-based services—such as social networking sites, wikis, communication tools, and folksonomies—that emphasize online collaboration and sharing among users." (from Wikipedia definition.)

The focus is on people, communication, collaboration, sharing - rather than events and automation. The one is not better or more important than the other, that's not the point.

But - I think the most exciting idea for the millions of people out there in corporate world [and elsewhere!] who depend on information to make complex decisions that are beyond automation and process automation - is the very idea that they can create and be creative themselves around information - just like consumers can create and be creative in Facebook and SecondLife.

I would like to argue that this super focus on the data and ITs ability to deliver one version of the truth is getting in the way of letting people out there use their smarts (!) applied to company data to create new, clever, intelligent, creative approaches to how to solve business problems.

YouTube, MySpace, Facebook, and other user created spaces are great because they allow anybody to pull together their views, ideas, and creativity into content that can quickly be shared and disseminated, without (!) top down supervision. This has been so revolutionary that people call it Web2.0. Of course there are challenges in terms of copyright, etc. - but it certainly hasn't stopped major corporations like News Corp and Google from adopting the approaches. We just recently learned that the intelligence community now runs a Wikipedia version called Intellipedia, completely bottoms up created.

So - what does this mean for Business Intelligence?! Is it time to go beyond IT controlled and disseminated reports, data warehouses painstakingly put together, and predefined queries?

Is is time to declare BI2.0? A world of BI where the focus is on People not data! People not IT! It's about People being able to create and share! It's about People being able to talk to their data and have it talk back!

A new world with user driven creation and creativity (like MySpace, YouTube, and Facebook!) rather than IT driven information control and delivery.

It was very good hear him say "we have now put in place all the infrastructure, the core applications, the databases, the reporting, the portals, etc. Now it's about what we can do with the data generated from these systems. That's how we will differentiate ourselves".

Have you gotten to that insight yet? It's about how you take advantage of data in your systems (or from your supply chain, partners, customers, vendors, etc.) that you will stand out. Not by your SAP implementation that everybody else also have.

Basically a nice idea: "A mashup is a website or web application that seamlessly combines content from more than one source into an integrated experience."

Applying this to analytics is brilliant. Think about it - in analytics the whole idea is to take data from all kinds of sources, pull it together, and out comes brilliant insights. Easier said than done?

Google Finance has been noted by many for the nice widgets for showing stock performance. The really neat part is how it interelates stock performance with news releases - taking two different data sources and putting them in a very nicely pulled together user interface that allows for rapid cross referenceing.

Now, the challenge is that is that pulling together data from many sources, putting it in one experience, provide a great set of user interfaces to it, and let you (or anybody!) go explore for insights is probably easier said than done using traditional manners. Do you really want to go off programming for this?

I'd like to challenge you to think about how we can help you there - building Google Finance like experiences for your data and business challenges. How quickly can we help you build the experience? Put us to the test.

http://www.spotfire.com/spotfire_downloads/case_studies/EastportCS.PDF

Ah well - in all seriousness it's pretty exciting - IRS using information visualization technology to look for tax evasion activity associated with offshore credit
cards, e.g. not the average person activities. And it's great to see the Federal Government to step up into taking advantage of new ways of dealing with large amounts of information.

The customer provocatively (in a great way) asked me "how early do you start train the future Spotfire user generation?". I always thought about the university programs we run serving this purpose - but she kicked back and said "you need to start earlier, start with 15 year olds - they need to be prepped for a world of looking at data holistically".

And she's right. Perhaps it's already back in school things go wrong with us all looking at problems in silo orientation - not looking at enough angles and not turning the data stones upside down to look for the unexpected to be able to make break-throughs. Question is what's the high value/interest data that 15 year olds will get all excited about. Fantasy football? Probably not their pension portfolio anyway...

Let me use an example to prove my point. There's an awesome dataset out there of 65,000M records (yes, a few more than the 65,000 rows possible to load into Excel) and probably some 20-30 columns of data. It has both numbers, text, and imagery associated with it, and has all kinds of smart algorithms, graphs, etc.

Not only this - it's used by many many people. Way more than any "statistics package" - and by literally any type of user.

And not only this - they use it to create a competitive advantage against both their counterparty in high value transactions, but also to minimize the value of the middle man in complex negotiations

Ok enough of guessing - who am I talking about? Zillow of course! Great real estate site - if you haven't already, you must try it http://www.zillow.com. Literally arming anybody to do quite sophisticated analytics. And why does it work? Because of awesome user interface design and visualization - that allows anybody to use it. Not just to browse data, but to actually become the smarter party in a really important transcation - competing on analytics!

So please, forget this silly argument about analytics being only for a few people. It's a pitiful excuse. Now think about if you had the tools to design/build Zillow-like experiences in minutes and hours rather than months and years? For anybody in your organization, real fast. Wouldn't that be game changing?