September 2, 2008
Catching up on recent news
I have not blogged for some time. Not that I had given up, I just found little time to craft posts that I thought were interesting enough to share. So this is a very overdue post.
Why get back on the horse now? That's simple to say. Over the last few weeks my mailbox has been full of questions about recent developments in BI and DW at Microsoft - so a new blog post seemed like a great way to provide a generic answer to the many questioners, especially where they were just asking for opinions rather than hard facts.
For those of you who prefer audio, I have cast my pod twice recently with the fine folks at the B-Eye-Network: one interview with Jill Dyché and one with Colin White. Both podcasts discuss recent events at Microsoft. As always, my views are not Microsoft's official position. They are my own, and often enough not even shared by both sides of my brain at once.
So what has been happening in Redmond that spurred all the questions? For those who need a reminder, here at least 4 issues.
The acquisition of DatAllegro
The acquisition of Zoomix
The release of SQL Server 2008
Bill Baker's leaving Microsoft
I'll be brief on each of these here, but do listen to the podcasts for more.
DatAllegro.
This acquisition made a big splash, if only because so many people had been eyeing the accelerator and appliance market waiting for the first signs of consolidation with BI or RDBMS vendors. However, although MS is first in the fray here, we have not really bought either an accelerator or an appliance - we have bought a significant step forward on our roadmap to greater scalability for SQL Server. The SQL Server team have been continuously improving scalability over the years - with some very effective case studies and proof points. DatAllegro just moves us effectively and efficiently along.
Two points have really got the BI bloggers chatting. The first is the sum Microsoft paid. (At this point, my legal rep is sweating and readying an email to donald.farmer reminding me that I cannot discuss this.) I cannot discuss this. But really, in a month or two this is a footnote. The most important thing about the price is how little it matters in the big picture. The second point the bloggers have enjoyed discussing is how long it will take Microsoft to integrate the DatAllegro codebase, while migrating it away from its open source roots. Again, I can't discuss in detail, but I can tell you that we bought a shortcut to massive scale implementations and a shortcut it will be.
It's a pretty exciting prospect: we'll be playing in a fascinating space. Other teams - management tools, ETL, reporting - also have challenges arising from this shortcut; we now need to ensure other elements of the stack are ready for the massively scaled deployments that we will support. In truth, there will be fewer problems here than you might expect. Our ETL product, SQL Server Integration Services, has already set a world record for ETL. See: http://tinyurl.com/5olqc6. Our Reporting Services in 2008 handles huge data volumes very effectively.
Nevertheless, it is interesting to me that few bloggers picked up on these issues at all: there was a very narrow view of the DatAllegro solution itself, rather than a broader consideration of how this would fit into the wider infrastructure of DW, reporting, performance management etc. This is pretty typical of the way in which we discuss the appliance and accelerator market as an industry: we tend to look only at the implications of massive scale for the database, without considering how the data is to consumed practically and efficiently.
Those of you follow the various BI blogs will of course be aware that there is actually a third issue that popped up around the DatAllegro acquisition - a legal action about some IP. Now that is something I surely cannot discuss - and I know nothing about it anyway. I'll say only one thing: my first reaction was simply "Here we go again." You may be surprised at how often this sort of thing happens, at MS and any other company with deep pockets. It's a bore, but it keeps lawyers busy and well-paid. (Which I like, because one of my attorney friends throws the best barbeques and wine-tastings.)
Zoomix.
I love Zoomix. They are a great wee Israeli company who addressed the problems of data quality in a new way, and did an awesome job of it. They were deservedly a Gartner cool vendor - if they were any cooler they could have solved global warming. I love the self learning capabilities and they have smart smart people on board. I first proposed the Zoomix acquisition, so I'm doubly pleased to see it happen. I moved teams meanwhile after that first proposition, and my old Microsoft team in Integration Services completed the acquisition. Great to see that happen, and great work by them to see the acquisition through - and I'm looking forward to being able to work more and more with a great data quality stack from Microsoft. And for sure I'll be writing a lot more about this technology in the future.
SQL Server 2008.
We shipped! In fact, we have our ship party this Friday. It's always a great feeling. Listen to my podcast for details of some of the great features: great end-user reporting, data profiling, best practices alerts for OLAP design etc.
The question I have had in my mailbox repeatedly about the release is: didn't you guys launch in February? Well, yes we did. That was the marketing launch of the product along with the other big releases of 2008: Windows Server 2008 and Visual Studio. It made a lot of sense to roll all these big releases into a single (and very successful) worldwide roadshow. I am not in marketing, but I did appreciate the three-in-one launch when I was presenting on the roadshow. It made a big juicy story and gave an opportunity to tell a very integrated and compelling message to IT, developers and BI alike.
Bill Baker
Today is Bill Baker's last official day at Microsoft. Those of you who know me, or who have read my blog, know how much I admire Bill. I really owe a great deal to him, as do many others in Microsoft. He'll be sorely missed for sure, although he does leave us in a good state to carry on what he started when he first came to MS.
There have been many good things said about Bill over the weeks since he announced his departure. Aside from the personal friendship and mentoring, I would like to mention just two things that I think Bill did that made a huge difference.
Firstly, Bill really understands the BI industry - the customers, the vendors, the consultants and, of course, the technology. He is passionate about Business Intelligence, in a completely non-partisan manner. Any customer who speaks with him, comes away fired up with enthusiasm for how they can transform their business with this technology. Bill has a real feeling for this transformative power of and he can talk with anyone from accountants to zookeepers about specific, actionable and achievable steps they can take to improve their decision-making.
Secondly, Bill had a real knack for building a community of users. At any major Microsoft conference, the Business Intelligence community is present and has a genuine sense of shared purpose, and we enjoy a lot of fun together. When you look at the growth of Microsoft BI over the years, one of the most significant factors is how much of that growth was driven by customers who are new to BI. These new customers learned BI on MS technologies and built a strong and common experience together. Fortunately, one thing at which Bill also excelled was "making others great." The result is that, the MS BI community will be largely self-sustaining and I expect it to be sparky and vibrant for a long time to come.
So, thanks Bill, for all you have done here. I'm really looking forward to what you do next at Visible Technologies - they are now, more than ever, a name to watch.
Finaly, if you are interested, you can follow my current adventures on Twitter. I am enjoying micro-blogging, and for sure I'll be doing more macro-blogging too!
Posted by Donald Farmer at 1:15 PM | Comments (1)
August 27, 2007
I'm biased. And so are you.
Earlier this year, I changed teams and moved offices within Microsoft. This interrupted a little habit I had developed: pinning up my “Cognitive Bias of the Week” outside my office.
Cognitive biases are somewhat like optical illusions, but they affect our thinking rather than our vision. A well known example is confirmation bias; we tend to give more weight to positive observations that confirm our beliefs rather than negative observations. Fortune-tellers may appear successful when people remember one or two correct predictions more readily than the many that were off the mark.
Of course, you wouldn’t make such an error, would you? Think again. Like an optical illusion, many biases are extremely difficult to shake even when you are aware of the effect. In fact, some biases are most effective when we try to think most logically.
I believe it’s important for those of in the BI world to understand these biases. We represent data and analytic conclusions in highly persuasive ways. We help our customers to get it right or to get it wrong - and at times our influence may be inadvertently malign. With that in mind, I’m going to translate my “Cognitive Bias of the Week” posters to occasional blog posts on particular biases. I hope you’ll find these interesting, and relevant. Let me know.
Here’s one to start with. It’s about risk, and it has some revealing insights into how we consider the impact of risk in our decisions. It’s often called “The Pseudocertainty Effect” and it was first examined by Tversky and Kahneman.
Imagine that the US is at risk from a new disease spreading from Asia. Without treatment, it will kill 600 people, but we have two treatments to choose from.
• With Program A, 200 people will certainly live.
• With Program B there is a 1/3 probability that all 600 people will leave. However, there is also a 2/3 probability that they will all die.
Program A is positive – you’re certainly going to save some people. Program B potentially has a better outcome, but it is way less than certain. What treatment program do you recommend?
In the original study, 72% recommended Program A, and only 28% preferred Program B.
Let’s flip the problem round.
• With Program A, 400 people will certainly die.
• With Program B there is a 1/3 probability that no-one will die. However, there is also a 2/3 probability that all 600 people will die.
Now, Program A is negative: 400 people will certainly die. Program B is still uncertain: there is a risk it will all go wrong. However, if you do nothing 600 will die anyway, and if you follow Program A, 400 will certainly die. With Program B you have a chance of saving everyone. In the original study, when presented in this way to a different sample, 78% chose Program B.
That’s pretty remarkable. Exactly the same choices, presented in a different way, led to a complete inversion of preferences.
From this example, you can perhaps see why I consider cognitive biases to be an important study for BI analysts and developers. We may think of ourselves, or our users, as super-rational objective analysts of complex data; but in reality we are subject to these same biases. Also, we will tend to fall back on these biases, shortcuts and heuristics when we are making decisions under stress.
As BI becomes ever more pervasive, emergency planners probably would use our tools and techniques to handle an epidemic. But we could also be discussing customer churn rather than a deadly disease. The specific KPIs we choose, the manner in which we present them – the ways in which they influence decisions may be subtle, but the impact can be dramatic.
I’ll try to keep up a regular posting of biases, with examples relevant to the BI world.
Posted by Donald Farmer at 11:32 AM | Comments (2)
August 12, 2007
Data visualization - in a music video
Not quite BI, but how often do I get the chance to post a link to a data visualization music video?
If you think you recognize the music, you're probably right. It's playing in the background of the Geico caveman advert when he's on the moving walkway in the airport.
My colleague Olivier Matrat points out that the video production is by a French design firm H5 who also made this excellent visualization for a nuclear services company.
Enjoy.
Share:
Posted by Donald Farmer at 11:22 AM | Comments (0)
