« August 2007 | Main
September 20, 2008
PC? Me?
Looking back over the last week, two main topics have dominated my conversations.
Firstly, that strange campaign, where an unlikely pair awkwardly and unconvincingly pretend to relate to ordinary people, leaving many incredulous and some amused. No, not McCain-Palin ye jaundiced readers; I refer to the Microsoft commercials with Bill Gates and Jerry Seinfeld, and the subsequent “I’m a PC” slots. As an exercise in branding the Microsoft campaign certainly had one important effect – everyone was talking about it.
Secondofly, within Microsoft, more specifically within the SQL Server Analysis Services team, we have been much focused on the annual company meeting. I guess this also revolves around Microsoft’s sense of identity. My my, haven’t I been the corporate guy this week?
It still feels strange for me to part of something as big as Microsoft. When I joined the Analysis Services team in 2001, “Picasso,” as we called it, numbered around 50 people: quite the largest organization for which I had ever worked. Now we must be twice that size and we fit into an organization that has expanded at every level: within SQL Server Business Intelligence, SQL Server itself, the Server and Tools Business and Microsoft itself. Nevertheless, despite our huge customer base, and the cautious processes that surround so much of our work, “Picasso” on our best days still feels like a start-up. We still generate that edgy excitement within the team and, with some of our recent work, in the wider company too.
This was why we focused so much on the company meeting this week. We had the opportunity to demonstrate our latest innovations to the entire company; at least the 20,000 or so gathered in Safeco Field for the event. When I say “we” demonstrated, I should clarify - our Senior VP Bob Muglia actually did the demonstration. Some people really fret about doing demonstrations, especially in front of large crowds. Some people would be rigid in fear if they had to show their barely coded software, not even at beta 1, in front of 2,000 people, never mind 20,000. Believe me: having your Senior VP do the demo is even more nerve-wracking. On the day, he did a great job, to resounding applause from the crowd, and to the delight and relief of our team. Even Steve Ballmer called it out as one of the highlights of the meeting.
So what are we working on, that gets such attention? Needless to say, “it” is something incredibly cool. Pete Loshin will buy Vista just to be able to run it. More detail? I hate to be a tease, but at this point, I cannot say, because we are not talking about it publicly yet. No I will not even whisper it – I am being corporate guy this week, remember? We have to manage the PR, the image, and the branding – all that stuff that we are getting newly engaged with in our new campaigns.
I make fun of being the corporate guy, but I do feel quite torn about it. I did not go to the company meeting, but stayed back in the office and watched it on video. I am personally very uncomfortable with being in a large crowd with such a directed motivation. The techniques of bringing a common impetus to such a mass quite disturb me, however wholesome the intent. When the attempt fails and falls flat, it can be terribly awkward; yet, when the attempt succeeds it makes me uneasy to see how willingly we can aggregate our experiences and ourselves.
Do not think that I am not proud of working at Microsoft. The very reason I am there is because I admired the company’s drive and its unmatched ability to bring the benefits that I enjoy from technology to countless others.
In that spirit, I rather like our new “I’m a PC” commercials. Of course, some segments are a little precious, and some a little corny. Still, they do capture the sheer diversity of the consumers for which we cater: the people we have to think about when we are designing, developing, testing, supporting … yes, and marketing.
It is an exciting time to be doing what I am doing. Keep an eye on the Microsoft BI Conference in October – it will be fun to talk more about this after that even, when we will be showing our new work to you too. There will not be 20,000 at that event – we could only wish! – but the impact will be felt very much wider.
Posted by Donald Farmer at 10:30 PM
September 7, 2008
The Cloud of Unknowing - experimenting with predictive analytics on the web
The Cloud of Unknowing is a quite beautiful mystical text from 13th century England. It emphasizes the importance of direct experience over intellectual knowledge. I often think about it when considering the complexities of implementing successful predictive analytics. My customers often have an almost mystical goal of foreknowledge and insight, but find themselves caught up with technical details of model building and validation.
Daunted by Quants? Flaunt this taunt.
All this can be overwhelming for the non-technical user trying to get to grips with the technology. But equally, when technical specialists are brought in, the business sponsor can feel out blinded with science. The analytic centre of excellence can quickly be seen as the centre of arrogance.
The problem here is that no amount of testing, validation or statistical reflection can compensate for lack of business understanding. There was a great article in the New York Times technology blog about the failure of technology in the sub prime crisis. Jim Goodnight, the CEO of SAS, put it succinctly when describing the role of analysts in the crisis: It was the quants, and they were using our software. But they didn’t understand the underlying vehicle when computing the risk.
My personal goal for Microsoft's Data Mining technology, and the goal which the team has been pursuing long before I joined, is simple: to put predictive analysis in the hands of people who do understand the business. To help clear the cloud of unknowing.
Of course, the specialist will have a ready reprimand for this approach. Surely, there is a danger of the user creating the wrong model? Well, perhaps that is a danger, but as George Box once put it All models are wrong, but some are useful. In fact, Box went further: The practical question is how wrong do they have to be to not be useful. The answer to that question is often not to be found in the charts and stats of the quants, but in the direct experience of the savvy business user.
Joining the cloud crowd
Even a quick survey of the predictive analytics market will show that its not only the traditional need for specialists that is a barrier to adoption. The cost of software is a significant factor too. (There is an excellent TDWI survey, certainly not done quickly, by Wayne Eckerson here.)
At Microsoft, as so often, we're trying to bring capabilities well within the reach of all users. Data mining has been part of the SQL Server product since 2000. Nevertheless, there is still a barrier to just experimenting with predictive analytics. Even our very successful Data Mining Add-ins for Excel need a server to host the modelling technology.
So here's what's new - and it really is quite unique, a lot of fun, and useful too. One of our developers, Bogdan Crivat, has been working over the Summer on an incubation - to adapt our simple Table Analysis data mining tools for hosting on the cloud. This is now available for anyone to try - no server infrastructure required, just a plain old web browser. (Yes Shawn Rogers, you could try this even in Chrome.) You can try it here.
You can use our sample data, or upload your own *.csv file. Folks have been having a lot of fun with this. For example Brent Ozar blogged enthusiastically and in helpful detail here.
Bogdan has done a super job of this experimental version. And, of course, it is important to emphasise that this is still an experiment at a very early stage. There are plenty of issues to be sorted out before this would be available as a full cloud service.
Yet I am sure there is a future here - enabling users to try before they buy at the very least. Another possibility is that users with only occasional need for predictive analytics - preparing a semi-annual campaign, or analyzing survey results - will love the ability to experiment and perhaps even pay ad-hoc for a simple analytic service.
On premises or on promises?
One of the issues of which I am very aware regarding cloud services is whether businesses are really willing to rely on them for mission-critical applications. I call this the choice between on-premises and on-promises. Once you choose the cloud, you are really in the hands of your service provider and that can be a difficult choice to make for risk-averse executives.
Of course, for these users I still see us delivering server software and powerful desktop clients. But do try this first foray into the cloud. I am sure you will be intrigued.
Feel free to follow me on Twitter.
Technorati Tags: data mining predictive analytics microsoft donald farmer cloud computing
Posted by Donald Farmer at 7:45 PM
September 2, 2008
Catching up on recent news
I have not blogged for some time. Not that I had given up, I just found little time to craft posts that I thought were interesting enough to share. So this is a very overdue post.
Why get back on the horse now? That's simple to say. Over the last few weeks my mailbox has been full of questions about recent developments in BI and DW at Microsoft - so a new blog post seemed like a great way to provide a generic answer to the many questioners, especially where they were just asking for opinions rather than hard facts.
For those of you who prefer audio, I have cast my pod twice recently with the fine folks at the B-Eye-Network: one interview with Jill Dyché and one with Colin White. Both podcasts discuss recent events at Microsoft. As always, my views are not Microsoft's official position. They are my own, and often enough not even shared by both sides of my brain at once.
So what has been happening in Redmond that spurred all the questions? For those who need a reminder, here at least 4 issues.
The acquisition of DatAllegro
The acquisition of Zoomix
The release of SQL Server 2008
Bill Baker's leaving Microsoft
I'll be brief on each of these here, but do listen to the podcasts for more.
DatAllegro.
This acquisition made a big splash, if only because so many people had been eyeing the accelerator and appliance market waiting for the first signs of consolidation with BI or RDBMS vendors. However, although MS is first in the fray here, we have not really bought either an accelerator or an appliance - we have bought a significant step forward on our roadmap to greater scalability for SQL Server. The SQL Server team have been continuously improving scalability over the years - with some very effective case studies and proof points. DatAllegro just moves us effectively and efficiently along.
Two points have really got the BI bloggers chatting. The first is the sum Microsoft paid. (At this point, my legal rep is sweating and readying an email to donald.farmer reminding me that I cannot discuss this.) I cannot discuss this. But really, in a month or two this is a footnote. The most important thing about the price is how little it matters in the big picture. The second point the bloggers have enjoyed discussing is how long it will take Microsoft to integrate the DatAllegro codebase, while migrating it away from its open source roots. Again, I can't discuss in detail, but I can tell you that we bought a shortcut to massive scale implementations and a shortcut it will be.
It's a pretty exciting prospect: we'll be playing in a fascinating space. Other teams - management tools, ETL, reporting - also have challenges arising from this shortcut; we now need to ensure other elements of the stack are ready for the massively scaled deployments that we will support. In truth, there will be fewer problems here than you might expect. Our ETL product, SQL Server Integration Services, has already set a world record for ETL. See: http://tinyurl.com/5olqc6. Our Reporting Services in 2008 handles huge data volumes very effectively.
Nevertheless, it is interesting to me that few bloggers picked up on these issues at all: there was a very narrow view of the DatAllegro solution itself, rather than a broader consideration of how this would fit into the wider infrastructure of DW, reporting, performance management etc. This is pretty typical of the way in which we discuss the appliance and accelerator market as an industry: we tend to look only at the implications of massive scale for the database, without considering how the data is to consumed practically and efficiently.
Those of you follow the various BI blogs will of course be aware that there is actually a third issue that popped up around the DatAllegro acquisition - a legal action about some IP. Now that is something I surely cannot discuss - and I know nothing about it anyway. I'll say only one thing: my first reaction was simply "Here we go again." You may be surprised at how often this sort of thing happens, at MS and any other company with deep pockets. It's a bore, but it keeps lawyers busy and well-paid. (Which I like, because one of my attorney friends throws the best barbeques and wine-tastings.)
Zoomix.
I love Zoomix. They are a great wee Israeli company who addressed the problems of data quality in a new way, and did an awesome job of it. They were deservedly a Gartner cool vendor - if they were any cooler they could have solved global warming. I love the self learning capabilities and they have smart smart people on board. I first proposed the Zoomix acquisition, so I'm doubly pleased to see it happen. I moved teams meanwhile after that first proposition, and my old Microsoft team in Integration Services completed the acquisition. Great to see that happen, and great work by them to see the acquisition through - and I'm looking forward to being able to work more and more with a great data quality stack from Microsoft. And for sure I'll be writing a lot more about this technology in the future.
SQL Server 2008.
We shipped! In fact, we have our ship party this Friday. It's always a great feeling. Listen to my podcast for details of some of the great features: great end-user reporting, data profiling, best practices alerts for OLAP design etc.
The question I have had in my mailbox repeatedly about the release is: didn't you guys launch in February? Well, yes we did. That was the marketing launch of the product along with the other big releases of 2008: Windows Server 2008 and Visual Studio. It made a lot of sense to roll all these big releases into a single (and very successful) worldwide roadshow. I am not in marketing, but I did appreciate the three-in-one launch when I was presenting on the roadshow. It made a big juicy story and gave an opportunity to tell a very integrated and compelling message to IT, developers and BI alike.
Bill Baker
Today is Bill Baker's last official day at Microsoft. Those of you who know me, or who have read my blog, know how much I admire Bill. I really owe a great deal to him, as do many others in Microsoft. He'll be sorely missed for sure, although he does leave us in a good state to carry on what he started when he first came to MS.
There have been many good things said about Bill over the weeks since he announced his departure. Aside from the personal friendship and mentoring, I would like to mention just two things that I think Bill did that made a huge difference.
Firstly, Bill really understands the BI industry - the customers, the vendors, the consultants and, of course, the technology. He is passionate about Business Intelligence, in a completely non-partisan manner. Any customer who speaks with him, comes away fired up with enthusiasm for how they can transform their business with this technology. Bill has a real feeling for this transformative power of and he can talk with anyone from accountants to zookeepers about specific, actionable and achievable steps they can take to improve their decision-making.
Secondly, Bill had a real knack for building a community of users. At any major Microsoft conference, the Business Intelligence community is present and has a genuine sense of shared purpose, and we enjoy a lot of fun together. When you look at the growth of Microsoft BI over the years, one of the most significant factors is how much of that growth was driven by customers who are new to BI. These new customers learned BI on MS technologies and built a strong and common experience together. Fortunately, one thing at which Bill also excelled was "making others great." The result is that, the MS BI community will be largely self-sustaining and I expect it to be sparky and vibrant for a long time to come.
So, thanks Bill, for all you have done here. I'm really looking forward to what you do next at Visible Technologies - they are now, more than ever, a name to watch.
Finaly, if you are interested, you can follow my current adventures on Twitter. I am enjoying micro-blogging, and for sure I'll be doing more macro-blogging too!
Posted by Donald Farmer at 1:15 PM | Comments (1)
