Donald Farmer: Foraging in the Data Forest

Main | July 2006 »

June 26, 2006

Too strange ...

I think I am pretty much inured to the strangeness of the internet. But every now and then you come across something so peculiar that it defies accurate description ... or at least accurate description that anyone would believe.

This is quite suitable for the fainthearted, so do not worry on that account. But after this, neither exercise videos or language lessons will be the same again ...

Enjoy!

Posted by Donald Farmer at 11:30 PM | Comments (0)

June 11, 2006

Customers and users

One aim of this blog will be to discuss choices facing designers of data integration software. It's easy to talk about "knowing the customer" when developing a product, and naturally every designer works hard to do this. Nevertheless, now and then I find myself refocusing on an aspect of our market position that is easy to overlook - the distinction between users and customers. This week, prompted by various happenstances, I have been doing just that.

The distinction can be summarized simply enough. Customers are those who make purchasing decisions about which software their enterprise will use: users work with the software to implement solutions. In many cases these may be the same people, acting in different roles from day to day. In other circumstances - in many, I would hope - users have huge influence in purchasing decisions. Nevertheless, there are numerous enterprises where decisions are made top-down and users just have to get on with it: and, of course, outsourcing is an increasing trend where the user may well have little or no say in the choice of toolset.

As product designers we therefore face two issues: we must appeal to customers in order to be selected as the tool or platform of choice, and we must meet the needs of users to make that choice work. Let's start with users, and in another post I'll cover how we address our customers.

Here at Microsoft, the development teams are encouraged to engage very directly with our community of users, partly through online forums where the public can post any issue for us to help out with. I was applying some business intelligence to the forum usage statistics, in particular to analyze activity on our Integration Services forum. That was quite an eye-opener. We had almost 1000 active members in the last financial quarter, and almost 10000 posts. At around 100 posts a day, that places us among the most active forums at Microsoft, coming in just after programming languages such as C#. That's pretty remarkable for a data integration application and it tells me two things, at least.

First of all, and simply, we clearly have a lot of users out there, actively exercising the product and asking questions. Scaling design issues to cover such large numbers is a unique, and stimulating, aspect of designing products at MS. It is also difficult, because it requires great depth of focus - on the needs, and feedback, of individual users, while also addressing the general case of thousands of slightly differing scenarios.

For example, we released a little shared-source metadata toolkit to the web. Within a week it had around 1000 downloads. The program manager was delighted to see 10x more instances of his work in the wild in one week than he had seen at his previous company in years. But our unit manager put it in context. Her background was working on the WebData data access stack for Microsoft, where she saw 1000 downloads per day for new releases. Working at that scale - well, it's a different world.

Secondofly (love that word!), we should consider who these thousands of users are. That's not too difficult to work out - they are by and large developers, active in the same fields as the C# or VisualBasic developers whose forums we most resemble. The market segment we see reflected in our online community is a demand for data integration as a general developer activity. Not data integration for Business Intelligence specifically (although we are being successful there too) but data integration as something that everyone who works with data has to do. This makes a lot of sense - we all have a ton of data, but it is rarely in the format, or of the quality, or as integrated with other relevant data, as we would like. We talk often of BI for the masses. Well, this is data integration for the masses, and it is clear that the masses need it.

In fact, Colin White's detailed report on data integration for TDWI showed that custom coding is still a most significant solution to data integration problems. Living as we do, at the confluence of hand-coded development and code-free applications, this is fertile ground for us.

In another post, I'll write about customers and the demands they make of us - and how we strive to resolve those with the demands of users. Meanwhile, I would be delighted to red any comments you may have on the distinction I have been making here - in my experience, It's a somewhat neglected area.

Posted by Donald Farmer at 2:45 PM | Comments (0)

June 4, 2006

Introductions - or, who is this guy and why is he blogging?

I'm terrible at introductions.

In formal business, Donald Farmer, Group Program Manager for Microsoft SQL Server Integration Services says all that needs to be said for many people. I am just what my name card says, and there's no need to know any more. I can rarely be so reserved. Customers and partners soon learn more than they need to know about my travels, my garden, my trips to the supermarket for shampoo, and so on.
�
Thankfully, blogs, even when about business, are relaxed. B-EYE blogs in particular are more like an evening event at TDWI, albeit without the drinks and nibbles - though you could supply your own whilst reading. There are good friends here, and colleagues, and other vendors, and - although we're mostly talking shop about business intelligence - we can also catch up on recent events, or share some stories and smiles. In that setting, I'm not just a program manager at Microsoft. I'm husband of a gifted artist, and proud father of my son. My childhood in the Scottish highlands and islands is as likely to sidetrack the discussion as my practices in data warehousing. My passions for food, wine, enlightenment philosophy, medieval history, modern art or wildflowers are side-by-side with a restless curiosity about information quality, performance tuning, data modeling, or unstructured data.
�
So, here in my B-EYE blog, don't expect formality. Chances are, I'll be writing this with a glass of wine to hand. Often I'll be writing at the end of a long day of difficult design decisions or working through customer issues. On other days, I'll just have put down a new book of poetry, or have come in from the garden, and those will be bubbling in my mind.
�
Nevertheless, I will be writing mostly about business intelligence: more particularly about data integration and the daily work involved in delivering a data integration platform to market. I trust I will give some insight to the decisions we have to make as platform designers and developers: how we make those choices, how we prioritize and resolve and explain and rethink: how we look at the market: how we emerge into the daylight when a product has shipped to find real people using it to solve real problems, often in ways we never intended.

I am anxious that it should be interesting - but I know I'm among friends, and I should just relax and get blogging.

Posted by Donald Farmer at 2:45 PM | Comments (1)