« The CEO's perspective: why companies are choosing expressor | Main | American Tower turbocharges complex data warehousing application with expressor »
October 15, 2009
Applying parallel concepts to application development
I believe that the answer to the question posed on our LinkedIn group recently ('Should business users be involved in data integration development?') is that enabling business users to be involved in the data integration process is a fundamental requirement in enabling information management to respond to the building demands of the business.
Over the past several years we've all seen a dramatic increase in the volumes of data we've had to process. From a technology point of view, we've largely risen to the challenge by using parallel processing techniques such as pipeline and data partitioning parallelism along with advances in the hardware and algorithms used by these parallel processes. This has allowed us to perform more of the activities in parallel faster.
Unfortunately our businesses needs have also grown at compounding speeds, requiring information management to deliver solutions faster, cheaper, more reliably but at the same time make them nimble enough to respond to the accelerated rate of change in the business.
The ideas we've implemented on the technology side aren't completely radical since even early ETL tools recommended 'chunking' an application to make them more manageable - unfortunately with most of the early tools we needed land the data and then manage a ever growing array of small 'jobs' and interfaces that comprised an application - typically without providing any easy way to manage the menagerie.
We seem to be positioned to take a similar breakthrough step with respect to the business side of application development that we took for the technical side. It's not unusual for advances in the development side to follow advances in the technical side - in fact, it's more the rule than the exception. By allowing multiple contributors to work on the application in parallel and facilitating an efficient communication mechanism between them we are effectively introducing the parallel processing techniques we just discussed to the development environment.
This approach is not entirely new - after all we've had modelers managing the logical view of our information through modeling applications and business users creating Word documents and spreadsheets with specifications, but there was significant impedance between the operations and communication vehicles since they were forced to use different terminologies, constantly convert the definitions that were used between the physical world and the business world, and there was limited interoperability between these tools.
Role-based development, and in particular business users building and managing the 'business rules,' allows us to potentially separate the activities involved in managing these 'business rules' from the activities associated with moving the data (data flow programming). This allows users most familiar with specific aspects of the development cycle to manage and contribute the portions of the application that they are most equipped to work on - the business side can build, test, and manage certain business rules while the developer can build, test, and manage the data movement and handling aspects of the application.
Of course the interfaces that each of these contributor's uses will pose different requirements and some of the requirements will be common to multiple contributors - they must balance providing the appropriate amount of functionality while managing the complexity for each contributor, while allowing interaction in the event the scope of a particular activity extends beyond the contributor's capability. This is much like our technology advances in the hardware and algorithms employed by parallel data processing.
- Mike Ruland, field engineering
Posted by expressor software at October 15, 2009 12:45 PM
Twitter 
LinkedIn 
YouTube