« expressor uniquely positioned to address 'big data' challenges of SQL Server community | Main | Is your company's process of collecting and analyzing clinical trial study data too manual and time consuming? »
November 11, 2009
after purchasing, please read the documentation
My first job out of college was working for a company that made what they called the 'bag.' The 'bag' was a cellular/GPS device designed to track parolees and warn when they might be doing something they should not.
It was 1995 and I could not have asked for a better first job. The system was completely real-time. The operating system was Windows NT 3.51. We were writing our own 'bag' operating system and here I was right out of school.
Then reality hit. The GPS record format.
Nowadays it might be easier (I have not kept up on it), but 15 years ago it was a binary fiasco. The use of 3-byte and 5-byte integers for altitude, time, latitude and longitude information, among other fields, was used to reduce the size of the data frame. They did not teach this stuff in school!
For those of you not intimately familiar with integer sizes and class, there are no common computer languages that handle odd-length integer values greater than 1. Meaning that 1, 2, 4 and 8 bytes are handled natively and 3, 5 and 7 bytes must be handled with conversion code. In fact, only two ETL vendors that I know of can handle this format, expressor software being one of them.
The reality is that most data integration tools can only handle relational, XML (barely), and delimited/fixed flat files. Sure, they can handle queues and SalesForce but these are all just hacks. If you read the documentation it is easy to see the limitations and there are many caveats and limitations when not using RDBMS and flat files.
In fact, it is not until you actual buy the product until you begin to discover all the things you simply just cannot do even though you were told you could. My favorite example, and this is so true, is that most data integration vendors include a statement such as, 'If you need to sort, perform the sort in the RDBMS as it is faster than our sort.' Really? You're kidding right? Well what about this flat file? What about this XML? What about my 3-byte integers? What about my complex HL7, EDI or legacy data?
-------
Dear Customer,
Sorry we were not completely honest.
Regards,
Your Data Integration Software Vendor
p.s. Please come to *****World in the Fall!
--------
Actually, there is really no need to worry about formats like HL7, EDI, and SalesForce data. The reason being is that these are all add-on products for which you need to pay even more. And when you buy them and read the documentation you find the truth. That they really don't work as advertised.
- John Russell, chief scientist and co-founder
Posted by expressor software at November 11, 2009 8:30 AM
Twitter 
LinkedIn 
YouTube