The main driver for choosing what to implement in this version has been to improve the analytical capabilities in general and to simplify handling of wide data sets in particular.

We started out just about a year ago by mining our internal knowledge base with enhancement requests and unresolved issues. This gave us a lot of ideas of areas of the product, where improved functionality would give a lot of customer value.

With the resources available, we decided that we should invest in improving the analytical capabilities by:

• Simplify handling of a huge number of columns in the data; to achieve this, we set up the goal that it should be possible to analyze data with 10000 columns.
• Improving the handling of color ranges and categorical coloring.
• Improve label drawing in plots.
• Improve overall performance.

These high level goals resulted in a major refactoring of many components that have been around for almost a decade now and we are now well positioned for improving the product even further in future releases.

We have also exposed quite a lot of new exciting capabilities for our API users.

Regards,
Jonas L

This release includes a lot of good stuff, which will make life a lot easier for end users and developers extending the platform. Stay tuned for the formal announcement on http://www.spotfire.com.

/Jonas

/Jonas

• Microsoft .Net 1.1
• Microsoft .Net 2.0
• A mix of Microsoft .Net and C
• Java

From a technical point of view we were convinced that the Microsoft .Net 2.0 framework had the greatest potential; some of the advantages we saw were:

• Tight integration with COM enabled applications possible
• Many attractive language features that may increase development productivity
• Good track record for end user adoption of previous .Net releases
• New software deployment features in .Net 2.0 enables better control over centralized deployment

After struggling with bugs in beta releases of platform and development tools, we still believe that we made the best choice. I’m confident that the analytics platform that we have built for Spotfire DXP will be just as successful as the Spotfire DecisionSite platform has been.

The key features of the Microsoft .Net 2.0 platform that we have made great use of are:

C# language feature like:

• Generics
• Anonymous delegates

• Improved remoremoting>
• Web services
• Windows Forms enhaenhancements
• Internationalization
• 64 bit platform support

Regards,
Jonas

Regards,
Jonas

The data handling of Spotfire DXP is one of the most exiting parts of the product, it relies heavily on experiences learned from the Spotfire DecisionSite product, but we have also spent quite some time on investigating the needs of the business analytics market. These findings gave us some interesting ideas that we have twisted around until something with a Spotfire feeling came out.

Many of the capabilities of the data handling of Spotfire DXP can be found in other products, but we have made it possible to start with a data set, add more data from various data sources and then visualizing multiple aspects (views) of this data simultaneously, with full interaction and tight coupling in between different data views.

In Spotfire DecisionSite we have visualizations that have specific columns assigned to their axes, and some plots have built in aggregation of these axes, i.e. bar chart and pie chart. In Spotfire DXP this has been taken much further. Each axis in the visualization is now assigned a data view, which can be dynamically calculated from one or more column in the data set.

The data view can be calculated based on the columns exposed by the original data set imported into Spotfire DXP or any other view derived from that view. Data views can also be based on subsets of other views, e.g. filtered rows or marked rows.

Just as in Spotfire DecisionSite, DXP has the ability to create new columns based on other columns. Spotfire DXP extends this concept with the ability to use the expression language to calculate aggregated columns, cumulative sums and hierarchies.

All data views and calculated columns are automatically recalculated when required, e.g. when the filtering is changed and when new rows are added to the data set.

Another important feature which has been enabled by Spotfire DXP is the ability to have different sets of data filters apply to different visualization. This is enabled by the ability to create new data subsets for different views. In Spotfire DXP 1.0 this is used to make it possible to have query device panels on different pages that work independently.

The hierarchy handling is another really important new feature, I will describe this in a later posting.

Happy analysis,
/Jonas

The product will be called Spotfire DXP and it is a completely new foundation for
interactive business analytics. Spotfire DXP embraces the strengths of the Spotfire
DecisionSite and adds a huge set of new capabilities never seen together in one
product before.

We have talked to customers and investigated all aspects of the data analytics market
to come up with a completely novel way of interact with data.

The main capabilities of Spotfire DXP are:

• Centrally managed data access via the Spotfire Information Services
• Centrally managed product deployment
• Interactive analysis of large data sets - a new take of the unique Spotfire way
  of allowing the and user to analyze business critical data
• End user friendly packaging of complex analytical tools
• Extensibility of product

Compared to the Spotfire DecisionSite product you will notice many differences.
The main features that are visible to the user are:

• New user interface, which allows a much more effective analysis of data, and gives the user direct access to all common operations via drag drop and on screen roperty
  controls
• Undo and redo of changes to the analysis session
• Addition of new data rows and columns in a much more flexible way
• Visualization of multiple views of data in parallel
• Analysis data with hierarchies

All of these new features are based on a completely new product foundation developed using the Microsoft .Net 2.0 framework.

Stay tuned for more details on architecture and exiting challenges encountered.

Regards,
Jonas

Spotfire DecisionSite started out as a groundbreaking tool for doing interactive ad-hoc data analytics and over the time Spotfire DecisionSite has added a number of different methods that can be used to customize the product to meet the needs of different end users:

• A rich COM API that enables automation of repetitive task
  
• Numerous extension APIs that enables addition of new functionality in many places of the product
  
• Central management of software deployment and user based access to functionality
  
• Central management of data access via the Spotfire Information Services, a middle layer that manages data access from a large selection of data bases
  
• Workflow recording and sharing via Guides and DecisionSite Library
  
• Sharing of results and online collaboration via DecisionSite Posters
  

Much time and effort has been put into enabling the usage of the tools best suited for a particular problem. We also try really hard to "eat our own dog food", almost all additions we do to the product now are done using the extension APIs of the product.

In later postings I will describe how we have evolved the APIs of Spotfire to become as versatile as it is today. In this entry I will give an overview of what different technologies Spotfire is using to add new features to the product.

The core of Spotfire DecisionSite was originally developed in C-Plus-Plus, C-Plus-Plus is still used for some development, but the focus has shifted to other technologies.

Over the decade that we have been developing the DecisionSite product the used platforms have been:

• Year 1-3 C-Plus-Plus
  
• Year 4-6 Jscript, Html, Java, C-Plus-Plus
  
• Year 7-10 C#, Jscript, Java, C-Plus-Plus
  

The main reason for these technology shifts has been to improve productivity and ease of implementation of new features. So far we find the C# as the compromise in between code quality, ease of implementation and efficiency, but there are a number of hurdles that must be overcome to build an application that mixes COM, Jscript, C-Plus-Plus and C# to the extent that we do, more on this in later postings.

Regards,
Jonas

When importing SAS data to Spotfire we have five different methods to choose in between for conversion:

1. Use the raw SAS data without conversion (in string or numeric format)
   
2. Use SAS output formatters to convert the raw values to strings formatted in the same fashion as the SAS user is used to see the data
   
3. Use SAS output formatters to convert the raw data to string, this string is then parsed to one of the six native Spotfire types
   
4. Convert the raw SAS data directly to a native Spotfire type
   
5. Use the raw SAS data in Spotfire DecisionSite and apply a Spotfire data formatter
   

In the Spotfire DecisionSite 8.2.1 version of the SAS data file access we have used all of these methods except number five. The choice of what method is used is controlled by what SAS output formatting is stored in the SAS data file. The user also have the option to use the raw data (option one) for all columns.

To set up the rules for what type of conversion to apply for different SAS formats was one of the most difficult tasks during the development of the SAS data file importer.

In most cases we could find good ways to map SAS formats to Spotfire types, the biggest issue we found hard to resolve is the handling of SAS time based formats. The main culprit was that the Spotfire time data type only has a valid range of 00:00:00 to 24:00:00 and the SAS time type supports both negative times and more than 24 hours. This was one of the cases where we have choose to fallback to importing the raw data, to make sure that no information is lost. Converting it to string would yield nice output but the distance (and possibly order) would be lost.

More information on how the type mapping is done can be found in the help system of Spotfire DecisionSite 8.2.1 :-).

Regards,
Jonas

The goal for the development of the SAS data file ntegration into Spotfire DecisionSite was:

• Make data from SAS data files (.SAS7BDAT) as easy to use in Spotfire DecisionSite as data from text files and excel files. It must be possible to open SAS files via File/Open, File/Import and drag and drop. It must also be possible to add new columns of data from SAS files
• Allow usage of SAS data files as targets for linked data in a Spotfire DecisionSite analysis
• Import SAS data in a format that is as similar as possible to the experience to how the data is presented in SAS
• The imported data should be useful for analysis in Spotfire DecisionSite, i.e. numerical data should still be numerical etc.
• No information should be lost during import

The technology platform that we choose for the integration is the local data provider, from the SAS Providers for OLE DB, which is available for free download from SAS. This OLE DB provider supports reading data from SAS data files, which are generated by SAS version 6 and newer.

More on type systems in the next article.

Regards,
Jonas

One of the most exciting new features is the seamless integration of data import from SAS data files into the product. I will cover this feature more in a later blog entry.

/Jonas

This month Spotfire celebrates its 10 year anniversary of Spotfire as a company. 10 years ago, we were 5 entrepreneurs with a great idea of how interactive visualizations should save the world sub-renting a small office in Gothenburg (Sweden), now we’re a global company with 40 times more employees.

My primary focus has always been to get high quality products that address business problems in the hands of customers. To achieve this goal I've had quite a few different roles at Spotfire, developer, software architect, project manager, VP Engineering, product management, IT support, … (I'm also the one to blame for the name Spotfire)

I will use this space to share some ideas and reflections that have come up during this time.

I invite everyone to come with feedback both on our products and on ideas of how to apply data visualization on business analytics.

See you later,
/Jonas