Sunday, May 16, 2010

Data Warehousing: The Agile Method

One of the more insightful sessions I attended at TDWI's 2010 World Conference in Chicago covered the topic of development methodology. In this case, the presenter recommended that we basically forget everything we know about the software development process and consider a new way of doing things.

Listening to what was described during this day long session, I came to realize that the way I had been doing things already had a name: the agile method. In fact, the presenter (Ralph Hughes) had already written a book about it titled "Agile Data Warehousing".

Rather than approaching development as an overly bureaucratic and planned out process, the agile method suggests that we drop the burden of extensive planning and focus on delivering smaller and more frequent deliverables. By following the agile method, we can approach a more ideal solution while providing small and useful pieces each step of the way.

For example, I have been involved in projects previously where the planning process was so painstaking and overbearing that the project never actually got off the ground. Instead of putting our heads down and getting to work, our heads were up in the clouds creating plans and contingencies for mitigating risk should plans not go as intended. Instead, by the agile method, some smaller piece (like a single dashboard or report) is selected for development, and this piece is developed as though the final deadline for the entire project lie just around the corner.

It's an exciting environment to work in, and it puts aside a lot of the fiscal concerns of project costs, as within weeks of kicking off a new project, something of value is delivered and implemented for end users.

If you're interested in learning more, click the image of the cover to order the book from Amazon's website. And for a quick overview of the best practices for agile method development, take a look here.

Thursday, May 13, 2010

BI User Adoption

Imagine yourself in the cockpit of a Boeing 747. You are the pilot, and you've got some difficult decisions to make. The plane is running low on fuel, with just enough to make it to your destination and land the plane. You've lost an engine, and you're carrying a fairly heavy load (okay, so you'll have to imagine yourself as a pilot prior to baggage checking fees coming into effect). Your co-pilot is, well, pre-occupied with a young flight attendant and a dry martini.

Having never flown a plane before, you stare down at the instrumentation panel and cluster of gauges in front of you, and you're confused.

This is often how the end users of BI implementations feel when they look down at their screens. While the content on the screen may provide the solutions to all business problems and more, the tools aren't easy to use and they're hard on the eyes.

BI vendors would all have you believing that their software suite provides the perfect solution to this ongoing problem. With all the glitz of glossy marketing materials, they insist that they are providing the best looking and easiest to use software in the business. What they're not telling you is that it's their job to lie...err...I mean sell.

This is one of those problems that money doesn't solve. You can throw tons of capital at your reports and dashboards and bring in every vendor under the BI umbrella, but at the end of the day, you'll still be left with an ugly and useless hunk of trash (exaggeration for emphasis).

In order to address this issue, you'll need some creative resources. Find someone with the ability to place themselves in the shoes of a non-technical computer user and identify the shortfalls of your user interface, yet knowledgable regarding the technical capabilities of the software. Find another person who has an eye for design and knows what will be pleasing to the eye while providing meaningful insights to the end user. BI is not just about data warehousing, coding and report authoring. We need to reconnect with the creative side we were so in touch with as children.

Don't believe it? Take a look at the average kindergarden student using an Apple product. Upon picking it up, they can almost immediately make it work for them and find the information (or iPhone game) that they're after. While they may not be landing airplanes, we can benefit by taking a page or two from their book.

Not sure where to find the latest edition? You may want to start with Stephen Few's "Information Dashboard Design: The Effective Communication of Data".

Tuesday, May 11, 2010

Did You Know?

At Cindi Howson's "Cool BI: The Latest Innovations" at TDWI's 2010 Spring Conference in Chicago, the video below was shown during the session break. It's a fascinating look at some of the demographic information that will create new challenges and shift the balance of global power in our lifetimes. Worth noting is the emphasis on the speed at which information and knowledge are expanding and how quickly new skills will become obsolete in the future. It is for this reason that a strong commitment to the advancement of educational attainment will be critical to our future successes.


Sunday, January 24, 2010

new partnership

hIghQ BI Consulting Services is excited to announce a partnership with Data Mining Tools Inc.

Data Mining Tools is a provider of a wealth of useful materials in the practice of a core function of business intelligence: data mining. Their website ( contains a variety of resources from tutorials, certification opportunities, instructional videos, and sample data sets.

Their services are subscription based, but well worth the small price paid. With the depth and breadth of resources available on the site, users receive valuable and cutting edge insights into best practices for data mining.

If you're interested in exploring how Data Mining Tools can assist you in learning and growing as a BI professional, please visit their website at:

hIghQ is currently analyzing sample data sets from Data Mining Tools and will provide examples of data mining techniques taught through their subscription service in the very near future.

Monday, January 11, 2010

partnership announcement

 Data Applied

hIghQ BI Consulting Services is pleased to announce a new partnership with Data Applied.

Data Applied is a small technology startup located in Western Washington State. They are passionate about distributed computing, data mining algorithms, applied statistics, and data visualization.

For years, large companies (ex: banks, insurers, retailers) have reaped the benefits of data mining to optimize their business. However, because of tremendous complexity and high costs, automated analysis has remained out of reach for small- and medium-sized businesses. Data Applied is changing the rules of the game by offering affordable, Web-based Data Mining and Business Intelligence solutions.

Data Applied revolutionizes data-driven decision making by integrating rich analytics, data mining, and information visualization capabilities - all using a zero footprint Web interface, collaboration features, and a secure XML Web API. By extracting valuable knowledge from data in domains as varied as Web Analytics, Sales, Marketing, Engineering, Social Sciences or Non-Profit, this fantastic software is helping organizations make better data-driven decisions and improve efficiency.

If you're interested in exploring how Data Applied's web-based software can help you take back the reins on your business performance, you can visit their website here:

hIghQ is rapidly developing their expertise within the Data Applied BI Suite, and will be ready to assist you with your implementation very soon.

Saturday, January 9, 2010

toyota's business intelligence

Just around the time of the bubble (i.e. 1999-2000), Toyota discovered that their business intelligence systems were, well, unintelligent. As is often said, garbage in, garbage out. While the company had set up a data warehouse intended to capture real-time, accurate data for decision making, they had failed to set up the controls required to eliminate human error, and as a result, they missed out on the true value of BI. While they had the right ideas in place, they had the wrong technology for the job.

When it comes to choosing a business intelligence platform, it's important to assess the robustness of the implementation. In many cases, being a "first-mover" is a positive characteristic, but when it comes to implementing a BI solution, it is often best to shop around and see what others are using. Talk to contacts, check the web, and take a look around. A costly BI system that doesn't work isn't worth the dollars you'll spend. While this might sound like an elementary lesson, Toyota learned the hard way.

Years after their failed BI implementation, they determined the wrong technology was in place. After a second costly reimplementation, their BI systems finally proved their value. Toyota uncovered an $800,000 billing error from a shipping company in the initial days after the implementation was completed. Over the next 4 years, they increased the volume of cars they handled by 40% while head count increased by a marginal 3%. Productivity gains such as these run rampant across firms who adopt business intelligence systems that work. In fact, the systems more than pay for themselves, and quite handsomely. Many case studies assess the ROI of such investments at the double digit percentage level, which is above the majority of firms' policy requirements.

So, in an attempt to make this blog more interesting, we'd like to ask for your feedback. What kind of information sources do you have at your disposal within your organization? Does each department have individual data sources? Or is everyone querying a centralized warehouse? Are key decision makers waiting unnecessarily for information? We'd love to hear your stories, so don't be shy.