One of the constant challenges for any Business Intelligence (BI) professional is that of balancing delivery outputs and timescales. We might spend weeks mapping out data sources and developing extraction and transformation techniques, however none of this will give the user a dashboard or report to start using now, which can lead to users losing interest.
I’m currently looking at ways to challenge more traditional approaches to BI develop and embrace a more evolutionary development process.
Many traditional organisations use a tried and tested method of development aligned to a sequential design and development process known as 'waterfall'. This model has been around since at least the early 1980's and as the name implies follows phases from one to the other like a waterfall:
- Conception
- Initiation
- Analysis
- Design
- Development
- Testing
- Implementation
- Maintenance
The process assumes that the costs of moving backward (trying to swim up the proverbial waterfall) are too high so it only flows in one direction. A typical BI project might translate the steps above into the following activities:
- Requirements gathering through user workshops and interviews
- Development of technical requirements - detail user / system analysis
- Develop data models and map out sources systems
- Develop extract, transform and load processes
- Constructs reporting environment including reports and dashboards
- Complete testing and sign off
However, following this process in only one direction can often lead to batching of the activity. If we are dealing with a large project without introducing phases the batches can become large, which for anyone familiar with lean processes will know is a barrier to flow and leads to delays!
|
The process also restricts any adjustment to approach or redevelopment of the solution as more details are uncovered. In many cases the process steps followed lead to the development of solutions not fit for purpose and require weeks of redevelopment. The questions to ask are 'Why did this not get spotted earlier on in the process?' 'How did we miss those requirements?' To break down the batches and improve the quality of each phase some organisations adopt an iterative development approach. These applications of iterative development usually involve breaking each of the above steps down into a series of iterations. Here is a sample breakdown of what each iteration would look like: Iteration 1 : Requirements gathering through user workshops and interviews Iteration 2 : Development of technical requirements - detail user / system analysis |
How can you quickly develop intelligent solutions that are "fit for purpose"? |
Iteration 3 : Develop data models and map out sources systems
Iteration 4 : Develop extract, transform and load processes
Iteration 5 : Refine requirements gathering through user workshops and interviews
Iteration 6 : Refine technical requirements - detail user / system analysis
Iteration 7 : Refine data models and map out sources systems
Iteration 8 : Refine extract, transform and load processes
Iteration 9 : Constructs reporting environment including reports and dashboards
Iteration 10: Complete testing and sign off
Whilst this does help to develop a better quality solution and may cut down some of the time elements it doesn't actually deliver a useable product until all the process steps have been completed. Agile development is a methodology that BI solutions can learn from as this philosophy aims to develop solutions to users in usable chunks.
So how can we embrace agile in BI development?
One of the key areas is to take the initial 6 steps identified earlier in a typical BI project and integrate them. In doing so individual steps can be brought closer together through use of integrated user / developer / analyst working groups to 'story board' solutions from 'data to dashboard / report'. Iterations are still being used however being delivered in series of drops, where each drop represents a full cycle of development from definition to implementation i.e. At the end of drop the user will be presented with a usable dashboard or report they can start using to add value to the business.
The key elements to enabling this type of rapid BI development are:
- Break down the BI project into a series of drops to deliver tangible outputs to the business and the customers of the project. This may involve delivery a preliminary drop to create core foundation services needed for the rest of the project. E.g. The initial setup of an environment or data warehouse or reporting platform.
- Allocate defined delivery timeframes for each drop, such as 6 week outputs. This will ensure the project maintains pace and gives the business clear visibility of what will be available and when. This forms a key part of keeping stakeholders engaged by continually delivering something visible to the business.
- Streamline processes and documentation through multi disciplinary team collaboration and story boarding. Storyboards should be used to replace the traditional requirements and detailed specifications documents to capture end to end giving visibility of all activity and interactions.
- Keep the stakeholders involved in the process though out the development of the solution with regular review, testing and improvement.