Realtime BI

Sometimes it's easy to forget just how different things are in the Internet world – but I was reminded yesterday. I was attending a gathering of technology executives, and the presentation was on business intelligence. The gentleman doing the presentation was the CIO at a local insurance company, which had spent a good chunk of the last six years building out their data warehouse. From what I could tell, most of it looked pretty well done (i.e., neither too much Inmon nor too much Kimball).

The interesting bit for me was when he happened to mention the frequency of their data warehouse loads, which took place once a month. Somebody else in the audience piped up with a question: "Has anybody here experimented with more realtime sorts of BI? I've heard some companies load their data warehouses as often as once a week." One other guy made a comment that led me to believe they had experimented with loading their warehouse as often as once a day, but nobody else seemed to think that was necessary, or even reasonable.

So I had to chuckle to myself. Zango – like most Internet companies – is addicted to realtime data. And by "realtime," we don't mean "once a week" or "once a day." Our data warehouse loads and processes every hour. Sometimes it gets behind, and that's a pain: at the moment, we're nearly two hours behind in our loads, and it sucks. It feels like we're flying blind – or like I'm going through withdrawal. For example, we made a change to one of our websites two hours ago, and I haven't been able to analyze what sort of an impact, if any, that change has had. I can't imagine the friction that would be injected into our world if we had to wait a week, or even a full day, to see whether we had just screwed something up, or fixed a particular problem.

Of course, sometimes you can get too much realtime. It can be tempting to sit in front of ProClarity and hit "Refresh" all day long, waiting for the latest revenue or install numbers to come trundling in. Because we can tune campaigns manually, we can neglect better, automated campaign optimization solutions. And worse, it really is a pain to get those realtime loads working. You can't sling two or three terabytes of data around without hitting something. We've spent a lot of time and money tuning our systems to get those loads working right, and part of me wonders if that was the best use of those resources. If we'd thrown those same people at the task of automating our campaign targeting, rather than reporting on it hourly, we might be further along than we are now.

But then I pull ProClarity back up, click refresh and take another hit.