Why an open data store ? - HELP!

[mzimmpa]

Hello,

I am a BI consultant on a new project which exists to integrate data from outside enterprises as well as providing the data to users for analytic apps.

I find it incredible, but I am in the postion of having to justify why we need to include in our architecture not
just a data warehouse, but I need to justify why we would want to keep processed data in an independent data store (format does not matter) instead of keeping SOLELY in a proprietary data structure, (which is technically a stovepipe). The argument against me is that (1) we will deliver data in very small chunks via discreet queries to small databases required by the apps, and (2) we will do that via java-based API's from the proprietary database.

I need help in digging up rationales that my whole BI career has been built upon. I require a more academic approach.

Any idea where to find a series of bullets or do you have any?

Thanks Much!

- Lost in a stovepipe design

 

[johnherman]

I need to think about your situation a bit, but when in a pinch, I always turn to two Kimball books, the DW Lifecycle Toolkit, and the other book he has with some sort of Web title. Another goodie is Adelman's Impossible Data Warehouse Situations.

I think you need to better understand their approach. Is it based on security? Are they afraid to "open up" their systems too much in case of cybertheft or attack?

One argument I use when dealing with vendors or customers outside my firewall - have a separate data store for their data. That way you always have a definitive record of what they sent you (or what you sent them). This exchange repository does not need to be open, however.

In fact, there is no good reason for building a data warehouse on open systems if data or system security is such a big issue, that it is more important than the application itself.

Good Luck.

Sometimes the grass is greener on the other side because there is more manure there - original.

 

[mzimmpa]

Thanks

-- the rationale behind it has nothing to do with security.
It is (IMHO) an over-zealous outlook on an analytical tool
which supplies fantastic data associations, however, the data cannot be extracted except in very small and specific queries, unless viewed via the tool's UI (the associative data is vastly compressed, in binary format and will explode into cartesian products).

I need to justify why I would possible need to keep the data
other than in this proprietary tool/database.

Looking for some "powerpoint bullets".

Read all the DW books, but they are not helping me (or I am not finding) in zapping this one argument.

thanks!!

 

[johnherman]

If, in the business eyes (user requirements), the sole purpose of the DW is to feed said analysis tool, you cannot justify additional data structures AT THIS TIME. You can argue that there could be savings in investing in such structures now if they will be needed later. You can also argue that industry studies have shown that once DW's are in production, users often ask questions which are beyond the scope of the initial DW requirements, questions which cannot be easily answered unless the original design provides flexibility and room for growth. However, getting back to the point....if the business sees the DW only as the data store for that analysis tool, you are probably stuck.

I would look for white papers which summarize benefits and capabilities of DW if you still want to push this issue. Let mw know if I can help more or if you think I still don't understand the issues at hand. Best of luck.

Sometimes the grass is greener on the other side because there is more manure there - original.

 

[entaroadun]

It sounds like the existing business requirements (or at least what the line of business thinks are the requirements) don't justify a separate data warehouse.

If I were you, I would concede the battle after making sure everyone is clear on my doubts and reservations. Namely:

- Requirements will grow. Independent data warehouses are agnostic, and are therefore flexible if another analytical tool is chosen.

- Once users see the power of reporting, they will start hitting the environment hard. Performance will degrade if the platform is not tuned for analyzing large volumes of records.

- It may be impossible with the current "stovepipe" to validate and reconcile the data. Data discrepancies undermine the reputation of the platform and limit user adoption. Without a way to reconcile to other sources, esp. the G/L, the system is dead before it launches. An independent data warehouse allows the reconciliation of information through the entire system.

Again, if I were you, I would make sure everyone knows these issues. Therefore, in the near future when people realize the limitations of the platform they build, you can be the one to spearhead/champion the new data warehousing initiative.

 

[mzimmpa]

I GREATLY appreciate your concise answers.

Just the bullets I was looking for.

What concerns me is IMHO the business requirements require,
at the minimum, a separate data store in order to facilitate user requirements other than the tool which is being viewed as the final and only data store.

I am simply told that the vendor of the tool concerned will
gladly meet all requirements for data usability given enough development time. (Vendor is small 10-12 man outfit desparate for capital).

Thus, my plea for help in creating an argument for an additional data store.

Thanks again!!

mzimmpa

 

[MDXer]

Ask them if the Vendor is willing to put a garuntee in writing. 10-20 man shop sounds like a small development company, actually in sounds like a company I consulted for in the bay area, what happens if the company goes bust? Given enough time almost anything is possible, but how much time can your company afford? How critical is this system to your company if you hit a road block can your company afford days or weeks for the vendor to work through the problem. Having your company be the development scenario isn't a nice situation to be in.

"Shoot Me! Shoot Me NOW!!!"
- Daffy Duck