Tek-Tips is the largest IT community on the Internet today!

Members share and learn making Tek-Tips Forums the best source of peer-reviewed technical information on the Internet!

  • Congratulations strongm on being selected by the Tek-Tips community for having the most helpful posts in the forums last week. Way to Go!

ETL Tools selection 10

Status
Not open for further replies.

ADB1

Programmer
Aug 24, 2001
235
GB
Has anybody looked at or evaluated Business Objects Data Integrator? I am also looking at Ascential Datastage for ETL too. Has anyone compared these two tools?

Thanks,

Adam.
 
I would like to put in my 2 cents about Data Stage. DS is good for handling sequential files. I think Microsoft DTS is a better product, even if you need to buy a separate Windows server and SQL Server database to run DTS. I have not even seen a demo of the BO product.

I know that with the next version of SQL Server (which includes DTS), code named Yukon, Microsoft will be going after the market leader, Informatica with respect to ETL tools.

Sometimes the grass is greener on the other side because there is more manure there - original.
 
Does anyone know about ETL tool throughput rates? Impressions or links would be useful, as I may have to make that decision if our CRM warehouse uses a vendor that doesn't have an ETL tool - like a Microstrategy. They pretty much recommend Microsoft DTS, while the Hyperion rep admitted that all of the BI reporting company's ETL tools are not that sophisticated - in the the MS DTS class in fact.

We're not really looking for sophistication in an ETS tool - just how much data can you move and how fast?
 
We looked at both those tools and also Informatica PowerCenter, which we bought. Ascential's strong point is it's metadata repository - their ETL tool has some good pieces, but overall - it makes you totally change how you look at data transfer (a single table and multiple lookups - you can't do a table join - as far as I saw in the demo). You gotta tilt your head 90 degrees to use this one! ;-)

BO is the 3rd stringer - they're trying to sell an entire package - it integrates tightly with their BI tool, but doesn't stand on it's own that well. They either bought the product or the company that made it. It did have some nice features, but not enough to make it stand out above the other two. And it wasn't that much less in price, as I recall.

PowerCenter only makes you tilt your head around 45 degrees (like the RCA dog) to get a handle on their model. :) It's UI is rather quirky in 6.x. We're getting 7.1 set up real soon now to see if they fixed their less than stellar integration between modules. They do give you the best flexibility - at a price, of course. I think because they're hungry, we still got a comparable price to Ascential.

Please note: DTS can't touch Informatica (at least this round). Informatica provides native connections to all the major databases, along with ODBC for the rest. DTS will only go native to SQL Server - but that may not be an issue for you.

As far as throughput, it processes ~200k rows from SQL Server in only a few seconds. We were pleasantly surprised at it's speed - even though that's not a big issue with us - we don't move terabytes of data.



"I think we're all Bozos on this bus!" - Firesign Theatre [jester]
 
Has anyone looked at the DTS product from Scionet? We just started looking at it. It's not got the polish of these big timers but our quote worked out less than 25% of any other and the pilot is working and was really quick to get started. I think they're pretty new but I didn't talk to them direct.
I've heard some good things about BO but have not tested it and it wasn't on our 'list' for some reason so I'd really like to hear any feedback on implementing it.
With MS DTS, I've heard some nasty stories about failures during processing requiring a complete restart but those issues are apparently fixed in the next version.

 
Were looking at Business Objects Data Integrator 6.5 as well as Pervasive Data Integrator (formerly data junction)... from a price point they blow Informatica and Ascential out of the water!

I've used Informatica before and don't like it... lots of maintenance issues; difficult to use; steep learning curve and the list goes on... this is suppose to save you money??


Pervasive and Business Objects Data Integrator are easy to use (huge huge points for this!!) offer great flexibility (from initial demo)

Were getting BO in house and playing with it... I'll follow up in 2 weeks time and let everybody know how good it is...
 
" I've used Informatica before and don't like it... lots of maintenance issues; difficult to use; steep learning curve and the list goes on... this is suppose to save you money??
"


I've come across plenty of tools, but never seen such a polished product as Powermart/Powercenter by INFA. trouble is that they lack a light-version that is affordable, but I dread the day that I will no longer be able to work with it. I have also VERY good experiences with INFA as a vendor, whereas BO is a vendor from hell (and I love their products too...)

T. Blom
Information analyst
tbl@shimano-eu.com
 
INFA is very expensive and it's also got some performance issues; also you need their on site consultants to make things work well... from what I gathered with BO it gives you all the power + real time and simplicity without the price... we are getting it in house soon for eval purposes...

as far as BO tech support - we have BO in house and they have been simply amazing at providing phone support....

Have you used BO Data Integrator... if so, what do u think of it?

regards,
Peter Martinakis
 
INFA is very expensive and it's also got some performance issues; also you need their on site consultants to make things work well... from what I gathered with BO it gives you all the power + real time and simplicity without the price... we are getting it in house soon for eval purposes...

PowerCenter is expensive comparatively (and so is Ascential) - it all depends on your needs. Not sure what performance issues you've encountered - it works fine for us, but we don't move gigabytes around. It does help to have their consultants assist in the first setup, but after that, it's not too complicated to modify.

The one thing I remember from our meeting with BO is that you MUST write the SQL joins between tables - there is no graphical connection ability - even lowly MS Access has that! Maybe they've changed, but I'm not interested in using a glorified SQL generator. It did have some nice features - it all depends on what your requirements are.

"I think we're all Bozos on this bus!" - Firesign Theatre [jester]
 
Hey ArtiChoke,

No joins between tables... hmm interesting... what version of Data Integrator were you using... DI is engine based; it's not an sql generator; one of the benefits of DI is that you can specify where the work gets done source our target... refered to as ELT; not many tools have this feature and this saves your butt on network traffic... let me get it in house and do a proper eval and we'll talk after that... INFA is big $$$ and depending on what you are doing the learning curve can be steep.

Keep an eye on Pervasive Data Integrator... very inexpensive and growing quickly.
 
icecore,
We did our evaluation last june, so whatever version was available then was used. I know it's more than just a SQL generator, but making us write the join conditions is a bit much - we wanted an ETL tool to hide the low-level stuff unless you need to do it. What version are you getting? Maybe they've made some significant changes in the last 9 months.

We wrote up a comparison of all three tools, but I can't find it right now. It would be interesting to see what we thought it's strengths and weaknesses were.

I think we saw data integrator also - if it's the one I'm thinking of, it's very inexpensive, but ya gotta do a lot of work - it's like a super version of Access. Coding macros and events - ouch! I'd rather write VB programs if I'm going to code - easier to follow the flow - and that's what we did in the past. We wanted to get away from custom coding, where only a select few can do it (and get stuck maintaining it - doh!).

Oh well, I guess it's just a matter of taste (and budget :)). Plus, for us - we can train some of our business process people to use it, since in most cases there's no coding to do. And when there is, us experts get involved.

"I think we're all Bozos on this bus!" - Firesign Theatre [jester]
 
ArtieChoke,

thx for the info... writing the joins? I'm trying to figure out what you mean by that... if you are joining tables you are probably creating an SQL statement/result set with constraints to move data to the next step in your transformation... so your saying there is no visual table joiner to join relatinal sources? interesting... We are evaluating version 6.5 and I'll give you some feedback on that next week; on the other hand writing SQL statements to represent a step in the process isn't that bad; eventually you would have to document each step and provide sql for it anyways no?

regards,
Peter
 
so you're saying there is no visual table joiner to join relational sources?

That is correct. You don't have to create the select statements - just the join conditions. It will do the rest. The only reason to override the SQL would be for special circumstances - and you'd definitely want to document that!

"I think we're all Bozos on this bus!" - Firesign Theatre [jester]
 
ArtieChoke,

thx for the info... I'll make a note to examine this more carefully when BO comes in.

We just had a presentation from Hummingbird ETL... an interesting product!

-Killer interface! - the best I've seen so far... I'll post more on it as I evaluate.

regards,
Peter
 
There is considerable room for consolidation in the ETL tools market. There was a similar situation with Client-Server front-ends in the early 1990's. Of the 20 or so front-ends back then, only two remain today (Powerbuilder and Progress) while Visual Basic emerged from nowhere to be extremely competitive.

The same thing is or will be happening with ETL tools. As I mentioned earlier in this thread, Microsoft is going after the market leader, Informatica, with Yukon, the next version of DTS (Data Transformation Services). From what I have seen and heard, this new version of DTS will be a considerable technical advancement, and with the price of DTS being reasonable (the cost of SQL Server database and client licenses = say, about $30k), Informatica, Data Stage, and all those other ETL tools will be forced to cut pricing or lose market share.

When Microsoft has made up its mind to enter a particular market, it invariably gets considerable market share. Witness Windows CE versus Palm OS - Palm is a much smaller and better OS, but Microsoft has market share. Remember Netscape Navigator and what Microsoft did to it with Internet Explorer. The same thing is going to happen with Microsoft DTS and the rest of the ETL tool world. The tool you pick today could well be obsolete in 5-7 years. If that is not a concern for you, so be it. If you are worried about the obsolescence of your ETL tool, unless you are with Informatica already,I would go with Microsoft DTS now. Stay away from the niche products such as those offered by BO, Cognos, SAS, Harte-Hanks, etc.

Mark my words. In ten years, the only major players in the ETL tool space will be Microsoft, Informatica (or the company that buys them), Data Stage (or the company that buys them), and perhaps one or two others "sleepers" such as ab Initio.

Sometimes the grass is greener on the other side because there is more manure there - original.
 
Ten years out... gosh... that's a huge timeline to look at things... we need to perform now! and in the next 6 months!

That's the wrong approach... buy DTS cause they will get better... I don't think so... you buy what is proven today to achieve your business objectives within the next 6 months... YUKON will be full of bugs... (you'll see) It's gonna be one hell of a product... maybe 2 years from now...

ETL is highly specialized tool... and ETL only companies have much to be worried about... companies like BO on the other hand offer a complete BI solution and it makes sense to buy their product because it fully integrates with their product suite... they are leaders in BI and will only grow their business cause managers will only have to deal with one vendor (this is a huge point)...

Also, microsoft sql server (even yukon) is far away from penetrating the big boys; by the time they mature their product so will the competition.

You buy what is proven today from a stable company that offers support and high performance technology... not "what might be" tomorrow.

Your analogy with Palm and microsoft as it relates to ETL tools is a bad one... your comparing apples to oranges...

ETL companies to watch for in the next 3 years:

Pervasive (formerly data junction)
HummingBird ETL (kick ass interface but needs to mature)
BO Data Integrator (robust and industrial strength ETL with full report integration)

all of the above will take market share by introducing a lower price point... Pervasive is in so many shops simply because of price and features... their next version currently in the works is certainly impresive to say the least....

as far as Informatica and Datastage - managers usually pick these products as a "safe" decision.. .costs are high...learning curve is steep and maintenance is a nightmare (in terms of support and administration) - I use to manage a BI competency center overseeing groups of informatica developers and I can tell you the product is not simple to use... often it's overkill.

Just my opinion.

regards,
Peter


 
Well icecore, you are certainly as welcome to your opinion as I am to mine. I still stand by my marked words. Ten years from now, and quite probably sooner, M$ will have control of the ETL market and will drive out the competition by killing their profit margins as a result of including it free with the database.

My analogy of Palm and Win CE against ETL tools is not wrong. When M$ decides to enter a market place, the enter it with the goal of achieving market dominance, and will cut prices, sacrifice profits, and spread Fear Uncertainty and Doubt about their competitors. They are very good at it. They did it with OS/2 versus Windows, and it is widely held that OS/2 is the better technology. They did it with IE against Netscape. They did it with MS-DOS against PC-DOS and DR-DOS. They used the competitive upgrade strategy and FUD to crush the Lotus and Borland office suites. Even today, they are using FUD to attack Linux. I am not wrong about Microsoft and its tactics.

As it is now, DTS is a very powerful ETL tool. The fact that it is getting better is an added bonus, not a deciding factor. If your planning horizon is only 2-3 years, go ahead and spend your half million! Most IT Management types are long gone before the mistakes of their decision making are felt.

Sometimes the grass is greener on the other side because there is more manure there - original.
 
JohnHerman,

your analogy of palm vs MS is wrong! each case of market competition is very specific; you make it seem as if it is generic... you must look at the details... anyways we are getting off topic..

ETL tools are complex, require training and by nature of their product embed themselves into a company's culture; switching ETL tools and re-writing processes is not an option;

DTS is powerful? really? I don't see it as having any industrial strength features such as failover, self-aware parallel process technology and currently it is lacking many transformation objects that others have...

2-3 years... I plan to implement successfullyin 6 months! and not by spending half a million but by $50 thousand on BO Data Integrator... we are a BI and Crystal shop and it has a perfect fit for us... you have to ask the question of how MS will dislodge existing environments that use sybase and oracle...??? it isn't going to happen because these ETL tools already function within the client and they are not upgrading to Yukon (if anything people upgrade to Oracle)... I think your missing the point... if Microsoft is smart they will BUY an existing ETL vendor and incorporate into Yukon!

regards,
Peter
 
You are correct that failover must be handled (coded) manually. DTS is quite capable of parallel processing and its ability to use VB, C, and Java in transformation routines means it can do anything it needs in that area.

I am sure you are making the best decision based on the information you have and on the strategic and tactical (for the 6 month part) levels. In addition, you have stated your desire to remain close to a single vendor solution, which has its advantages in reduced finger pointing but carries the risk of having all ones' eggs in one basket (and dangerously exposed to, for instance, a CA takeover).

As far as M$ goes, give me a ring on tek tips in a couple or three years and we'll see how our predictions are going. In fact, I think I'll put it on my Outlook calendar.

Bon Chance

Sometimes the grass is greener on the other side because there is more manure there - original.
 
John,

if you are using vb, c or java it means you have the wrong etl tool... you are wasting valuable time coding in c or java (unless of course your problems is too large for the SQL language to solve from a performance point of view)

All the eggs in one basket... hmmm I think it's a good thing cause if somebody buys them they will keep their existing technology because it works so well (hence the reason for buyout) BO 11 is coming soon and that is a merger between BO and Crystal which are 2 fantastic reporting/BI platforms... typically you buy an ETL tool to build data marts/datawarehouses... why do you build data marts/datawarehouses -> for reporting! and what better company to use than BO at this time... their service offering is incredible but getting back to ETL I really don't think Microsoft will be a key player in this area in the near future... they simply will have "some ETL" functionality to entice existing users into upgrading to Yukon; definitely won't challenge the existing ETL players out there cause they are a very weak ETL engine to say the least...

it's not about finger pointing...we thoroughly evaluate all products before buying them; to date we have never pointed the finger at BO or Crystal cause their tech support has been first class.

In 3 years... hmmm what about for people on this forum looking for a solution in the next 12 months??!! - you have to go with what is available now... not what might be in 3 years (you'll be out of a job by then; LOL..)

As far as predictions go M$ will enter the ETL market... they will probably buy somebody elses tool... but in 3 years think about how advanced the other players will be like Informatica, Ascential and BO Data Integrator (which is definitely the best bang for the buck... by a factor of 10 for that matter!!!) - These ETL companies are already growing their real-time functionality which is something you ususally have to spend money on in the future...
In fact BO Date integrator has an established real-time engine that is industrial strength right out of the box...

so in 3 years you are correct...M$ will be in ETL but the other ETL players will be so far advanced and have a huge customer base that nobody will want to switch - think about it... once you pick a certain database or technology the winds of change don't force you to go buy a new product because of the huge conversation effort ($$$) required...

talk to you in 3 years.

regards,
Peter
 
Status
Not open for further replies.

Part and Inventory Search

Sponsor

Back
Top