How long will DataStage still live?

ogmios · Post by **ogmios** » Wed Dec 22, 2004 6:13 am

This is not really a technical question. I would like to compare some 'feelings' with other DataStage developers/administrators.

<rant on>

How long do you think DataStage will still live and in which form? Most of us hope that DataStage will be around for a long time, but what if it isn't: compared to Oracle the income of Ascential is a 'rounding error'.

- Did anyone already think of a migration path in case DataStage ceases to exist, I made an estimate of the amount of effort to convert all 'our' DataStage jobs to some other tool (I based myself on a migration to Informatica), with around 10000 DataStage jobs management kind of dropped from their seats when I gave them my estimate.

- Are you using all functionality of DataStage, or only the minimal amount to get the job done. If you only have SQL input queries being run and output just being insert/updated in another database, chances are large that moving to another tool is a 1-to-1 migration.

- Will DataStage still exist in the current form in 5 years or will Ascential try to do something as SeeBeyond, which did quite a paradigm shift going from their own internal language in one version to using Java as scripting language in their next version. I personally feel UV is at the end of its life and don't know what will become of it in 5 years time, on the other hand I don't see an easy migration path for DataStage to use something else, as DataStage is only a 'shell' above UV converting GUI jobs in BASIC code.

- Wouldn't developing your own little framework e.g. in perl or java and using that to make jobs be cheaper in the long run? If you don't need a SAP interface, ... You would need some very technical people to get the thing going, but at 100K+ DataStage licensing costs... Why are there no good open source initiatives for this?

<rant off>

Ogmios

kcbland · Post by **kcbland** » Wed Dec 22, 2004 10:41 am

PX is the future platform, not Server. It's only a matter of time. I sure hope they make it more friendly before they bury Server. The Designer stages are, IMHO, unfriendly and unwieldy. The job designing is arduous, whereas Server is so easy to use. I think once the Windoze version of PX is released, the push will be on to retire Server as there's no Unix-only platform limitation at that point. All of the other tools (Profile, Quality, etc) are built on the PX technology, so there's your answer.

datastage · Post by **datastage** » Wed Dec 22, 2004 11:00 am

kcbland wrote:PX is the future platform, not Server. It's only a matter of time. I sure hope they make it more friendly before they bury Server. The Designer stages are, IMHO, unfriendly and unwieldy. The job designing is arduous, whereas Server is so easy to use. I think once the Windoze version of PX is released, the push will be on to retire Server as there's no Unix-only platform limitation at that point. All of the other tools (Profile, Quality, etc) are built on the PX technology, so there's your answer.

Any idea as to whether the Windoze PX will priced similarly to Windoze Server or can we expect a noticeable price increase eliminating DataStage from the SME marketplace?

Also, Ogmios didn't go specifically into a purchase of Ascential, but I am curious if it were to happen in the next 2.5 years which company would be the most likely to make an aquisition of Ascential?

kduke · Post by **kduke** » Wed Dec 22, 2004 11:18 am

I doubt if anyone can afford Ascential at over a billion. The return on investment is not there. Eventually Oracle and Microsoft will have good alternatives. Unless something drastic happens then 5 years is easy to see a good future. They have locked in SAP and PeopleSoft which makes for a very good future by themselves. The connectivity also makes Ascential stand out. I doubt if anyone can build something to connect to IBM mainframes and eveythign else the DataStage does. I am not sure they need to. Those are niche markets which usually helps SAP and the big software companies integrate better. Look for the little guys like Embarcedaro or whatever else is out there to get bought by someone. DTS and the Oracle product have got to improve. They are both an embaressment to their companies.

The metadata needs may change all this. These products plus the reporting tools either OLAP or otherwise need to co-exist better at the metadata level. Look for these tools and Web Methods and tools like that to merge. I do not think the database companies need tools like this to stay around but if they had good versions then Oracle along could seriously hurt Ascential. Most of the places I have worked are Oracle shops. If Oracle had a really good product then people would not buy DataStage.

ogmios · Post by **ogmios** » Wed Dec 22, 2004 11:34 am

One of the big "upcoming" contenders in the niche would be SAS with their end-to-end capabilities (and also including mainframe).

If anyone is going to buy Ascential it would probably be for a fade-out scenario as IBM did/is doing with Informix. So they'd better not get bought :D

In a worst case scenario we'd probably get about 1 or 2 years to convert any DataStage job to something else, until the surrounding technology (Oracle, OS) is no longer supported by DataStage. And although it's not likely to happen, it can. I've been hit by similar things in the past where tool-vendors just go belly up.

Ogmios

kcbland · Post by **kcbland** » Wed Dec 22, 2004 11:45 am

Also look at how all of the aspects of ETL are being positioned to make it a "Universal Data Connector". There's that funny IBM commerical where this massive black box is sitting on a conference table surrounded by senior executives of some nameless software company. The black box has every kind of plug hanging out of it, as well as zillions of ports and receptacles. I think I even saw a kitchen faucet with knobs attached, serial ports, electrical outlets, cell phone jacks, parallel ports, 7pin/15pin/25pin din, fibre channel slots, etc.

The CEO is saying: What the heck is that? The veeps are smiling and saying it's a univeral data connector. You can plug anything into it and attach anything to any plug coming out of it. The magic happens inside the box, nice and neat. Anything connects to anything and the box takes care of all of the hard work, analysis, programming, etc. The ultimate solution because it's also infinitely scalable and as simple as just a black box.

The commercial ends with the CEO saying: We'll be rich.

newtier · Post by **newtier** » Wed Dec 22, 2004 2:15 pm

Gee...DataStage as "middleware"....what a concept!

If it only worked with "older" middleware, like MQSeries...Oh wait, it does!

As an architect for a corporation, some questions you should always consider with any technology which has proprietary languages (DataStage, bi tools, code generators, etc.) or limited target architectures (ms's products only running on ms platforms, big O's tools only running against their db, etc.) are:
1) "How much are we as a company willing to invest in building business logic into a product that will make it difficult to transition out at a later point?"
2) "Are we willing to commit to this technology for a long period of time, knowing that someday we will have to incur a cost, whatever that is, to replace the technology and the business functionality that was built into it?"
3) "How can we minimize our dependence on this technology and still get significant ROI from its use?"
4) "Does the technology have 'openings' that allow us to use non- or less-proprietary products to build the business logic (such as Java)?"
5) "Does the vendor have a long-term vision and strategy (and sales to support it) that give us reason to believe they will evolve with the market-demands and not lag too far behind?"
6) "Does this vendor show a pattern of protecting customer investments by building/buying replacement technologies for the next technology cycle, and providing translators from the 'old' to the 'new' technology?"
7) "Is this technology likely to have a big enough customer base that a future replacement technology vendor can justify writing a translator to help preserve as much of the business logic investment as possible without directly rewriting?"

And if you'll notice, Ascential is positioning itself and its offerings as more than just a "tool vendor". They are putting together a "strategy" behind their suite, and building it around a meta data architecture (a trend that is still in its early stages). This should allow it a great opportunity to keep pace with future market demands and technology innovations.

MetaStage as a "product" will be going away, and the repository will just "be there" with a (hopefully) much friendlier interface for native manipulation where it makes sense.

kduke · Post by **kduke** » Wed Dec 22, 2004 2:38 pm

I think all of these are excellent questions. I also think Ascential has a good answer for all of them at least as good as any player in the marketplace now. I think IBM would benefit the most from owning DataStage but they had the chance and let them go. It would take an IBM or Oracle or someone of that stature to unseat Ascential. They have lots of momentum. When next year's version hits and if it works well then they will definitely have a strong future. If it comes out buggy then they could also hurt themselves.

Athorne · Post by **Athorne** » Wed Dec 22, 2004 4:34 pm

I'm not sure if you all are aware or not but Oracle just basically purchased PeopleSoft last week. (Not sure how anyone in the field wouldn't know about this by now.)

Point is, I'm sure it won't be long before Oracle muscles out Ascential and puts in their own tools. That will certainly take a large chunk of revenue that Ascential would have recieved from the PeopleSoft customer base.

Anyone that was in the field around the time of the browser wars between Microsoft and Netscape can probably see the horrific vision I think of where Oracle puts poison pills into it's Database Platform to make Ascential slower.

That would be an extreme scenario but I wouldn't doubt they could put proprietary technology in place to make pure Oracle products more effecient against their DB platforms.

I predict an increase in DB2 implementations in about a year when Oracle tells their purchased PeopleSoft customer base that they will now be running Oracle for their ERP and Database needs.

Don't mind me, just the ramblings of a fool with nothing better to do during the Christmas Holiday week. :D

mhester · Post by **mhester** » Wed Dec 22, 2004 5:35 pm

I think Ascential's marketshare is not likely to decrease because of such acquisitions and the goal is not to be an ETL vendor anyway, but a vendor that can help manage data no matter where the data comes from or where it is going. This view is a corporate view of which ETL plays a small (but critically important) role and without robust tools, processes and methodologies the management of this data is almost impossible.

It's a more "global" view rather than specific and this is something that MS and Oracle do not yet have as far as Information Asset Management so I still think Ascential is leading the market and will probably continue to be a major player for many years.

kduke · Post by **kduke** » Wed Dec 22, 2004 9:54 pm

I agree but what if DTS all of a sudden was robust and as easy to use as DataStage, Maybe the next release of DS was not as easy to use or only as easy to use as PX is now. What if Oracle developed a product as easy to use and robust and included it with the price of Oracle. Maybe it could only talk to Oracle and ODBC but it is free instead of hundreds of thousands of dollars. It could seriously damage Ascential if either of these happened.

Our focus would change from DataStage experts to data warehouse experts. I think the likelyhood of one or both of these happening is strong. Most of these tools are price driven and not based on wise forward thinking by corporate IT professionals. Most corporations have little or no vision to metadata or even the power of OLAP. These are hard to prove ROI numbers. Metadata is almost impossible to prove. The idea of documenting ETL is lost even among DSX users. Documenting ETL is a form of metadata. Most are only concerned about making it work then making it run faster. If Oracle or MS had tools included with their databases which worked well and ran fast then look out DataStage. A lot of shops use only one database provided by these two.

I say it will happen in both databases and in the next 5 years. DTS will not be a joke forever. As soon as DTS is a solid product then Oracle will answer. How much would a company pay for DataStage when they have a good solid product already? How many companies pay for SQL Navigator when Toad is free. These companies know how to narrow the field to themselves and as few others as possible. Very few companies use anything besides MS Office.

vmcburney · Post by **vmcburney** » Thu Dec 23, 2004 6:13 am

I think much of the change in DataStage over the next few years will be to make a suite that is easier to use. 2006 is going to be a big year for Ascential when they potentially have a suite on the market that no one else can match.

Regarding DataStage PeopleSoft Edition - this will be around for a long time. Oracle cannot replace it with OWB because that tool is specialised towards Oracle warehouse building while PeopleSoft EPM needs to support other database types such as DB2. DataStage is also being used in PeopleSoft upgrades, DataStage is an allrounder and OWB cannot take on this role.

Part of the future of DataStage should be a DataStage-lite server plus a pro-active push of job components to those servers on an as needed basis. Eg. Instead of installing jobs on each server the jobs can be pushed out by the central server when it detects an available machine that does not already have that job.

OWB has got a lot better in the last release and is a real threat. DTS is a bit of a disgrace but has been completely rebuilt for the latest version. Be interesting to see how it goes. These tools are to DataStage what the Sharepoint portal is to the Enterprise Portal market. They will win over departments within companies but they are not an enterprise solution.

Regarding Standard Edition, there was no evidence at Ascential World that it was falling out of favour. I wonder if they will eventually blend the two, either putting PX functionality into Standard Edition or putting hash files, BASIC language and transformers into parallel jobs.

mhester · Post by **mhester** » Thu Dec 23, 2004 8:01 am

Kim,

I agree with your opinion that we need to be Data Warehouse centric instead of Ascential DataStage centric. I think it would be a mistake to tie yourself to a tool rather than an industry. Since leaving Ascential I have tried to bolster my skills as a BI architect rather than just an Ascential ETL developer. I don't really care what tools are used because in truth, the principles of Data Warehousing apply no matter what tool(s) are used.

Regards,

ogmios · Post by **ogmios** » Thu Dec 23, 2004 8:35 am

Question remains: Assume the worst happens, Ascential goes belly up because of fierce competition and no company wants to take over DataStage (it's a niche market, it doesn't pay to get into this kind of stuff under fierce competition). What are you going to do with 10.000 DataStages jobs that use the full functionality of DataStage, is someone out there taking this currently into account?

DataStage is enormously good at locking in people... ever tried converting a complex DataStage job to Informatica?

I can already see it before me: "We would need at least 150 DataStage/Informatica Developers for 1 year to do the conversion of DataStage to something else... assuming the brick-theory holds", I can already see a few of management drop off their chair.

Ogmios

kduke · Post by **kduke** » Thu Dec 23, 2004 10:32 am

I agree you have a problem converting 10,000 jobs but I doubt if Informatica is going to out last DataStage. Also automating a conversion from DataStage to some other product would be difficult. I think DataStage excels in transformation. Even automating a conversion from server to PX would be difficult or 390. Forget about it. Only straight table copies could be converted. That would be about 80 percent of most jobs as a wild guess.

I do not think DataStage is going away soon. It may only survive at mercy of the giants like Oracle and MS. I do think it could always have a niche market with the 390 and other unique connection capabilites. Vincent is correct though in 2006 they should see jump in business. If all the pieces work together well then it will be an awesome product. It is a very expensive product though and they bet the farm on this release. How do they pay for all this if they get into a price war with free or cheap products from Oracle and MS.

If I was Ascential I would want us, the developers, to be so knowledgable and so loyal that we would beg not to leave. I would get Ray's book to market as fast as possible. I would lock us in as fast as possible. If we jump ship to other products who is going to convince major corporations to buy an expensive tool like DataStage when DTS or OWB can do the job for less and with less expensive talent. Talent trained at the local university and by super cheap books at the local bookstore. Lets face it we are also a part of solution and an expensive one too. I think Ab Inito is really hurting themselves by limiting the number of people that know the product. They try to corner the market on consulting talent. To drive up the cost of all phases of your product is not wise. I think customers get tired of paying more for everything. I think that hurts SAP as well. You can only claim to be the best for so long. Soon you will hear Oracle's and MS's products are not so bad. MS Office was never the best. Probably the same for Oracle could be said. But the overall balance of things it was good enough and a lot more cost effective. Plus more people understood it.

Come to think of it, the best solution would be for SAP to buy Ascential. SAP is not going away for a long time. It costs too much to install that monster. Companies cannot afford to get off of it. SAP may need to become a database company to compete with Oracle. SAP could also take Business Objects or any OLAP tool off the market as well. Where would Oracle be then. MS on the low end and SAP on the high end.