Hey, no one is using this forum, so I thought I'd stir the pot a little bit. I'm really curious as to how people have used/are using Datastage.
I thought I'd share a little of my experiences using DataStage, as a way to start a discussion as to its usage. Almost all of my projects were traditional batch ETL for data warehouses. I did have one major client for which my team used it both for ETL for the warehouse and data marts, and as a cyclical batch data bridge full duplex between Siebel and Peoplesoft. But, most clients have been daily batch processes.
With the incorporation of RTI, I know the old school method of using DataStage Server/PX for near-realtime type of work is going to probably change, but I am curious.
How is DataStage Server/PX being utilized out there
How is DataStage Server/PX being utilized out there
Kenneth Bland
Rank: Sempai
Belt: First degree black
Fight name: Captain Hook
Signature knockout: right upper cut followed by left hook
Signature submission: Crucifix combined with leg triangle
Rank: Sempai
Belt: First degree black
Fight name: Captain Hook
Signature knockout: right upper cut followed by left hook
Signature submission: Crucifix combined with leg triangle
We are presently using the tool for a single collective philosophy -- where it is believed that a single massive database design can compliment the entire range of clients that we have (not just in one section of the market... ALL MARKETS).
We've used ETL where we used Pro*C for ODS development and it taught us the shortcoming of using ETL for such a purpose. We devised a Sandbox philosophy for everything before ODS, just cleanse the data, and use a singular massive load program (one for each section - Brand, Store, ... et cetera).
Then we have the transfer to Data Warehouse. Here, it's 100% PX ETL, even the transfer. Well, okay, there may be a couple of PL/SQL scripts here and there, but only because we hired DBA folks who most strongly prefers to stick to PL/SQL for their work, not those pukey ETL stuff.
Even our Data Marts are now 100% PX ETL driven.
Pushing to do everything under PX when it's in its infancy at 6.0 really taught us the limits far earlier than most people would learn them. Within two week of getting the tool, we were already struggling on figuring out how to output a pristine fixed format file, something that is so laughable easy using Server.
It was expected by our big folks that we would be releasing our largest clients (which is a TRUE Data Warehouse sized database) version within approximately a couple months after getting the tools. Last Christmas was horrible, where I had to cancel a trip to my family in order to get the program running reasonably well.
And here we are, my Thanksgiving trip cancelled... for the same client.
Mind you, a number of other clients were completed and released, but they are about 3-10 times smaller in scope, size, and coverage compared to this particular client.
At least we are at the phase where we are finally stabilized on most of the ODS and Data Warehouse programs (producing accurate data in a reasonable time... not good, but... reasonable), and I definitely plans on seeing my poor Dad this Christmas.
Of course, it was all done without MetaStage, which we have but have not used beyond the most basic attempts, QualityStage, which I think we have an evaluation licence only for, AuditStage, which is nonexistant here, and whatever else that may be out there that we should be checking out. It was one of those thing where managment were oversold the tools, and thought that it was the magic pixie dust that would work immediately. So damn the need to do R&D and evaluate the tools, and understanding what to do. It's full speed ahead.
There is a lot of interesting experinces over the past year with this tool, many of it our fault, with quite a bit DataStage's fault. Growing pains, the need of a paradigm shift, misconception of the product, corners being cut, and so on forth. Thank god I went through this -- I learned so much.
But I'll kick the ass of the Data Warehouse Architect if that person would drag me through this process again without a good plan!
-T.J.
We've used ETL where we used Pro*C for ODS development and it taught us the shortcoming of using ETL for such a purpose. We devised a Sandbox philosophy for everything before ODS, just cleanse the data, and use a singular massive load program (one for each section - Brand, Store, ... et cetera).
Then we have the transfer to Data Warehouse. Here, it's 100% PX ETL, even the transfer. Well, okay, there may be a couple of PL/SQL scripts here and there, but only because we hired DBA folks who most strongly prefers to stick to PL/SQL for their work, not those pukey ETL stuff.
Even our Data Marts are now 100% PX ETL driven.
Pushing to do everything under PX when it's in its infancy at 6.0 really taught us the limits far earlier than most people would learn them. Within two week of getting the tool, we were already struggling on figuring out how to output a pristine fixed format file, something that is so laughable easy using Server.
It was expected by our big folks that we would be releasing our largest clients (which is a TRUE Data Warehouse sized database) version within approximately a couple months after getting the tools. Last Christmas was horrible, where I had to cancel a trip to my family in order to get the program running reasonably well.
And here we are, my Thanksgiving trip cancelled... for the same client.
Mind you, a number of other clients were completed and released, but they are about 3-10 times smaller in scope, size, and coverage compared to this particular client.At least we are at the phase where we are finally stabilized on most of the ODS and Data Warehouse programs (producing accurate data in a reasonable time... not good, but... reasonable), and I definitely plans on seeing my poor Dad this Christmas.
Of course, it was all done without MetaStage, which we have but have not used beyond the most basic attempts, QualityStage, which I think we have an evaluation licence only for, AuditStage, which is nonexistant here, and whatever else that may be out there that we should be checking out. It was one of those thing where managment were oversold the tools, and thought that it was the magic pixie dust that would work immediately. So damn the need to do R&D and evaluate the tools, and understanding what to do. It's full speed ahead.
There is a lot of interesting experinces over the past year with this tool, many of it our fault, with quite a bit DataStage's fault. Growing pains, the need of a paradigm shift, misconception of the product, corners being cut, and so on forth. Thank god I went through this -- I learned so much.
But I'll kick the ass of the Data Warehouse Architect if that person would drag me through this process again without a good plan!
-T.J.
Developer of DataStage Parallel Engine (Orchestrate).
A while back we had a brief discussion about realtime usage in a data warehouse environment. Here's an article that says what I was trying to say but in a lot better english.
http://www.intelligententerprise.com/03 ... ?/data_int
http://www.intelligententerprise.com/03 ... ?/data_int
This is just a continuation of my request for commentary on how DataStage may have been used.
http://www.intelligententerprise.com/03 ... ?/data_int
http://www.intelligententerprise.com/03 ... ?/data_int
This is just a continuation of my request for commentary on how DataStage may have been used.
Kenneth Bland
Rank: Sempai
Belt: First degree black
Fight name: Captain Hook
Signature knockout: right upper cut followed by left hook
Signature submission: Crucifix combined with leg triangle
Rank: Sempai
Belt: First degree black
Fight name: Captain Hook
Signature knockout: right upper cut followed by left hook
Signature submission: Crucifix combined with leg triangle