Datastage V3.6r2 (ODBC V3.11) on Compag ( Dec ) Alpha runnin
Datastage V3.6r2 (ODBC V3.11) on Compag ( Dec ) Alpha runnin
This is a topic for an orphaned message.
Hello, I am using DS V3.6 running a DEC ALPHA using ODBC against a RedBrick Database.
What I have found is that it depends on how much you are trying to do in your jobs. I had some jobs that had about 15 hash file look ups, 4 transformer stages and an ODBC connection which was pulling out between 4 - 13 million rows of data from the RedBrick database. What I had to do was split the job into multiple steps. The first one pulls the data from the database with the ODBC and creates a sequential file. The sequential file becomes the input for the second job which has about 7 hash file look ups and just one transformer stage, this job created a sequential file that became the input for a third job and so on. I ended up with 4 jobs.
The old way the jobs processed 20 - 80 rows a second and took days to run. The new way the first job with the ODBC gets up as high as 900 rows a second and the other steps get as high at times as 1500 rows a second.
Im not sure what you have in your jobs but you might want to split them up. The ODBC is the slowest thing you can use and if you add additional transformer stages and hash file look ups you are just compounding the problem.
Hope this helps.
Mark Griffin
AMP Sr. Application Developer
Data Warehouse Systems
CIGNA Healthcare
Two College Park Drive, 454
Hooksett, NH 03106
Phone: 603-430-7610
603-268-7239
Fax: 603-268-7909
email: griffimk@hlthsrc.com
>>> LeonK@bsbs.co.nz 08/02 5:54 PM >>>
Does anyone have experience with running Datastage V3.6 r2 on a Dec Alpha running TRU64 Unix V5.0(Rev9.10). Using ODBC V3.11 ( Intersolv )
Most importantly we need to improve the throughput.
We are also having problems with ODBC warning 6066:[INTERSOLV][ODBC 20101 driver]6060.
Has anyone tried using Informix ODBC drivers instead of the ODBC drivers supplied with datastage? If so which driver manager did you use and where did you get it from? We cannot get the informix ODBC driver to work with Datastage. It does work with demoodbc.branded.
We have tried the Informix CLI Plugin, this is just as slow as ODBC. CLI also has other issues like, not supporting text datatype, single database error returning as multiple warning messages in datastage. Informix CLI Version 1.0 ( For Tru64)
On average our throughput is about 20 rows per second, although initial loading is < 10 rows per second.
Details of our System setup follow.
Datastage Server.
Alpha running TRU64 Unix V5.0 (Rev.910),
Datastage V3.6r2 on Compag ( Dec )
Informix CLI Version 1.0 ( For Tru64)
Intersolv ODBC Driver 3.11 (Driver manager v2.x????)
Informix ODBC Driver 3.32 (driver manager v2.x ????)
Ods / Datamart Server
Alpha running TRU64 Unix V5.0 (Rev.910),
Datastage V3.6r2 on Compag ( Dec )
Informix 9.21 (Also on Dec Alpha)
Data Source Server
JDE
Oracle V8
AIX RS6000
Thanks in Advance
Hong / Leon
------------------------------------------------------------------------------
CONFIDENTIALITY NOTICE: If you have received this e-mail in error, please immediately notify the sender by e-mail at the address shown. This e-mail transmission may contain confidential information. This information is intended only for the use of the individual(s) or entity of whom it is intended even if addressed incorrectly. Please delete from your files if you are not the intended recipient. Thank you for your compliance.
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
What I have found is that it depends on how much you are trying to do in your jobs. I had some jobs that had about 15 hash file look ups, 4 transformer stages and an ODBC connection which was pulling out between 4 - 13 million rows of data from the RedBrick database. What I had to do was split the job into multiple steps. The first one pulls the data from the database with the ODBC and creates a sequential file. The sequential file becomes the input for the second job which has about 7 hash file look ups and just one transformer stage, this job created a sequential file that became the input for a third job and so on. I ended up with 4 jobs.
The old way the jobs processed 20 - 80 rows a second and took days to run. The new way the first job with the ODBC gets up as high as 900 rows a second and the other steps get as high at times as 1500 rows a second.
Im not sure what you have in your jobs but you might want to split them up. The ODBC is the slowest thing you can use and if you add additional transformer stages and hash file look ups you are just compounding the problem.
Hope this helps.
Mark Griffin
AMP Sr. Application Developer
Data Warehouse Systems
CIGNA Healthcare
Two College Park Drive, 454
Hooksett, NH 03106
Phone: 603-430-7610
603-268-7239
Fax: 603-268-7909
email: griffimk@hlthsrc.com
>>> LeonK@bsbs.co.nz 08/02 5:54 PM >>>
Does anyone have experience with running Datastage V3.6 r2 on a Dec Alpha running TRU64 Unix V5.0(Rev9.10). Using ODBC V3.11 ( Intersolv )
Most importantly we need to improve the throughput.
We are also having problems with ODBC warning 6066:[INTERSOLV][ODBC 20101 driver]6060.
Has anyone tried using Informix ODBC drivers instead of the ODBC drivers supplied with datastage? If so which driver manager did you use and where did you get it from? We cannot get the informix ODBC driver to work with Datastage. It does work with demoodbc.branded.
We have tried the Informix CLI Plugin, this is just as slow as ODBC. CLI also has other issues like, not supporting text datatype, single database error returning as multiple warning messages in datastage. Informix CLI Version 1.0 ( For Tru64)
On average our throughput is about 20 rows per second, although initial loading is < 10 rows per second.
Details of our System setup follow.
Datastage Server.
Alpha running TRU64 Unix V5.0 (Rev.910),
Datastage V3.6r2 on Compag ( Dec )
Informix CLI Version 1.0 ( For Tru64)
Intersolv ODBC Driver 3.11 (Driver manager v2.x????)
Informix ODBC Driver 3.32 (driver manager v2.x ????)
Ods / Datamart Server
Alpha running TRU64 Unix V5.0 (Rev.910),
Datastage V3.6r2 on Compag ( Dec )
Informix 9.21 (Also on Dec Alpha)
Data Source Server
JDE
Oracle V8
AIX RS6000
Thanks in Advance
Hong / Leon
------------------------------------------------------------------------------
CONFIDENTIALITY NOTICE: If you have received this e-mail in error, please immediately notify the sender by e-mail at the address shown. This e-mail transmission may contain confidential information. This information is intended only for the use of the individual(s) or entity of whom it is intended even if addressed incorrectly. Please delete from your files if you are not the intended recipient. Thank you for your compliance.
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++