Search found 1304 matches

by PaulVL
Wed Jul 14, 2021 6:23 am
Forum: IBM<sup>®</sup> DataStage Enterprise Edition (Formerly Parallel Extender/PX)
Topic: Join Stage slowness in parallel job
Replies: 6
Views: 230

Re: Join Stage slowness in parallel job

1,000,000 / 200 = 5,000.

Try that as your array size.
by PaulVL
Mon Jul 12, 2021 7:16 am
Forum: IBM<sup>®</sup> DataStage Enterprise Edition (Formerly Parallel Extender/PX)
Topic: Join Stage slowness in parallel job
Replies: 6
Views: 230

Re: Join Stage slowness in parallel job

Don't go boosting that array size without taking into account what it is doing to your memory buffers.

Here's my rule of thumb. Each stage has (by default) about 3MB of buffer space. I always recommend calculating your row length, then taking a third of that buffer space and see how many rows fit into that space. so 1,000,000 / row length = X. Round X down to the nearest 100 value. Use that number for your array size. That way, you will not cause datastage to write temporary buffer files to your TMPDIR location. Each time you read and write to disk you slow the process down.

Next is the very misleading rows per second you may be seeing when doing a monitor activity. It misleads to you. Plain and simple. I always look at the live data update counts. Set your refresh to 10 seconds. Write down the value you see on your desired stage. Wait for the refresh then write down the new value. New Value - Old Valud = Num of rows in that 10 seconds. Then divide by 10. That is your CURRENT rows per second on that stage.

If you just look at the right hand side rows per second value, it does not reflect your current speed of processing data in that stage because the calculation is based upon START TIME of job, not the start time in which that stage got it's first row.
by PaulVL
Fri Jul 09, 2021 7:10 am
Forum: IBM<sup>®</sup> DataStage Enterprise Edition (Formerly Parallel Extender/PX)
Topic: How to set the path for the heap dump files via DS project environment variables
Replies: 3
Views: 171

Re: How to set the path for the heap dump files via DS project environment variables

I would experiment with changing these two environment variables. $HOME and/or CWD. Set those at a job level. Force your job to fail and see where the dump goes.
by PaulVL
Mon Jun 14, 2021 8:19 am
Forum: IBM<sup>®</sup> DataStage Enterprise Edition (Formerly Parallel Extender/PX)
Topic: Using sql file to insert data in the table.
Replies: 2
Views: 465

Re: Using sql file to insert data in the table.

Perhaps you can post how you resolved it so that future generations digging into the archives can also find the solution. :lol:
by PaulVL
Thu Jun 10, 2021 6:09 am
Forum: IBM<sup>®</sup> DataStage Enterprise Edition (Formerly Parallel Extender/PX)
Topic: Exporting executables - best practice
Replies: 2
Views: 454

Re: Exporting executables - best practice

Best is like saying Chocolate is way better than Vanilla. It's a matter of taste. But... Standard Practice... Yes. I would say migrating objects with executables is Standard Practice.

This allows for a full Unix side automated migration without any manual intervention or super fancy call to a windows client compile command. This would facilitate a CICD deployment. Do not underestimate that buzzword, managers and higher ups find that it is all the rage.

When you migrate an object that has the executable in it, you most likely pulled it from an external source control repo (github). If you are not doing that, then it's a silly process that should be rewritten because disasters happen and you need to be able to rebuild or back out code to a prior version level.

The Object coming from the repo will most likely be placed into a higher tier environment (sit/stage/uat/prod) where regular folks do not have developer authority due to security compliance reasons. Typically only an admin or a batch ID / code migration ID has authority to load code and/or compile. OPS folks should never have compile authority in anything other than DEV.

If you are working in a large shop, you are bound to get an audit done. They will ask who has access to compile, edit, migrate code into prod and what security measures are in place to control that. If any John Doe can edit and compile, you've pretty much allowed them keys to the kingdom and any type of code migration rules are thrown out the window.

From an admin point of view, I only allow Admins and Batch IDs, where the application teams do not have the password, to be able to load code into sit/stage/uat/prod. If I ever allowed an application team full access, they would "just make a quick fix" in prod and never reapply that back to baseline dev. Then the next time the edit in dev, they do not have that fix, and it breaks prod again.

There will be a war room situation where an admin has to do an emergency fix and compile with application team guidance, but that fix would then be loaded back to dev because it would be tracked.

Nope... I've been an admin in some of the biggest shops around... never going to allow users compile access in prod. I'd rather abdicate ownership of that datastage setup rather than do that.
by PaulVL
Mon May 03, 2021 5:50 pm
Forum: General
Topic: Hierarchy stage not working with flash gone now
Replies: 1
Views: 829

Re: Hierarchy stage not working with flash gone now

Reach out to IBM to see what they can work out with your company. They have worked with others.
Depends how big you are of course.
by PaulVL
Mon Aug 24, 2020 6:01 am
Forum: IBM<sup>®</sup> DataStage Enterprise Edition (Formerly Parallel Extender/PX)
Topic: Datastage on Azure
Replies: 2
Views: 1038

Those are all questions that you need to be asking IBM via a Case ticket (formerly PMR ticket).
by PaulVL
Tue Jul 21, 2020 11:19 am
Forum: IBM<sup>®</sup> Infosphere DataStage Server Edition
Topic: IBM Datastage / db2 connection Issue
Replies: 2
Views: 1826

Welcome to DSXchange Jinaanoo, just to be clear, nobody here is your official support team. We HELP however we can.

You should reach out to your local DataStage admin team.
You should reach out to your local DB2 DBA team.
You should reach out to your IBM support if the first two do not work out. Typically that task would be for your DataStage admins to coordinate.

That being said, here is what I would do (Note: I'm a DataStage admin).

I would start by trying your connectivity test from the Unix command line. Use the installed db2 client and try that connection (from unix DataStage host).
I see that you are using a SERVER job, please make sure of that since a Parallel job "might" have been dispatched to a different host if your APT file told it to.
by PaulVL
Sun Jul 19, 2020 3:17 pm
Forum: General
Topic: DS created database connections
Replies: 1
Views: 855

are you doing a read or a write action?

Partitioned Read?

Anything in your SQL statement to pump up the connections?

Nodes in the config file will not matter. You can drop the Oracle Connector stage to sequential mode vs parallel, but, you have to look at those settings.

Is this a Multi Instance job?
Do you have a "before SQL" statement or a before job routine that is also establishing a connection?
by PaulVL
Sun Jul 19, 2020 3:11 pm
Forum: General
Topic: DataStage Cloud Pak
Replies: 0
Views: 830

DataStage Cloud Pak

Is anyone using Information Server (DataStage) Cloud Pak module?

Would like to get your opinion on it. Please msg me privately.

Thanks
by PaulVL
Mon Jul 13, 2020 8:07 am
Forum: General
Topic: Export/Import environment variables from the command line
Replies: 2
Views: 1079

I would NOT recommend copying the dsparms file from one project to another. You do not know if the difference in release level of datastage also constitutes new or removed entries in the dsparms file.

You could script it, but then again, you will have trouble separating the user parms vs the default project (tool) parms.

As an administrator of many years... I would not script that (and I love scripting stuff). Use the GUI admin tool, highlight the user section and copy those entries. You can export it into a file and load into the other host/project.

Ask your admin if there were any other non user entries that have been changed from their default values and edit those accordingly.
by PaulVL
Tue Jun 02, 2020 9:16 am
Forum: IBM<sup>®</sup> Infosphere DataStage Server Edition
Topic: Server Jobs Deprecated
Replies: 6
Views: 3532

Is there an official link to an IBM statement on this? Need to forward to some internal customers.