Datastage slow performance

Post questions here relative to DataStage Server Edition for such areas as Server job design, DS Basic, Routines, Job Sequences, etc.

Moderators: chulett, rschirm, roy

Post Reply
sreesuku2
Participant
Posts: 45
Joined: Tue Oct 22, 2013 11:45 am

Datastage slow performance

Post by sreesuku2 »

Hi Team,

From that last 2 days all the Datastage jobs are running slow in our server.
Even extraction from sequential files seems to be slow.

We have around 60 GB Free space in the Drive where DS is installed. All Hashfiles and output files are kept in separate Drive which has around 200 GB Free space.

We are trying to clear the job logs using Director. Is there any way to clear all logs ?

What all other steps we should perform to make the jobs run faster

Regards
Sree
sreesuku2
Participant
Posts: 45
Joined: Tue Oct 22, 2013 11:45 am

Post by sreesuku2 »

Can I delete the RT_Log files from the project directory ?

What are RT_Status and RT_BP Files ?
priyadarshikunal
Premium Member
Premium Member
Posts: 1735
Joined: Thu Mar 01, 2007 5:44 am
Location: Troy, MI

Post by priyadarshikunal »

Do not delete RT_BPnnn or any file as such from there, RT_LOG* or RT_STATUS* can be cleared but for that use CLEAR.FILE <file name> instead from dssh/uvsh or from command window of administrator client.
Priyadarshi Kunal

Genius may have its limitations, but stupidity is not thus handicapped. :wink:
jerome_rajan
Premium Member
Premium Member
Posts: 376
Joined: Sat Jan 07, 2012 12:25 pm
Location: Piscataway

Post by jerome_rajan »

Also have a peek into the &PH& directory to ensure that the number of files isn't too much. You decide how much is too much.
Jerome
Data Integration Consultant at AWS
Connect With Me On LinkedIn

Life is really simple, but we insist on making it complicated.
sreesuku2
Participant
Posts: 45
Joined: Tue Oct 22, 2013 11:45 am

Post by sreesuku2 »

Hi,

Thanks for input. Is there any way to clear multiple log files using the Clear.File command ?
chulett
Charter Member
Charter Member
Posts: 43085
Joined: Tue Nov 12, 2002 4:34 pm
Location: Denver, CO

Post by chulett »

No. And it's CLEAR.FILE as in all CAPS as was posted.
sreesuku2 wrote:Can I delete the RT_Log files from the project directory ?
Also wanted to emphasis the previous answer: NO. What makes you think they are the problem? Are they 'large' and if so, do you not have auto-purge enabled for your job logs? This also assumes that you have your logging going to the Universe repository rather than XMETA, any idea what your LOGGING settings are?
-craig

"You can never have too many knives" -- Logan Nine Fingers
sreesuku2
Participant
Posts: 45
Joined: Tue Oct 22, 2013 11:45 am

Post by sreesuku2 »

Hi,

I don't have the auto purging enabled. One project have the log size close to 20 GB (Thats the maximum size).
Where can I see my LOGGING settings ?
chulett
Charter Member
Charter Member
Posts: 43085
Joined: Tue Nov 12, 2002 4:34 pm
Location: Denver, CO

Post by chulett »

Enable auto purge in the Director. Now. Also found a decent discussion that should help here.
-craig

"You can never have too many knives" -- Logan Nine Fingers
sreesuku2
Participant
Posts: 45
Joined: Tue Oct 22, 2013 11:45 am

Post by sreesuku2 »

Hi,

I have read somewhere Auto Purging is only possible for the the jobs which are getting created newly. Is it possible to do Auto-purge for existing jobs ?
PaulVL
Premium Member
Premium Member
Posts: 1315
Joined: Fri Dec 17, 2010 4:36 pm

Post by PaulVL »

mass recompile would do the trick.
chulett
Charter Member
Charter Member
Posts: 43085
Joined: Tue Nov 12, 2002 4:34 pm
Location: Denver, CO

Post by chulett »

sreesuku2 wrote:Is it possible to do Auto-purge for existing jobs ?
Of course. What you read was than in older versions Auto Purge settings / changes would only be automatically applied to new jobs. You can always enable it one by one for any job you desire. Newer versions will propagate Auto Purge settings to all jobs, something easy enough to test to see for yourself. What version are you running?
-craig

"You can never have too many knives" -- Logan Nine Fingers
sreesuku2
Participant
Posts: 45
Joined: Tue Oct 22, 2013 11:45 am

Post by sreesuku2 »

I am on V 8.5
chulett
Charter Member
Charter Member
Posts: 43085
Joined: Tue Nov 12, 2002 4:34 pm
Location: Denver, CO

Post by chulett »

I'm fairly certain that version will take your project level Auto Purge settings and propagate them down to all jobs automagically. Give it a try, let us know!
-craig

"You can never have too many knives" -- Logan Nine Fingers
PaulVL
Premium Member
Premium Member
Posts: 1315
Joined: Fri Dec 17, 2010 4:36 pm

Post by PaulVL »

Just remember that the auto purge only happens when you actually update a job log entry.
chulett
Charter Member
Charter Member
Posts: 43085
Joined: Tue Nov 12, 2002 4:34 pm
Location: Denver, CO

Post by chulett »

Right - it will start purging them the next time it runs, assuming the job does not fail. Oh, and by 'let us know' I meant check a job via the Director to see what the current Auto Purge settings are that it has, no need to run the job just to check that.
-craig

"You can never have too many knives" -- Logan Nine Fingers
Post Reply