Datastage server directory running out of space

nishant_prakash · Post by **nishant_prakash** » Tue Nov 14, 2006 9:19 pm

Hi Friends,

I am facing this issue of Datastage server directory on UNIX "Ascential/Datastage" running out of space frequently. My system administrator keeps increasing the size of the directory but in a matter of 10 days the directory has grown from 8 GB to 11 GB in size!!! I am baffled because I cant understand the reason for this. The file sytem for writing the datafiles is completely different, so there is no chance of data files being written to datastage project directory.

I have also eneabled Auto purge job log feature in datastage admin. So the log files hould not create a prob either..

Can any of you shed some light on this peculiar issue? Your help will be highly appreciated.

DeepakCorning · Post by **DeepakCorning** » Tue Nov 14, 2006 9:21 pm

Not sure but huge Hashed files??

nishant_prakash · Post by **nishant_prakash** » Tue Nov 14, 2006 9:24 pm

Hi Deepak, as i told earlier the files are written to a completely different directory meant for storing all kinds of data and hashed files.. So why should the datastage direcoty get filled up?

loveojha2 · Post by **loveojha2** » Tue Nov 14, 2006 9:34 pm

Purge the log files periodically, you can set the options also from the Administrator of auto purge.

loveojha2 · Post by **loveojha2** » Tue Nov 14, 2006 9:36 pm

Are there any jobs running with large number of warnings, if yes, then try to modify the jobs in such a way that you don't get warnings at all.

nishant_prakash · Post by **nishant_prakash** » Tue Nov 14, 2006 9:41 pm

loveojha2 wrote:Are there any jobs running with large number of warnings, if yes, then try to modify the jobs in such a way that you don't get warnings at all.

Hi Loveojha2,

As long as the Auto purge feature is enabled in ds admin, it dsnt matter even if the job gives warnings.. I have already tried all that..

Any other suggestions?

loveojha2 · Post by **loveojha2** » Tue Nov 14, 2006 9:46 pm

Check which files are relatively big in size and what is their type (log,RTBP,temp file)?

Just classify the space utilized by the different types of file, that may lead to some clue.

kduke · Post by **kduke** » Tue Nov 14, 2006 9:56 pm

Code: Select all

#!/bin/ksh
# 2GB
find . -size +2000000000c -exec ls -l {} \;

Adjust the size down for smaller than 2GB files.

nishant_prakash · Post by **nishant_prakash** » Tue Nov 14, 2006 9:56 pm

loveojha2 wrote:Check which files are relatively big in size and what is their type (log,RTBP,temp file)?

Just classify the space utilized by the different types of file, that may lead to some clue.

There are a lot of 'TEMP*', 'RT_LOG*' and 'RT_STATUS*' files in the project directory, But I feel these files are generated internally by ds processes and are required.. Do you mean to say that I can DELETE these files?

loveojha2 · Post by **loveojha2** » Tue Nov 14, 2006 9:58 pm

Do you mean to say that I can DELETE these files?

No

Otherwise you will lose your project

nishant_prakash · Post by **nishant_prakash** » Tue Nov 14, 2006 10:01 pm

loveojha2 wrote:
Do you mean to say that I can DELETE these files?
No
Otherwise you will lose your project

Then what are you exaclty suggesting...
I would appreciate if you be clear in your suggestions...

loveojha2 · Post by **loveojha2** » Tue Nov 14, 2006 10:03 pm

How many jobs are there in the project?
If they are not hugh in numbers.
My advise could be taking export of the entire project with the executable and creating a different project with that, then you can make a comparison actually, since you have said that you are auto purging the log and you are not maintaining Hashed Files in the project directory, so there should not be much of a difference.

nishant_prakash · Post by **nishant_prakash** » Tue Nov 14, 2006 10:12 pm

loveojha2 wrote:How many jobs are there in the project?
If they are not hugh in numbers.
My advise could be taking export of the entire project with the executable and creating a different project with that, then you can make a comparison actually, since you have said that you are auto purging the log and you are not maintaining Hashed Files in the project directory, so there should not be much of a difference.

Ok.. can you tell me if it is usual for a project with approx 2000 jobs and sequences to occupy 11 GB of space????

loveojha2 · Post by **loveojha2** » Tue Nov 14, 2006 10:22 pm

Ok.. can you tell me if it is usual for a project with approx 2000 jobs and sequences to occupy 11 GB of space????

We have around 10K maps in one project with around 1 GB of space.

nishant_prakash · Post by **nishant_prakash** » Tue Nov 14, 2006 10:29 pm

loveojha2 wrote:
Ok.. can you tell me if it is usual for a project with approx 2000 jobs and sequences to occupy 11 GB of space????
We have around 10K maps in one project with around 1 GB of space.

So you mean to say that it is normal?

DSXchange

Datastage server directory running out of space

Datastage server directory running out of space

Re: Datastage server directory running out of space