DS Version Control and UFS filesystem limitation

Post questions here relative to DataStage Server Edition for such areas as Server job design, DS Basic, Routines, Job Sequences, etc.

Moderators: chulett, rschirm, roy

Post Reply
bjkilleen
Premium Member
Premium Member
Posts: 18
Joined: Tue Oct 28, 2003 1:27 pm

DS Version Control and UFS filesystem limitation

Post by bjkilleen »

While working with the DS Version Control product, we began experiencing the following error:

Error while creating job SeqGrrServiceStat^1_4
Error on CREATE.FILE commenat (Creating file "DS_TEMP5455" as Type 30.
mkdbfile: cannot create file DS_TEMP544 Unable to create operating system
"DS_TEMP5455" Process cannot continue.)

We believe it may be related to a UFS filesystem limitation on the number of directories/files, 32768. Is there anyone else that has experienced this? What exists as a resolution?

We have tried to "delete" DS components from DS Version Control, but this has not resolved the issue.

Thanks in advance!!!
ray.wurlod
Participant
Posts: 54607
Joined: Wed Oct 23, 2002 10:52 pm
Location: Sydney, Australia
Contact:

Post by ray.wurlod »

I have seen this limit on Solaris; max 32K subdirectories in a directory. In that case it was also the VERSION project (for Version Control); we got around it by converting dynamic hashed files (directories) into static hashed files (O/S files).

Given that each version of each job creates dynamic hashed files, the VERSION project is the most likely to hit this limit.

Another possibility is to archive older versions of jobs, and/or to use multiple version control projects.

Find out how many entries there are in the directory. Of these, how many are subdirectories? ls -l | grep ^d | wc -l
IBM Software Services Group
Any contribution to this forum is my own opinion and does not necessarily reflect any position that IBM may hold.
bjkilleen
Premium Member
Premium Member
Posts: 18
Joined: Tue Oct 28, 2003 1:27 pm

Post by bjkilleen »

Thanks Ray.

How does one convert the dynamic hashed files into static hashed files? After doing this does DS Version Control continue to use(create) the static hashed files?

We have definitely thought about creating multiple version control projects but feel that we may run into the same situation.

When you state " archive older versions of jobs", you are meaning to export the older versions of jobs and then delete them from DS Manager, right?

When I used the command provided, the result was 32765.

Again Ray, thanks for the information.

Brad
ArndW
Participant
Posts: 16318
Joined: Tue Nov 16, 2004 9:08 am
Location: Germany
Contact:

Post by ArndW »

The conversion of one type of a hashed file to another is through the RESIZE command. It is important to ensure that the files are not in use when you do a RESIZE and the use of the "CONCURRENT" option is generally discouraged. If a file is going to be very small (such as the RT_LOGnnn or RT_STATUSnnn files in VCS) then you can do a simple "RESIZE RT_STATUS001 2 1 1" to convert the file to a type 2, modulo 1 and separation of 1.

BTW, this 32Kb inode limit is also present under AIX
chulett
Charter Member
Charter Member
Posts: 43085
Joined: Tue Nov 12, 2002 4:34 pm
Location: Denver, CO

Post by chulett »

ArndW wrote:BTW, this 32Kb inode limit is also present under AIX
:shock: Yikes... have to keep that in mind. They are contemplating moving some of the DS processing from H-PUX over to AIX here and this could definitely be an issue.
-craig

"You can never have too many knives" -- Logan Nine Fingers
ArndW
Participant
Posts: 16318
Joined: Tue Nov 16, 2004 9:08 am
Location: Germany
Contact:

Post by ArndW »

The site that I am at has not seen the problem in the version control project(s), only in certain development projects. But the limitation is present and diagnosing the error when it occurs still takes up time. Converting all RT_STATUS<n>, RT_CONFIG<n> files to static will do the trick.
ray.wurlod
Participant
Posts: 54607
Joined: Wed Oct 23, 2002 10:52 pm
Location: Sydney, Australia
Contact:

Post by ray.wurlod »

RT_STATUSnn and RT_CONFIGnn are good choices because they are always small and static.

To get a rough idea of how big to make the static hashed files, look at how big the DATA.30 and OVER.30 files are. For example:

Code: Select all

ls -l RT_CONFIG11
total 56
-rw-rw-r--   1 aoprxw   dsdevt         4096 Jan 24 11:48 DATA.30
-rw-rw-r--   1 aoprxw   dsdevt        22528 Jan 20 09:36 OVER.30

ls -l RT_STATUS11
total 16
-rw-rw-r--   1 aoprxw   dsdevt         4096 Jan 24 08:55 DATA.30
-rw-rw-r--   1 aoprxw   dsdevt         2048 Jan 10 10:48 OVER.30
Add the two values and subtract 4096 (the two header blocks). So, in the above example, the RT_CONFIG11 static hashed file would need to be 22528 bytes in size, and the RT_STATUS11 static hashed file would need to be 2048 bytes in size.

Use type 18 and separation 4, and calculate modulo by dividing bytes by 2048. Thus the modulo needed for RT_CONFIG11 here would be 11 and for RT_STATUS11 would be 1. The commands would be:

Code: Select all

RESIZE RT_CONFIG11 18 11 4
RESIZE RT_STATUS11 18 1 4
IBM Software Services Group
Any contribution to this forum is my own opinion and does not necessarily reflect any position that IBM may hold.
bjkilleen
Premium Member
Premium Member
Posts: 18
Joined: Tue Oct 28, 2003 1:27 pm

Post by bjkilleen »

Ray,

What do you mean by "archive older versions of jobs"?

I want to make sure about this, because I have exported older versions of jobs to a dsx file, and then deleted the component from the VERSION project. But, when I went back into DS Version Control, the component was still present.

What have I actually done? Did I actually delete the component from DS Version Control? How do you refresh DS Version Control so that it recognizes that versions of a DS component are no longer present?

Thanks!
ray.wurlod
Participant
Posts: 54607
Joined: Wed Oct 23, 2002 10:52 pm
Location: Sydney, Australia
Contact:

Post by ray.wurlod »

Export doesn't delete anything - it's just another way to take a backup.

Once you've taken a backup - or two - you actually need to start deleting things. And therefore you have to find out which things you want to delete. You can start by getting the job numbers from DS_JOBS.

Code: Select all

SELECT NAME,JOBNO FROM DS_JOBS WHERE NAME NOT LIKE '\\%' ORDER BY NAME;
IBM Software Services Group
Any contribution to this forum is my own opinion and does not necessarily reflect any position that IBM may hold.
bjkilleen
Premium Member
Premium Member
Posts: 18
Joined: Tue Oct 28, 2003 1:27 pm

Post by bjkilleen »

Okay, I have my export, and I have also been able to identify the job numbers with the code provided.

What is the next step?

Isn't deleting the DS components from DS Manager removing them from the VERSION project? I have done this, but the DS components still show in DS Version Control. Is there a "refresh" process for the VERSION project, since it is not your typical DS project?

Thanks!
Post Reply