How to avoid overruns on Hash Files, FileSets, & DataSet

chandankambli · Post by **chandankambli** » Sat Jan 13, 2007 9:18 pm

Hello Experts:

How to avoid overruns on Hash Files, FileSets, & DataSets?

What measures to be taken when overruns takes place during the job execution?

Pls. this is very important for me to understand.

DSguru2B · Post by **DSguru2B** » Sat Jan 13, 2007 10:10 pm

What do you mean by overruns. If you mean re-running the same job then it depends upon what option you specify. For example for hashed files if you specify "clear file" then it will do as it says, clear the file and then load it again.

I_Server_Whale · Post by **I_Server_Whale** » Sat Jan 13, 2007 10:40 pm

DSguru2B wrote:What do you mean by overruns.

Exactly. Even I didn't get what the OP meant by 'overrun'?

Chandan, Could you explain what exactly you mean by overrun?

Whale.

narasimha · Post by **narasimha** » Sat Jan 13, 2007 11:36 pm

Are you looking into situtions which can fill up the allocated space for a Hash file, FileSets, & DataSets?
If yes these conditions are specific to your OS.
Of course there are ways to alter the defaults.

kumar_s · Post by **kumar_s** » Sun Jan 14, 2007 1:05 am

If you referring to over writing the created file during other flows, you have a option to re write it after clearing and etc.,
If you referring to Memory overflow when loaded into Physical memory, it is again depends on your input data size and available memory size.
If you are referring to running out of space, its again need to be decided based on the input data size.
With this vague question, posters are prompted to reply at their own observation. So as pls give the specific information to get specific answer.

chandankambli · Post by **chandankambli** » Sun Jan 14, 2007 6:36 pm

I am sorry to get back to the forum due to weekend. Anyways, I am refering to the max size that these stages allow say 2gb.

I have a server/parallel jobs and we get source data from Mainframes in a
Seq.file -> Hash File/DataSet/FileSet -> Database.

how to take care of the situations when during the execution time (production job) it goes beyond that limits?

What measures to be taken when overruns takes place during the job execution?

What check we can take to avoid situations?

kcbland · Post by **kcbland** » Sun Jan 14, 2007 6:51 pm

You need to identify your bottlenecks. Is your data volume changing? What are your server utilizations?

narasimha · Post by **narasimha** » Sun Jan 14, 2007 6:52 pm

You need to estimate the size, before hand.
For example if you need to store more than 2 Gb of data in a hashed file, you can resize the file to 64 Bit.

You can use HFC calculator to get estimates of the size of your hashed files.

ray.wurlod · Post by **ray.wurlod** » Sun Jan 14, 2007 7:34 pm

Hashed files can be created or resized with 64-bit internal pointers to avoid the 2GB limit.

Data Sets and File Sets automatically adjust, by creating additional segment files.

chandankambli · Post by **chandankambli** » Sun Jan 14, 2007 7:52 pm

During the production we get data sometimes more than 2gb.

Can you explain more about the "HFC calculator", Pls.?

How do we keep track if the load gets bigger than 2gb during productoion jobs.

Can you provide ideas wrt HF/DS/FS, pls?

narasimha · Post by **narasimha** » Sun Jan 14, 2007 8:04 pm

During the production we get data sometimes more than 2gb.

If you are not sure if your data is going to be below 2 Gb, you need to go for a 64-bit, to be on the safer side.

Can you explain more about the "HFC calculator", Pls.?

HFC.exe is available on your DS installation CD. You can search in the forum for more details on it.

How do we keep track if the load gets bigger than 2gb during productoion jobs.

If the data gets bigger than 2 Gb, you will have problems.

Can you provide ideas wrt HF/DS/FS, pls?

Ray has explained above for Data Sets and File Sets.

chandankambli · Post by **chandankambli** » Sun Jan 14, 2007 9:02 pm

Perfect answers provided by all the experts. I appreciate your response.

DSguru2B · Post by **DSguru2B** » Sun Jan 14, 2007 9:18 pm

So overruns actually meant over-loads. If you are satisfied with the answers and have a clear direction of action, you can mark the thread as Resolved.