Log files

[DSmamta]

One more thing:

The messages in the file were 244,000.

I tried to view the log for the job, but on the Director, it says "Retrieving" and the process hangs. This is possibly due to a huge log file.

In future we will have to run the jobs with 600,000 Messages in Message queue and the possibility of the job being aborted is high, please help.

Thanks,
Mamta

[ray.wurlod]

It doesn't actually hang, but it does take a long time to retrieve even the most recent 100 records from RT_LOGnnn (because it uses a full table scan to determine which ones they are).

The preventive measure is to resize the RT_LOGnnn file so that it uses 64-bit addressing internally. Make sure you have the correct nnn value!

Code: Select all

RESIZE RT_LOGnnn * * * 64BIT USING directory

You can omit USING directory, in which case the current (project) directory will be used for temporary workspace. Otherwise directory is used for temporary workspace. Clearly, RT_LOGnnn must not be in use when the resize is performed.

Having a hashed file capable of storing more than 2GB will not make retrieval by Director any faster. Chances are it will be slower.

Regular purging is probably essential in your case.

Setting the log's MINIMUM.MODULUS prior to resizing may prove slightly beneficial to run-time performance. This pre-sizes the log, so that the overhead of enlarging it does not occur at run-time.

[DSmamta]

Thanks Ray:

But can you tell me how to avoid these warnings coming in the Logs...? Is there a defect in the row splitter...?

Errors in log file are as follows:
JobName..Row_Splitter_For_Table_Name_1: read_delimited() - row 270, line 1, too many columns in record

JobName..Row_Splitter_For_Table_Name_2: read_delimited() - row 270, line 1, too many columns in record

JobName..Row_Splitter_For_Table_Name_n: read_delimited() - row 271, line 1, too many columns in record

My Design is as follows:

MQ WebSphere
|
Transformer ---> SEQ File(Header)
|
Seq File (Data)
|
Transformer
|
Row Splitter(RS1)
|
Transformer(T1)
|
Seq File (SF1)

The MQ pluggin is connected to the MQ Websphere server. In staging area I use the transformer stage to connect to MQ WebSphere stage. In transformer one column is the Message ID column and I create a new column called New Column in which I put my data. The Message Headers is sent to one seq file. The data is sent to another seq file. The data seq file is connected to transformer here I call a routine that reads the data seperated by delimiter. The transformer is connected to a row Splitter. I load the meta data here and the row is broken into number of columns and from there it goes to a transformer. The Transformer matches the source columns with target files. The target has more columns then source files and in Transformer I match the extra columns with @NULL.

Done unit testing and Integration testing everything is fine except for Log file. Chances of it becoming toooo big are there because the volumn of data in source can go upto 600,000 data.

These people here do not want to clear the log file nor do they want to write a script to clear the log file or dump the log to a different file and then view this file via vi editor.

They think its the Row-Splitter. What do you say...?

Thanks in Advance.

MJ

[ray.wurlod]

The messages are certainly being generated by the Row Splitter stage. It "believes" (if I may be anthropomorphic temporarily) that there are too many columns in the data.

This can arise, for example, if there are instances of the delimiter characters used within the row splitter stage (specified on the format tab) in your actual data.

Right now I don't have access to DataStage, so cannot check my guess.

[DSmamta]

Hello Ray:

Yes in the row splitter I load the table definition and it does read characters that are delimited and put in the respective columns.

Thanks,
MJ