I've been working on a client site whose DS jobs have required some serious diagnosis in order to work out which jobs have been major culprits in terms of poor performance. One of the actions I undertook was to perform an end-of-processing dump of the Job Log to enable analysis of which jobs are contributing to the poor performance. Up until now I have been using the commands available within UNIX to dump the log but with in excess of 1000 job instances in their DS environment (ugly is a word that springs to mind), it has been taking in excess of 45 minutes to get a daily dump of the logs.
I therefore set myself the task of writing something within DataStage that would obtain same the information but in a much shorter elapsed time. This post contains the exported XML of the objects I created and I'm posting them here on the basis that they may be useful for others in the DSXchange community.
I'll post the objects as a sequence of replies to this topic so that this particular message doesn't get too long and hopefully it will be easier for anyone to copy and save the XML from the other messages within this topic. With each message I'll post a short blurb about the purpose of the given object(s). Bear with me and refrain from replying to this topic until I get all messages posted (which, with my dinner soon to be on the table, may be over the period of an hour or two).
Note: this solution has been trialled on DS 6.x only, running under a Unix server. I have provided an object that I believe will achieve the same results with a Windows server but that hasn't been trialled in anger. If someone wishes to trial this in 7.x and indicate whether or not the objects work, that would be helpful.
Hopefully the community will find these objects useful.
David.
Job Log - get the entire content using DS job
Moderators: chulett, rschirm, roy
Job Log - get the entire content using DS job
Last edited by djm on Mon Oct 03, 2005 11:30 pm, edited 1 time in total.
Something from nothing!
The purpose of the first three objects is to create something from nothing! There is a generic version, a Unix version and a Windows version.
The premise of providing three objects is that whichever O/S your DS server is, take a copy of the O/S specific version and rename it to be SingleRow. The current version of SingleRow is for Unix thus if you are running on a Unix server, you need not copy the Unix version and replace SingleRow step.
Firstly the generic version:
[/color]
Now the Unix version (which is the same as the generic but included for completeness):
And now the Windows version:
If you save those as three distinct XML files, they can be imported using DataStage Manager.
The premise of providing three objects is that whichever O/S your DS server is, take a copy of the O/S specific version and rename it to be SingleRow. The current version of SingleRow is for Unix thus if you are running on a Unix server, you need not copy the Unix version and replace SingleRow step.
Firstly the generic version:
Code: Select all
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE DSExport [
<!ELEMENT DSExport (Header, (Job |
SharedContainer |
DataElements |
Routines |
StageTypes |
TableDefinitions |
Transforms)+
) >
<!ELEMENT Header EMPTY >
<!ATTLIST Header
CharacterSet CDATA #IMPLIED
ExportingTool CDATA #IMPLIED
ToolVersion CDATA #REQUIRED
ServerName CDATA #IMPLIED
ToolInstanceID CDATA #IMPLIED
Date CDATA #IMPLIED
Time CDATA #IMPLIED
>
<!ELEMENT Job (Record*) >
<!ATTLIST Job
Identifier CDATA #REQUIRED
DateModified CDATA #IMPLIED
TimeModified CDATA #IMPLIED
>
<!ELEMENT SharedContainer (Record*) >
<!ATTLIST SharedContainer
Identifier CDATA #REQUIRED
DateModified CDATA #IMPLIED
TimeModified CDATA #IMPLIED
>
<!ELEMENT DataElements (Record*) >
<!ELEMENT Routines (Record*) >
<!ELEMENT StageTypes (Record*) >
<!ELEMENT TableDefinitions (Record*) >
<!ELEMENT Transforms (Record*) >
<!ELEMENT Record ((Property | Collection)*) >
<!ATTLIST Record
Identifier CDATA #REQUIRED
DateModified CDATA #IMPLIED
TimeModified CDATA #IMPLIED
Type CDATA #REQUIRED
Readonly (0|1) '0'
>
<!ELEMENT Property (#PCDATA| ListItem)* >
<!ATTLIST Property
Name CDATA #REQUIRED
PreFormatted (0|1) '0'
xml:space (default|preserve) 'preserve'
>
<!ELEMENT ListItem (#PCDATA)* >
<!ATTLIST ListItem
DisplayName CDATA #REQUIRED
xml:space (default|preserve) 'preserve'
>
<!ELEMENT Collection (SubRecord*) >
<!ATTLIST Collection
Name CDATA #REQUIRED
Type CDATA #REQUIRED
>
<!ELEMENT SubRecord (Property*) >
]>
<DSExport>
<Header CharacterSet="ENGLISH" ExportingTool="Ascential DataStage Export" ToolVersion="4" ServerName="djm" ToolInstanceID="dev" Date="2005-10-03" Time="18.04.34"/>
<SharedContainer Identifier="SingleRow" DateModified="2005-10-03" TimeModified="17.51.29">
<Record Identifier="ROOT" Type="ContainerDefn" Readonly="0">
<Property Name="Name">SingleRow</Property>
<Property Name="Description">Generate a single row of output from nothing.</Property>
<Property Name="ContainerType">0</Property>
<Property Name="View">V1</Property>
<Property Name="Version">25.0.0</Property>
<Property Name="Category">RunLog</Property>
<Property Name="NextID">2</Property>
</Record>
<Record Identifier="V0S32" Type="SeqFileStage" Readonly="0">
<Property Name="Name">single_row_sf</Property>
<Property Name="Description">Execute a command to generate a single row with a single column without taking input from any data soruce.</Property>
<Property Name="NextID">3</Property>
<Property Name="OutputPins">V0S32P2</Property>
<Property Name="UnixFormat">Unix</Property>
<Property Name="PipeStage">File</Property>
<Property Name="UnicodeBOM">1</Property>
<Property Name="UnicodeSwapped">1</Property>
<Property Name="WithFilter">1</Property>
</Record>
<Record Identifier="V0S32P2" Type="SeqOutput" Readonly="0">
<Property Name="Name">single_row_l</Property>
<Property Name="Partner">V1</Property>
<Property Name="FileName">/dev/null|echo 1</Property>
<Property Name="ColDelim">,</Property>
<Property Name="QuoteChar">"</Property>
<Property Name="ColHeaders">0</Property>
<Property Name="FixedWidth">0</Property>
<Property Name="ColSpace">0</Property>
<Property Name="EnforceMetaData">Pad with SQL NULL</Property>
<Property Name="Readtimeout">0</Property>
<Collection Name="Columns" Type="OutputColumn">
<SubRecord>
<Property Name="Name">dummy</Property>
<Property Name="Description">The unary column for the unary row!</Property>
<Property Name="DataType">Number</Property>
<Property Name="SqlType">Integer</Property>
<Property Name="Precision">1</Property>
<Property Name="KeyPosition">Yes</Property>
<Property Name="DisplaySize">1</Property>
<Property Name="AllowCRLF">No</Property>
<Property Name="PadChar"/>
</SubRecord>
</Collection>
<Property Name="Waitforwriter">0</Property>
<Property Name="PadChar">#</Property>
<Property Name="LeftTextPos">150</Property>
<Property Name="TopTextPos">146</Property>
<Property Name="SuppressTruncWarn">0</Property>
</Record>
<Record Identifier="V1" Type="ContainerView" Readonly="0">
<Property Name="Name">DataStageJobLogContent</Property>
<Property Name="NextID">5</Property>
<Property Name="InputPins">V1P4</Property>
<Property Name="IsTopLevel">0</Property>
<Property Name="StageList">V0S32|V1</Property>
<Property Name="StageXPos">120|120</Property>
<Property Name="StageYPos">72|216</Property>
<Property Name="StageTypes">CSeqFileStage|ID_PALETTEOUTPUT</Property>
<Property Name="NextStageID">42</Property>
<Property Name="SnapToGrid">1</Property>
<Property Name="GridLines">0</Property>
<Property Name="ZoomValue">100</Property>
<Property Name="StageXSize">48|49</Property>
<Property Name="StageYSize">48|49</Property>
<Property Name="ContainerViewSizing">0022 0022 0740 0251 0000 0001 0000 0000</Property>
</Record>
<Record Identifier="V1P4" Type="StdPin" Readonly="0">
<Property Name="Name">single_row_l</Property>
<Property Name="Partner">V0S32</Property>
<Property Name="LinkType">1</Property>
</Record>
</SharedContainer>
</DSExport>
Now the Unix version (which is the same as the generic but included for completeness):
Code: Select all
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE DSExport [
<!ELEMENT DSExport (Header, (Job |
SharedContainer |
DataElements |
Routines |
StageTypes |
TableDefinitions |
Transforms)+
) >
<!ELEMENT Header EMPTY >
<!ATTLIST Header
CharacterSet CDATA #IMPLIED
ExportingTool CDATA #IMPLIED
ToolVersion CDATA #REQUIRED
ServerName CDATA #IMPLIED
ToolInstanceID CDATA #IMPLIED
Date CDATA #IMPLIED
Time CDATA #IMPLIED
>
<!ELEMENT Job (Record*) >
<!ATTLIST Job
Identifier CDATA #REQUIRED
DateModified CDATA #IMPLIED
TimeModified CDATA #IMPLIED
>
<!ELEMENT SharedContainer (Record*) >
<!ATTLIST SharedContainer
Identifier CDATA #REQUIRED
DateModified CDATA #IMPLIED
TimeModified CDATA #IMPLIED
>
<!ELEMENT DataElements (Record*) >
<!ELEMENT Routines (Record*) >
<!ELEMENT StageTypes (Record*) >
<!ELEMENT TableDefinitions (Record*) >
<!ELEMENT Transforms (Record*) >
<!ELEMENT Record ((Property | Collection)*) >
<!ATTLIST Record
Identifier CDATA #REQUIRED
DateModified CDATA #IMPLIED
TimeModified CDATA #IMPLIED
Type CDATA #REQUIRED
Readonly (0|1) '0'
>
<!ELEMENT Property (#PCDATA| ListItem)* >
<!ATTLIST Property
Name CDATA #REQUIRED
PreFormatted (0|1) '0'
xml:space (default|preserve) 'preserve'
>
<!ELEMENT ListItem (#PCDATA)* >
<!ATTLIST ListItem
DisplayName CDATA #REQUIRED
xml:space (default|preserve) 'preserve'
>
<!ELEMENT Collection (SubRecord*) >
<!ATTLIST Collection
Name CDATA #REQUIRED
Type CDATA #REQUIRED
>
<!ELEMENT SubRecord (Property*) >
]>
<DSExport>
<Header CharacterSet="ENGLISH" ExportingTool="Ascential DataStage Export" ToolVersion="4" ServerName="djm" ToolInstanceID="dev" Date="2005-10-03" Time="18.04.39"/>
<SharedContainer Identifier="SingleRowUnix" DateModified="2005-10-03" TimeModified="17.51.53">
<Record Identifier="ROOT" Type="ContainerDefn" Readonly="0">
<Property Name="Name">SingleRowUnix</Property>
<Property Name="Description">Generate a single row of output from nothing. Solution specific to Unix.</Property>
<Property Name="ContainerType">0</Property>
<Property Name="View">V1</Property>
<Property Name="Version">25.0.0</Property>
<Property Name="Category">RunLog</Property>
<Property Name="NextID">2</Property>
</Record>
<Record Identifier="V0S32" Type="SeqFileStage" Readonly="0">
<Property Name="Name">single_row_sf</Property>
<Property Name="Description">Execute a command to generate a single row with a single column without taking input from any data soruce.</Property>
<Property Name="NextID">3</Property>
<Property Name="OutputPins">V0S32P2</Property>
<Property Name="UnixFormat">Unix</Property>
<Property Name="PipeStage">File</Property>
<Property Name="UnicodeBOM">1</Property>
<Property Name="UnicodeSwapped">1</Property>
<Property Name="WithFilter">1</Property>
</Record>
<Record Identifier="V0S32P2" Type="SeqOutput" Readonly="0">
<Property Name="Name">single_row_l</Property>
<Property Name="Partner">V1</Property>
<Property Name="FileName">/dev/null|echo 1</Property>
<Property Name="ColDelim">,</Property>
<Property Name="QuoteChar">"</Property>
<Property Name="ColHeaders">0</Property>
<Property Name="FixedWidth">0</Property>
<Property Name="ColSpace">0</Property>
<Property Name="EnforceMetaData">Pad with SQL NULL</Property>
<Property Name="Readtimeout">0</Property>
<Collection Name="Columns" Type="OutputColumn">
<SubRecord>
<Property Name="Name">dummy</Property>
<Property Name="Description">The unary column for the unary row!</Property>
<Property Name="DataType">Number</Property>
<Property Name="SqlType">Integer</Property>
<Property Name="Precision">1</Property>
<Property Name="KeyPosition">Yes</Property>
<Property Name="DisplaySize">1</Property>
<Property Name="AllowCRLF">No</Property>
<Property Name="PadChar"/>
</SubRecord>
</Collection>
<Property Name="Waitforwriter">0</Property>
<Property Name="PadChar">#</Property>
<Property Name="LeftTextPos">150</Property>
<Property Name="TopTextPos">146</Property>
<Property Name="SuppressTruncWarn">0</Property>
</Record>
<Record Identifier="V1" Type="ContainerView" Readonly="0">
<Property Name="Name">DataStageJobLogContent</Property>
<Property Name="NextID">5</Property>
<Property Name="InputPins">V1P4</Property>
<Property Name="IsTopLevel">0</Property>
<Property Name="StageList">V0S32|V1</Property>
<Property Name="StageXPos">120|120</Property>
<Property Name="StageYPos">72|216</Property>
<Property Name="StageTypes">CSeqFileStage|ID_PALETTEOUTPUT</Property>
<Property Name="NextStageID">42</Property>
<Property Name="SnapToGrid">1</Property>
<Property Name="GridLines">0</Property>
<Property Name="ZoomValue">100</Property>
<Property Name="StageXSize">48|49</Property>
<Property Name="StageYSize">48|49</Property>
<Property Name="ContainerViewSizing">0022 0022 0740 0251 0000 0001 0000 0000</Property>
</Record>
<Record Identifier="V1P4" Type="StdPin" Readonly="0">
<Property Name="Name">single_row_l</Property>
<Property Name="Partner">V0S32</Property>
<Property Name="LinkType">1</Property>
</Record>
</SharedContainer>
</DSExport>
Code: Select all
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE DSExport [
<!ELEMENT DSExport (Header, (Job |
SharedContainer |
DataElements |
Routines |
StageTypes |
TableDefinitions |
Transforms)+
) >
<!ELEMENT Header EMPTY >
<!ATTLIST Header
CharacterSet CDATA #IMPLIED
ExportingTool CDATA #IMPLIED
ToolVersion CDATA #REQUIRED
ServerName CDATA #IMPLIED
ToolInstanceID CDATA #IMPLIED
Date CDATA #IMPLIED
Time CDATA #IMPLIED
>
<!ELEMENT Job (Record*) >
<!ATTLIST Job
Identifier CDATA #REQUIRED
DateModified CDATA #IMPLIED
TimeModified CDATA #IMPLIED
>
<!ELEMENT SharedContainer (Record*) >
<!ATTLIST SharedContainer
Identifier CDATA #REQUIRED
DateModified CDATA #IMPLIED
TimeModified CDATA #IMPLIED
>
<!ELEMENT DataElements (Record*) >
<!ELEMENT Routines (Record*) >
<!ELEMENT StageTypes (Record*) >
<!ELEMENT TableDefinitions (Record*) >
<!ELEMENT Transforms (Record*) >
<!ELEMENT Record ((Property | Collection)*) >
<!ATTLIST Record
Identifier CDATA #REQUIRED
DateModified CDATA #IMPLIED
TimeModified CDATA #IMPLIED
Type CDATA #REQUIRED
Readonly (0|1) '0'
>
<!ELEMENT Property (#PCDATA| ListItem)* >
<!ATTLIST Property
Name CDATA #REQUIRED
PreFormatted (0|1) '0'
xml:space (default|preserve) 'preserve'
>
<!ELEMENT ListItem (#PCDATA)* >
<!ATTLIST ListItem
DisplayName CDATA #REQUIRED
xml:space (default|preserve) 'preserve'
>
<!ELEMENT Collection (SubRecord*) >
<!ATTLIST Collection
Name CDATA #REQUIRED
Type CDATA #REQUIRED
>
<!ELEMENT SubRecord (Property*) >
]>
<DSExport>
<Header CharacterSet="ENGLISH" ExportingTool="Ascential DataStage Export" ToolVersion="4" ServerName="djm" ToolInstanceID="dev" Date="2005-10-03" Time="18.04.46"/>
<SharedContainer Identifier="SingleRowWindows" DateModified="2005-10-03" TimeModified="17.52.10">
<Record Identifier="ROOT" Type="ContainerDefn" Readonly="0">
<Property Name="Name">SingleRowWindows</Property>
<Property Name="Description">Generate a single row of output from nothing. Solution specific to Windows.</Property>
<Property Name="ContainerType">0</Property>
<Property Name="View">V1</Property>
<Property Name="Version">25.0.0</Property>
<Property Name="Category">RunLog</Property>
<Property Name="NextID">2</Property>
</Record>
<Record Identifier="V0S32" Type="SeqFileStage" Readonly="0">
<Property Name="Name">single_row_sf</Property>
<Property Name="Description">Execute a command to generate a single row with a single column without taking input from any data soruce.</Property>
<Property Name="NextID">3</Property>
<Property Name="OutputPins">V0S32P2</Property>
<Property Name="UnixFormat">Unix</Property>
<Property Name="PipeStage">File</Property>
<Property Name="UnicodeBOM">1</Property>
<Property Name="UnicodeSwapped">1</Property>
<Property Name="WithFilter">1</Property>
</Record>
<Record Identifier="V0S32P2" Type="SeqOutput" Readonly="0">
<Property Name="Name">single_row_l</Property>
<Property Name="Partner">V1</Property>
<Property Name="FileName">NUL|cmd.exe -c echo 1</Property>
<Property Name="ColDelim">,</Property>
<Property Name="QuoteChar">"</Property>
<Property Name="ColHeaders">0</Property>
<Property Name="FixedWidth">0</Property>
<Property Name="ColSpace">0</Property>
<Property Name="EnforceMetaData">Pad with SQL NULL</Property>
<Property Name="Readtimeout">0</Property>
<Collection Name="Columns" Type="OutputColumn">
<SubRecord>
<Property Name="Name">dummy</Property>
<Property Name="Description">The unary column for the unary row!</Property>
<Property Name="DataType">Number</Property>
<Property Name="SqlType">Integer</Property>
<Property Name="Precision">1</Property>
<Property Name="KeyPosition">Yes</Property>
<Property Name="DisplaySize">1</Property>
<Property Name="AllowCRLF">No</Property>
<Property Name="PadChar"/>
</SubRecord>
</Collection>
<Property Name="Waitforwriter">0</Property>
<Property Name="PadChar">#</Property>
<Property Name="LeftTextPos">150</Property>
<Property Name="TopTextPos">146</Property>
<Property Name="SuppressTruncWarn">0</Property>
</Record>
<Record Identifier="V1" Type="ContainerView" Readonly="0">
<Property Name="Name">DataStageJobLogContent</Property>
<Property Name="NextID">5</Property>
<Property Name="InputPins">V1P4</Property>
<Property Name="IsTopLevel">0</Property>
<Property Name="StageList">V0S32|V1</Property>
<Property Name="StageXPos">120|120</Property>
<Property Name="StageYPos">72|216</Property>
<Property Name="StageTypes">CSeqFileStage|ID_PALETTEOUTPUT</Property>
<Property Name="NextStageID">42</Property>
<Property Name="SnapToGrid">1</Property>
<Property Name="GridLines">0</Property>
<Property Name="ZoomValue">100</Property>
<Property Name="StageXSize">48|49</Property>
<Property Name="StageYSize">48|49</Property>
<Property Name="ContainerViewSizing">0022 0022 0740 0251 0000 0001 0000 0000</Property>
</Record>
<Record Identifier="V1P4" Type="StdPin" Readonly="0">
<Property Name="Name">single_row_l</Property>
<Property Name="Partner">V0S32</Property>
<Property Name="LinkType">1</Property>
</Record>
</SharedContainer>
</DSExport>
Routine to get Job Log Details
This next object is a routine named JobLogGet that obtains the job log details for a given job.
Edit (2005-10-07): Corrected a couple of bugs in the routine and replaced original code with resulting code.
Edit (2006-03-04): Code replaced with updated version at the end of this post.
Code: Select all
Edit (2006-03-04): Code replaced with updated version at the end of this post.
Last edited by djm on Sat Mar 04, 2006 3:10 am, edited 4 times in total.
The next object does the guts of the work. It creates a shared container called DataStageJobLogContent. Note that within the shared container, a couple of hash files are used. Prior to running the job, it would pay to check whether their names clash with any existing hash files in your project. If so, rename them! Further, the hash files use the Account rather than Directory mechansim, so if this differs from your project's standards, make sure you change them to use the Directory mechanism.
Code: Select all
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE DSExport [
<!ELEMENT DSExport (Header, (Job |
SharedContainer |
DataElements |
Routines |
StageTypes |
TableDefinitions |
Transforms)+
) >
<!ELEMENT Header EMPTY >
<!ATTLIST Header
CharacterSet CDATA #IMPLIED
ExportingTool CDATA #IMPLIED
ToolVersion CDATA #REQUIRED
ServerName CDATA #IMPLIED
ToolInstanceID CDATA #IMPLIED
Date CDATA #IMPLIED
Time CDATA #IMPLIED
>
<!ELEMENT Job (Record*) >
<!ATTLIST Job
Identifier CDATA #REQUIRED
DateModified CDATA #IMPLIED
TimeModified CDATA #IMPLIED
>
<!ELEMENT SharedContainer (Record*) >
<!ATTLIST SharedContainer
Identifier CDATA #REQUIRED
DateModified CDATA #IMPLIED
TimeModified CDATA #IMPLIED
>
<!ELEMENT DataElements (Record*) >
<!ELEMENT Routines (Record*) >
<!ELEMENT StageTypes (Record*) >
<!ELEMENT TableDefinitions (Record*) >
<!ELEMENT Transforms (Record*) >
<!ELEMENT Record ((Property | Collection)*) >
<!ATTLIST Record
Identifier CDATA #REQUIRED
DateModified CDATA #IMPLIED
TimeModified CDATA #IMPLIED
Type CDATA #REQUIRED
Readonly (0|1) '0'
>
<!ELEMENT Property (#PCDATA| ListItem)* >
<!ATTLIST Property
Name CDATA #REQUIRED
PreFormatted (0|1) '0'
xml:space (default|preserve) 'preserve'
>
<!ELEMENT ListItem (#PCDATA)* >
<!ATTLIST ListItem
DisplayName CDATA #REQUIRED
xml:space (default|preserve) 'preserve'
>
<!ELEMENT Collection (SubRecord*) >
<!ATTLIST Collection
Name CDATA #REQUIRED
Type CDATA #REQUIRED
>
<!ELEMENT SubRecord (Property*) >
]>
<DSExport>
<Header CharacterSet="ENGLISH" ExportingTool="Ascential DataStage Export" ToolVersion="4" ServerName="djm" ToolInstanceID="dev" Date="2005-10-03" Time="18.04.55"/>
<SharedContainer Identifier="DataStageJobLogContent" DateModified="2005-10-03" TimeModified="17.52.30">
<Record Identifier="C3" Type="ContainerStage" Readonly="0">
<Property Name="Name">single_row_sc</Property>
<Property Name="NextID">2</Property>
<Property Name="OutputPins">C3P1</Property>
<Property Name="ContainerType">0</Property>
<Property Name="ContainerName">SingleRow</Property>
</Record>
<Record Identifier="C3P1" Type="StdOutput" Readonly="0">
<Property Name="Name">single_row_l</Property>
<Property Name="Partner">V0S33</Property>
<Collection Name="Columns" Type="OutputColumn">
<SubRecord>
<Property Name="Name">dummy</Property>
<Property Name="Description">The unary column for the unary row!</Property>
<Property Name="DataType">Number</Property>
<Property Name="SqlType">Integer</Property>
<Property Name="Precision">1</Property>
<Property Name="Nullable">No</Property>
<Property Name="KeyPosition">Yes</Property>
<Property Name="DisplaySize">1</Property>
<Property Name="Transform">\(1B)</Property>
<Property Name="AllowCRLF">No</Property>
<Property Name="PadNulls">Error</Property>
<Property Name="PadChar"/>
<Property Name="ExtendedPrecision">0</Property>
<Property Name="TaggedSubrec">0</Property>
<Property Name="OccursVarying">0</Property>
</SubRecord>
</Collection>
<Property Name="LeftTextPos">72</Property>
<Property Name="TopTextPos">115</Property>
<Property Name="PartnerLink">single_row_l</Property>
</Record>
<Record Identifier="ROOT" Type="ContainerDefn" Readonly="0">
<Property Name="Name">DataStageJobLogContent</Property>
<Property Name="Description">Obtain the run log for those jobs with output in the DataStage job log.</Property>
<Property Name="ContainerType">0</Property>
<Property Name="View">V1</Property>
<Property Name="FullDescription" PreFormatted="1">Obtain the current content of the DataStage Job Log (i.e. the details visible within DataStage Director).
Parameters provide the following functionality:
JOB_NAME_MATCH - Identifies a pattern against which a job's name should match in order to be processed within the job.
EXCLUDED_JOB_LIST - Exclude a particular list of jobs from being processed
EXCLUDED_JOBS_PATTERN - Exclude any job whose name mataches a particular pattern.
The output generated is an list of job names and details, one row per job log entry ordered by Job Name and event sequence.
Note that the job in which this container is placed will not be included in the output. Unconfirmed - there appears to be some restriction within the DS API that prevents the Job Log details to be obtained for the currently running job.</Property>
<Property Name="Version">25.0.0</Property>
<Property Name="Category">RunLog</Property>
<Property Name="NextID">4</Property>
<Collection Name="Parameters" Type="Parameters">
<SubRecord>
<Property Name="Name">JOB_NAME_MATCH</Property>
<Property Name="Prompt">Pattern matching the name of the jobs to be included in the log dump.</Property>
<Property Name="Default">%</Property>
<Property Name="HelpTxt">Pattern against which job names should match to be considered for inclusion in the output.\(20)</Property>
<Property Name="ParamType">String</Property>
<Property Name="ParamLength">0</Property>
<Property Name="ParamScale">0</Property>
</SubRecord>
<SubRecord>
<Property Name="Name">EXCLUDED_JOBS_LIST</Property>
<Property Name="Prompt">List of jobs that should not be included.</Property>
<Property Name="Default">'ArchiveClearDSJOBLog'</Property>
<Property Name="HelpTxt">Comma separated list of job names to exclude from processing from those jobs identified via the matching job names. Leave empty if a particular job names should not be used for exclusion purposes. If both a job name list and a pattern are specified, the exlcuded list of job will take precedence and the pattern will be ignored.</Property>
<Property Name="ParamType">String</Property>
<Property Name="ParamLength">0</Property>
<Property Name="ParamScale">0</Property>
</SubRecord>
<SubRecord>
<Property Name="Name">EXCLUDED_JOBS_PATTERN</Property>
<Property Name="HelpTxt">Pattern against if a job name matches it will be excluded from processing from those jobs identified via the matching job names. Leave empty if job names should not be matched against a pattern for exclusion purposes. If both a job name list and a pattern are specified, the exlcuded list of job will take precedence and the pattern will be ignored.</Property>
<Property Name="ParamType">String</Property>
<Property Name="ParamLength">0</Property>
<Property Name="ParamScale">0</Property>
</SubRecord>
</Collection>
</Record>
<Record Identifier="V0S1" Type="TransformerStage" Readonly="0">
<Property Name="Name">ignore_particular_jobs_t</Property>
<Property Name="NextID">11</Property>
<Property Name="InputPins">V0S1P10</Property>
<Property Name="OutputPins">V0S1P2</Property>
<Collection Name="StageVars" Type="StageVar">
<SubRecord>
<Property Name="Name">ExcludedJobsList</Property>
<Property Name="Description">List of comma delimited job names of jobs to not process. Empty if there are no explicit exclusions.</Property>
<Property Name="InitialValue">IF (len(EXCLUDED_JOBS_LIST) > 0) then ',' : trim(EXCLUDED_JOBS_LIST , ',' , 'B') : ',' else ''</Property>
<Property Name="Transform">\(1B)</Property>
<Property Name="Precision">18</Property>
<Property Name="ColScale">4</Property>
</SubRecord>
<SubRecord>
<Property Name="Name">MatchesExcludedJobsPattern</Property>
<Property Name="Description">Indicates whether or not the job name names any job name exclusion pattern.</Property>
<Property Name="Expression" PreFormatted="1">if (len(ExcludedJobsList) > 0) then
@FALSE
else
if (len(EXCLUDED_JOBS_PATTERN) > 0) then
(job_list_l.job_name Matches EXCLUDED_JOBS_PATTERN)
else
@FALSE</Property>
<Property Name="ParsedExpression"> if (len(ExcludedJobsList) > 0) then @FALSE else if (len(EXCLUDED_JOBS_PATTERN) > 0) then (job_list_l.job_name Matches EXCLUDED_JOBS_PATTERN) else @FALSE</Property>
<Property Name="SourceColumn">job_list_l.job_name</Property>
<Property Name="StageVars">ExcludedJobsList</Property>
<Property Name="Transform">\(1B)</Property>
<Property Name="Precision">18</Property>
<Property Name="ColScale">4</Property>
</SubRecord>
<SubRecord>
<Property Name="Name">ProcessJob</Property>
<Property Name="Description">Indicates whether any processing of the job should be done.</Property>
<Property Name="Expression" PreFormatted="1">if (NOT (job_list_l.job_name Matches JOB_NAME_MATCH)) then
@FALSE
else
((index(ExcludedJobsList , ',' : job_list_l.job_name : ',' , 1) = 0)
and (NOT (MatchesExcludedJobsPattern)))</Property>
<Property Name="InitialValue">@FALSE</Property>
<Property Name="ParsedExpression"> if (NOT(job_list_l.job_name Matches JOB_NAME_MATCH)) then @FALSE else ((index(ExcludedJobsList, ',' : job_list_l.job_name : ',', 1) = 0) and (NOT(MatchesExcludedJobsPattern)))</Property>
<Property Name="SourceColumn">job_list_l.job_name</Property>
<Property Name="StageVars">ExcludedJobsList;MatchesExcludedJobsPattern</Property>
<Property Name="Transform">\(1B)</Property>
<Property Name="Precision">18</Property>
<Property Name="ColScale">4</Property>
</SubRecord>
</Collection>
</Record>
<Record Identifier="V0S18" Type="HashedFileStage" Readonly="0">
<Property Name="Name">expand_log_details_h</Property>
<Property Name="Description">Expands the job log from a single record into a row per event in the job log.</Property>
<Property Name="NextID">7</Property>
<Property Name="InputPins">V0S18P6</Property>
<Property Name="OutputPins">V0S18P5</Property>
<Property Name="UVcompatible">0</Property>
<Property Name="SQLNull">0</Property>
</Record>
<Record Identifier="V0S18P5" Type="HashedOutput" Readonly="0">
<Property Name="Name">line_per_event_message_l</Property>
<Property Name="Partner">V0S30</Property>
<Property Name="FileName">job_log</Property>
<Property Name="InMemory">0</Property>
<Property Name="Normalize">log_content</Property>
<Property Name="LeftTextPos">72</Property>
<Property Name="TopTextPos">716</Property>
<Property Name="RecordLevelRead">0</Property>
<Collection Name="Columns" Type="OutputColumn">
<SubRecord>
<Property Name="Name">job_name</Property>
<Property Name="SqlType">VarChar</Property>
<Property Name="Precision">20</Property>
<Property Name="Nullable">No</Property>
<Property Name="KeyPosition">Yes</Property>
<Property Name="Group">No</Property>
<Property Name="FieldType">S</Property>
<Property Name="AllowCRLF">No</Property>
<Property Name="PadNulls">Error</Property>
<Property Name="SortingOrder">Ascending</Property>
<Property Name="PadChar"/>
<Property Name="ExtendedPrecision">0</Property>
<Property Name="TaggedSubrec">0</Property>
<Property Name="OccursVarying">0</Property>
</SubRecord>
<SubRecord>
<Property Name="Name">log_content</Property>
<Property Name="SqlType">LongVarChar</Property>
<Property Name="Nullable">No</Property>
<Property Name="KeyPosition">No</Property>
<Property Name="Group">No</Property>
<Property Name="Association">log_content</Property>
<Property Name="FieldType">MV</Property>
<Property Name="AllowCRLF">No</Property>
<Property Name="PadNulls">Error</Property>
<Property Name="SortingOrder">Ascending</Property>
<Property Name="PadChar"/>
<Property Name="ExtendedPrecision">0</Property>
<Property Name="TaggedSubrec">0</Property>
<Property Name="OccursVarying">0</Property>
</SubRecord>
</Collection>
</Record>
<Record Identifier="V0S18P6" Type="HashedInput" Readonly="0">
<Property Name="Name">non_empty_logs_l</Property>
<Property Name="Partner">V0S37</Property>
<Property Name="FileName">job_log</Property>
<Property Name="ClearFile">1</Property>
<Property Name="Backup">0</Property>
<Property Name="DisableCW">1</Property>
<Property Name="CreateFileOptions">DYNAMIC</Property>
</Record>
<Record Identifier="V0S1P10" Type="TrxInput" Readonly="0">
<Property Name="Name">job_list_l</Property>
<Property Name="Partner">V0S35</Property>
<Property Name="LinkType">1</Property>
</Record>
<Record Identifier="V0S1P2" Type="TrxOutput" Readonly="0">
<Property Name="Name">job_list_restricted_l</Property>
<Property Name="Partner">V0S37</Property>
<Property Name="Constraint">( (ProcessJob) and (job_list_l.job_name <> DSJobName))</Property>
<Property Name="Reject">No</Property>
<Property Name="RowLimit">0</Property>
<Property Name="ParsedConstraint">((ProcessJob) and (job_list_l.job_name <> DSJobName))</Property>
<Property Name="SourceColumns">job_list_l.job_name</Property>
<Property Name="RoutineTransforms">\(1B)</Property>
<Collection Name="Columns" Type="OutputColumn">
<SubRecord>
<Property Name="Name">job_name</Property>
<Property Name="SqlType">VarChar</Property>
<Property Name="Precision">20</Property>
<Property Name="Nullable">No</Property>
<Property Name="KeyPosition">Yes</Property>
<Property Name="Derivation">job_list_l.job_name</Property>
<Property Name="Group">No</Property>
<Property Name="ParsedDerivation">job_list_l.job_name</Property>
<Property Name="SourceColumn">job_list_l.job_name</Property>
<Property Name="Transform">\(1B)</Property>
<Property Name="TableDef">djm\djm\djm_hash_file</Property>
<Property Name="AllowCRLF">No</Property>
<Property Name="PadNulls">Error</Property>
<Property Name="SortingOrder">Ascending</Property>
<Property Name="PadChar"/>
<Property Name="ColumnReference">Name</Property>
<Property Name="ExtendedPrecision">0</Property>
<Property Name="TaggedSubrec">0</Property>
<Property Name="OccursVarying">0</Property>
</SubRecord>
</Collection>
<Property Name="StageVars">ProcessJob</Property>
<Property Name="LeftTextPos">72</Property>
<Property Name="TopTextPos">474</Property>
</Record>
<Record Identifier="V0S30" Type="TransformerStage" Readonly="0">
<Property Name="Name">split_event_detail_l</Property>
<Property Name="NextID">4</Property>
<Property Name="InputPins">V0S30P1</Property>
<Property Name="OutputPins">V0S30P3</Property>
<Collection Name="StageVars" Type="StageVar">
<SubRecord>
<Property Name="Name">EventId</Property>
<Property Name="Expression">Field(line_per_event_message_l.log_content,@TM,1)</Property>
<Property Name="ParsedExpression">Field(line_per_event_message_l.log_content, @TM, 1)</Property>
<Property Name="SourceColumn">line_per_event_message_l.log_content</Property>
<Property Name="Transform">\(1B)</Property>
<Property Name="Precision">0</Property>
<Property Name="ColScale">0</Property>
</SubRecord>
<SubRecord>
<Property Name="Name">EventTimestamp</Property>
<Property Name="Expression">Field(line_per_event_message_l.log_content,@TM,2)</Property>
<Property Name="ParsedExpression">Field(line_per_event_message_l.log_content, @TM, 2)</Property>
<Property Name="SourceColumn">line_per_event_message_l.log_content</Property>
<Property Name="Transform">\(1B)</Property>
<Property Name="Precision">0</Property>
<Property Name="ColScale">0</Property>
</SubRecord>
<SubRecord>
<Property Name="Name">EventType</Property>
<Property Name="Expression">Field(line_per_event_message_l.log_content,@TM,3)</Property>
<Property Name="ParsedExpression">Field(line_per_event_message_l.log_content, @TM, 3)</Property>
<Property Name="SourceColumn">line_per_event_message_l.log_content</Property>
<Property Name="Transform">\(1B)</Property>
<Property Name="Precision">0</Property>
<Property Name="ColScale">0</Property>
</SubRecord>
<SubRecord>
<Property Name="Name">MultipleInstanceDetail</Property>
<Property Name="Expression">Field(line_per_event_message_l.log_content,@TM,4)</Property>
<Property Name="ParsedExpression">Field(line_per_event_message_l.log_content, @TM, 4)</Property>
<Property Name="SourceColumn">line_per_event_message_l.log_content</Property>
<Property Name="Transform">\(1B)</Property>
<Property Name="Precision">18</Property>
<Property Name="ColScale">4</Property>
</SubRecord>
<SubRecord>
<Property Name="Name">EventJobInstance</Property>
<Property Name="Expression">Field(line_per_event_message_l.log_content,@TM,5)</Property>
<Property Name="ParsedExpression">Field(line_per_event_message_l.log_content, @TM, 5)</Property>
<Property Name="SourceColumn">line_per_event_message_l.log_content</Property>
<Property Name="Transform">\(1B)</Property>
<Property Name="Precision">18</Property>
<Property Name="ColScale">4</Property>
</SubRecord>
<SubRecord>
<Property Name="Name">EventMessages</Property>
<Property Name="Expression">Field(line_per_event_message_l.log_content,@TM,6)</Property>
<Property Name="ParsedExpression">Field(line_per_event_message_l.log_content, @TM, 6)</Property>
<Property Name="SourceColumn">line_per_event_message_l.log_content</Property>
<Property Name="Transform">\(1B)</Property>
<Property Name="Precision">0</Property>
<Property Name="ColScale">0</Property>
</SubRecord>
</Collection>
</Record>
<Record Identifier="V0S30P1" Type="TrxInput" Readonly="0">
<Property Name="Name">line_per_event_message_l</Property>
<Property Name="Partner">V0S18</Property>
<Property Name="LinkType">1</Property>
</Record>
<Record Identifier="V0S30P3" Type="TrxOutput" Readonly="0">
<Property Name="Name">event_detail_unsorted_l</Property>
<Property Name="Partner">V1S41</Property>
<Property Name="Reject">No</Property>
<Property Name="RowLimit">0</Property>
<Property Name="RoutineTransforms">\(1B)</Property>
<Collection Name="Columns" Type="OutputColumn">
<SubRecord>
<Property Name="Name">job_name</Property>
<Property Name="SqlType">VarChar</Property>
<Property Name="Precision">20</Property>
<Property Name="Nullable">No</Property>
<Property Name="KeyPosition">Yes</Property>
<Property Name="Derivation">line_per_event_message_l.job_name</Property>
<Property Name="Group">No</Property>
<Property Name="ParsedDerivation">line_per_event_message_l.job_name</Property>
<Property Name="SourceColumn">line_per_event_message_l.job_name</Property>
<Property Name="Transform">\(1B)</Property>
<Property Name="FieldType">S</Property>
<Property Name="AllowCRLF">No</Property>
<Property Name="PadNulls">Error</Property>
<Property Name="SortingOrder">Ascending</Property>
<Property Name="PadChar"/>
<Property Name="ExtendedPrecision">0</Property>
<Property Name="TaggedSubrec">0</Property>
<Property Name="OccursVarying">0</Property>
</SubRecord>
<SubRecord>
<Property Name="Name">event_id</Property>
<Property Name="SqlType">Integer</Property>
<Property Name="Precision">5</Property>
<Property Name="Nullable">Yes</Property>
<Property Name="KeyPosition">Yes</Property>
<Property Name="Derivation">EventId</Property>
<Property Name="Group">No</Property>
<Property Name="ParsedDerivation">EventId</Property>
<Property Name="Transform">\(1B)</Property>
<Property Name="AllowCRLF">No</Property>
<Property Name="PadNulls">Error</Property>
<Property Name="SortingOrder">Ascending</Property>
<Property Name="StageVars">EventId</Property>
<Property Name="PadChar"/>
</SubRecord>
<SubRecord>
<Property Name="Name">event_timestamp</Property>
<Property Name="SqlType">VarChar</Property>
<Property Name="Precision">19</Property>
<Property Name="Nullable">Yes</Property>
<Property Name="KeyPosition">No</Property>
<Property Name="Derivation">EventTimestamp</Property>
<Property Name="Group">No</Property>
<Property Name="ParsedDerivation">EventTimestamp</Property>
<Property Name="Transform">\(1B)</Property>
<Property Name="AllowCRLF">No</Property>
<Property Name="PadNulls">Error</Property>
<Property Name="SortingOrder">Ascending</Property>
<Property Name="StageVars">EventTimestamp</Property>
<Property Name="PadChar"/>
</SubRecord>
<SubRecord>
<Property Name="Name">event_type</Property>
<Property Name="SqlType">VarChar</Property>
<Property Name="Precision">2</Property>
<Property Name="Nullable">Yes</Property>
<Property Name="KeyPosition">No</Property>
<Property Name="Derivation">EventType</Property>
<Property Name="Group">No</Property>
<Property Name="ParsedDerivation">EventType</Property>
<Property Name="Transform">\(1B)</Property>
<Property Name="AllowCRLF">No</Property>
<Property Name="PadNulls">Error</Property>
<Property Name="SortingOrder">Ascending</Property>
<Property Name="StageVars">EventType</Property>
<Property Name="PadChar"/>
</SubRecord>
<SubRecord>
<Property Name="Name">multiple_instance_detail</Property>
<Property Name="SqlType">VarChar</Property>
<Property Name="Precision">255</Property>
<Property Name="Nullable">Yes</Property>
<Property Name="KeyPosition">No</Property>
<Property Name="Derivation">MultipleInstanceDetail</Property>
<Property Name="Group">No</Property>
<Property Name="ParsedDerivation">MultipleInstanceDetail</Property>
<Property Name="Transform">\(1B)</Property>
<Property Name="AllowCRLF">No</Property>
<Property Name="PadNulls">Error</Property>
<Property Name="SortingOrder">Ascending</Property>
<Property Name="StageVars">MultipleInstanceDetail</Property>
<Property Name="PadChar"/>
</SubRecord>
<SubRecord>
<Property Name="Name">job_instance</Property>
<Property Name="SqlType">VarChar</Property>
<Property Name="Precision">4</Property>
<Property Name="Nullable">Yes</Property>
<Property Name="KeyPosition">No</Property>
<Property Name="Derivation">EventJobInstance</Property>
<Property Name="Group">No</Property>
<Property Name="ParsedDerivation">EventJobInstance</Property>
<Property Name="Transform">\(1B)</Property>
<Property Name="AllowCRLF">No</Property>
<Property Name="PadNulls">Error</Property>
<Property Name="SortingOrder">Ascending</Property>
<Property Name="StageVars">EventJobInstance</Property>
<Property Name="PadChar"/>
</SubRecord>
<SubRecord>
<Property Name="Name">event_message</Property>
<Property Name="SqlType">LongVarChar</Property>
<Property Name="Nullable">Yes</Property>
<Property Name="KeyPosition">No</Property>
<Property Name="Derivation">EventMessages</Property>
<Property Name="Group">No</Property>
<Property Name="ParsedDerivation">EventMessages</Property>
<Property Name="Transform">\(1B)</Property>
<Property Name="AllowCRLF">No</Property>
<Property Name="PadNulls">Error</Property>
<Property Name="SortingOrder">Ascending</Property>
<Property Name="StageVars">EventMessages</Property>
<Property Name="PadChar"/>
</SubRecord>
</Collection>
<Property Name="LeftTextPos">73</Property>
<Property Name="TopTextPos">836</Property>
</Record>
<Record Identifier="V0S33" Type="TransformerStage" Readonly="0">
<Property Name="Name">job_name_list_create_t</Property>
<Property Name="NextID">3</Property>
<Property Name="InputPins">V0S33P1</Property>
<Property Name="OutputPins">V0S33P2</Property>
<Collection Name="StageVars" Type="StageVar">
<SubRecord>
<Property Name="Name">JobListss</Property>
<Property Name="Description">List of all the jobs in the project, comma separated.</Property>
<Property Name="Expression">DSGetProjectInfo(DSJ.JOBLIST)</Property>
<Property Name="ParsedExpression">DSGetProjectInfo(DSJ.JOBLIST)</Property>
<Property Name="Transform">\(1B)</Property>
<Property Name="Precision">0</Property>
<Property Name="ColScale">0</Property>
</SubRecord>
<SubRecord>
<Property Name="Name">JobListDelimitedss</Property>
<Property Name="Description">List of all jobs in the project, "Value Mark" delimited.</Property>
<Property Name="Expression">Change(JobListss,',',@VM)</Property>
<Property Name="ParsedExpression">Change(JobListss, ',', @VM)</Property>
<Property Name="StageVars">JobListss</Property>
<Property Name="Transform">\(1B)</Property>
<Property Name="Precision">0</Property>
<Property Name="ColScale">0</Property>
</SubRecord>
</Collection>
</Record>
<Record Identifier="V0S33P1" Type="TrxInput" Readonly="0">
<Property Name="Name">single_row_l</Property>
<Property Name="Partner">C3</Property>
<Property Name="LinkType">1</Property>
</Record>
<Record Identifier="V0S33P2" Type="TrxOutput" Readonly="0">
<Property Name="Name">job_name_list_l</Property>
<Property Name="Partner">V0S35</Property>
<Property Name="RowLimit">0</Property>
<Property Name="RoutineTransforms">\(1B)</Property>
<Collection Name="Columns" Type="OutputColumn">
<SubRecord>
<Property Name="Name">pseudo_key</Property>
<Property Name="SqlType">SmallInt</Property>
<Property Name="Nullable">No</Property>
<Property Name="KeyPosition">Yes</Property>
<Property Name="Derivation">single_row_l.dummy</Property>
<Property Name="ParsedDerivation">single_row_l.dummy</Property>
<Property Name="SourceColumn">single_row_l.dummy</Property>
<Property Name="Transform">\(1B)</Property>
<Property Name="PadChar"/>
<Property Name="ColumnReference">pseudo_key</Property>
<Property Name="ExtendedPrecision">0</Property>
<Property Name="TaggedSubrec">0</Property>
<Property Name="OccursVarying">0</Property>
</SubRecord>
<SubRecord>
<Property Name="Name">job_list</Property>
<Property Name="SqlType">LongVarChar</Property>
<Property Name="Derivation">JobListDelimitedss</Property>
<Property Name="ParsedDerivation">JobListDelimitedss</Property>
<Property Name="Transform">\(1B)</Property>
<Property Name="StageVars">JobListDelimitedss</Property>
<Property Name="PadChar"/>
</SubRecord>
</Collection>
<Property Name="LeftTextPos">72</Property>
<Property Name="TopTextPos">231</Property>
</Record>
<Record Identifier="V0S35" Type="HashedFileStage" Readonly="0">
<Property Name="Name">expand_job_list</Property>
<Property Name="Description">Takes a single row of input containing a list of jobs and expands the data into one row per job in the list.</Property>
<Property Name="NextID">3</Property>
<Property Name="InputPins">V0S35P1</Property>
<Property Name="OutputPins">V0S35P2</Property>
<Property Name="UVcompatible">0</Property>
<Property Name="SQLNull">0</Property>
</Record>
<Record Identifier="V0S35P1" Type="HashedInput" Readonly="0">
<Property Name="Name">job_name_list_l</Property>
<Property Name="Partner">V0S33</Property>
<Property Name="FileName">job_list_1_to_n_h</Property>
<Property Name="ClearFile">1</Property>
<Property Name="Backup">0</Property>
<Property Name="DisableCW">1</Property>
<Property Name="CreateFileOptions">DYNAMIC</Property>
</Record>
<Record Identifier="V0S35P2" Type="HashedOutput" Readonly="0">
<Property Name="Name">job_list_l</Property>
<Property Name="Description">Normalize the jobs on the job_name to expand a single row into one row per job name.</Property>
<Property Name="Partner">V0S1</Property>
<Property Name="FileName">job_list_1_to_n_h</Property>
<Property Name="InMemory">0</Property>
<Property Name="Normalize">job_name</Property>
<Property Name="LeftTextPos">72</Property>
<Property Name="TopTextPos">354</Property>
<Property Name="RecordLevelRead">0</Property>
<Collection Name="Columns" Type="OutputColumn">
<SubRecord>
<Property Name="Name">pseudo_key</Property>
<Property Name="SqlType">SmallInt</Property>
<Property Name="KeyPosition">Yes</Property>
<Property Name="AllowCRLF">No</Property>
<Property Name="PadChar"/>
</SubRecord>
<SubRecord>
<Property Name="Name">job_name</Property>
<Property Name="SqlType">VarChar</Property>
<Property Name="Precision">20</Property>
<Property Name="Association">job_name</Property>
<Property Name="FieldType">MV</Property>
<Property Name="AllowCRLF">No</Property>
<Property Name="PadChar"/>
</SubRecord>
</Collection>
</Record>
<Record Identifier="V0S37" Type="TransformerStage" Readonly="0">
<Property Name="Name">log_detail_get_t</Property>
<Property Name="NextID">3</Property>
<Property Name="InputPins">V0S37P1</Property>
<Property Name="OutputPins">V0S37P2</Property>
<Collection Name="StageVars" Type="StageVar">
<SubRecord>
<Property Name="Name">LogContent</Property>
<Property Name="Description">Content of the job log for a given job.</Property>
<Property Name="Expression">JobLogGet(job_list_restricted_l.job_name)</Property>
<Property Name="ParsedExpression">JobLogGet(job_list_restricted_l.job_name)</Property>
<Property Name="SourceColumn">job_list_restricted_l.job_name</Property>
<Property Name="Transform">JobLogGet\(1B)</Property>
<Property Name="Precision">18</Property>
<Property Name="ColScale">4</Property>
</SubRecord>
<SubRecord>
<Property Name="Name">ProcessJob</Property>
<Property Name="Description">Indicates whether or not to process the job.</Property>
<Property Name="Expression">(Len(LogContent) > 0)</Property>
<Property Name="ParsedExpression">(Len(LogContent) > 0)</Property>
<Property Name="StageVars">LogContent</Property>
<Property Name="Transform">\(1B)</Property>
<Property Name="Precision">18</Property>
<Property Name="ColScale">4</Property>
</SubRecord>
</Collection>
</Record>
<Record Identifier="V0S37P1" Type="TrxInput" Readonly="0">
<Property Name="Name">job_list_restricted_l</Property>
<Property Name="Partner">V0S1</Property>
<Property Name="LinkType">1</Property>
</Record>
<Record Identifier="V0S37P2" Type="TrxOutput" Readonly="0">
<Property Name="Name">non_empty_logs_l</Property>
<Property Name="Partner">V0S18</Property>
<Property Name="Constraint">ProcessJob</Property>
<Property Name="Reject">No</Property>
<Property Name="RowLimit">0</Property>
<Property Name="ParsedConstraint">ProcessJob</Property>
<Property Name="RoutineTransforms">\(1B)</Property>
<Collection Name="Columns" Type="OutputColumn">
<SubRecord>
<Property Name="Name">job_name</Property>
<Property Name="SqlType">VarChar</Property>
<Property Name="Precision">20</Property>
<Property Name="Nullable">No</Property>
<Property Name="KeyPosition">Yes</Property>
<Property Name="Derivation">job_list_restricted_l.job_name</Property>
<Property Name="ParsedDerivation">job_list_restricted_l.job_name</Property>
<Property Name="SourceColumn">job_list_restricted_l.job_name</Property>
<Property Name="Transform">\(1B)</Property>
<Property Name="PadChar"/>
<Property Name="ColumnReference">name</Property>
<Property Name="ExtendedPrecision">0</Property>
<Property Name="TaggedSubrec">0</Property>
<Property Name="OccursVarying">0</Property>
</SubRecord>
<SubRecord>
<Property Name="Name">log_content</Property>
<Property Name="SqlType">LongVarChar</Property>
<Property Name="Derivation">LogContent</Property>
<Property Name="ParsedDerivation">LogContent</Property>
<Property Name="Transform">\(1B)</Property>
<Property Name="StageVars">LogContent</Property>
<Property Name="PadChar"/>
</SubRecord>
</Collection>
<Property Name="StageVars">ProcessJob</Property>
<Property Name="LeftTextPos">72</Property>
<Property Name="TopTextPos">594</Property>
</Record>
<Record Identifier="V1" Type="ContainerView" Readonly="0">
<Property Name="Name">DataStageJobLogContent</Property>
<Property Name="NextID">4</Property>
<Property Name="InputPins">V1P3</Property>
<Property Name="IsTopLevel">0</Property>
<Property Name="StageList">V1|V0S30|V0S18|V0S37|V0S35|V0S33|V0S1|V1S41|C3</Property>
<Property Name="StageXPos">48|48|48|48|48|48|48|48|48</Property>
<Property Name="StageYPos">1008|768|648|528|288|168|408|888|48</Property>
<Property Name="StageTypes">ID_PALETTEOUTPUT|CTransformerStage|CHashedFileStage|CTransformerStage|CHashedFileStage|CTransformerStage|CTransformerStage|CCustomStage|ID_PALETTESHAREDCONTAINER</Property>
<Property Name="NextStageID">44</Property>
<Property Name="SnapToGrid">1</Property>
<Property Name="GridLines">0</Property>
<Property Name="ZoomValue">100</Property>
<Property Name="StageXSize">49|48|48|48|48|48|48|49|48</Property>
<Property Name="StageYSize">49|48|48|48|48|48|48|49|48</Property>
<Property Name="ContainerViewSizing">0022 0022 0740 0251 0000 0001 0000 0720</Property>
</Record>
<Record Identifier="V1P3" Type="StdPin" Readonly="0">
<Property Name="Name">job_log_sorted_l</Property>
<Property Name="Partner">V1S41</Property>
<Property Name="LinkType">1</Property>
</Record>
<Record Identifier="V1S41" Type="CustomStage" Readonly="0">
<Property Name="Name">order_job_log_details_s</Property>
<Property Name="NextID">3</Property>
<Property Name="InputPins">V1S41P2</Property>
<Property Name="OutputPins">V1S41P1</Property>
<Property Name="StageType">sort</Property>
<Collection Name="Properties" Type="CustomProperty">
<SubRecord>
<Property Name="Name">SORTSPEC</Property>
<Property Name="Value">job_name asc, event_id asc</Property>
</SubRecord>
<SubRecord>
<Property Name="Name">MAXROWSINVM</Property>
<Property Name="Value">10000</Property>
</SubRecord>
<SubRecord>
<Property Name="Name">TEMPDIR</Property>
</SubRecord>
<SubRecord>
<Property Name="Name">ESCCHAR</Property>
<Property Name="Value">\</Property>
</SubRecord>
<SubRecord>
<Property Name="Name">TRACLVL</Property>
<Property Name="Value">0</Property>
</SubRecord>
<SubRecord>
<Property Name="Name">STABLESORT</Property>
<Property Name="Value">no</Property>
</SubRecord>
<SubRecord>
<Property Name="Name">COLSEP</Property>
<Property Name="Value">,</Property>
</SubRecord>
<SubRecord>
<Property Name="Name">MAXOPENFILES</Property>
<Property Name="Value">10</Property>
</SubRecord>
</Collection>
</Record>
<Record Identifier="V1S41P1" Type="CustomOutput" Readonly="0">
<Property Name="Name">job_log_sorted_l</Property>
<Property Name="Partner">V1</Property>
<Collection Name="Columns" Type="OutputColumn">
<SubRecord>
<Property Name="Name">job_name</Property>
<Property Name="SqlType">VarChar</Property>
<Property Name="Precision">20</Property>
<Property Name="Nullable">No</Property>
<Property Name="KeyPosition">Yes</Property>
<Property Name="Group">No</Property>
<Property Name="Transform">\(1B)</Property>
<Property Name="TableDef">Saved\sort_39\job_log_l</Property>
<Property Name="FieldType">S</Property>
<Property Name="AllowCRLF">No</Property>
<Property Name="PadNulls">Error</Property>
<Property Name="SortingOrder">Ascending</Property>
<Property Name="PadChar"/>
<Property Name="ColumnReference">name</Property>
<Property Name="ExtendedPrecision">0</Property>
<Property Name="TaggedSubrec">0</Property>
<Property Name="OccursVarying">0</Property>
</SubRecord>
<SubRecord>
<Property Name="Name">event_id</Property>
<Property Name="SqlType">Integer</Property>
<Property Name="Precision">5</Property>
<Property Name="Nullable">No</Property>
<Property Name="KeyPosition">Yes</Property>
<Property Name="Group">No</Property>
<Property Name="Transform">\(1B)</Property>
<Property Name="AllowCRLF">No</Property>
<Property Name="PadNulls">Error</Property>
<Property Name="SortingOrder">Ascending</Property>
<Property Name="PadChar"/>
</SubRecord>
<SubRecord>
<Property Name="Name">event_timestamp</Property>
<Property Name="SqlType">VarChar</Property>
<Property Name="Precision">19</Property>
<Property Name="Transform">\(1B)</Property>
<Property Name="TableDef">Saved\sort_39\job_log_l</Property>
<Property Name="PadChar"/>
<Property Name="ColumnReference">event_timestamp</Property>
</SubRecord>
<SubRecord>
<Property Name="Name">event_type</Property>
<Property Name="SqlType">VarChar</Property>
<Property Name="Precision">2</Property>
<Property Name="Nullable">No</Property>
<Property Name="KeyPosition">No</Property>
<Property Name="Group">No</Property>
<Property Name="Transform">\(1B)</Property>
<Property Name="AllowCRLF">No</Property>
<Property Name="PadNulls">Error</Property>
<Property Name="SortingOrder">Ascending</Property>
<Property Name="PadChar"/>
</SubRecord>
<SubRecord>
<Property Name="Name">multiple_instance_detail</Property>
<Property Name="SqlType">VarChar</Property>
<Property Name="Precision">255</Property>
<Property Name="Nullable">Yes</Property>
<Property Name="KeyPosition">No</Property>
<Property Name="Group">No</Property>
<Property Name="Transform">\(1B)</Property>
<Property Name="TableDef">Saved\sort_39\job_log_l</Property>
<Property Name="AllowCRLF">No</Property>
<Property Name="PadNulls">Error</Property>
<Property Name="SortingOrder">Ascending</Property>
<Property Name="PadChar"/>
<Property Name="ColumnReference">event_unknown_column</Property>
</SubRecord>
<SubRecord>
<Property Name="Name">job_instance</Property>
<Property Name="SqlType">VarChar</Property>
<Property Name="Precision">4</Property>
<Property Name="Nullable">No</Property>
<Property Name="KeyPosition">No</Property>
<Property Name="Group">No</Property>
<Property Name="Transform">\(1B)</Property>
<Property Name="AllowCRLF">No</Property>
<Property Name="PadNulls">Error</Property>
<Property Name="SortingOrder">Ascending</Property>
<Property Name="PadChar"/>
</SubRecord>
<SubRecord>
<Property Name="Name">event_message</Property>
<Property Name="SqlType">LongVarChar</Property>
<Property Name="Nullable">Yes</Property>
<Property Name="KeyPosition">No</Property>
<Property Name="Group">No</Property>
<Property Name="Transform">\(1B)</Property>
<Property Name="AllowCRLF">No</Property>
<Property Name="PadNulls">Error</Property>
<Property Name="SortingOrder">Ascending</Property>
<Property Name="PadChar"/>
</SubRecord>
</Collection>
<Property Name="LeftTextPos">72</Property>
<Property Name="TopTextPos">956</Property>
</Record>
<Record Identifier="V1S41P2" Type="CustomInput" Readonly="0">
<Property Name="Name">event_detail_unsorted_l</Property>
<Property Name="Partner">V0S30</Property>
<Property Name="LinkType">1</Property>
<Property Name="TXNBehaviour">0</Property>
<Property Name="EnableTxGroup">0</Property>
</Record>
</SharedContainer>
</DSExport>
The final object is a job that actually uses the shared container. I'll post this, have some dinner, and then follow up with a brief description of using this job.
<edit: slight error in original code. updated with correction>
Code: Select all
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE DSExport [
<!ELEMENT DSExport (Header, (Job |
SharedContainer |
DataElements |
Routines |
StageTypes |
TableDefinitions |
Transforms)+
) >
<!ELEMENT Header EMPTY >
<!ATTLIST Header
CharacterSet CDATA #IMPLIED
ExportingTool CDATA #IMPLIED
ToolVersion CDATA #REQUIRED
ServerName CDATA #IMPLIED
ToolInstanceID CDATA #IMPLIED
Date CDATA #IMPLIED
Time CDATA #IMPLIED
>
<!ELEMENT Job (Record*) >
<!ATTLIST Job
Identifier CDATA #REQUIRED
DateModified CDATA #IMPLIED
TimeModified CDATA #IMPLIED
>
<!ELEMENT SharedContainer (Record*) >
<!ATTLIST SharedContainer
Identifier CDATA #REQUIRED
DateModified CDATA #IMPLIED
TimeModified CDATA #IMPLIED
>
<!ELEMENT DataElements (Record*) >
<!ELEMENT Routines (Record*) >
<!ELEMENT StageTypes (Record*) >
<!ELEMENT TableDefinitions (Record*) >
<!ELEMENT Transforms (Record*) >
<!ELEMENT Record ((Property | Collection)*) >
<!ATTLIST Record
Identifier CDATA #REQUIRED
DateModified CDATA #IMPLIED
TimeModified CDATA #IMPLIED
Type CDATA #REQUIRED
Readonly (0|1) '0'
>
<!ELEMENT Property (#PCDATA| ListItem)* >
<!ATTLIST Property
Name CDATA #REQUIRED
PreFormatted (0|1) '0'
xml:space (default|preserve) 'preserve'
>
<!ELEMENT ListItem (#PCDATA)* >
<!ATTLIST ListItem
DisplayName CDATA #REQUIRED
xml:space (default|preserve) 'preserve'
>
<!ELEMENT Collection (SubRecord*) >
<!ATTLIST Collection
Name CDATA #REQUIRED
Type CDATA #REQUIRED
>
<!ELEMENT SubRecord (Property*) >
]>
<DSExport>
<Header CharacterSet="ENGLISH" ExportingTool="Ascential DataStage Export" ToolVersion="4" ServerName="djm" ToolInstanceID="dev" Date="2005-10-03" Time="18.05.51"/>
<Job Identifier="JobLogDump" DateModified="2005-10-03" TimeModified="17.58.03">
<Record Identifier="C9" Type="ContainerStage" Readonly="0">
<Property Name="Name">job_log_content_sc</Property>
<Property Name="Description">Obtain full content of Job Log.</Property>
<Property Name="NextID">2</Property>
<Property Name="OutputPins">C9P1</Property>
<Collection Name="ParameterValues" Type="ParamValues">
<SubRecord>
<Property Name="Name">JOB_NAME_MATCH</Property>
<Property Name="Description">#JOB_NAME_MATCH#</Property>
</SubRecord>
<SubRecord>
<Property Name="Name">EXCLUDED_JOBS_LIST</Property>
<Property Name="Description">#EXCLUDED_JOBS_LIST#</Property>
</SubRecord>
<SubRecord>
<Property Name="Name">EXCLUDED_JOBS_PATTERN</Property>
<Property Name="Description">#EXCLUDED_JOBS_PATTERN#</Property>
</SubRecord>
</Collection>
<Property Name="ContainerType">0</Property>
<Property Name="ContainerName">DataStageJobLogContent</Property>
</Record>
<Record Identifier="C9P1" Type="StdOutput" Readonly="0">
<Property Name="Name">job_log_sorted_l</Property>
<Property Name="Partner">V0S28</Property>
<Collection Name="Columns" Type="OutputColumn">
<SubRecord>
<Property Name="Name">job_name</Property>
<Property Name="SqlType">VarChar</Property>
<Property Name="Precision">20</Property>
<Property Name="Nullable">No</Property>
<Property Name="KeyPosition">Yes</Property>
<Property Name="FieldType">S</Property>
<Property Name="AllowCRLF">No</Property>
<Property Name="PadNulls">Error</Property>
<Property Name="PadChar"/>
<Property Name="ExtendedPrecision">0</Property>
<Property Name="TaggedSubrec">0</Property>
<Property Name="OccursVarying">0</Property>
</SubRecord>
<SubRecord>
<Property Name="Name">event_id</Property>
<Property Name="SqlType">Integer</Property>
<Property Name="Precision">5</Property>
<Property Name="Nullable">No</Property>
<Property Name="KeyPosition">Yes</Property>
<Property Name="Transform">\(1B)</Property>
<Property Name="AllowCRLF">No</Property>
<Property Name="PadNulls">Error</Property>
<Property Name="PadChar"/>
<Property Name="ExtendedPrecision">0</Property>
<Property Name="TaggedSubrec">0</Property>
<Property Name="OccursVarying">0</Property>
</SubRecord>
<SubRecord>
<Property Name="Name">event_timestamp</Property>
<Property Name="SqlType">VarChar</Property>
<Property Name="Precision">19</Property>
<Property Name="Nullable">No</Property>
<Property Name="KeyPosition">No</Property>
<Property Name="Transform">\(1B)</Property>
<Property Name="AllowCRLF">No</Property>
<Property Name="PadNulls">Error</Property>
<Property Name="PadChar"/>
<Property Name="ExtendedPrecision">0</Property>
<Property Name="TaggedSubrec">0</Property>
<Property Name="OccursVarying">0</Property>
</SubRecord>
<SubRecord>
<Property Name="Name">event_type</Property>
<Property Name="SqlType">VarChar</Property>
<Property Name="Precision">2</Property>
<Property Name="Nullable">No</Property>
<Property Name="KeyPosition">No</Property>
<Property Name="Transform">\(1B)</Property>
<Property Name="AllowCRLF">No</Property>
<Property Name="PadNulls">Error</Property>
<Property Name="PadChar"/>
<Property Name="ExtendedPrecision">0</Property>
<Property Name="TaggedSubrec">0</Property>
<Property Name="OccursVarying">0</Property>
</SubRecord>
<SubRecord>
<Property Name="Name">multiple_instance_detail</Property>
<Property Name="SqlType">VarChar</Property>
<Property Name="Precision">255</Property>
<Property Name="Nullable">Yes</Property>
<Property Name="KeyPosition">No</Property>
<Property Name="Transform">\(1B)</Property>
<Property Name="AllowCRLF">No</Property>
<Property Name="PadNulls">Error</Property>
<Property Name="PadChar"/>
<Property Name="ExtendedPrecision">0</Property>
<Property Name="TaggedSubrec">0</Property>
<Property Name="OccursVarying">0</Property>
</SubRecord>
<SubRecord>
<Property Name="Name">job_instance</Property>
<Property Name="SqlType">VarChar</Property>
<Property Name="Precision">4</Property>
<Property Name="Nullable">No</Property>
<Property Name="KeyPosition">No</Property>
<Property Name="Transform">\(1B)</Property>
<Property Name="AllowCRLF">No</Property>
<Property Name="PadNulls">Error</Property>
<Property Name="PadChar"/>
<Property Name="ExtendedPrecision">0</Property>
<Property Name="TaggedSubrec">0</Property>
<Property Name="OccursVarying">0</Property>
</SubRecord>
<SubRecord>
<Property Name="Name">event_message</Property>
<Property Name="SqlType">LongVarChar</Property>
<Property Name="Nullable">Yes</Property>
<Property Name="KeyPosition">No</Property>
<Property Name="Transform">\(1B)</Property>
<Property Name="AllowCRLF">No</Property>
<Property Name="PadNulls">Error</Property>
<Property Name="PadChar"/>
<Property Name="ExtendedPrecision">0</Property>
<Property Name="TaggedSubrec">0</Property>
<Property Name="OccursVarying">0</Property>
</SubRecord>
</Collection>
<Property Name="LeftTextPos">120</Property>
<Property Name="TopTextPos">114</Property>
<Property Name="PartnerLink">job_log_sorted_l</Property>
</Record>
<Record Identifier="ROOT" Type="JobDefn" Readonly="0">
<Property Name="Name">JobLogDump</Property>
<Property Name="Description">Create a flat file of the DataStage run-time log for input into the run-time analysis spreadsheet.</Property>
<Property Name="NextID">10</Property>
<Property Name="Container">V0</Property>
<Property Name="FullDescription" PreFormatted="1">Obtain the content of the DataStage run log for a particular job, matching jobs or all job. Create a flat file suitably formatted for input into the run-time analysis spreadsheet.
Note the job is laid out in the "Magill"-style vertical fashion (i.e. primary input at the top, primary output at the bottom, secondary inputs on the left, secondary outputs on the right).
</Property>
<Property Name="JobVersion">25.0.0</Property>
<Property Name="ControlAfterSubr">0</Property>
<Collection Name="Parameters" Type="Parameters">
<SubRecord>
<Property Name="Name">JOB_NAME_MATCH</Property>
<Property Name="Prompt">Pattern matching the name of the jobs to be included in the log dump.</Property>
<Property Name="Default">...</Property>
<Property Name="HelpTxt">Pattern matching follows DataStage pattern matching syntax e.g. 0X</Property>
<Property Name="ParamType">String</Property>
<Property Name="ParamLength">0</Property>
<Property Name="ParamScale">0</Property>
</SubRecord>
<SubRecord>
<Property Name="Name">EXCLUDED_JOBS_LIST</Property>
<Property Name="Prompt">Comma separated list of job names to exclude from processing.</Property>
<Property Name="Default">ArchiveClearDSJOBLog</Property>
<Property Name="HelpTxt">List of job names to exclude from processing from those jobs identified via the matching job names. Leave empty if a particular job names should not be used for exclusion purposes. If both a job name list and a pattern are specified, the exlcuded list of job will take precedence and the pattern will be ignored.</Property>
<Property Name="ParamType">String</Property>
<Property Name="ParamLength">0</Property>
<Property Name="ParamScale">0</Property>
</SubRecord>
<SubRecord>
<Property Name="Name">EXCLUDED_JOBS_PATTERN</Property>
<Property Name="Prompt">Pattern identifying job names to exclude from processing.</Property>
<Property Name="HelpTxt">Pattern against which if a job name matches, it will be excluded from processing from those jobs identified via the matching job names. Leave empty if job names should not be matched against a pattern for exclusion purposes. If both a job name list and a pattern are specified, the exlcuded list of job will take precedence and the pattern will be ignored.</Property>
<Property Name="ParamType">String</Property>
</SubRecord>
<SubRecord>
<Property Name="Name">DESTINATION_DIRECTORY</Property>
<Property Name="Prompt">Directory in which the job output should be placed.</Property>
<Property Name="Default">/tmp</Property>
<Property Name="ParamType">String</Property>
<Property Name="ParamLength">0</Property>
<Property Name="ParamScale">0</Property>
</SubRecord>
<SubRecord>
<Property Name="Name">OUTPUT_FILE_NAME</Property>
<Property Name="Prompt">Name of the file to which output should be written.</Property>
<Property Name="Default">ds_log.txt</Property>
<Property Name="ParamType">String</Property>
<Property Name="ParamLength">0</Property>
<Property Name="ParamScale">0</Property>
</SubRecord>
</Collection>
<Property Name="NLSLocale" PreFormatted="1">Time=(Project default)
Numeric=(Project default)
Currency=(Project default)
CType=(Project default)
Collate=(Project default)</Property>
<Property Name="JobType">Server</Property>
<Property Name="Category">UTIL\ADMIN</Property>
<Property Name="CenturyBreakYear">30</Property>
<Property Name="NextAliasID">2</Property>
<Property Name="ParameterFileDDName">DD00001</Property>
<Property Name="ReservedWordCheck">Yes</Property>
<Property Name="ValidationStatus">0</Property>
<Property Name="Uploadable">0</Property>
<Property Name="PgmCustomizationFlag">0</Property>
<Property Name="JobReportFlag">0</Property>
<Property Name="AllowMultipleInvocations">0</Property>
<Property Name="Act2ActOverideDefaults">0</Property>
<Property Name="Act2ActEnableRowBuffer">0</Property>
<Property Name="Act2ActUseIPC">0</Property>
<Property Name="EnableCacheSharing">0</Property>
<Property Name="RuntimeColumnPropagation">0</Property>
</Record>
<Record Identifier="V0" Type="ContainerView" Readonly="0">
<Property Name="Name">Job</Property>
<Property Name="NextID">1</Property>
<Property Name="IsTopLevel">0</Property>
<Property Name="StageList">V0S28|C9</Property>
<Property Name="StageXPos">96|96</Property>
<Property Name="StageYPos">168|48</Property>
<Property Name="StageTypes">CSeqFileStage|ID_PALETTESHAREDCONTAINER</Property>
<Property Name="NextStageID">60</Property>
<Property Name="SnapToGrid">1</Property>
<Property Name="GridLines">0</Property>
<Property Name="ZoomValue">100</Property>
<Property Name="StageXSize">49|48</Property>
<Property Name="StageYSize">49|48</Property>
<Property Name="ContainerViewSizing">0000 0290 0727 0580 0000 0001 0000 0000</Property>
</Record>
<Record Identifier="V0S28" Type="SeqFileStage" Readonly="0">
<Property Name="Name">job_analysis_log_sf</Property>
<Property Name="NextID">6</Property>
<Property Name="InputPins">V0S28P5</Property>
<Property Name="UnixFormat">Unix</Property>
<Property Name="PipeStage">File</Property>
<Property Name="UnicodeBOM">1</Property>
<Property Name="UnicodeSwapped">1</Property>
<Property Name="WithFilter">0</Property>
</Record>
<Record Identifier="V0S28P5" Type="SeqInput" Readonly="0">
<Property Name="Name">job_log_sorted_l</Property>
<Property Name="Partner">C9</Property>
<Property Name="FileName">#OUTPUT_FILE#</Property>
<Property Name="Delimiter">,</Property>
<Property Name="QuoteChar">"</Property>
<Property Name="ColHeaders" PreFormatted="0">0</Property>
<Property Name="OmitNewLine" PreFormatted="0">0</Property>
<Property Name="Append" PreFormatted="0">0</Property>
<Property Name="Backup" PreFormatted="0">0</Property>
<Property Name="FixedWidth" PreFormatted="0">0</Property>
<Property Name="ColSpace" PreFormatted="0">0</Property>
<Property Name="EnforceMetaData">0</Property>
<Property Name="Writetime">0</Property>
<Property Name="WaitforReader">0</Property>
<Property Name="Flushrow">0</Property>
<Property Name="PadChar">#</Property>
</Record>
</Job>
</DSExport>
Last edited by djm on Mon Oct 03, 2005 9:35 pm, edited 1 time in total.
Using this job is (meant to be) fairly straight forward. There are five parameters:
JOB_NAME_MATCH
Specifies a pattern which jobs' name should match to be included in the output. Use ... to obtain all jobs.
EXCLUDED_JOBS_LIST
A comma separated list of job names that are excluded from the output.
EXCLUDED_JOBS_PATTERN
A pattern that identifies job names which, if they match, are excluded from the output.
DESTINATION_DIRECTORY
The directory in which the output should be written.
OUTPUT_FILE_NAME
The file to which the output should be written.
Note that the parameters EXCLUDED_JOBS_LIST and EXCLUDED_JOBS_PATTERN are mutually-exclusice i.e. you can have one or the other but not both. If you specify both, the list of job names takes precedence over the pattern.
And that's about it.
Hopefully someone out there may find this useful. If you have any corrections to this job, by all means reply to this topic.
David.
JOB_NAME_MATCH
Specifies a pattern which jobs' name should match to be included in the output. Use ... to obtain all jobs.
EXCLUDED_JOBS_LIST
A comma separated list of job names that are excluded from the output.
EXCLUDED_JOBS_PATTERN
A pattern that identifies job names which, if they match, are excluded from the output.
DESTINATION_DIRECTORY
The directory in which the output should be written.
OUTPUT_FILE_NAME
The file to which the output should be written.
Note that the parameters EXCLUDED_JOBS_LIST and EXCLUDED_JOBS_PATTERN are mutually-exclusice i.e. you can have one or the other but not both. If you specify both, the list of job names takes precedence over the pattern.
And that's about it.
Hopefully someone out there may find this useful. If you have any corrections to this job, by all means reply to this topic.
David.
-
htrisakti3
- Charter Member
- Posts: 36
- Joined: Thu Jun 10, 2004 11:22 pm
return (-13)
I tried this on v7.5
sometimes it works, but most of the time, JobGetLog returns (-13)
what does this mean ? I saw that the job in question finished normally & I can see the logs in director
sometimes it works, but most of the time, JobGetLog returns (-13)
what does this mean ? I saw that the job in question finished normally & I can see the logs in director
Check (a) which routine version and (b) only job running
I don't see -13s any more. I did at one stage but I think it was a bug in the initial version(s?) of JobLogGet. I deployed an update to it on Oct 7th. Did you obtain a version prior to then? Is so, could you try getting the most recent version of the routine and see if it helps? Plus there was a minor tweak to the job that uses the shared container output and writes it to the flat file, so you may want to check you have the up-to-date version for that too.
I've also noticed that the jobs may head off to cloud-cuckooland if other jobs are running when it is running. My suposition is that this is a feature (it is documented!) of DSAttachJob not allowing multiple connections to the same job. I'm also looking into seeing whether there is anything smarter than can be done within JobLogGet to deal with uncompiled jobs, as at the moment it spits the dummy on such jobs.
I appreciate you being a guinea pig.
David
I've also noticed that the jobs may head off to cloud-cuckooland if other jobs are running when it is running. My suposition is that this is a feature (it is documented!) of DSAttachJob not allowing multiple connections to the same job. I'm also looking into seeing whether there is anything smarter than can be done within JobLogGet to deal with uncompiled jobs, as at the moment it spits the dummy on such jobs.
I appreciate you being a guinea pig.
David
Require your assistance!
Hi there D,
I have been trying to something similar to what you have done but i am getting no where.I am new to DS. I need to get the log dumps and then i came across your forum. I am not sure on how to use in DS, whether i create it a function or job etc. Could you help me kindly. I sense what you have placed here is very useful.
Expecting your assistance.
Thank you and have a nice day.
Regards,
Holymac
I have been trying to something similar to what you have done but i am getting no where.I am new to DS. I need to get the log dumps and then i came across your forum. I am not sure on how to use in DS, whether i create it a function or job etc. Could you help me kindly. I sense what you have placed here is very useful.
Expecting your assistance.
Thank you and have a nice day.
Regards,
Holymac
Hi Holymac,
Let's see if the following helps you.
1) For each of the "code" blocks in the above thread, copy the text and paste it into a distinctly named file with a file name suffix of ".xml".
2) Run the DataStage Manager tool and select the menu option "Import/DataStage Components (XML) ...".
3) Import each of the files saved in 1), one at a time.
4) Compile the Routine found in \Routines\Custom\JobLog
5) Run DataStage Designer and open the job \Jobs\Util\Admin\JobLogDump". Compile the job.
6) The compiled job should then be able to run.
As an aside, a bug has been identified in the code when the job logs are partially cleared but not reset (i.e. the event numbers continue to increase but don't start at 0). I'm working on a fix but I have yet to fit it into my schedule. By and large, it is unlikely that this bug will cause a problem if you follow the recommended practice of resetting the job logs within a project.
David.
Let's see if the following helps you.
1) For each of the "code" blocks in the above thread, copy the text and paste it into a distinctly named file with a file name suffix of ".xml".
2) Run the DataStage Manager tool and select the menu option "Import/DataStage Components (XML) ...".
3) Import each of the files saved in 1), one at a time.
4) Compile the Routine found in \Routines\Custom\JobLog
5) Run DataStage Designer and open the job \Jobs\Util\Admin\JobLogDump". Compile the job.
6) The compiled job should then be able to run.
As an aside, a bug has been identified in the code when the job logs are partially cleared but not reset (i.e. the event numbers continue to increase but don't start at 0). I'm working on a fix but I have yet to fit it into my schedule. By and large, it is unlikely that this bug will cause a problem if you follow the recommended practice of resetting the job logs within a project.
David.
Correction to JobLogGet
Update to JobLogGet routine to resolved issue identified by vaby in the original code. Thank to vladimir for persevering while I found the time to correct the problem.
The complete code in this post can be downloaded as part of a zip file from this web page.This may be easier than having to copy each piece of code from within this post and saving each one with a distinct file name. Within the zip file are two additional items; an additional Data Stage Object and an Excel workbook template. These two objects provide the ability to snapshot the time line of jobs for the purpose of analysing the profile of schedules (as outlined in another post on this forum).
If you already have every object and simply want the updated code, just copy the code listed below and import it into Data Stage.
David
The complete code in this post can be downloaded as part of a zip file from this web page.This may be easier than having to copy each piece of code from within this post and saving each one with a distinct file name. Within the zip file are two additional items; an additional Data Stage Object and an Excel workbook template. These two objects provide the ability to snapshot the time line of jobs for the purpose of analysing the profile of schedules (as outlined in another post on this forum).
If you already have every object and simply want the updated code, just copy the code listed below and import it into Data Stage.
David
Code: Select all
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE DSExport [
<!ELEMENT DSExport (Header, (Job |
SharedContainer |
DataElements |
Routines |
StageTypes |
TableDefinitions |
Transforms)+
) >
<!ELEMENT Header EMPTY >
<!ATTLIST Header
CharacterSet CDATA #IMPLIED
ExportingTool CDATA #IMPLIED
ToolVersion CDATA #REQUIRED
ServerName CDATA #IMPLIED
ToolInstanceID CDATA #IMPLIED
Date CDATA #IMPLIED
Time CDATA #IMPLIED
>
<!ELEMENT Job (Record*) >
<!ATTLIST Job
Identifier CDATA #REQUIRED
DateModified CDATA #IMPLIED
TimeModified CDATA #IMPLIED
>
<!ELEMENT SharedContainer (Record*) >
<!ATTLIST SharedContainer
Identifier CDATA #REQUIRED
DateModified CDATA #IMPLIED
TimeModified CDATA #IMPLIED
>
<!ELEMENT DataElements (Record*) >
<!ELEMENT Routines (Record*) >
<!ELEMENT StageTypes (Record*) >
<!ELEMENT TableDefinitions (Record*) >
<!ELEMENT Transforms (Record*) >
<!ELEMENT Record ((Property | Collection)*) >
<!ATTLIST Record
Identifier CDATA #REQUIRED
DateModified CDATA #IMPLIED
TimeModified CDATA #IMPLIED
Type CDATA #REQUIRED
Readonly (0|1) '0'
>
<!ELEMENT Property (#PCDATA| ListItem)* >
<!ATTLIST Property
Name CDATA #REQUIRED
PreFormatted (0|1) '0'
xml:space (default|preserve) 'preserve'
>
<!ELEMENT ListItem (#PCDATA)* >
<!ATTLIST ListItem
DisplayName CDATA #REQUIRED
xml:space (default|preserve) 'preserve'
>
<!ELEMENT Collection (SubRecord*) >
<!ATTLIST Collection
Name CDATA #REQUIRED
Type CDATA #REQUIRED
>
<!ELEMENT SubRecord (Property*) >
]>
<DSExport>
<Header CharacterSet="ENGLISH" ExportingTool="Ascential DataStage Export" ToolVersion="4" ServerName="djm" ToolInstanceID="dev" Date="2006-03-03" Time="09.30.28"/>
<Routines>
<Record Identifier="JobLogGet" DateModified="2006-02-02" TimeModified="13.06.43" Type="Routine" Readonly="0">
<Property Name="Category">Custom\JobLog</Property>
<Property Name="ShortDesc">Obtain the job log for a given job.</Property>
<Property Name="Description" PreFormatted="1">Given the name of a job (parameter job_name_av), use the DataStage API to obtain the details from the job log.
The result is returned as a value marker (@VM) separated fields, each sub-divied into separate sub-fields by value markers (@VM).
</Property>
<Property Name="RoutineType">Transform Function</Property>
<Property Name="ArgCount">1</Property>
<Property Name="Built">1</Property>
<Property Name="SourceFlag">1</Property>
<Property Name="Author">David Manley</Property>
<Property Name="Version">2.0.0</Property>
<Property Name="TestValues">LoadFConnectionDeviceType2 LoadDwCustDim4UpdExisting djmDumpDwCustDim blarg CreateDBUser JobLogAnalysis djmDwCustDimDump djmPpDistInvDetDump</Property>
<Property Name="Source" PreFormatted="1">FUNCTION JobLogGet(job_name_av)
* ----------------------------------------------------------------------------
* Date By Comment
* 2005-12-28 David Manley Better handle situation where the first
* event id is not zero.
* 2005-10-07 David Manley Corrections to deal with an empty event
* log and obtaining timestamps.
* 2005-10-05 David Manley Switched from ignoring errors to recording
* errors.
* 2005-09-20 David Manley Refined approach to obtain job details.
* 2005-02-?? David Manley Initial release.
* ----------------------------------------------------------------------------
$INCLUDE DSINCLUDE JOBCONTROL.H
* Establish the job handle for the named job.
job_handle_li = DSAttachJob( job_name_av , DSJ.ERRNONE )
* Assume a default of failing to determine job details.
Ans = ""
* If the job handle could not be established ...
if ( not ( job_handle_li ) ) then
* ... don't do any more processing.
return (Ans)
end
* Get the highest event id for the job.
event_max_li = DSGetNewestLogId ( job_handle_li , DSJ.LOGANY)
* If no event was found ...
if ((event_max_li <= 0) or (event_max_li = DSJE.BADHANDLE) or (event_max_li = DSJE.BADTYPE)) then
* ... simulate the error log content.
summary_detail_ls = ""
end
else
* Obtain all of the log details.
summary_detail_ls = DSGetLogSummary ( job_handle_li , DSJ.LOGANY,
"" , "" , 0)
end
* Finished obtaining information about the job so detach from the job.
detach_result_li = DSDetachJob( job_handle_li )
* The job log details are held in a dynamic array, with each entry having
* 4 backslash separated entries. Manipulate the dynamic array in one go
* to:
* (a) Replace the first three backslahes with a text marker. Don't
* replace all backslashes since any backslash characters in the
* fourth field (which is the event message) should be left unchanged.
* (b) Replace any field marker character with the value marker character.
* Manipulating the dynamic array in this way will allow the data to then
* be loaded into a single column in a hash file but treated as a multi-
* value column.
Ans = change ( change (summary_detail_ls , "\" , @TM) , @FM , @TM : @VM )
RETURN(Ans)</Property>
<Property Name="Catalog">DSU.JobLogGet</Property>
<Property Name="JobType">Server</Property>
<Collection Name="Arguments" Type="RtnArgument">
<SubRecord>
<Property Name="Name">job_name_av</Property>
<Property Name="Desc">Name of the job</Property>
<Property Name="IOType">I</Property>
<Property Name="Nullable">No</Property>
<Property Name="KeyPosition">No</Property>
</SubRecord>
</Collection>
</Record>
</Routines>
</DSExport>
(Previously known as D)
Be alturistic and donate your spare CPU cycles to research. http://www.worldcommunitygrid.org/team/ ... TZ9H4CGVP1
Be alturistic and donate your spare CPU cycles to research. http://www.worldcommunitygrid.org/team/ ... TZ9H4CGVP1
Has anyone gotten this to work lately?
I'm trying to get it to work, (I was able to download the XML files, compile, run) but the SingleRow job keeps blowing up. I get the following error
JobLogDump..job_log_content_sc.single_row_sc.single_row_sf.single_row_l: ds_seqopen() - Error in filter command "echo '1'" -
The sequential file stage "single_row_sf" shows "/dev/null" as the file name (in the Outputs...General Tab) and "echo 1" in the Filter Command box. When I took out "echo 1" I was able to get it to run (once) to completion, but no output file was created (tmp folders are created, but no rows written). I see that the original code was "/dev/null | echo 1". I tried putting all that in the file name box, no luck. I am using DS 7.5.
I am running DS 7.5 on Unix.
Any help would be greatly appreciated.
Thank you.
Jacob Metzger
I'm trying to get it to work, (I was able to download the XML files, compile, run) but the SingleRow job keeps blowing up. I get the following error
JobLogDump..job_log_content_sc.single_row_sc.single_row_sf.single_row_l: ds_seqopen() - Error in filter command "echo '1'" -
The sequential file stage "single_row_sf" shows "/dev/null" as the file name (in the Outputs...General Tab) and "echo 1" in the Filter Command box. When I took out "echo 1" I was able to get it to run (once) to completion, but no output file was created (tmp folders are created, but no rows written). I see that the original code was "/dev/null | echo 1". I tried putting all that in the file name box, no luck. I am using DS 7.5.
I am running DS 7.5 on Unix.
Any help would be greatly appreciated.
Thank you.
Jacob Metzger