Sequential file stage with schemafile crops text

Post questions here relative to DataStage Enterprise/PX Edition for such areas as Parallel job design, Parallel datasets, BuildOps, Wrappers, etc.

Moderators: chulett, rschirm, roy

Post Reply
ArndW
Participant
Posts: 16318
Joined: Tue Nov 16, 2004 9:08 am
Location: Germany
Contact:

Post by ArndW »

That is an interesting question indeed, but unfortunately the answer is that the extra length isn't detected or flagged with your scenario using a schema file and RCP.
Depending upon how important this is to you, there are some methods that I can think of to get this information, but they do involve quite a bit of work. You can read the data in using a dummy schema defining the columns as varchar with no length, apply an aggregator to get the maximum length and then compare that with the appropriate values in the schema file to output a flag on whether a column's length has been exceeded. It is neither simple nor efficient if you have large amounts of data.
UCDI
Premium Member
Premium Member
Posts: 383
Joined: Mon Mar 21, 2016 2:00 pm

Post by UCDI »

You could make an assumption that any string that runs out the full length "may have been" cropped and raise a flag when that happens with very simple transformer logic (for example have 2 outputs from a transformer, normal path as already happens and second path that writes all the records that have length = max to a dataset for evaluation). You can read the evaluation dataset to see if the vast majority of these are cropped records and try to find a way to handle it from there.
csphere
Premium Member
Premium Member
Posts: 4
Joined: Sat Feb 11, 2012 1:05 pm

Post by csphere »

Hi ArndW and UCDI,

Thanks for the reply's well the main thing is that I should build something that can do this checking, I was afraid that I missed "something" :wink:

Cheers!
Mike
Premium Member
Premium Member
Posts: 1021
Joined: Sun Mar 03, 2002 6:01 pm
Location: Tampa, FL

Post by Mike »

The transformer stage is frequently the way to avoid implicit type conversion warnings, transforming larger strings into smaller strings, and other such things as might require an explicit transformation.

You could try replacing the transformer with a copy or a modify to see if the warnings you want to see will appear.

Mike
Teej
Participant
Posts: 677
Joined: Fri Aug 08, 2003 9:26 am
Location: USA

Post by Teej »

You need to upgrade to 11.3.1.2 with the latest patch or 11.5 in order to get this feature.

We added this based on a customer's complaint recently. I do not have the specific APAR handy at the moment, but did discuss this in a recent ticket.

-T.J.
Post Reply