Datastage -- Produces More Rows after each run

Post questions here relative to DataStage Enterprise/PX Edition for such areas as Parallel job design, Parallel datasets, BuildOps, Wrappers, etc.

Moderators: chulett, rschirm, roy

Post Reply
irvinew
Participant
Posts: 15
Joined: Mon Jun 18, 2018 8:52 am
Location: Regina, SK
Contact:

Datastage -- Produces More Rows after each run

Post by irvinew »

My job is working though it has a peculiar property.

It seems to produce more rows with each execution.

I doesn't produce too many rows; but with each turn it can read more rows until it can consume the whole file.

I have hierarchical stage that reads xml, in that stage it produces at the beginning about 150 rows. After some executions it can produce 213 rows. I think there should be about 270.

Is this a memory issue? it has to be a config issue somewhere. I tried increasing the java heap size but it doesn't seem to have any effect.

Sorry for the bizarre question; I haven't found the answer yet. The xml file is about 250K
Will
UCDI
Premium Member
Premium Member
Posts: 383
Joined: Mon Mar 21, 2016 2:00 pm

Post by UCDI »

we have an outstanding ticket with IBM about the H-stage dropping a record here and there at random. We can run the same data 3 times and get {one record missed, all the data, different record missed} as the result.

Whether you are seeing something similar or not, I can't say from your problem statement. Why do you think 270 is true and 213 is not? You need to run some simple tests with a few records (arguably 300 or so is few) but you need to know exactly what you expect in the output and compare that to what you got.
irvinew
Participant
Posts: 15
Joined: Mon Jun 18, 2018 8:52 am
Location: Regina, SK
Contact:

Post by irvinew »

For the one case; there is exactly 11 students in the xml file; resulting in 257 rows in total. It seems I have to work my way up from some random number like 60 then it will go to 85 then 114 and so on until it finally reaches the total.

I will continue to debug; thanks for the heads up
Will
Post Reply