Waves with Hierarchical Stages

[wpkalsow]

We took a perfectly fine working job that uses hierarchical stage to parse transactions. We broke it by adding a Wave Generator stage in an attempt to get the Hierarchical stage to parse a single file at a time rather than an entire batch of 5000 files.

Our favorite error is currently: Xfm_EOM_Group,0: Failure during execution of operator logic.

Xfm_EOM_Group is the name of a transform following a funnel stage. This transform adds some default values and calculates a row number. The output goes to another Hierarchical stage that composes an XML file that is landed on a drive.

[chulett]

I'll save Ray a little work the next time he shows up and ask... do you have an actual question for us? Something specific you are looking for or just generic advice on your situation / error?

[eostic]

I often use end-of-wave to "release" xml when I need independent documents based on some key value. ...but end-of-wave has different impacts on all kinds of Stages.....it has to be tested carefully with each to understand what its flushing behavior will mean.

Ernie

[wpkalsow]

To achieve independent documents for XML from the Composer step, we use a Regroup step prior to the Composer step in the assembly. The Regroup does a great job of separating the XML into appropriate levels/files.

I would like to use the waves through the first 95% of the job and then do the last 5% as a continuous stream.

Process individual XML files for the first 95% of the job.
Separate the new documents based on line item count limits on the final 5% of the job.

Would I be able to add a second Wave Generator with a higher record count to effectively erase the end-of-waves from the initial Wave Generator?

[eostic]

Sorry. Don't know. Never tried it...... not sure what it will do.....

Ernie