Sequential File Stage in PX

srikanthd1978 · Post by **srikanthd1978** » Wed Jun 22, 2005 8:48 am

people,

When using a sequential stage with the read method as 'File Pattern', i cannot use "Number of Readers Per Node" option as they are mutually exclusive..

can someone explain why they are mutually exculusive ??

thanks

gh_amitava · Post by **gh_amitava** » Wed Jun 22, 2005 1:36 pm

They are mutually exclusive because DataStage will read Sequential file in sequential mode.. Row by Row.. Only one node will be used to read the entire file.

Regards
Amitava

vmcburney · Post by **vmcburney** » Wed Jun 22, 2005 4:12 pm

The number of Readers Per Node has a complex start up involving counting the size of the files, dividing the file across the specified number of readers to partition your data. I'm guessing that they don't allow this option for File Patterns due to complexity it adds to the stage code. It may be an option that is much easier to apply programmatically when you have a fully defined set of files rather then a fuzzy file pattern.

You can get around the problem by retrieving your file list at the sequence job level and running the job multiple times with different file names.

ray.wurlod · Post by **ray.wurlod** » Wed Jun 22, 2005 5:29 pm

As well as everything Vincent said, note that Readers Per Node is intended only for fixed-width format files. It's fully described in the Parallel Job Developer's Guide

bcarlson · Post by **bcarlson** » Thu Jun 23, 2005 11:10 am

One final note - the goal of multiple readers is to take advantage of multiple nodes - instead of one single stream of data, you have as many as there are readers. On the other hand, if you are using either filesets or the file pattern, you will already be taking advantage of multiple nodes - one per file

Now, if you could specify multiple readers to be use for each file (fixed length, of course), that would be great - but I believe that would be a discussion for another forum ( PX enhancements?)