Single,variable length seq file to be read in parallel

sharmabhavesh · Post by **sharmabhavesh** » Wed Jun 27, 2012 1:18 am

Hi,
I have a single sequential file which I want to read in parallel. How can I achieve that? I went through multiple previous posts on the same topic but could not find a solution. There are many contradiciting answers in many posts.

pandeesh · Post by **pandeesh** » Wed Jun 27, 2012 3:45 am

Reading parallel from different jobs?

sharmabhavesh · Post by **sharmabhavesh** » Wed Jun 27, 2012 4:19 am

I am reading a single sequential file from a single parallel job.

ntr · Post by **ntr** » Wed Jun 27, 2012 6:13 am

by making read method as file pattern or by increasing of readers

sharmabhavesh · Post by **sharmabhavesh** » Wed Jun 27, 2012 9:02 am

Hi,
Can you please elaborate a bit?
I have heard that number of readers per node can be set only for fixed length files.
Also, please elaborate the first option you have specified.

zulfi123786 · Post by **zulfi123786** » Wed Jun 27, 2012 10:01 am

sharmabhavesh wrote:Also, please elaborate the first option you have specified.

The first option is not applicable in your case. when you have multiple files to be read and all share a specific pattern, only then you can go ahead with file pattern which is not in your case

chulett · Post by **chulett** » Wed Jun 27, 2012 1:04 pm

There's nothing stopping someone from using a file pattern to read a single file. And I believe you may find that the 'only for fixed-width files' mulitple readers advise is for older versions and that restriction was lifted in newer versions. Hopefully someone can confirm / deny...

ray.wurlod · Post by **ray.wurlod** » Wed Jun 27, 2012 5:35 pm

What version? Recent versions allow multiple readers per node to work with delimited formats.

Kryt0n · Post by **Kryt0n** » Wed Jun 27, 2012 5:49 pm

chulett wrote: And I believe you may find that the 'only for fixed-width files' mulitple readers advise is for older versions and that restriction was lifted in newer versions. Hopefully someone can confirm / deny...

That's good to know... didn't realise it had changed. Although your source file mustn't have a header row as it doesn't like them with multiple readers...

chulett · Post by **chulett** » Wed Jun 27, 2012 10:09 pm

Headers are still an issue for multiple files read using a file pattern, but I don't believe they would have an adverse effect on multiple readers on a single file, however.

Kryt0n · Post by **Kryt0n** » Wed Jun 27, 2012 11:36 pm

You wouldn't have thought so... but the options are mutually exclusive...

chandra.shekhar@tcs.com · Thu Jun 28, 2012 2:32 am

In version 8.1 and newer versions it works like

For delimited Files ---> Num of readers per node.
For Fixed Width Files -> Read from Multiple nodes.

These options are also mutually exclusive.

sharmabhavesh · Post by **sharmabhavesh** » Thu Jun 28, 2012 1:40 pm

Thanks for your quick responses. Things look more clear now.

DSXchange

Single,variable length seq file to be read in parallel

Single,variable length seq file to be read in parallel

Re: Single,variable length seq file to be read in parallel

Re: Single,variable length seq file to be read in parallel

Re: Single,variable length seq file to be read in parallel