Parallel Extender data flow

[akash_nitj]

Hi Datascions,
I have query regarding how px internally process the records.

My query is based upon the following design :

1. You read a input from a table having three coloumns ( table
definition : command_type, timestamp, data )

2. Input is presorted based upon timestamp value

3. Then you have transformer stage which have three ouptuts based
upon command_type value (C/U/D).

4. The data read is passed to three o/ps of transformer based upon
command_type and finally loaded to table XYZ.

5. All the links load data to same table XYZ.



QUERY : Will the data be loaded in the XYZ table in the same sequence ( based on timestamp value as it is read in ascending order of timestamp)as it is read assuming no record is rejected by any of intermediate stage .


My current understanding is datastage process each record one by one i.e is one record is read from input and written to o/p. Though Parrallelism is possible but in that case also records are written to o/p in same sequence as they are in input.

Please validate the same and also update my understanding if it is wrong


Regards
Akash

[cyh]

The data that loading into table XYZ have not particular, unless, you perform sorting explicitly. It is because DS will create data partition for the inputs, example :

Records in Table
A
B
C
D

Records during intermediate processing (with natural partitioning)

Code: Select all

Partition 1       Partition 2
A                 B
C                 D

The sequence of records in output tables cannot be determined. Possible sequence :

A, C, B, D
A, B, D, C
B, A, C, D ....

You have to re-sort the data before loading back to table ....


[/code]