DSXchange: DataStage and IBM Websphere Data Integration Forum
View next topic
View previous topic
Add To Favorites
Author Message
DSFreddie
Participant



Joined: 25 Nov 2009
Posts: 129

Points: 1378

Post Posted: Tue Sep 04, 2018 1:37 pm Reply with quote    Back to top    

DataStage® Release: 9x
Job Type: Parallel
OS: Unix
Hi All -

I am facing issue in one Datastage job that reads the data ( Has fields with JSON data, defined as BLOB datatype in the Oracle table) and performs BULK load to the Oracle table. Note that this table doesnt have any constraints defined, so ideally its supposed to perform the load run faster.

The job takes 15 minutes to load 5 Million records. The commit count/Array size gets defaulted to 1 due to the LongVarBinary field. Can someone pls suggest a better way to handle this scenario so that the loads run faster ?


Flow :

Sequential File ----> Oracle Load ( using Oracle Connector)
JSON data ---> Defined as LongVarBinary in Datastage

Thanks
Freddie
ray.wurlod

Premium Poster
Participant

Group memberships:
Premium Members, Inner Circle, Australia Usergroup, Server to Parallel Transition Group

Joined: 23 Oct 2002
Posts: 54398
Location: Sydney, Australia
Points: 295054

Post Posted: Tue Sep 04, 2018 4:26 pm Reply with quote    Back to top    

JSON is usually pure text, so could be defined as CLOB and treated by DataStage as Long VarChar data type. Then you could probably boost your granularity.

_________________
RXP Services Ltd
Melbourne | Canberra | Sydney | Hong Kong | Hobart | Brisbane
currently hiring: Canberra, Sydney and Melbourne (especially seeking good business analysts)
Rate this response:  
Not yet rated
DSFreddie
Participant



Joined: 25 Nov 2009
Posts: 129

Points: 1378

Post Posted: Tue Sep 11, 2018 1:14 pm Reply with quote    Back to top    

Thanks for your reply Ray. One of the reason why it is BLOB is to bring in more flexibility in terms of what we would store in the future.

Are there any other ways in which we can perform a Bulk Load operation against Oracle Exadata table with BLOB fields in it.

Pls help.

Thanks
Rate this response:  
Not yet rated
ray.wurlod

Premium Poster
Participant

Group memberships:
Premium Members, Inner Circle, Australia Usergroup, Server to Parallel Transition Group

Joined: 23 Oct 2002
Posts: 54398
Location: Sydney, Australia
Points: 295054

Post Posted: Thu Sep 13, 2018 6:58 pm Reply with quote    Back to top    

Probably, but not through DataStage. BLOB is not a supported data type for DataStage.

_________________
RXP Services Ltd
Melbourne | Canberra | Sydney | Hong Kong | Hobart | Brisbane
currently hiring: Canberra, Sydney and Melbourne (especially seeking good business analysts)
Rate this response:  
Not yet rated
Display posts from previous:       

Add To Favorites
View next topic
View previous topic
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum



Powered by phpBB © 2001, 2002 phpBB Group
Theme & Graphics by Daz :: Portal by Smartor
All times are GMT - 6 Hours