DSXchange: DataStage and IBM Websphere Data Integration Forum
View next topic
View previous topic
Add To Favorites
Author Message
drkumar
Participant



Joined: 04 Mar 2014
Posts: 2
Location: Chennai
Points: 15

Post Posted: Tue Sep 11, 2018 9:58 pm Reply with quote    Back to top    

DataStage® Release: 11x
Job Type: Parallel
OS: Unix
Hi All
I have Source file which is pipe delimited, it's load to Teradata table, Here my source file structure will change dynamically its should handle below scenarios


1. My ETL Job should not fail it has to process with existing columns.

2. If any changes in source file structure, Need to get EMAIL notification.


Thanks
Ratna Kumar

_________________
Thanks
Ratna Kumar
chulett

Premium Poster


since January 2006

Group memberships:
Premium Members, Inner Circle, Server to Parallel Transition Group

Joined: 12 Nov 2002
Posts: 42790
Location: Denver, CO
Points: 220550

Post Posted: Wed Sep 12, 2018 5:55 am Reply with quote    Back to top    

Confused

In what world does a source file "change dynamically" and can you give us some ideas as to what exactly that may mean? Specifically wondering what kind of changes you are expecting. With the mention of "existing columns" are we talking about possible new columns being added to the end of the record? That's something that can be both checked for and fairly easily handled. But if we go full dynamic here - columns can be swapped around, new columns added in random spots, those kind of things - then that's a whole different kettle of fish.

Please clarify for us.

_________________
-craig

"I don't mind you comin' here and wastin' all my time time"
Rate this response:  
Not yet rated
ray.wurlod

Premium Poster
Participant

Group memberships:
Premium Members, Inner Circle, Australia Usergroup, Server to Parallel Transition Group

Joined: 23 Oct 2002
Posts: 54407
Location: Sydney, Australia
Points: 295092

Post Posted: Thu Sep 13, 2018 7:00 pm Reply with quote    Back to top    

How do you propose to detect a change?

That will affect how your processing runs.

_________________
RXP Services Ltd
Melbourne | Canberra | Sydney | Hong Kong | Hobart | Brisbane
currently hiring: Canberra, Sydney and Melbourne (especially seeking good business analysts)
Rate this response:  
Not yet rated
drkumar
Participant



Joined: 04 Mar 2014
Posts: 2
Location: Chennai
Points: 15

Post Posted: Fri Sep 14, 2018 8:42 am Reply with quote    Back to top    

Thank you Craig..
The new columns will be added at end of the record.

_________________
Thanks
Ratna Kumar
Rate this response:  
Not yet rated
chulett

Premium Poster


since January 2006

Group memberships:
Premium Members, Inner Circle, Server to Parallel Transition Group

Joined: 12 Nov 2002
Posts: 42790
Location: Denver, CO
Points: 220550

Post Posted: Fri Sep 14, 2018 9:06 am Reply with quote    Back to top    

Then it seems to me you will need to have a pre-check process for the presence of new columns and it sounds like that's as simple as counting the number of pipe delimiters in any record. There are multiple ways to handle it, could be a DataStage job but seems to me a script would be perfectly acceptable as well. When it finds more than the expected number of pipes, sound the alarm.

Then continue to run the job with the expected number of columns and see what kind of warnings / errors you get when there is a "short read" i.e. there are additional columns not included in the metadata. From what I recall, there was a checkbox on the Server side to suppress them, not sure how PX handles it but I would guess not gracefully. Build a small test harness to see and post your findings... unless someone chimes in with an answer for you before you get that together. Wink

_________________
-craig

"I don't mind you comin' here and wastin' all my time time"
Rate this response:  
Not yet rated
rrcr
Participant



Joined: 06 Jul 2017
Posts: 4

Points: 34

Post Posted: Mon Sep 24, 2018 5:30 am Reply with quote    Back to top    

Like chulett suggested we need to have before job script in which we can check for the number of columns . If the columns are more/less than the expected then send a mail notification.

coming to column definations we have to use schema file with RCP.
the schema file needs to be updated in before job script based on the number of columns

thanks,
Ramireddy Ch
Rate this response:  
Not yet rated
Display posts from previous:       

Add To Favorites
View next topic
View previous topic
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum



Powered by phpBB © 2001, 2002 phpBB Group
Theme & Graphics by Daz :: Portal by Smartor
All times are GMT - 6 Hours