DSXchange: DataStage and IBM Websphere Data Integration Forum
View next topic
View previous topic
Add To Favorites
Author Message
mdbatra



Group memberships:
Premium Members

Joined: 22 Oct 2008
Posts: 174
Location: City of London
Points: 1276

Post Posted: Wed May 17, 2017 1:52 am Reply with quote    Back to top    

DataStage® Release: 11x
Job Type: Parallel
OS: Unix
Additional info: Issues post upgrade
Hi All
we are performing some initial testing in order to upgrade our existing DataStage v8.7 to v11.5. we migrated jobs to the new server and used connector migration tool which replaced all DRS stages (of v8.7) to DRS Connector stages (of v11.5).

But while running the jobs on new server, we have run into number of issues(both warnings and fatal ones which were not there in v8.7). And mostly all of them are in relation to DRS connector stage. Few examples are as below:

1. User-defined queries having columns without aliases in select clause doesn’t work i.e.

Upper(ColName) doesn't work - worked fine in v8.7 though.
Upper(ColName) as Col1 works.


2. Implicit conversion from source type "dfloat" to result type "sfloat": Possible range/precision limitation ( * though this could be handled via project message handler).

3. Other implicit conversions warnings (varying from job to job) like ustring to Int[32], ustring[max 50] to ustring[max 1], WVARCHAR to VARCHAR etc.

4. For Auto-generated queries, directly providing Aliases/different column name (other than what's in table metadata) on Columns tab doesn't work.

5. On Join stage, if reference table doesn't have a proper alias for a hard-coded column, the job fails. [in old version, providing a name on Column tab directly used to work fine]

There were no special environment variables or message handlers in old server to handle such things and jobs were running smoothly.

As this is not possible to test each & every job for obvious reasons, we cannot even get a definite list of warning/errors.

Did anybody encounter these as part of their upgrade or could somebody please advise how could these be tackled at a bigger (project) scale.

Any help would be appreciated.


Regards
Mohit

_________________
Rgds,
MB
chulett

Premium Poster


since January 2006

Group memberships:
Premium Members, Inner Circle, Server to Parallel Transition Group

Joined: 12 Nov 2002
Posts: 41840
Location: Denver, CO
Points: 214660

Post Posted: Wed May 17, 2017 10:13 am Reply with quote    Back to top    

Let me make a few points, not from having experienced any of this myself but rather from reading every dang post made here since 2002 and seeing this over and over when people upgrade from the 8.x world.

By "this" I mean the issue of things that "worked" in 8 that no longer do. The strongly typed framework that PX relies upon was far too lenient when it came to enforcing the rules back in the day, allowing people to get away with things that fell outside of any Best Practice. Like implicit conversions. These "loopholes" get closed as you move up the version releases, not sure if it was better to incrementally upgrade and find new ones as you went or make the long leap that you did from one end of the train to the other. [shrug] Regardless, you got some fix'n to do.

Your issues with the DRS stage are odd, however, and sound like something to take up with support. I'd also be curious why you are using it. From what I recall it was added to have a bullet point on a list of capabilities to compete with other products, not that it was actually such a great idea. Unless, of course, you actually have jobs a need to switch relational databases "dynamically"? Assuming no need for that, IMHO, you would be far better off switching to a dedicated native connector.

Not sure what your "obvious reasons" would be for not testing everything, but when you hopped into your time machine and instantly jumped 10 or 11 years into the future of the product I would have thought would have been a given. Wink

_________________
-craig

The black queen chants the funeral march, the cracked brass bells will ring
To summon back the fire witch to the court of the crimson king
Rate this response:  
mdbatra



Group memberships:
Premium Members

Joined: 22 Oct 2008
Posts: 174
Location: City of London
Points: 1276

Post Posted: Thu May 18, 2017 3:48 am Reply with quote    Back to top    

Hi Craig

Thanks for sharing the reason around implicit conversion issues and other coding practices while upgrading to newer versions. Wasn't aware of this thing.

You're right in stating that there's no actual need of using DRS stages but just to set the context - we overtook the support sometime back from another vendor for our Client. They had developed jobs using DRS stages. And connector migration can convert them only to DRS connector stages. so we've got to live with that Smile Also client has SQL-Server database for which i don't see a native connector stage in v11.5 they have.


On testing, what i meant with 'we cannot test every thing for Obvious reason' is that it would hugely time-consuming activity. we have around 3000 jobs out of which around 1400 jobs use DRS stage. Testing, identifying issues and then fixing them for each & every job, to be honest, is not really an option for us/client .

we'll take this up with IBM support as you suggested, that might be helpful - hopefully.

Thanks once again for insights.

_________________
Rgds,
MB
Rate this response:  
sinhasaurabh014
Participant



Joined: 02 Apr 2008
Posts: 86
Location: Bangalore
Points: 1014

Post Posted: Thu May 18, 2017 4:26 am Reply with quote    Back to top    

Hey Mohit, Good to see you here..Doing Good?
I never used DRS stage, instead used ODBC connector to connect to various databases if a native stage is not found.
These errors that you have highlighted around not using aliases for column names like (upper(x)) as XY or 'hardcoded' as HRDCD, we have always faced this issue. So when you raise it here now, I wonder how come it worked earlier!!


I have done migrations from 8.5 to 8.7 to 9.1 to 11.5 and practically have not faced any issue so far.
Rate this response:  
chulett

Premium Poster


since January 2006

Group memberships:
Premium Members, Inner Circle, Server to Parallel Transition Group

Joined: 12 Nov 2002
Posts: 41840
Location: Denver, CO
Points: 214660

Post Posted: Thu May 18, 2017 12:29 pm Reply with quote    Back to top    

Didn't mean you literally had to test every single job (a bit of hyperbole on my part), just that a significant testing effort should be part of your migration. I would try to put together a list of job types, jobs with the same combination of stages, for example. Or jobs that use a certain stage in a certain way; say DRS as source or target for example. Then check representative samples of each type and be prepared to handle any issues that crop up in the group / set.

_________________
-craig

The black queen chants the funeral march, the cracked brass bells will ring
To summon back the fire witch to the court of the crimson king
Rate this response:  
chulett

Premium Poster


since January 2006

Group memberships:
Premium Members, Inner Circle, Server to Parallel Transition Group

Joined: 12 Nov 2002
Posts: 41840
Location: Denver, CO
Points: 214660

Post Posted: Thu May 18, 2017 12:35 pm Reply with quote    Back to top    

sinhasaurabh014 wrote:
I have done migrations from 8.5 to 8.7 to 9.1 to 11.5 and practically have not faced any issue so far.

Sounds like you followed a Best Practice path more closely. That would definitely help minimize the issues the OP is finding.

_________________
-craig

The black queen chants the funeral march, the cracked brass bells will ring
To summon back the fire witch to the court of the crimson king
Rate this response:  
mdbatra



Group memberships:
Premium Members

Joined: 22 Oct 2008
Posts: 174
Location: City of London
Points: 1276

Post Posted: Fri May 19, 2017 12:47 am Reply with quote    Back to top    

Hey Saurabh...i'm doing well. Hope same for you too. On the subject, perhaps KARMA does exist Very Happy

we have anyhow initiated a Problem ticket with IBM.

_________________
Rgds,
MB
Rate this response:  
Display posts from previous:       

Add To Favorites
View next topic
View previous topic
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum



Powered by phpBB © 2001, 2002 phpBB Group
Theme & Graphics by Daz :: Portal by Smartor
All times are GMT - 6 Hours