DataStage upgrade from v8.7 to v11.5

Post questions here relative to DataStage Enterprise/PX Edition for such areas as Parallel job design, Parallel datasets, BuildOps, Wrappers, etc.

Moderators: chulett, rschirm, roy

Post Reply
chulett
Charter Member
Charter Member
Posts: 43085
Joined: Tue Nov 12, 2002 4:34 pm
Location: Denver, CO

Post by chulett »

Let me make a few points, not from having experienced any of this myself but rather from reading every dang post made here since 2002 and seeing this over and over when people upgrade from the 8.x world.

By "this" I mean the issue of things that "worked" in 8 that no longer do. The strongly typed framework that PX relies upon was far too lenient when it came to enforcing the rules back in the day, allowing people to get away with things that fell outside of any Best Practice. Like implicit conversions. These "loopholes" get closed as you move up the version releases, not sure if it was better to incrementally upgrade and find new ones as you went or make the long leap that you did from one end of the train to the other. [shrug] Regardless, you got some fix'n to do.

Your issues with the DRS stage are odd, however, and sound like something to take up with support. I'd also be curious why you are using it. From what I recall it was added to have a bullet point on a list of capabilities to compete with other products, not that it was actually such a great idea. Unless, of course, you actually have jobs a need to switch relational databases "dynamically"? Assuming no need for that, IMHO, you would be far better off switching to a dedicated native connector.

Not sure what your "obvious reasons" would be for not testing everything, but when you hopped into your time machine and instantly jumped 10 or 11 years into the future of the product I would have thought would have been a given. :wink:
-craig

"You can never have too many knives" -- Logan Nine Fingers
mdbatra
Premium Member
Premium Member
Posts: 175
Joined: Wed Oct 22, 2008 10:01 am
Location: City of London

Post by mdbatra »

Hi Craig

Thanks for sharing the reason around implicit conversion issues and other coding practices while upgrading to newer versions. Wasn't aware of this thing.

You're right in stating that there's no actual need of using DRS stages but just to set the context - we overtook the support sometime back from another vendor for our Client. They had developed jobs using DRS stages. And connector migration can convert them only to DRS connector stages. so we've got to live with that :) Also client has SQL-Server database for which i don't see a native connector stage in v11.5 they have.


On testing, what i meant with 'we cannot test every thing for Obvious reason' is that it would hugely time-consuming activity. we have around 3000 jobs out of which around 1400 jobs use DRS stage. Testing, identifying issues and then fixing them for each & every job, to be honest, is not really an option for us/client .

we'll take this up with IBM support as you suggested, that might be helpful - hopefully.

Thanks once again for insights.
Rgds,
MB
sinhasaurabh014
Participant
Posts: 86
Joined: Wed Apr 02, 2008 2:32 am
Location: Bangalore

always faced this

Post by sinhasaurabh014 »

Hey Mohit, Good to see you here..Doing Good?
I never used DRS stage, instead used ODBC connector to connect to various databases if a native stage is not found.
These errors that you have highlighted around not using aliases for column names like (upper(x)) as XY or 'hardcoded' as HRDCD, we have always faced this issue. So when you raise it here now, I wonder how come it worked earlier!!


I have done migrations from 8.5 to 8.7 to 9.1 to 11.5 and practically have not faced any issue so far.
chulett
Charter Member
Charter Member
Posts: 43085
Joined: Tue Nov 12, 2002 4:34 pm
Location: Denver, CO

Post by chulett »

Didn't mean you literally had to test every single job (a bit of hyperbole on my part), just that a significant testing effort should be part of your migration. I would try to put together a list of job types, jobs with the same combination of stages, for example. Or jobs that use a certain stage in a certain way; say DRS as source or target for example. Then check representative samples of each type and be prepared to handle any issues that crop up in the group / set.
-craig

"You can never have too many knives" -- Logan Nine Fingers
chulett
Charter Member
Charter Member
Posts: 43085
Joined: Tue Nov 12, 2002 4:34 pm
Location: Denver, CO

Re: always faced this

Post by chulett »

sinhasaurabh014 wrote:I have done migrations from 8.5 to 8.7 to 9.1 to 11.5 and practically have not faced any issue so far.
Sounds like you followed a Best Practice path more closely. That would definitely help minimize the issues the OP is finding.
-craig

"You can never have too many knives" -- Logan Nine Fingers
mdbatra
Premium Member
Premium Member
Posts: 175
Joined: Wed Oct 22, 2008 10:01 am
Location: City of London

Re: always faced this

Post by mdbatra »

Hey Saurabh...i'm doing well. Hope same for you too. On the subject, perhaps KARMA does exist :D

we have anyhow initiated a Problem ticket with IBM.
Rgds,
MB
mdbatra
Premium Member
Premium Member
Posts: 175
Joined: Wed Oct 22, 2008 10:01 am
Location: City of London

Post by mdbatra »

and IBM seems to acknowledge this (the alias issue especially) officially:

https://www.ibm.com/support/knowledgece ... RS063.html


Excerpt from above article:

"The DRS Connector stage configured for the Oracle database type and the Dynamic RDBMS stage allow the use of SELECT statements where items in the list do not match column names on the output link"


so, if your underlying DB is oracle, you won't face any issue with upgrade from v8.7 to v11.5 around aliases and also if you were not using DRS stage at all in previous version.
Rgds,
MB
Post Reply