From Server Jobs to PX jobs

Post questions here relative to DataStage Enterprise/PX Edition for such areas as Parallel job design, Parallel datasets, BuildOps, Wrappers, etc.

Moderators: chulett, rschirm, roy

Post Reply
ariear
Participant
Posts: 237
Joined: Thu Dec 26, 2002 2:19 pm

From Server Jobs to PX jobs

Post by ariear »

Hi all,

I'm facing my first PX challange armed with my DataStage server knowledge (quite a lot). Am I to design/think as i'm implementing a server job or should i change my way of thinking ? like I usually full table scanning My RDBMS sources manipulate entities using local hash files, adict for multi-instance jobs, allways sorting before aggregating and so on..
I would like a some tips from you guys who already knows the good/weak spots and differences

Thanks,

ArieAR
kcbland
Participant
Posts: 5208
Joined: Wed Jan 15, 2003 8:56 am
Location: Lutz, FL
Contact:

Post by kcbland »

To quote Yoda:

Code: Select all

You must unlearn, what you have learned.
Server to PX is not like a Corvette to a Ferrari. It's like an Humvee to a Ferrari. A Humvee is highly versatile, goes just about anywhere, doesn't break down. Just about anyone can drive one.

A Ferrari is a high-performance vehicle. You don't want to take it off smooth pavement. You have to tune the engine every 7500 miles at an approved dealer. You have to have specialized training just to be able to drive one (don't believe me? I saw pictures of some idiot millionaire who took his brand new lamborghini for his first drive on a race track and crashed/totalled it on the first turn).

PX is the future of the product. You must approach using the product with a fresh perspective. Keep your design goals firmly entrenched in your mind: audit trails, restartable, modularity, etc. Resist super-intergalactic death-star-sized job designs, as that will be the tendency. Exploit the power, but stay grounded in simple designs.
Kenneth Bland

Rank: Sempai
Belt: First degree black
Fight name: Captain Hook
Signature knockout: right upper cut followed by left hook
Signature submission: Crucifix combined with leg triangle
Teej
Participant
Posts: 677
Joined: Fri Aug 08, 2003 9:26 am
Location: USA

Post by Teej »

Honestly, I can not add much more to what Ken said.

In PX -- the transform stage is to be shunned as much as you can, while in Server, it is beloved as the do-all-be-all.

In Server, it is easy to keep track of how many processes are going on. On PX, well... even after 18 months of this product, I am still understanding how tricky timing of PX stage execution can be.

There are tricks you can pull from Server experience that can apply to PX, but there are always cavets.

Instead of copying the behavior of your Server job to PX jobs, you have to back up to the level of the data movement -- what goes where and when, and WHY.

Eventually, you will be tuning your scratch space usage, when to sort, when to land data, where to land it, which data to land, and...

And most of all, Upgrade to 7.0.1 ASAP. Server is very mature, even back in 5.2, which is why you still see a lot of folks using it. But PX is practically at version 2.0.1 now. Far better than 1.0, as I'm seeing right now, but even at 7.0.1, there are some rough edges. Ascential Support won't like me for the next few weeks (submitted 7 issues to them today, I believe).

-T.J.
Developer of DataStage Parallel Engine (Orchestrate).
ariear
Participant
Posts: 237
Joined: Thu Dec 26, 2002 2:19 pm

Post by ariear »

Thanks Guys !
Post Reply