Px Jobs are importing in uncompiled state

Madhumitha_Raghunathan · Fri Sep 28, 2012 8:24 am

Hi All,

We are trying to import job designs and executable (dsx files) using the DSXImportService.sh

Before we run the script we are making sure that there are no locks or datastage jobs running through the Cleanup Resources on Director.

After all this the jobs are imported in not-compiled state most of the time (sometimes it is in compiled state) and we need to run the multi-job compile which takes a couple of hours considering the number of jobs we have.

Has anyone faced this situation? Is there anything we are missing?

Madhumitha_Raghunathan · Tue Dec 18, 2012 10:55 am

Hay anyone faced this issue? We are having this issue when deploying the jobs to PROD and end up wasting a couple of hours just for re-compiling the jobs. Would be grateful if anyone can help me out.

rameshrr3 · Post by **rameshrr3** » Wed Dec 19, 2012 12:44 pm

Its not wasted time . The best practice is to import jobs and recompile them in the new environment using multiple job compile. Otherwise you will run into design time issues.
Any differences between servers can screw up things.

You also save some time in exporting and importing ONLY job designs.

ArndW · Post by **ArndW** » Wed Dec 19, 2012 1:16 pm

I disagree about the best practice of importing uncompiled jobs.

Most production environments I have used and know don't have a c++ compiler installed.

rameshrr3 · Post by **rameshrr3** » Wed Dec 19, 2012 1:26 pm

Im really concerned if a PROD environment doesn't have a compiler installed , and the operations team need to make a quick fix on a transformer stage issue or a custom operator/build op issue. All environments I have worked have at least a single user compiler license for non Development environments.

SURA · Post by **SURA** » Wed Dec 19, 2012 2:01 pm

I too disagree in term of best practice. But its about the decision by the project team.

But i personally feel safe to export the dsx rather than exe to PROD.

Can't do any hot-fix and hard for day to day life until it got matured.

Post by **daignault** » Wed Dec 19, 2012 2:18 pm

We have the development teams break up the datastage jobs to a single element in the SCCS and the job must have been compiled in the Development system only, not the QA environment.

1 Datastage component (EE Job, Sequencer, Shell Script, or Parameter set) and it's object is migrated into the system at a time. We have an automated import process on the UNIX system that reads all dsx entries in a directory and then execute the .../ASBNode/bin/DSXImportService.sh to import each element into the system.

Hope this helps.

Ray D

Madhumitha_Raghunathan · Wed Dec 19, 2012 3:07 pm

Thanks Ray,

We are also doing the exact same thing. We are importing individual jobs and using a unix script to loop through and import each element. We also ensure we run a multi job compile before we export from dev. Still the jobs get imported in uncompiled state.

Madhumitha_Raghunathan · Wed Dec 19, 2012 3:17 pm

rameshrr3 wrote:Im really concerned if a PROD environment doesn't have a compiler installed , and the operations team need to make a quick fix on a transformer stage issue or a custom operator/build op issue. All environments I have worked have at least a single user compiler license for non Development environments.

Hi Ramesh,

Though we are able to compile the jobs on PROD, we dont do quick fixes in PROD. The general process that we have followed so far in all my previous projects is to use a lower env (not necessarily DEV.. It could be INT or QA) and get it tested by the QA team and then move to PROD. As per our policy the design in PROD should be Read-Only.
Also I am thinking that if I am exporting both the design and the executables it should not import in uncompiled state though we are taking all the precautions of checking for locks and ensuring that jobs are not running. unless we missed something because it has not happened in the prevous projects i have been in.

rameshrr3 · Post by **rameshrr3** » Wed Dec 19, 2012 4:09 pm

Murphy's law states that certain issues will only arise in PROD when business needs the data without delay and that the developer has taken a vacation . I assume this has been factored into your design and operations strategy.

One example is a data volume issue - you may need to process 40 million row lookup in datastage with custom lookup compilation options , and you simply don't have that much data in your fully isolated DEV /QA or SIT environment. Or that many organizations cannot affort a dedicated SIT environment ( System Integration test) -given time and cost limitations.

btw - DataStage Parallel extender is notorious for throwing errors that cannot be replicated on another environment.

Madhumitha_Raghunathan · Wed Dec 19, 2012 5:44 pm

Thanks Boss for that note! What I gave you is my current situation and how the environment and process was and is in the projects I have been in. I did not generalize and neither did i say that no one is fixing code in PROD. When your Murphy's law does come into action it is handled differently. Anyways the issue here is not best practice but why the jobs are imported in not compiled state.. If you have an idea on that please let me know.

ray.wurlod · Post by **ray.wurlod** » Wed Dec 19, 2012 7:11 pm

Murphy was an optimist.

Muphry's Law applies to typesetting.

rameshrr3 · Post by **rameshrr3** » Thu Dec 20, 2012 11:57 am

Can you post the syntax of the DSXImportService command you are using in the script ? Was this working earlier and now it has stopped importing executables ? I presume the export file does contain executables. Are only some executables missed out ?

Madhumitha_Raghunathan · Thu Dec 20, 2012 1:54 pm

Hi Ramesh,

Following is the synatx we are using:
DSXImportService.sh -ISFile /opt/IBM/IS/InformationServer/ASBNode/conf/dsimportconnection.conf -DSProject <PROJ_NAME> -DSXFile <File>.dsx -Overwrite -Verbose

Sometimes it imports all the jobs in compiled state but sometimes it doesnt. The export file contains the job designs and the executables. We check to ensure no jobs are running and there are not locks while exporting and importing. We run this for each job which individually in a loop.

Sometimes we get the following error when it fails: An unexpected error has occurred.DataStage Import of <file>.dsx Failed with Errors.
But most of the time we dont get this error even if the jobs dont compile.
When we manually recompile the jobs thru multiple job compile things go fine.

Thanks for your help on this.

rameshrr3 · Post by **rameshrr3** » Thu Dec 20, 2012 3:56 pm

I created a dsx file ( exported with 'executables' ) and could successfully import it using DSXImportService.sh after ftp to AIX server m/c

The 2 jobs were imported in 'compiled' status.
I did not find time to create a conf file , i put the command directly like this (I launched it from the dsx export file location - my O/S is AIX 6.1, Datastage version is 8.7 FP1)

Code: Select all

$ASBHOME/bin/DSXImportService.sh -ISHost qdsaasnshmim01.a1-core.q1.mycompany.net:9080 -ISUser dsadm -ISPassword $3cr3T -DSProject  FIELDCUST_Dev -DSXFile /cidba/home/dsadm/testfiles/importcheck.dsx -Overwrite -Verbose

Here is how my import log looks

Code: Select all

Server import initiated.
Attempting authentication.......
Authentication successful..
Processing Jobs.............
Processing Job Executables....
Completing Import
Import completed.
Design item imported successfully. Item Type: Job. Identifier: JITM0536ODBCinqPolUPDnewCASHVALUEDT.
Design item imported successfully. Item Type: Job. Identifier: JITM0536ODBCinqPolUPDnewFIELDS.
[color=darkred]Runtime item imported successfully. Item Type: Job. Identifier: JITM0536ODBCinqPolUPDnewCASHVALUEDT.
Runtime item imported successfully. Item Type: Job. Identifier: JITM0536ODBCinqPolUPDnewFIELDS.[/color]
IS Host = qdsaasnshmim01.a1-core.q1.mycompany.net
IS Port = 9080
IS User = dsadm
DataStage Project = FIELDCUST_Dev
Imported file = /cidba/home/dsadm/testfiles/importcheck.dsx
Total items = 4
Items imported = 4
Items not imported = 0
Total import time = 00:00:23

As the log shows executables were successfully imported