Px Jobs are importing in uncompiled state
Moderators: chulett, rschirm, roy
-
- Premium Member
- Posts: 59
- Joined: Fri Apr 22, 2011 8:02 am
Px Jobs are importing in uncompiled state
Hi All,
We are trying to import job designs and executable (dsx files) using the DSXImportService.sh
Before we run the script we are making sure that there are no locks or datastage jobs running through the Cleanup Resources on Director.
After all this the jobs are imported in not-compiled state most of the time (sometimes it is in compiled state) and we need to run the multi-job compile which takes a couple of hours considering the number of jobs we have.
Has anyone faced this situation? Is there anything we are missing?
We are trying to import job designs and executable (dsx files) using the DSXImportService.sh
Before we run the script we are making sure that there are no locks or datastage jobs running through the Cleanup Resources on Director.
After all this the jobs are imported in not-compiled state most of the time (sometimes it is in compiled state) and we need to run the multi-job compile which takes a couple of hours considering the number of jobs we have.
Has anyone faced this situation? Is there anything we are missing?
Thanks,
Madhumitha
Madhumitha
-
- Premium Member
- Posts: 59
- Joined: Fri Apr 22, 2011 8:02 am
I disagree about the best practice of importing uncompiled jobs.
Most production environments I have used and know don't have a c++ compiler installed.
Most production environments I have used and know don't have a c++ compiler installed.
<a href=http://www.worldcommunitygrid.org/team/ ... TZ9H4CGVP1 target="WCGWin">
</a>
</a>
Im really concerned if a PROD environment doesn't have a compiler installed , and the operations team need to make a quick fix on a transformer stage issue or a custom operator/build op issue. All environments I have worked have at least a single user compiler license for non Development environments.
I too disagree in term of best practice. But its about the decision by the project team.
But i personally feel safe to export the dsx rather than exe to PROD.
Can't do any hot-fix and hard for day to day life until it got matured.
But i personally feel safe to export the dsx rather than exe to PROD.
Can't do any hot-fix and hard for day to day life until it got matured.
Thanks
Ram
----------------------------------
Revealing your ignorance is fine, because you get a chance to learn.
Ram
----------------------------------
Revealing your ignorance is fine, because you get a chance to learn.
We have the development teams break up the datastage jobs to a single element in the SCCS and the job must have been compiled in the Development system only, not the QA environment.
1 Datastage component (EE Job, Sequencer, Shell Script, or Parameter set) and it's object is migrated into the system at a time. We have an automated import process on the UNIX system that reads all dsx entries in a directory and then execute the .../ASBNode/bin/DSXImportService.sh to import each element into the system.
Hope this helps.
Ray D
1 Datastage component (EE Job, Sequencer, Shell Script, or Parameter set) and it's object is migrated into the system at a time. We have an automated import process on the UNIX system that reads all dsx entries in a directory and then execute the .../ASBNode/bin/DSXImportService.sh to import each element into the system.
Hope this helps.
Ray D
-
- Premium Member
- Posts: 59
- Joined: Fri Apr 22, 2011 8:02 am
-
- Premium Member
- Posts: 59
- Joined: Fri Apr 22, 2011 8:02 am
Hi Ramesh,rameshrr3 wrote:Im really concerned if a PROD environment doesn't have a compiler installed , and the operations team need to make a quick fix on a transformer stage issue or a custom operator/build op issue. All environments I have worked have at least a single user compiler license for non Development environments.
Though we are able to compile the jobs on PROD, we dont do quick fixes in PROD. The general process that we have followed so far in all my previous projects is to use a lower env (not necessarily DEV.. It could be INT or QA) and get it tested by the QA team and then move to PROD. As per our policy the design in PROD should be Read-Only.
Also I am thinking that if I am exporting both the design and the executables it should not import in uncompiled state though we are taking all the precautions of checking for locks and ensuring that jobs are not running. unless we missed something because it has not happened in the prevous projects i have been in.
Thanks,
Madhumitha
Madhumitha
Murphy's law states that certain issues will only arise in PROD when business needs the data without delay and that the developer has taken a vacation . I assume this has been factored into your design and operations strategy.
One example is a data volume issue - you may need to process 40 million row lookup in datastage with custom lookup compilation options , and you simply don't have that much data in your fully isolated DEV /QA or SIT environment. Or that many organizations cannot affort a dedicated SIT environment ( System Integration test) -given time and cost limitations.
btw - DataStage Parallel extender is notorious for throwing errors that cannot be replicated on another environment.
One example is a data volume issue - you may need to process 40 million row lookup in datastage with custom lookup compilation options , and you simply don't have that much data in your fully isolated DEV /QA or SIT environment. Or that many organizations cannot affort a dedicated SIT environment ( System Integration test) -given time and cost limitations.
btw - DataStage Parallel extender is notorious for throwing errors that cannot be replicated on another environment.
-
- Premium Member
- Posts: 59
- Joined: Fri Apr 22, 2011 8:02 am
Thanks Boss for that note! What I gave you is my current situation and how the environment and process was and is in the projects I have been in. I did not generalize and neither did i say that no one is fixing code in PROD. When your Murphy's law does come into action it is handled differently. Anyways the issue here is not best practice but why the jobs are imported in not compiled state.. If you have an idea on that please let me know.
Thanks,
Madhumitha
Madhumitha
-
- Participant
- Posts: 54607
- Joined: Wed Oct 23, 2002 10:52 pm
- Location: Sydney, Australia
- Contact:
-
- Premium Member
- Posts: 59
- Joined: Fri Apr 22, 2011 8:02 am
Hi Ramesh,
Following is the synatx we are using:
DSXImportService.sh -ISFile /opt/IBM/IS/InformationServer/ASBNode/conf/dsimportconnection.conf -DSProject <PROJ_NAME> -DSXFile <File>.dsx -Overwrite -Verbose
Sometimes it imports all the jobs in compiled state but sometimes it doesnt. The export file contains the job designs and the executables. We check to ensure no jobs are running and there are not locks while exporting and importing. We run this for each job which individually in a loop.
Sometimes we get the following error when it fails: An unexpected error has occurred.DataStage Import of <file>.dsx Failed with Errors.
But most of the time we dont get this error even if the jobs dont compile.
When we manually recompile the jobs thru multiple job compile things go fine.
Thanks for your help on this.
Following is the synatx we are using:
DSXImportService.sh -ISFile /opt/IBM/IS/InformationServer/ASBNode/conf/dsimportconnection.conf -DSProject <PROJ_NAME> -DSXFile <File>.dsx -Overwrite -Verbose
Sometimes it imports all the jobs in compiled state but sometimes it doesnt. The export file contains the job designs and the executables. We check to ensure no jobs are running and there are not locks while exporting and importing. We run this for each job which individually in a loop.
Sometimes we get the following error when it fails: An unexpected error has occurred.DataStage Import of <file>.dsx Failed with Errors.
But most of the time we dont get this error even if the jobs dont compile.
When we manually recompile the jobs thru multiple job compile things go fine.
Thanks for your help on this.
Thanks,
Madhumitha
Madhumitha
I created a dsx file ( exported with 'executables' ) and could successfully import it using DSXImportService.sh after ftp to AIX server m/c
The 2 jobs were imported in 'compiled' status.
I did not find time to create a conf file , i put the command directly like this (I launched it from the dsx export file location - my O/S is AIX 6.1, Datastage version is 8.7 FP1)
Here is how my import log looks
As the log shows executables were successfully imported
The 2 jobs were imported in 'compiled' status.
I did not find time to create a conf file , i put the command directly like this (I launched it from the dsx export file location - my O/S is AIX 6.1, Datastage version is 8.7 FP1)
Code: Select all
$ASBHOME/bin/DSXImportService.sh -ISHost qdsaasnshmim01.a1-core.q1.mycompany.net:9080 -ISUser dsadm -ISPassword $3cr3T -DSProject FIELDCUST_Dev -DSXFile /cidba/home/dsadm/testfiles/importcheck.dsx -Overwrite -Verbose
Here is how my import log looks
Code: Select all
Server import initiated.
Attempting authentication.......
Authentication successful..
Processing Jobs.............
Processing Job Executables....
Completing Import
Import completed.
Design item imported successfully. Item Type: Job. Identifier: JITM0536ODBCinqPolUPDnewCASHVALUEDT.
Design item imported successfully. Item Type: Job. Identifier: JITM0536ODBCinqPolUPDnewFIELDS.
[color=darkred]Runtime item imported successfully. Item Type: Job. Identifier: JITM0536ODBCinqPolUPDnewCASHVALUEDT.
Runtime item imported successfully. Item Type: Job. Identifier: JITM0536ODBCinqPolUPDnewFIELDS.[/color]
IS Host = qdsaasnshmim01.a1-core.q1.mycompany.net
IS Port = 9080
IS User = dsadm
DataStage Project = FIELDCUST_Dev
Imported file = /cidba/home/dsadm/testfiles/importcheck.dsx
Total items = 4
Items imported = 4
Items not imported = 0
Total import time = 00:00:23
Last edited by rameshrr3 on Thu Dec 20, 2012 4:03 pm, edited 1 time in total.