DSXchange: DataStage and IBM Websphere Data Integration Forum
View next topic
View previous topic
Add To Favorites
This topic has been marked "Resolved."
Author Message
dj
Participant



Joined: 24 Aug 2006
Posts: 78
Location: india
Points: 1101

Post Posted: Thu Aug 25, 2016 11:54 am Reply with quote    Back to top    

DataStage® Release: 11x
Job Type: Parallel
OS: Unix
Hi,

Few of the web service enabled quality jobs which were running fine earlier aborted.
And the compilation failed with below error
Code:


> ISD_Address_Standardization - Failed <<
Failed to invoke GenRuntime using phantom process helper.
Internal data error.
  File '/path/IBM/InformationServer/Server/Projects/MDMQS/RT_STATUS1/OVER.30':
  Computed blink of 0x90000 does not match expected blink of 0xE800!
  Detected within group starting at address 0xA1000!
  Cannot find a job number 0
Attempting to Cleanup after ABORT raised in job 0
Attempting to Cleanup after ABORT raised in job 0

DataStage Phantom Aborting with @ABORT.CODE = 3



We will try to reimport the dsx or make copy of the jobs and try to recompile.
1. Would like to understand why the job status file got corrupted?
2. How to check administrator, there are no RT_LOGS number
3. The webservice jobs are using Isd stages. After server restart do these jobs try to run automatically, I could see multiple invocation id in abort status until any instance is in running state.

Thanks
qt_ky



Group memberships:
Premium Members

Joined: 03 Aug 2011
Posts: 2775
Location: USA
Points: 20907

Post Posted: Thu Aug 25, 2016 1:56 pm Reply with quote    Back to top    

I would guess your log file logged a HUGE number of warnings or possibly multi-instance aborts, probably exceeding 2 GB of job log details. I've run into that exact case and error before. Enable auto-purge settings.

RT_STATUS1 tells you the job ID is 1. You could try command COUNT RT_LOG1 to see how many detailed job log entries you've got.

To avoid crashed job status with ISD related jobs when stopping or bouncing the server, make sure to stop each ISD application first. This can be done from the command line and scripted. Do the opposite after startup.

_________________
Choose a job you love, and you will never have to work a day in your life. - Confucius
Rate this response:  
Not yet rated
ray.wurlod

Premium Poster
Participant

Group memberships:
Premium Members, Inner Circle, Australia Usergroup, Server to Parallel Transition Group

Joined: 23 Oct 2002
Posts: 54225
Location: Sydney, Australia
Points: 294087

Post Posted: Thu Aug 25, 2016 4:04 pm Reply with quote    Back to top    

Do clear the log. It's just good practice to do so. However, the hashed file that has become corrupted is RT_STATUS1, not RT_LOG1. If you have administrative tools enabled in Director client, y ...

_________________
RXP Services Ltd
Melbourne | Canberra | Sydney | Hong Kong | Hobart | Brisbane
currently hiring: Canberra, Sydney and Melbourne
Rate this response:  
Not yet rated
dj
Participant



Joined: 24 Aug 2006
Posts: 78
Location: india
Points: 1101

Post Posted: Fri Aug 26, 2016 8:25 am Reply with quote    Back to top    

thanks for the replies.

After trying to view the RT Log files which caused the issue, icould only find the below error in all the jobs.

Code:
Fatal Error: waitForWriteSignal(): Premature EOF on node server Bad file descriptor

1) Is this error is related to server down? we are just trying to identify what caused the huge logs.

2) What happens actually when the server restarts?

Before Server restart- To stop the jobs dsjob commands or do we need to undeploy them

After restart- Do we need re-deploy or just start commands would suffice?

Thanks
Rate this response:  
Not yet rated
qt_ky



Group memberships:
Premium Members

Joined: 03 Aug 2011
Posts: 2775
Location: USA
Points: 20907

Post Posted: Fri Aug 26, 2016 11:18 am Reply with quote    Back to top    

As Ray pointed out, it wasn't your job log that got filled up, rather its status file overflowed. I'm not sure what would cause that.

If you were to reach out and stop the DataStage job, ISD would notice and immediately spawn a new instance.

To script stops and starts, you need to undeploy and deploy. Here is a UNIX example of the syntax you can use within shell scripts.

ISDAdmin.sh -authfile <AUTH_FILE> -action undeploy -application <ISD_application_name>

ISDAdmin.sh -authfile <AUTH_FILE> -action deploy -project <ISD_project_name> -application <ISD_application_name>

_________________
Choose a job you love, and you will never have to work a day in your life. - Confucius
Rate this response:  
Not yet rated
dj
Participant



Joined: 24 Aug 2006
Posts: 78
Location: india
Points: 1101

Post Posted: Mon Aug 29, 2016 7:49 am Reply with quote    Back to top    

Thanks for the details.

In addition to the status file overflowed, for the ISD jobs we could also see the /tmp folder getting full.

What are these files? Are there any harm in removing these files and why do they go to tmp?

Are there any settings we could change the directory

ad1.proj.ISD_jobname.input
ad1.proj.ISD_jobname.output

Thanks
Rate this response:  
Not yet rated
qt_ky



Group memberships:
Premium Members

Joined: 03 Aug 2011
Posts: 2775
Location: USA
Points: 20907

Post Posted: Mon Aug 29, 2016 11:50 am Reply with quote    Back to top    

The TMPDIR environment variable may be set to any path you like. You can change it using DataStage Administrator, per project. After you set a new one and run the ISD jobs, or redeploy, then remove the old tmp files.

_________________
Choose a job you love, and you will never have to work a day in your life. - Confucius
Rate this response:  
Not yet rated
dj
Participant



Joined: 24 Aug 2006
Posts: 78
Location: india
Points: 1101

Post Posted: Tue Aug 30, 2016 8:28 am Reply with quote    Back to top    

we are working with PMR team to figure out why ther server went down.

After clearing up the tmp and logs files, imported the dsx again and the jobs got compiled.

Thanks.
Rate this response:  
Not yet rated
Display posts from previous:       

Add To Favorites
View next topic
View previous topic
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum



Powered by phpBB © 2001, 2002 phpBB Group
Theme & Graphics by Daz :: Portal by Smartor
All times are GMT - 6 Hours