DSXchange: DataStage and IBM Websphere Data Integration Forum
View next topic
View previous topic
Add To Favorites
This topic is not resolved, but there is a WORKAROUND.
Author Message
ArndW

Premium Poster
Participant

Group memberships:
Premium Members, Inner Circle, Australia Usergroup

Joined: 16 Nov 2004
Posts: 16301
Location: Germany
Points: 92436

Post Posted: Thu Sep 29, 2016 11:25 am Reply with quote    Back to top    

DataStage® Release: 11x
Job Type: Parallel
OS: Unix
At this site we collect operational metadata for data lineage. There are several jobs for which collection has been turned on (in the Information Governance Catalog) while it is turned off for all other jobs.

The normal process is for the job to run, to dump the operational metadata into the $DSHOME/../DSOMD/xml directory and for the OMDMonApp to process the files and then delete them.

Unfortunately, the rate at which the files get consumed is much less than the rate at which they get created - some of these jobs run every 15 minutes with 60 instances...

I haven't found any references to any settings in the configuration file (omdmonapp.config) which could increase the speed at which it processes items, but currently it only averages about 1-3 files per minute which seems awfully slow.

I should add that our repository is in Oracle; I recall that there were some known performance issues in the past with Oracle as the xmeta repository.

Does anyone know if it is possible to speed up the ingestion process somehow?

_________________

Image
qt_ky



Group memberships:
Premium Members

Joined: 03 Aug 2011
Posts: 2739
Location: USA
Points: 20502

Post Posted: Thu Sep 29, 2016 12:12 pm Reply with quote    Back to top    

I also recall the Oracle xmeta performance discussions; not sure if those are related.

The omdmonapp.config file has perhaps one relevant setting. In mine it is commented out by default:
#Default is currently 5 (seconds)
#OMDPollPeriod=5

There is a DSOMDMonApp.sh script under bin with a java call to start OMDMonApp. You could try increasing the memory available to see if it helps. I see a default argument of "-Xmx384m" in my script.

If tinkering doesn't help, then see what Support has to say.

_________________
Choose a job you love, and you will never have to work a day in your life. - Confucius
Rate this response:  
Not yet rated
eostic

Premium Poster



Group memberships:
Premium Members

Joined: 17 Oct 2005
Posts: 3749

Points: 30012

Post Posted: Thu Sep 29, 2016 1:29 pm Reply with quote    Back to top    

Hi Arnd You might want to consider just turning off the omdmon application and just using istool [ istool workbench importOMD ] to import the xml's. Not sure if it is the polling process caus ...

_________________
Ernie Ostic

blogit!
Open IGC is Here!
Rate this response:  
Not yet rated
ArndW

Premium Poster
Participant

Group memberships:
Premium Members, Inner Circle, Australia Usergroup

Joined: 16 Nov 2004
Posts: 16301
Location: Germany
Points: 92436

Post Posted: Fri Sep 30, 2016 3:02 am Reply with quote    Back to top    

Thanks for suggestions; I'll play with increasing memory for the OMDMonApp to see if that might make a difference, but I haven't seen that process consume significant system resources so I'm not sure that it is the bottleneck.
I'll also try manual import to see how long that takes per job so I can have a comparison between the two methods.
Thanks.

_________________

Image
Rate this response:  
Not yet rated
Display posts from previous:       

Add To Favorites
View next topic
View previous topic
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum



Powered by phpBB © 2001, 2002 phpBB Group
Theme & Graphics by Daz :: Portal by Smartor
All times are GMT - 6 Hours