DSXchange: DataStage and IBM Websphere Data Integration Forum
View next topic
View previous topic
Add To Favorites
This topic is not resolved, but there is a WORKAROUND.
Author Message
jackson.eyton



Group memberships:
Premium Members

Joined: 26 Oct 2017
Posts: 109

Points: 1819

Post Posted: Thu Mar 29, 2018 8:19 am Reply with quote    Back to top    

DataStage® Release: 11x
Job Type: Server
OS: Windows
Hi everyone,
I received a disk usage alert from my service tier server this morning. Upon further review of the disk I am seeing that the IBM\InformationServer\shared-open-source\kafka\logs directory is 12GB. It does not appear that any of the logs are bloated per se, just a steady accumulation of log files. Does anyone know how to address this? The logs go back for a couple years, I assume there must be a log retention setting that can be configured?

_________________
-Me
JRodriguez



Group memberships:
Premium Members

Joined: 19 Nov 2005
Posts: 413
Location: New York City
Points: 4531

Post Posted: Thu Mar 29, 2018 9:13 am Reply with quote    Back to top    

Two ways come to my mind:

1) change the retention period for events on the topic
$InstallPath/shared-open-source/kafka$ update-config.sh --zookeeper serverName:port --entity-type topics --alter --add-config retention-ms=500 --entity-name InfosphereEvents

2) delete the topic. It will be automatically recreated

_________________
Julio Rodriguez
ETL Developer by choice

"Sure we have lots of reasons for being rude - But no excuses
Rate this response:  
jackson.eyton



Group memberships:
Premium Members

Joined: 26 Oct 2017
Posts: 109

Points: 1819

Post Posted: Thu Mar 29, 2018 9:23 am Reply with quote    Back to top    

Thanks for the reply! I have been reading a bit about altering the log retention settings. I see you're calling the update-config script and passing in new retention attributes from there. Is this different than editing the server.properties file itself in $InstallPath\shared-open-source\kafka\conf ?

I'm on a windows server unfortunately. I'm also not entirely sure I understand what a topic is in this context. Embarassed

_________________
-Me
Rate this response:  
Not yet rated
JRodriguez



Group memberships:
Premium Members

Joined: 19 Nov 2005
Posts: 413
Location: New York City
Points: 4531

Post Posted: Thu Mar 29, 2018 10:17 am Reply with quote    Back to top    

It should be ok to add the new parameter to the configuration manually. My preference is to use the commands provided to do all the maintenance tasks

A Topic is just a stream of records, In the Kafka or streaming world is called topic. For Information Server the default name of the topic is InfosphereEvents

_________________
Julio Rodriguez
ETL Developer by choice

"Sure we have lots of reasons for being rude - But no excuses
Rate this response:  
jackson.eyton



Group memberships:
Premium Members

Joined: 26 Oct 2017
Posts: 109

Points: 1819

Post Posted: Mon Apr 02, 2018 4:14 pm Reply with quote    Back to top    

Official word thus far from IBM is to delete the log files. I did that back to January of this year for the kafka-server log files only. I am awaiting a response from them on proper configuration of server log file retention as it was indicated that those logs are not affected by the config file.

_________________
-Me
Rate this response:  
Not yet rated
jackson.eyton



Group memberships:
Premium Members

Joined: 26 Oct 2017
Posts: 109

Points: 1819

Post Posted: Thu Apr 05, 2018 12:16 pm Reply with quote    Back to top    

Official word back from IBM was that log retention configuration for kafka-server.log files cannot be done via any log files. These need to be manually cleaned or scripted to maintain a limited retention. I created a scheduled task that runs the following command:
Code:
cmd /c powershell -command "Get-ChildItem 'D:\IBM\InformationServer\shared-open-source\kafka\logs' -Recurse -File | Where { -not $_.PSIsContainer -and $_.LastWriteTime -lt (Get-Date).AddDays(-60) -and $_.Name -like 'kafka-server.log*'} | Remove-Item -force" && exit


Replace 'D:\IBM\InformationServer\shared-open-source\kafka\logs' with your IBM InformationServer directory for your logs.

_________________
-Me
Rate this response:  
Not yet rated
Display posts from previous:       

Add To Favorites
View next topic
View previous topic
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum



Powered by phpBB © 2001, 2002 phpBB Group
Theme & Graphics by Daz :: Portal by Smartor
All times are GMT - 6 Hours