Kafka logs on Service tier

A forum for discussing DataStage<sup>®</sup> basics. If you're not sure where your question goes, start here.

Moderators: chulett, rschirm, roy

Post Reply
jackson.eyton
Premium Member
Premium Member
Posts: 145
Joined: Thu Oct 26, 2017 10:43 am

Kafka logs on Service tier

Post by jackson.eyton »

Hi everyone,
I received a disk usage alert from my service tier server this morning. Upon further review of the disk I am seeing that the IBM\InformationServer\shared-open-source\kafka\logs directory is 12GB. It does not appear that any of the logs are bloated per se, just a steady accumulation of log files. Does anyone know how to address this? The logs go back for a couple years, I assume there must be a log retention setting that can be configured?
-Me
JRodriguez
Premium Member
Premium Member
Posts: 425
Joined: Sat Nov 19, 2005 9:26 am
Location: New York City
Contact:

Post by JRodriguez »

Two ways come to my mind:

1) change the retention period for events on the topic
$InstallPath/shared-open-source/kafka$ update-config.sh --zookeeper serverName:port --entity-type topics --alter --add-config retention-ms=500 --entity-name InfosphereEvents

2) delete the topic. It will be automatically recreated
Julio Rodriguez
ETL Developer by choice

"Sure we have lots of reasons for being rude - But no excuses
jackson.eyton
Premium Member
Premium Member
Posts: 145
Joined: Thu Oct 26, 2017 10:43 am

Post by jackson.eyton »

Thanks for the reply! I have been reading a bit about altering the log retention settings. I see you're calling the update-config script and passing in new retention attributes from there. Is this different than editing the server.properties file itself in $InstallPath\shared-open-source\kafka\conf ?

I'm on a windows server unfortunately. I'm also not entirely sure I understand what a topic is in this context. :oops:
-Me
JRodriguez
Premium Member
Premium Member
Posts: 425
Joined: Sat Nov 19, 2005 9:26 am
Location: New York City
Contact:

Post by JRodriguez »

It should be ok to add the new parameter to the configuration manually. My preference is to use the commands provided to do all the maintenance tasks

A Topic is just a stream of records, In the Kafka or streaming world is called topic. For Information Server the default name of the topic is InfosphereEvents
Julio Rodriguez
ETL Developer by choice

"Sure we have lots of reasons for being rude - But no excuses
jackson.eyton
Premium Member
Premium Member
Posts: 145
Joined: Thu Oct 26, 2017 10:43 am

Post by jackson.eyton »

Official word thus far from IBM is to delete the log files. I did that back to January of this year for the kafka-server log files only. I am awaiting a response from them on proper configuration of server log file retention as it was indicated that those logs are not affected by the config file.
-Me
jackson.eyton
Premium Member
Premium Member
Posts: 145
Joined: Thu Oct 26, 2017 10:43 am

Post by jackson.eyton »

Official word back from IBM was that log retention configuration for kafka-server.log files cannot be done via any log files. These need to be manually cleaned or scripted to maintain a limited retention. I created a scheduled task that runs the following command:

Code: Select all

cmd /c powershell -command "Get-ChildItem 'D:\IBM\InformationServer\shared-open-source\kafka\logs' -Recurse -File | Where { -not $_.PSIsContainer -and $_.LastWriteTime -lt (Get-Date).AddDays(-60) -and $_.Name -like 'kafka-server.log*'} | Remove-Item -force" && exit
Replace 'D:\IBM\InformationServer\shared-open-source\kafka\logs' with your IBM InformationServer directory for your logs.
-Me
Post Reply