DSXchange: DataStage and IBM Websphere Data Integration Forum
View next topic
View previous topic
Add To Favorites
Author Message
joycerecacho
Participant



Joined: 26 Aug 2008
Posts: 290

Points: 5351

Post Posted: Mon Jan 28, 2019 5:19 am Reply with quote    Back to top    

DataStage® Release: 11x
Job Type: Parallel
OS: Unix
Hi everybody.

DataStage here runs in a cluster and the '.apt' file has a dynamic configuration, which refers to 13 nodes with 2 instances each - except the 'conductor node'.

The thing is: when we generate a DataSet that already exists (datastage is suppose to overwrite it), we notice that the older '.ds' is renamed to: "<datasetName.ds>.being_deleted" and the data files stay in HDFS forever.
Since the 'descriptor' has changed, it is not possible to remove it through 'orchadmin' command.

Why do these data files are not efetively removed after '.ds' is renamed?
Is this an issue of Hadoop?

ps: '.ds' files are located at the linux of the 'conductor node' only, and the data files are located at HDFS, distributed between the nodes.

The environment variable $APT_EXECUTION_MODE = Parallel.

I´d really appreciate any tip/help.

Thanks in advance.

Best regards.

_________________
Joyce A. Recacho
Săo Paulo/SP
Brazil
Display posts from previous:       

Add To Favorites
View next topic
View previous topic
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum



Powered by phpBB © 2001, 2002 phpBB Group
Theme & Graphics by Daz :: Portal by Smartor
All times are GMT - 6 Hours