Not enough storage to import large XML job exports

A forum for discussing DataStage<sup>®</sup> basics. If you're not sure where your question goes, start here.

Moderators: chulett, rschirm, roy

Post Reply
sec105105
Participant
Posts: 44
Joined: Fri Mar 20, 2009 7:21 am
Location: Ottawa

Not enough storage to import large XML job exports

Post by sec105105 »

In the thread "Manully alter DS DSX or XML job exports", they discussed moving XML files into a project, but there was no mention of XML file size.

I have a project where the full project exported is > 350mb. Exporting to XML is no problem, but I get errors when I import elsewhere:
Run-time error:
'not enough storage to complete this operation'

My local DS admin team was not able to fix, and they suggested I export my project in segments, so each XML file is < 100mb. That seems to work - import is working.

However, what 'storage' is low, such that the import fails?
Although 100mb works, what is the magic number?
Are there any notes that describe this problem?

Any other suggestions?

[Note: title corrected to match topic - Andy]
chulett
Charter Member
Charter Member
Posts: 43085
Joined: Tue Nov 12, 2002 4:34 pm
Location: Denver, CO

Post by chulett »

My first suggestion? Don't use XML files. They add an extra layer to the import / export process as it must be converted to a dsx file before it can actually be imported... or converted to one after export which takes extra time and memory. Skip all of that conversion who-ha and just stick with dsx files. IMHO.
-craig

"You can never have too many knives" -- Logan Nine Fingers
eostic
Premium Member
Premium Member
Posts: 3838
Joined: Mon Oct 17, 2005 9:34 am

Post by eostic »

+ 1
Ernie Ostic

blogit!
<a href="https://dsrealtime.wordpress.com/2015/0 ... ere/">Open IGC is Here!</a>
sec105105
Participant
Posts: 44
Joined: Fri Mar 20, 2009 7:21 am
Location: Ottawa

Post by sec105105 »

Certainly XMLs seem to be more of a pain; however in our case we are finding it easier to manipulate the XML files (have to do several global changes, using a Visual basic script, then a Linux script). Embedded in a .DSX file are dozens of XML statements which can be several KB in length. These cause issues with our sed and awk commands.

So, it's the least path of resistance to use the XML. Apart from contacting IBM directly, has anyone experienced such size limitations, and has any workarounds or solutions, except for splitting up the exports.
chulett
Charter Member
Charter Member
Posts: 43085
Joined: Tue Nov 12, 2002 4:34 pm
Location: Denver, CO

Post by chulett »

Contact IBM. I'm assuming "storage" is memory and wondering if Java is involved, if so perhaps you can increase the size of the JVM being used.
-craig

"You can never have too many knives" -- Logan Nine Fingers
Post Reply