XML File size limit

[pillip]

Hi,

Do we have any limit on the size of the XML file which can be written from Datastage.


Thanks,
Prasanna

[dsdevper]

I tried to get that answer from IBM support.But they didn't say the exact size but, They Said 100 MB is preferable size.

I tried reading 250 Mb file, i was successful, But higher than that job was terminating abnormally.

Here is the Message i got from IBM

"" Found that the XML Input stage requires, on average, 5-7 times the size of the file in memory to process the document. The memory usage is based on
the actual structure and data within the document, and on the XPATH that defined in the job.
There is a risk of random job failures with large files, even when the memory usage is optimally configured.
The recommended solution is the input XML files should be kept as small as possible. The guideline is 100 MB or less for each file. ""

[pillip]

I am using XML output stage to create XMLs. For 100 MB what would be the approximate count of the records which can be put in it.


Thanks

[eostic]

There are too many variables. Size of tags, size of data...no real way to easily predict it. I've done as much as 500M in closed tests --- this is why the conservative recommendation that you received is 100M.

On targeting though, you can cheat. As you have mentioned in prior threads, just create single nodes and send those "out" of xmlOutput and into a sequential stage.....build the ultimate document in there by piling on thousands of nodes. I've created multi-gigabyte xml documents that way.

Ernie

[eostic]

There are too many variables. Size of tags, size of data...no real way to easily predict it. I've done as much as 500M in closed tests --- this is why the conservative recommendation that you received is 100M.

On targeting though, you can cheat. As you have mentioned in prior threads, just create single nodes and send those "out" of xmlOutput and into a sequential stage.....build the ultimate document in there by piling on thousands of nodes. I've created multi-gigabyte xml documents that way.

Ernie

[chulett]

Nobody wants to process LARGE xml files. Lots and lots of small ones, sure, but a small number of biggies? No thank you.

[ray.wurlod]

I recall an email sent back by one of the first Ardent techo people to go into China - "don't worry about 2GB as a file size, guys, here 2GB is a single transaction!"

The good news is that the next version of DataStage will have support for much larger XML file sizes, provided your hardware has the requisite memory available.

[chulett]

OK, maybe just no-one in their right mind.

I've told this before, but when sending XML to Google, they have a hard 100MB limit file size. 1 byte over on any file and the whole collection gets rejected. They didn't care what quantity you sent, but were quite emphatic about the individual sizes.

[pillip]

Do we have any option on the XML output stage to spilt the files based on the number of records.



Thanks

[chulett]

Yes. Specifically, you have the Trigger Column option which splits the file when the value there changes. It's up to you to make it change at the right time, in this case every X records. That's how we managed things with Google, picked a sufficiently small enough number that we knew it would 'never' be too big.

[arunkumarmm]

Hi,

Do we have any limit on the size of the XML file which can be written from Datastage.


Thanks,
Prasanna

Not much sure about the size limit but have created a file with >500MB. Only thing is that the job ran for 4 hours.