S3 connector error with transferring large files

Post questions here relative to DataStage Enterprise/PX Edition for such areas as Parallel job design, Parallel datasets, BuildOps, Wrappers, etc.

Moderators: chulett, rschirm, roy

Post Reply
mehimadri_123
Premium Member
Premium Member
Posts: 3
Joined: Sat Aug 26, 2017 7:55 pm

S3 connector error with transferring large files

Post by mehimadri_123 »

Hello Everyone,

We have a parallel Job reading DB2 AS400 table and writing out to AWS S3 bucket. Now, while this Job is working for all other relatively smaller tables, but for one large table with 200 million records and of size around 50 GB it is failing after few hours with the following error. We do not get this error if we limit the number of rows to lets say couple of thousands.

An exception occurred: com.amazonaws.services.s3.model.AmazonS3Exception: Part number must be an integer between 1 and 10000, inclusive

Now, based upon internet search, it appears that this error was resolved in other AWS S3 upload methods (not Datastage; could not find anything on DataStage) can be resolved by increasing the default chunk size (multipart-chunk-size-mb) from 15MB. to a larger value like 50 MB.

I am not sure what is the corresponding S3 connector property or environment variable to set in DataStage for the chunk size.

I tried with $APT_DEFAULT_TRANSPORT_BLOCK_SIZE and set to its maximum 1048576 but it did not help.

Please help me if you have faced this error before or any ideas on someone who has?
Post Reply