DSXchange: DataStage and IBM Websphere Data Integration Forum
View next topic
View previous topic
Add To Favorites
This topic has been marked "Resolved."
Author Message
FranklinE



Group memberships:
Premium Members

Joined: 25 Nov 2008
Posts: 628
Location: Malvern, PA
Points: 5952

Post Posted: Fri Sep 29, 2017 7:27 am Reply with quote    Back to top    

DataStage® Release: 11x
Job Type: Parallel
OS: Unix
Additional info: Ernie is my hero :)
We have a situation which we solved on the file transfer side, but the behavior of the parallel job which processes the xml file leaves me confused. I've checked the manual, and the wording of the description of the XML Input General tab is rather confusing.

EDIT: I forgot to add that due to auto-purge I don't have the Director messages from the cycle in which the file was incomplete. Sad

The file's transfer was interrupted mid-flight, resulting in an incomplete data string and no closing tag.

The first two stages are External Source and XML Input, both from the Px palette.

The job finishes okay, but no rows are processed.

Before I start experimenting, I'm looking for advice about what to expect. Is there a setting on either stage that will allow me to force the job to abort when there is one or more missing closing tag? We are already looking into using Unix scripting, but I like to have options.

Explanation of Additional Info: Ernie Ostic helped me get started on using the XML stages, and the job in question was the result, developed under 8x and migrated to 11x without remedial coding. Very Happy

_________________
Franklin Evans
"Shared pain is lessened, shared joy increased. Thus do we refute entropy." -- Spider Robinson

Using mainframe data FAQ: http://www.dsxchange.com/viewtopic.php?t=143596
PaulVL



Group memberships:
Premium Members

Joined: 17 Dec 2010
Posts: 1167

Points: 7709

Post Posted: Fri Sep 29, 2017 8:02 am Reply with quote    Back to top    

My advice: Craft a better handshaking strategy. Your XML parsing job should never have been kicked off if the file transfer was incomplete.

bad/incomplete data is worse than no data.
Rate this response:  
Not yet rated
FranklinE



Group memberships:
Premium Members

Joined: 25 Nov 2008
Posts: 628
Location: Malvern, PA
Points: 5952

Post Posted: Fri Sep 29, 2017 8:06 am Reply with quote    Back to top    

Yes, Paul, that's what we did. The file watcher was finding the file handle before the FTP completed. We fixed that.

I'm looking at this from a "generic" point of view. Every other file process I use gives me low-level control over the status of the file. XML processing is still new to me, in that I only have two jobs (out of 350) that use XML formats. I'm looking at that proportion to grow, and I like to use single-point solutions. Call me spoiled, you'd be accurate. Laughing

EDIT: It was a freak coincidence. The FTP itself was delayed a few minutes, and the file watcher was on a ten-minute cycle. Murphy's Law applies.

_________________
Franklin Evans
"Shared pain is lessened, shared joy increased. Thus do we refute entropy." -- Spider Robinson

Using mainframe data FAQ: http://www.dsxchange.com/viewtopic.php?t=143596
Rate this response:  
Not yet rated
UCDI



Group memberships:
Premium Members

Joined: 21 Mar 2016
Posts: 273

Points: 2756

Post Posted: Fri Sep 29, 2017 9:08 am Reply with quote    Back to top    

I have found the keep it local approach to be the best for that.
So what we do is drop the file to an intermediate folder from the network then do a unix move command once that is complete to the correct folder, where the detection can happen safely. If it is all on the same disk, you should not be able to catch a partial file as the file moves on the same disk do not move the data on the disk, they just re-arrange its entry in the file system.

you can do it many other ways -- 'we done yet' secondary files, or flip one of the obsolete or unused bits on the file (chmod), actual process communications, and more. The above is just one overly simple yet effective way.

you are new to xml so FYI, if you have bandwidth issues sending huge xml files around you can use bzip2 on them. That algorithm does magic to xml.
Rate this response:  
Not yet rated
FranklinE



Group memberships:
Premium Members

Joined: 25 Nov 2008
Posts: 628
Location: Malvern, PA
Points: 5952

Post Posted: Fri Sep 29, 2017 9:24 am Reply with quote    Back to top    

Thanks for the tips on moving files, but that is not my question. We've corrected the cause of the incomplete file, I'm looking for advice on how to handle incomplete xml files in DataStage.

Murphy's Law suggested corollary: you find one reason for a failure, and two more appear. I cannot code for every contingency, nor is that anything like best practice. I want my job to abort, and I'm looking for ways to code for it.

_________________
Franklin Evans
"Shared pain is lessened, shared joy increased. Thus do we refute entropy." -- Spider Robinson

Using mainframe data FAQ: http://www.dsxchange.com/viewtopic.php?t=143596
Rate this response:  
Not yet rated
eostic

Premium Poster



Group memberships:
Premium Members

Joined: 17 Oct 2005
Posts: 3724

Points: 29740

Post Posted: Mon Oct 02, 2017 4:12 am Reply with quote    Back to top    

In my experience, the xmlInput Stage will check for well-formed-ness successfully with its various options for validation and and reject...including missing end tags and such. It may depend on the ...

_________________
Ernie Ostic

blogit!
Open IGC is Here!
Rate this response:  
FranklinE



Group memberships:
Premium Members

Joined: 25 Nov 2008
Posts: 628
Location: Malvern, PA
Points: 5952

Post Posted: Mon Oct 02, 2017 8:31 am Reply with quote    Back to top    

Thanks, Ernie. I'd hoped to get away with minimal coding changes, but the hierarchical stage, being new to our design, looks like a promising approach.

_________________
Franklin Evans
"Shared pain is lessened, shared joy increased. Thus do we refute entropy." -- Spider Robinson

Using mainframe data FAQ: http://www.dsxchange.com/viewtopic.php?t=143596
Rate this response:  
Not yet rated
eostic

Premium Poster



Group memberships:
Premium Members

Joined: 17 Oct 2005
Posts: 3724

Points: 29740

Post Posted: Mon Oct 02, 2017 11:26 am Reply with quote    Back to top    

I am surprised that xmlInput isn't able to capture a missing end tag....it may depend on your particular document and location of the tag.....?

Ernie

_________________
Ernie Ostic

blogit!
Open IGC is Here!
Rate this response:  
Not yet rated
FranklinE



Group memberships:
Premium Members

Joined: 25 Nov 2008
Posts: 628
Location: Malvern, PA
Points: 5952

Post Posted: Mon Oct 02, 2017 11:41 am Reply with quote    Back to top    

The job was migrated 8.7 to 11.5. I paid it no further attention after the successful compile and regression testing. After be careful what you wish for is be careful what you don't know. Wink

_________________
Franklin Evans
"Shared pain is lessened, shared joy increased. Thus do we refute entropy." -- Spider Robinson

Using mainframe data FAQ: http://www.dsxchange.com/viewtopic.php?t=143596
Rate this response:  
Not yet rated
FranklinE



Group memberships:
Premium Members

Joined: 25 Nov 2008
Posts: 628
Location: Malvern, PA
Points: 5952

Post Posted: Mon Oct 02, 2017 11:46 am Reply with quote    Back to top    

Transformation error mappings current settings:
Fatal -- reject
Error -- reject
Warning -- warning

Looks like it was right in front of me, the usual place for things to successfully hide from me. I'll play with Fatal and Error to come up with what fulfills my requirements.

_________________
Franklin Evans
"Shared pain is lessened, shared joy increased. Thus do we refute entropy." -- Spider Robinson

Using mainframe data FAQ: http://www.dsxchange.com/viewtopic.php?t=143596
Rate this response:  
Not yet rated
eostic

Premium Poster



Group memberships:
Premium Members

Joined: 17 Oct 2005
Posts: 3724

Points: 29740

Post Posted: Mon Oct 02, 2017 3:56 pm Reply with quote    Back to top    

Yeah...it's actually a pretty cool feature. You don't even need to bother with an xsd or formal validation....not-well-formed will kick out most of the time. I like to choose the reject option, and ...

_________________
Ernie Ostic

blogit!
Open IGC is Here!
Rate this response:  
FranklinE



Group memberships:
Premium Members

Joined: 25 Nov 2008
Posts: 628
Location: Malvern, PA
Points: 5952

Post Posted: Mon Oct 09, 2017 8:29 am Reply with quote    Back to top    

Marking this resolved, as it isn't an open issue here at this point. I just need to be prepared for future repeats of the incident.

Thanks to all who responded.

_________________
Franklin Evans
"Shared pain is lessened, shared joy increased. Thus do we refute entropy." -- Spider Robinson

Using mainframe data FAQ: http://www.dsxchange.com/viewtopic.php?t=143596
Rate this response:  
Not yet rated
Display posts from previous:       

Add To Favorites
View next topic
View previous topic
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum



Powered by phpBB © 2001, 2002 phpBB Group
Theme & Graphics by Daz :: Portal by Smartor
All times are GMT - 6 Hours