DataStage w/ Linux CIFS Mounts

A forum for discussing DataStage<sup>®</sup> basics. If you're not sure where your question goes, start here.

Moderators: chulett, rschirm, roy

Post Reply
gsbrown
Premium Member
Premium Member
Posts: 148
Joined: Mon Sep 23, 2002 1:00 pm
Location: USA

DataStage w/ Linux CIFS Mounts

Post by gsbrown »

Our DataStage jobs integrate with many Windows shares from our SLES 11 Linux DataStage environment. For a long time, we've managed these mount points through /etc/fstab file which are connected on boot and then we have a cron job every hour that reconnects if any happened to disconnect.

This works all fine and well, except in the situation where a mount disconnects and a DataStage jobs starts before the 1hr cron job has a chance to catch the disconnect to reconnect. With that, I've been exploring either 1) increasing the frequency of the cron job to try and lessen the chance of failure or 2) implementing an auto mount features (ie. autofs)

I'm playing around with autofs, but the issue I'm running into is that it can take several seconds to "wake up" a CIFS mount point and DataStage isn't patient with that at all. DataStage job immediately fails with a "cannot write to file" error. It's fine if the mount point was already "woke" by a prior command or job, but I'm not comfortable with that insecurity. I'm also not open to the idea of coding all of our DataStage jobs to "wake" the mount point with a touch or cd command before attempting file processing.

With all that, it sounds like having a more frequent mounting cron script (ie. every minute?) would help alleviate the mount disconnect issue, but does anyone have experience with a more solid solution to keeping CIFS mounts alive on a Linux DataStage server. Appreciate any feedback or personal experiences of what's working for your team. Thank You!
PaulVL
Premium Member
Premium Member
Posts: 1315
Joined: Fri Dec 17, 2010 4:36 pm

Post by PaulVL »

Well, I would try to address the root cause to see why those mounts go belly up.

Your application teams COULD write a diagnostic job to detect if the mount is fubar. That job could then issue the mount command, sleep for a determined amount of time, retest, etc... It should then alert your staff that the mount can't recover, or proceed with the flow if everything is cool.

Since your mount point seems unstable, what do you think would happen to the flow if it gets un-mounted mid ETL stream?

Often times your DataStage jobs are kicked off from a shell script. That mount test/attach script could be imbedded within each of you scripts. TEST aspect would be very fast. The MOUNT aspect would be required if it failed. You would simply have to craft a locking procedure so that you don't spam the box with mount commands.
UCDI
Premium Member
Premium Member
Posts: 383
Joined: Mon Mar 21, 2016 2:00 pm

Post by UCDI »

you can do something cheezy like put a static text file out on the mount point in a shared location and all your jobs that touch that mount point can use a wait for file until it can see that, then proceed. That would wake it up.

This could be a hardware thing, on top of whatever config stuff. Some disks are put to sleep if idle for too long to save power and some of them take a few seconds to come back up.
Post Reply