Sequencer reading from just one folder

UAUITSBI · Post by **UAUITSBI** » Wed Mar 18, 2015 8:37 pm

Hello,

I have designed a sequencer which will run a simple unix command to count the number of files:

cd E:/DataSources/Export/Source/ | ls -l | wc -l | sed "s/ //g;"

For some reason Datastage is not reading files from this folder instead it is reading from the below folder:

Code: Select all

C:/IBM/InformationServer/Server/Projects/ETL1

No matter what folder I provide after the cd command it always reads the files from the ETL1 folder, sequencer job doesn't abort but simply provides output from ETL1folder.

Inorder to test, I have also moved the files to "C:/IBM/InformationServer/Server/Projects" to see if it reads from this folder but still it reads from only ETL1 folder.

This doesn't happen when I pass this folder ( E:/DataSources/Export/Source/) into the jobs.

Please let me know if I am missing something.

Note: I have also tried passing the folder name in quotes
cd "E:/DataSources/Export/Source/" | ls -l | wc -l | sed "s/ //g;" but there is no success.

Appreciate the help!!

ray.wurlod · Post by **ray.wurlod** » Wed Mar 18, 2015 8:52 pm

Try changing your logged drive in the command.

Code: Select all

E: ; cd E:/DataSources/Export/Source/ | ls -l | wc -l | sed "s/ //g;"

UAUITSBI · Post by **UAUITSBI** » Wed Mar 18, 2015 9:10 pm

Thanks for the tip, I have changed the command as per your recommendation it is not working below is the error:

Code: Select all

'E:' is not recognized as an internal or external command,
operable program or batch file.

Should it be:

Code: Select all

cd E: ; cd E:/DataSources/Export/Source/ | ls -l | wc -l | sed "s/ //g;"

?

UAUITSBI · Post by **UAUITSBI** » Wed Mar 18, 2015 9:14 pm

I have tried adding cd E:

Below is the error:

Code: Select all

The filename, directory name, or volume label syntax is incorrect.

It returned the number of files count from folder but still it is referring to the ETL1 folder that I mentioned in the earlier email.

ray.wurlod · Post by **ray.wurlod** » Thu Mar 19, 2015 1:18 am

Then how about a unique UNC pathname?

UAUITSBI · Post by **UAUITSBI** » Thu Mar 19, 2015 10:44 am

Hello Ray,

By UNC pathname do you mean something like \\server\path\filename ? How can I map the same in datastage, by passing this as parameter ? Is there a way to set UNC path in datastage ?

When passing the cd E:/DataSources/Export/Source/ folder in the job the files are being from that folder, but when it is in the sequencer the files are being read from the different folder (ETL1 -- folder).

UAUITSBI · Post by **UAUITSBI** » Thu Mar 19, 2015 1:37 pm

I have tried giving the shared drive path and below is the result:

Code: Select all

SEQ_TEST..JobControl (@Exec_XML_Files_Count): Executed: cd E: ; cd \\mmetl1\PIMExport\MDMExport\Source\ | ls -l | wc -l | sed "s/ //g;"
Reply=0
Output from command ====>
The system cannot find the path specified.
15896

It is again going to the default folder ETL1 and getting the count. In my source folder there are 3 files but the count here is 15896.

chulett · Post by **chulett** » Thu Mar 19, 2015 1:57 pm

Right... when your directory path cannot be found you are still in the CWD or Current Working Directory of the executing process. I have no idea if your UNC path is both valid and if the user executing the DataStage job has access to it. I'm suspecting it's not or it doesn't. Regardless, I'd suggest a couple of changes: drop the drive change to E: as it's no longer needed and get rid of the pipe after the UNC path, there's nothing to "pipe" at that point so it should just be a command separator:

Code: Select all

cd \\mmetl1\PIMExport\MDMExport\Source\ ; ls -l | wc -l | sed "s/ //g;"

If you were on a real UNIX system

I'd suggest you replace that semi-colon like so:

Code: Select all

cd \\mmetl1\PIMExport\MDMExport\Source\ && ls -l | wc -l | sed "s/ //g;"

That "&&" means to only execute the right command if the left command succeeds. The ";" is unconditional.

ps. When next you reply, use the Reply to topic link, please. This rather than "Reply with quote". That would be much appreciated.

chanaka · Post by **chanaka** » Thu Mar 19, 2015 3:00 pm

Hi,

Either you have to follow the work around that Chulett provided. Or else you have to map the network drive by loging in to the system as dsadm user and save the credentials to access the network drive path.

Please try this and let me know what happened.

UAUITSBI · Post by **UAUITSBI** » Thu Mar 19, 2015 3:42 pm

Hello Chullett,

I have checked the permissions on the shared drive and it provides FULL CONTROL (READ, WRITE etc..). I have tried your recommendation and still it doesn't work below is the error:

Code: Select all

SEQ_TEST..JobControl (@Exec_XML_Files_Count): Executed: cd \\mmetl1\PIMExport\MDMExport\Source\ ; ls -l | wc -l | sed "s/ //g;"
Reply=0
Output from command ====>
'\\mmetl1\PIMExport\MDMExport\Source\ ; ls -l'
CMD does not support UNC paths as current directories.
0

Then I have tried the Non-UNC path to check along with your recommendation, and got below error:

Code: Select all

SEQ_TEST..JobControl (@Exec_XML_Files_Count): Executed: cd E:\DataSources\PIMExport\MDMExport\Source\ ; ls -l | wc -l | sed "s/ //g;"
Reply=0
Output from command ====>
The system cannot find the path specified.
0

The intriguing part is when I pass "E:\DataSources\PIMExport\MDMExport\Source\" folder name into my ETL job it works, files are extracted and processed. But when I pass this in the sequencer job it is not picking up not sure about the reason for this behavior.

p.s I usually do "Reply with quote" when referring to a specific post in this case Ray's. If not I usually do "Reply Topic" will try to stick with "Reply Topic" unless it is a very specific reply. :D

chulett · Post by **chulett** » Thu Mar 19, 2015 3:57 pm

CMD does not support UNC paths as current directories

Ah... so forget the "cd" part and see if a restructured command like this works:

Code: Select all

ls -l \\mmetl1\PIMExport\MDMExport\Source\ | wc -l | sed "s/ //g;"

PaulVL · Post by **PaulVL** » Thu Mar 19, 2015 4:37 pm

Does the path exist?

How about passing the path in your LS command as such:

ls E:/DataSources/Export/Source/ | wc -l | sec "s/ //g;"

not sure why you have the -l on your LS command since you are just counting lines.

Also, does the user id executing the datastage job have E: mounted as a drive?

SURA · Post by **SURA** » Thu Mar 19, 2015 6:04 pm

Use the command activity stage, use echo command with the ls file path | wc -l . This will print the command in the Director you wish to execute. Go to director, copy it from the echo and execute it in the OS command line. This will helps you to find how the value looks like when while the DS executes the command.

chulett · Post by **chulett** » Fri Mar 20, 2015 8:48 am

PaulVL wrote:not sure why you have the -l on your LS command since you are just counting lines.

Guilty of this over the years for a couple of reasons. Force of habit mostly and it generally works just fine.

Now, perhaps it can contribute to the dreaded "argument list too long" error you'll see with a large number of files but after alll these years I'm not certain if that's strictly based on number or if the data in the listing is overflowing some internal buffer. It the latter, then you'd have more head room without a long listing.

UAUITSBI · Post by **UAUITSBI** » Fri Mar 20, 2015 1:34 pm

Chulett,

The LS command worked like a charm, thanks for that.

PaulV,

Yes the path exists when I pass this folder path into datastage job it works perfect it's only in the sequencer that I got this issue.
Yes E: is mounted as a drive, While defining the folder path rather than having it manually entered I browse to the folder path through Datastage.
Thanks for the suggestion, ls command worked.

Sura,

Thanks for the tip. I used the command activity stage and PWD inside it to determine from where the files are being read. I guess it has to do with the 'CD' command, I have minimal experience in the Windows environment probably the reason for this glitch.

P.S. Yeah as chulett pointed out it's just force of habit I always been guilty of using ls -l never ran into overflowing error probably that will make me stop using -l :D