DSXchange: DataStage and IBM Websphere Data Integration Forum
View next topic
View previous topic
Add To Favorites
Author Message
chandu123



Group memberships:
Premium Members

Joined: 12 Sep 2016
Posts: 40

Points: 411

Post Posted: Thu Sep 29, 2016 4:28 pm Reply with quote    Back to top    

DataStage® Release: 11x
Job Type: Parallel
OS: Windows
I am trying to read multiple files from a network location (not on the same server where the Datastage is installed). I am using sequential file stage:

File Pattern: //pd4p-ccbv-w01/d$/Cognos/Historical Volumes*.csv

Read Method: File Pattern

But it is returning 0 records.

I read in the forum that files should be on the same location as Datastage to read them using File Pattern method. But I couldn't find any solution to read them from the network location.

Is there any way to acheive this?

_________________
HB
SURA



Group memberships:
Premium Members

Joined: 14 Jul 2007
Posts: 1229
Location: Sydney
Points: 9005

Post Posted: Thu Sep 29, 2016 9:22 pm Reply with quote    Back to top    

Does it allow space in the folder name?

Did you tried to view the data or ...?

chandu123 wrote:
But it is returning 0 records.
Is there any way to acheive this?

When you execute the job in the director you can find the Datastage user name who triggered the job. The shared folder should have the access for this Datastage user.

_________________
Thanks
Ram
----------------------------------
Revealing your ignorance is fine, because you get a chance to learn.
Rate this response:  
Not yet rated
ArndW

Premium Poster
Participant

Group memberships:
Premium Members, Inner Circle, Australia Usergroup

Joined: 16 Nov 2004
Posts: 16305
Location: Germany
Points: 92459

Post Posted: Fri Sep 30, 2016 4:34 am Reply with quote    Back to top    

That file path must be as it is used on the DataStage engine machine. The path must be mounted/shared for the DataStage process and of course the process needs read access as well.

Try logging into your engine machine with the DataStage user (probably "dsadm") and see if you can list the files in that path.

_________________

Image
Rate this response:  
Not yet rated
FranklinE



Group memberships:
Premium Members

Joined: 25 Nov 2008
Posts: 679
Location: Malvern, PA
Points: 6425

Post Posted: Mon Oct 03, 2016 7:48 am Reply with quote    Back to top    

Dusting off some basics I learned about Windows networks, from a practical point of view.

Typically, a user's logon protocol will include a series of "net use" commands to establish a network link to remote servers (the one "local" server being your workstation's hard drive). This is (very) roughly equivalent to logging into a remote Unix server, most often lately using ssh. The "net use" commands will be accepted or rejected based on the user's authorized access to each server.

Our network security is perhaps more robust than most places, so grain of salt advice: if your process id (the DataStage appid or dsadm) has not been formally granted access to the remote server and the path to the files, you will not be able to access anything. Get a verbose set of log messages for your sequential file stage, and I will guess that the underlying failure is user authorization.

EDIT: Sorry, forgot to add that logging onto the remote server, or just trying to list files won't tell you anything unless you try the "net use" command first. It will tell you whether you have access to that server.

_________________
Franklin Evans
"Shared pain is lessened, shared joy increased. Thus do we refute entropy." -- Spider Robinson

Using mainframe data FAQ: http://www.dsxchange.com/viewtopic.php?t=143596 Using CFF FAQ: http://www.dsxchange.com/viewtopic.php?t=157872
Rate this response:  
Not yet rated
chandu123



Group memberships:
Premium Members

Joined: 12 Sep 2016
Posts: 40

Points: 411

Post Posted: Mon Oct 03, 2016 4:07 pm Reply with quote    Back to top    

I am able to access single file using sequential stage but not multiple files using File Pattern method. This makes me feel like there is no issue with the access to the directory on the network.

_________________
HB
Rate this response:  
Not yet rated
SURA



Group memberships:
Premium Members

Joined: 14 Jul 2007
Posts: 1229
Location: Sydney
Points: 9005

Post Posted: Mon Oct 03, 2016 11:54 pm Reply with quote    Back to top    

chandu123 wrote:
//pd4p-ccbv-w01/d$/Cognos/Historical Volumes*.csv


May be the space issue in the file.

try like below

Quote:
//pd4p-ccbv-w01/d$/Cognos/'Historical Volumes'*.csv


Or create few test files without space and try to read it.

_________________
Thanks
Ram
----------------------------------
Revealing your ignorance is fine, because you get a chance to learn.
Rate this response:  
Not yet rated
ray.wurlod

Premium Poster
Participant

Group memberships:
Premium Members, Inner Circle, Australia Usergroup, Server to Parallel Transition Group

Joined: 23 Oct 2002
Posts: 54221
Location: Sydney, Australia
Points: 294071

Post Posted: Tue Oct 04, 2016 12:59 am Reply with quote    Back to top    

A useful trick when pathnames have special characters, such as space, is to use the old "8+3" variant of the name.

_________________
RXP Services Ltd
Melbourne | Canberra | Sydney | Hong Kong | Hobart | Brisbane
currently hiring: Canberra, Sydney and Melbourne
Rate this response:  
chulett

Premium Poster


since January 2006

Group memberships:
Premium Members, Inner Circle, Server to Parallel Transition Group

Joined: 12 Nov 2002
Posts: 42570
Location: Denver, CO
Points: 219053

Post Posted: Tue Oct 04, 2016 6:53 am Reply with quote    Back to top    

Only if you're aware that they are still there under the fancy Windows 'long filenames' and how they work... tilde and all. Wink

_________________
-craig

I know I don't say this enough, but I like when you talk to me. It's much better than when nobody talks to me. Or when people that I don't like will not stop talking to me.
Rate this response:  
chandu123



Group memberships:
Premium Members

Joined: 12 Sep 2016
Posts: 40

Points: 411

Post Posted: Wed Oct 05, 2016 3:32 pm Reply with quote    Back to top    

I tried using below formats and still didn't work.

//pd4p-ccbv-w01/d$/Cognos/Data Manager/Historical Volumes*.csv
'//pd4p-ccbv-w01/d$/Cognos/Data Manager/Historical Volumes'*.csv
'//pd4p-ccbv-w01/d$/Cognos/Data Manager/Historical Volumes*.csv'
"//pd4p-ccbv-w01/d$/Cognos/Data Manager/Historical Volumes*.csv"

And above combinations with backslash(\) in place of front slash.

Ray,

Could you please elaborate the meaning of "8+3" format of file path?

_________________
HB
Rate this response:  
Not yet rated
chandu123



Group memberships:
Premium Members

Joined: 12 Sep 2016
Posts: 40

Points: 411

Post Posted: Wed Oct 05, 2016 3:33 pm Reply with quote    Back to top    

I also tried putting the file path in the parameter and using it which made no difference.

#FilePath#\Historical Volumes*.csv

_________________
HB
Rate this response:  
Not yet rated
chulett

Premium Poster


since January 2006

Group memberships:
Premium Members, Inner Circle, Server to Parallel Transition Group

Joined: 12 Nov 2002
Posts: 42570
Location: Denver, CO
Points: 219053

Post Posted: Wed Oct 05, 2016 3:49 pm Reply with quote    Back to top    

chandu123 wrote: Could you please elaborate the meaning of "8+3" format of file path? As suspected, a bit of arcane lore. https://en.wikipedia.org/wiki/8.3_filename The part you ...

_________________
-craig

I know I don't say this enough, but I like when you talk to me. It's much better than when nobody talks to me. Or when people that I don't like will not stop talking to me.
Rate this response:  
Not yet rated
SURA



Group memberships:
Premium Members

Joined: 14 Jul 2007
Posts: 1229
Location: Sydney
Points: 9005

Post Posted: Wed Oct 05, 2016 3:58 pm Reply with quote    Back to top    

try like below

Quote:
//pd4p-ccbv-w01/d$/Cognos/'Historical Volumes'*.csv

_________________
Thanks
Ram
----------------------------------
Revealing your ignorance is fine, because you get a chance to learn.
Rate this response:  
Not yet rated
chandu123



Group memberships:
Premium Members

Joined: 12 Sep 2016
Posts: 40

Points: 411

Post Posted: Wed Oct 05, 2016 4:39 pm Reply with quote    Back to top    

I forgot to mention that I tried Sura's suggested format of adding quotes and it didn't work.

Thanks for sharing me about 8.3 filename. I found though command line query that the short names for the files are something like this.

\\pd4p-ccbv-w01\d$\Cognos\DATAMA~1\SOURCE~1\FCUNIT~1\HISTOR~1.CSV
\\pd4p-ccbv-w01\d$\Cognos\DATAMA~1\SOURCE~1\FCUNIT~1\HISTOR~2.CSV
\\pd4p-ccbv-w01\d$\Cognos\DATAMA~1\SOURCE~1\FCUNIT~1\HI4B34~1.CSV
\\pd4p-ccbv-w01\d$\Cognos\DATAMA~1\SOURCE~1\FCUNIT~1\HI5B02~1.CSV
\\pd4p-ccbv-w01\d$\Cognos\DATAMA~1\SOURCE~1\FCUNIT~1\HI06DD~1.CSV

I tried giving below for file name:

HI*~?.CSV
HI~*.CSV
HISTOR~*.CSV

But resulted in 'Sequential_File_0: When checking operator: No files found for pattern.'

Any suggestions?

_________________
HB
Rate this response:  
Not yet rated
SURA



Group memberships:
Premium Members

Joined: 14 Jul 2007
Posts: 1229
Location: Sydney
Points: 9005

Post Posted: Wed Oct 05, 2016 4:47 pm Reply with quote    Back to top    

chandu123 wrote:
\\pd4p-ccbv-w01\d$\Cognos\DATAMA~1\SOURCE~1\FCUNIT~1\*.CSV

Can't you try like above?

_________________
Thanks
Ram
----------------------------------
Revealing your ignorance is fine, because you get a chance to learn.
Rate this response:  
Not yet rated
chandu123



Group memberships:
Premium Members

Joined: 12 Sep 2016
Posts: 40

Points: 411

Post Posted: Wed Oct 05, 2016 4:50 pm Reply with quote    Back to top    

I tried the above one and got the same message. I wonder if there some issue with the Datastage itself when reading FilePattern from network location? Since I am able to read single file successfully but not FilePattern.

_________________
HB
Rate this response:  
Not yet rated
Display posts from previous:       

Add To Favorites
View next topic
View previous topic
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum



Powered by phpBB © 2001, 2002 phpBB Group
Theme & Graphics by Daz :: Portal by Smartor
All times are GMT - 6 Hours