DSXchange: DataStage and IBM Websphere Data Integration Forum
View next topic
View previous topic
Add To Favorites
This topic has been marked "Resolved."
Author Message
rumu
Participant



Joined: 06 Jun 2005
Posts: 267

Points: 2694

Post Posted: Mon Jul 23, 2018 9:24 am Reply with quote    Back to top    

DataStage® Release: 11x
Job Type: Parallel
OS: Unix
We have copied Datasets within the same server. How can we copy Datasets from one server to another server ?

_________________
Rumu
IT Consultant
UCDI



Group memberships:
Premium Members

Joined: 21 Mar 2016
Posts: 338

Points: 3409

Post Posted: Mon Jul 23, 2018 12:18 pm Reply with quote    Back to top    

it is better if you do not do this. In an emergency, you can move the dataset AND its underlying files AND change the 'header' file (the one you interface to in datastage when you open or write a dataset) to point to the new locations. The header file is in some sort of unicodeish partial text partial binary format, so editing it and getting it working has a little risk. You can also mount the other server and re-create it there.

You can also write another type of file to transport, like a flat file, or land the data in a temporary database table, and fetch it on the other server.
Rate this response:  
Not yet rated
ray.wurlod

Premium Poster
Participant

Group memberships:
Premium Members, Inner Circle, Australia Usergroup, Server to Parallel Transition Group

Joined: 23 Oct 2002
Posts: 54431
Location: Sydney, Australia
Points: 295208

Post Posted: Mon Jul 23, 2018 9:58 pm Reply with quote    Back to top    

I too counsel against trying to copy Data Sets from one server to another. You could mount all the required disks from the new system, then create a configuration file that mentioned resource ...

_________________
RXP Services Ltd
Melbourne | Canberra | Sydney | Hong Kong | Hobart | Brisbane
currently hiring: Canberra, Sydney and Melbourne (especially seeking good business analysts)
Rate this response:  
Not yet rated
PaulVL



Group memberships:
Premium Members

Joined: 17 Dec 2010
Posts: 1257

Points: 8256

Post Posted: Tue Jul 24, 2018 9:27 am Reply with quote    Back to top    

I have written a custom set of jobs to dump a generic named dataset (parm to the job) into a sequential file + a schema file. The schema file was created with the orchadmin command to dump the schema, filter that via a few awk commands and echo statements and you can easily recreate the schema text file layout that a sequential stage will need to read that file in.

So, here is my generic process.

Step 1) Create a sequencer that has an execute stage and a parallel job.

Execute stage has the orchadmin command.

Job is the dataset to Sequential file process.

Step 2) Create a generic job for sequential file to dataset creation.

Parms to that job would contain the schema file created via that orchadmin command.


Now you should be able to fill in the blanks on the above process. But what this does is that it allows you to pass in ANY dataset and dump to a text file + schema file.

SFTP those over to your target system and run job #2 with takes a Sequential file + Schema file and creates a Dataset.


Here is my Execute Stage command:

echo "record" > #seqfilepath##ds_filename#.schema; echo "{record_delim='\n', final_delim=end,delim=',', quote=double}" >> #seqfilepath##ds_filename#.schema; $DSHOME/../PXEngine/bin/orchadmin describe -s #dataset_path##ds_filename# 2>/dev/null | sed 1,11d >> #seqfilepath##ds_filename#.schema; echo #seqfilepath##ds_filename#.schema


That should set you on the right path.

Please keep in mind that this is A method, not the only method. Some far craftier people out there have better ways of skinning the cat.
Rate this response:  
Not yet rated
rumu
Participant



Joined: 06 Jun 2005
Posts: 267

Points: 2694

Post Posted: Tue Jul 24, 2018 11:41 am Reply with quote    Back to top    

Thanks all for your response . We decided not to copy datasets at this point.

_________________
Rumu
IT Consultant
Rate this response:  
Not yet rated
asorrell
Site Admin

Group memberships:
Premium Members, DSXchange Team, Inner Circle, Server to Parallel Transition Group

Joined: 04 Apr 2003
Posts: 1694
Location: Colleyville, Texas
Points: 23057

Post Posted: Wed Jul 25, 2018 8:58 am Reply with quote    Back to top    

IBM's official answer:

How to move dataset from one server to another in IBM InfoSphere DataStage
https://www-01.ibm.com/support/docview.wss?uid=swg21392477

_________________
Andy Sorrell
Certified DataStage Consultant
IBM Analytics Champion 2009 - 2017
Rate this response:  
Not yet rated
PaulVL



Group memberships:
Premium Members

Joined: 17 Dec 2010
Posts: 1257

Points: 8256

Post Posted: Wed Jul 25, 2018 10:54 am Reply with quote    Back to top    

Andy, that method lacks the schema information that will be required for the dataset. The method I listed above is basically that + schema.

I'm (not) surprised that IBM left the schema information out of their explanation.

How the heck would you translate anything into a field other than a var char without the schema file?!?
Rate this response:  
Not yet rated
chulett

Premium Poster


since January 2006

Group memberships:
Premium Members, Inner Circle, Server to Parallel Transition Group

Joined: 12 Nov 2002
Posts: 42834
Location: Denver, CO
Points: 220807

Post Posted: Wed Jul 25, 2018 11:48 am Reply with quote    Back to top    

Isn't that what they mean by the 'header file'?

Quote:
Once the job has run, the data will be on the target system but the dataset header file for the target dataset will still be on the source server. You will have to move the header files to your target system (same paths).

_________________
-craig

Your brain is full of spiders, you've got garlic in your soul, Mr. Grinch!
I wouldn't touch you With a thirty-nine-and-a-half foot pole!
Rate this response:  
Not yet rated
PaulVL



Group memberships:
Premium Members

Joined: 17 Dec 2010
Posts: 1257

Points: 8256

Post Posted: Wed Jul 25, 2018 3:33 pm Reply with quote    Back to top    

if you make your own schema, then you do not have to worry about an APT file that has different node name info. You could even have a different degree of parallelism.

I'm dead set against someone copying over the .ds header file to another host. that is just plain silly. shame on IBM for recommending that.
Rate this response:  
Not yet rated
UCDI



Group memberships:
Premium Members

Joined: 21 Mar 2016
Posts: 338

Points: 3409

Post Posted: Thu Jul 26, 2018 2:04 pm Reply with quote    Back to top    

there is something that works for many small simple DS too:

orchadmin dump -delim ',' dsname.ds outputname.csv

but this is a quick fix for simple problems; there are times when it won't be sufficient, and its terrible for anything big.

it will also lose the schema. But you can export a table def for a ds, datastage can create those from the ds and you can send it alongside the csv.
Rate this response:  
Not yet rated
PaulVL



Group memberships:
Premium Members

Joined: 17 Dec 2010
Posts: 1257

Points: 8256

Post Posted: Mon Jul 30, 2018 9:27 am Reply with quote    Back to top    

The process I listed above works for us. Makes the schema for any dataset. Dumps to a sequential file... works great.
Rate this response:  
Not yet rated
ArndW

Premium Poster
Participant

Group memberships:
Premium Members, Inner Circle, Australia Usergroup

Joined: 16 Nov 2004
Posts: 16318
Location: Germany
Points: 92566

Post Posted: Tue Jul 31, 2018 3:52 am Reply with quote    Back to top    

UCDI wrote: there is something that works for many small simple DS too: orchadmin dump -delim ',' dsname.ds outputname.csv ... I seem to remember that you will be losing any NULL va ...

_________________

Image
Rate this response:  
Not yet rated
Display posts from previous:       

Add To Favorites
View next topic
View previous topic
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum



Powered by phpBB © 2001, 2002 phpBB Group
Theme & Graphics by Daz :: Portal by Smartor
All times are GMT - 6 Hours