DSXchange: DataStage and IBM Websphere Data Integration Forum
View next topic
View previous topic
Add To Favorites
Author Message
anusha
Participant



Joined: 28 Nov 2007
Posts: 37
Location: pune
Points: 451

Post Posted: Fri Dec 03, 2010 7:14 am Reply with quote    Back to top    

DataStage® Release: 8x
Job Type: Parallel
OS: Unix
We have set of nearly 50 CSV files out of which we have create same set of 50 XML files in a single datastage job. For this we are trying to make use of Schema file functionlaity for reading the CSV files. so My design would be like:

Code:
Sequential  --------> Copy Stage --------------->XML Output Stage


So will the XML out stage provide functionality of dynamically changing metadata definitions. Here am trying to avoid RCP enabled.

1. I had thought of using XSD files(50) for XML generation in XML transformer stage by parameterizing the XSD file names.Would this work, I can't test it now as I don't have DS environment setup.
Please suggest...
eostic

Premium Poster



Group memberships:
Premium Members

Joined: 17 Oct 2005
Posts: 3813

Points: 30674

Post Posted: Fri Dec 03, 2010 7:40 am Reply with quote    Back to top    

How dynamic do you need it to be? If you want to send the job an "externally" defined specification, then you will have to use xslt and supply that to the job at run time in the desired xml stage...

...then it's really more of an xsllt project, with the xslt conforming to DataStage and then doing whatever it is you need it to do...

Ernie

_________________
Ernie Ostic

blogit!
Open IGC is Here!
Rate this response:  
Not yet rated
chulett

Premium Poster


since January 2006

Group memberships:
Premium Members, Inner Circle, Server to Parallel Transition Group

Joined: 12 Nov 2002
Posts: 42926
Location: Denver, CO
Points: 221381

Post Posted: Fri Dec 03, 2010 7:52 am Reply with quote    Back to top    

"xsllt project"? Google is telling me that's a typo, so assuming you just meant "xslt project", yes?

_________________
-craig

The Old Ones were, the Old Ones are, and the Old Ones shall be. Not in the spaces we know, but between them. They walk serene and primal, undimensioned and to us unseen.
Rate this response:  
Not yet rated
anusha
Participant



Joined: 28 Nov 2007
Posts: 37
Location: pune
Points: 451

Post Posted: Fri Dec 03, 2010 1:14 pm Reply with quote    Back to top    

1. Will XSLT helps in converting relational/tabular data into XML data?, I thouhgt, it needs only XML data as input to do any processing...

2. If XSLT supports relational data to XML conversion, then,
I have 50 CSV files & am using 50 differernt corresponding schema files to read them in a single DS job. Can XML transformer stage supports runtime metadata functionality (as I have schema file for reading),so that for a file I will assign a specific Schema file & specific XSLT to generate an XML file. Will this functionality be possible.

Thanks,
Anusha
Rate this response:  
Not yet rated
eostic

Premium Poster



Group memberships:
Premium Members

Joined: 17 Oct 2005
Posts: 3813

Points: 30674

Post Posted: Fri Dec 03, 2010 1:33 pm Reply with quote    Back to top    

That was a typo up above. ; ) We're talking xslt here....

All I am implying with xslt (using either the XMLTransformer Stage or the new XML Stage in 8.5) is that xslt is the only way that you would be able to make the parsing and column work entirely dynamic..... and even then, it probably won't deliver exactly what you are looking for.

The XML Stages (unless you resort to the largely custom solution with XML Transformer and XSLT) in pre-8.5 use xpath that is "inside" the link. It is not dynamic --- you can't change it without a re-compilation of the Job. ...and in 8.5 a formal xml schema definition is required.

Take us higher up...what is your ultimate goal here? Having 50 different output (for different nodes, etc.) is entirely do-able within the Stage...but the links have to be defined and in your Job.....the Stage will take care of sending the output to the appropriate link depending on the incoming content.

Ernie

_________________
Ernie Ostic

blogit!
Open IGC is Here!
Rate this response:  
Not yet rated
anusha
Participant



Joined: 28 Nov 2007
Posts: 37
Location: pune
Points: 451

Post Posted: Sun Dec 05, 2010 12:47 am Reply with quote    Back to top    

Quote:
Take us higher up...what is your ultimate goal here? Having 50 different output (for different nodes, etc.) is entirely do-able within the Stage...but the links have to be defined and in your Job.....the Stage will take care of sending the output to the appropriate link depending on the incoming content.


Hope, we are not on the same page here. let me brief again.
Actually we have dynamically changing Input over that we have create respective XML files.

I have source files like CSV1,CSV2, CSV3...CSV50 and these should be converted to XML1,XML2,XML3.....XML50. All the Input is not part of a single node of single file but each details will part indepedent source files.
What we wanted here is 1:1 mapping should be built between CSV:XML.

And our query over this, Is this achievable in a single Datastage Job or we need 50 DS jobs specific to each files as each file has got it's own metadata defined in it.
Rate this response:  
Not yet rated
eostic

Premium Poster



Group memberships:
Premium Members

Joined: 17 Oct 2005
Posts: 3813

Points: 30674

Post Posted: Mon Dec 06, 2010 10:08 am Reply with quote    Back to top    

I'd consider other factors also before making a decision on which way to go here.....consider such things as:

Is the max number of files 50? Or is that just your current number, and it could eventually be 500?

Are the files large? or will they eventually be large? Will you need parallel processing of xml along the way?

Will you want/need formal validation against a schema as you build each xml?

Will you eventually be reading that xml with DataStage, and converting back into csv?

Fifty is a lot, but if that's the final number, developing 50 links or 50 separate Jobs won't be "too" bad, and then you will have all the benefits offered by the new or existing stage and some further future proofing against large volumes, the desire to utilize built-in xml functionality for validation, the assurance that a future DataStage developer can pick up and maintain your code, and better management of metadata.

Short of that, if most of your column transformation is upstream, you don't have any schema or concerns about xml validation, and you expect the number of interfaces to grow dramatically, and you have xslt or java expertise available now and in the future, the XMLTransformer or just JavaPack may be the best solution. Feed the transformed columns into either and then use custom xslt or a java class.

Ernie

_________________
Ernie Ostic

blogit!
Open IGC is Here!
Rate this response:  
Not yet rated
anusha
Participant



Joined: 28 Nov 2007
Posts: 37
Location: pune
Points: 451

Post Posted: Tue Dec 07, 2010 6:43 am Reply with quote    Back to top    

eostic,

Can you brief us on how to create and use Xslt in the Xml Transformer stage to create independent multiple XML files


Thanks..
Rate this response:  
Not yet rated
samyamkrishna



Group memberships:
Premium Members

Joined: 04 Jul 2006
Posts: 256
Location: Toronto
Points: 1577

Post Posted: Tue Dec 07, 2010 7:08 am Reply with quote    Back to top    

In the Stage Tarsformation settings.
click the radio button for Use trigger column and select a column.
Rate this response:  
Not yet rated
Aquilis
Participant



Joined: 05 Apr 2007
Posts: 204
Location: Bangalore
Points: 2317

Post Posted: Tue Dec 07, 2010 11:15 am Reply with quote    Back to top    

Verify options on UNIX, where few lists of commands will convert a CSV files into XML file, not even XSLT is required.

Base on your specifications or Requirements, it's betetr to use XSLT.
According to your requirement you don't need any XSLT usage, just try to play with csv2xml command on UNIX.

_________________
Aquilis
Rate this response:  
Not yet rated
sivadanam
Participant



Joined: 10 Jan 2014
Posts: 1

Points: 5

Post Posted: Thu Jan 10, 2019 11:40 am Reply with quote    Back to top    

Hi Anusha,
Have you achieved the above scenario in singlle job. If so Could you please help me in achieving it am facing issue
Rate this response:  
Not yet rated
chulett

Premium Poster


since January 2006

Group memberships:
Premium Members, Inner Circle, Server to Parallel Transition Group

Joined: 12 Nov 2002
Posts: 42926
Location: Denver, CO
Points: 221381

Post Posted: Thu Jan 10, 2019 1:05 pm Reply with quote    Back to top    

Exclamation sivadanam, sorry but you need to start your own post for something like this rather than jump on the end of something similar that's eight years old. That way we know what version you are running and you can fully explain what your requirement and/or issue is that you are having trouble with.

Thanks.

_________________
-craig

The Old Ones were, the Old Ones are, and the Old Ones shall be. Not in the spaces we know, but between them. They walk serene and primal, undimensioned and to us unseen.
Rate this response:  
Not yet rated
Display posts from previous:       

Add To Favorites
View next topic
View previous topic
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum



Powered by phpBB © 2001, 2002 phpBB Group
Theme & Graphics by Daz :: Portal by Smartor
All times are GMT - 6 Hours