Reading MQ data ?

PilotBaha · Post by **PilotBaha** » Sat Jan 03, 2009 8:05 am

I have to read data that is coming from mainframe that was put into an MQ.
The mainframe data has a lot of redefines in it and this is making the things a bit more complicated for me.
I have never worked with MQ before. After setting the values of the MQ connector i was able to view the data in the Connector stage but moving it into a text file has been a challange.
First off, the column definitions is a bit complicated for me. In the Connectivity Guide it's talking about

The remaining columns (MSGField1 and so forth) represent the logical column names of the fields contained in the WebSphere MQ message. A WebSphere MQ message is a string-formatted message of fixed-length records, except the last column, which can be shorter. Therefore, you should specify character data types and lengths for these columns.

Well, is there a way i can import the definition of the message columns from the queue? (My guess is no , since the message is a string of undefined lenght) Do I have to type in the columns into the definition of the fields? How do I handle the redefines in COBOL when the data is put in a queue?

eostic · Post by **eostic** » Sat Jan 03, 2009 8:34 pm

Alas, the MQ stage is not the CFF Stage...it can nicely read the buffer from MQ, but not do complex things with datatypes and re-defines, etc. You will need to save the message to disk in order to more "easily" work with it via CFF.....

...possibly (I'll let other EE experts chime in here) you could use the various Restructure stages to do your own chewing up of the complex record(s) in the message(s) after you receive them via MQ.

Ernie

vmcburney · Post by **vmcburney** » Sun Jan 04, 2009 7:18 pm

What type of MQ content is it? There is an MQ message and an MQ file, it could be they are just using MQ for file transfer in which case you save the file to disk and it's a normal CFF file that should have a cobol copy book to define it. If it's an MQ message then the redefines should have happened and you should be left with an XML message defined by an XML file.

PilotBaha · Post by **PilotBaha** » Mon Jan 05, 2009 2:38 pm

It's an MQ message.. If it's already done through the XML it should not be an issue then..

eostic · Post by **eostic** » Mon Jan 05, 2009 4:43 pm

An MQ message is just a "bucket of bytes." What it has in it is entirely up to the developer of the application and the program that did the initial "PUT". Generally speaking, nowadays people more commonly adopt 100% character data as their payload, for simpler conversions, and in 90% of those cases, it's XML. However, it could be anything, and it's very possible that this is a binary record, with complex internal record constructs --- or, as Vincent notes, it could be a "whole file" of simple or complex records, where MQ is simply being used as an ensured file transfer mechanism. It's all out there... The only choice here, if you don't have anyone on the application team that can tell you exactly what is in the payload, is to just dump a single message and look at it, preferably with an editor that can handle hex, like UEdit, just in case it's all mainframe numeric datatypes........

Ernie

PilotBaha · Post by **PilotBaha** » Tue Jan 06, 2009 1:56 pm

Ermie and Vince,
thanks for the help. I was able to view the data after little bit of a research.(message padding, etc. ) The data is coming with different format but they are in the expected forms. Splitting the data and then merging later will not be an issue.
What i am not able to get is how to get the data in the message payload and some other information that comes with the queue into a text file.
I looked into importing the metadata for the the queue but there isn't an option for that. I also tried hand coding the metadata of the queue and that didn't get me anywhere..
When I press 'view data' I am seeing columns like MsgTypeTag, Expiry, Format, etc. etc.
What's the easiest way to define the columns in MQ connector stage so that i can send this data to a transformer and get done with the E portion of ETL ?

PilotBaha · Post by **PilotBaha** » Tue Jan 06, 2009 2:05 pm

Ok I found the Data Element Column in Column definitions , so I am guessing that will be the way to import queue related information into the metadata i defined.. I am still looking for a way to load the message payload though.

eostic · Post by **eostic** » Tue Jan 06, 2009 2:49 pm

The MQ columns using the Data Elements are powerful, but may not yet be needed here..... Start with a job that simply has one column in the MQ connector's output link. Make it a longvarchar with a length of 99999 and see what you get. Send it to a sequential file with no delimiters and no quote characters.

Ernie

PilotBaha · Post by **PilotBaha** » Tue Jan 06, 2009 2:56 pm

Ernie,
there was a table def under Table Definitions -> Real Time -> Websphere MQ Connector named MQ message. I have a feeling like this is added under table def once the MQ connector is installed. It worked fine for me. As a matter of fact it had way more information than i needed.

Thanks,

Baha

eostic · Post by **eostic** » Tue Jan 06, 2009 8:21 pm

Yes...of course..that's how it works...."if" you need it. Your problem above doesn't need any (descriptor metadata).....I thought you were just trying to get your payload to a file.

Ernie

eostic · Post by **eostic** » Tue Jan 06, 2009 8:23 pm

Maybe I should be more specific...."Descriptor" metadata has nothing to do with the layout of your payload/message. It defines things like the message identifier, the put-time and put-date, etc. So, for the moment, it (all the metadata in the supplied table definition) is not meaningful to your task as you've outlined above.

Ernie

PilotBaha · Post by **PilotBaha** » Wed Jan 07, 2009 2:08 pm

Ernie,
the descriptor information might be needed for reporting purposes. you are correct that i needed the message load to be transferred and that's been accomplished now.
thanks for the help..

Baha