How to migrate a DS-Basic Job from SERVER to PARALLEL?

[Kerem2222]

Hello,

I am involved in a process of porting a system containing several DS-Jobs
from DataStage V7.5 Server Edition running on AIX 6.1
to DataStage V8.5 Parallel Edition running on SUSE-Linux.

Migrating the DS-Jobs was running good so far, but I have now serious problems with a DS-Job written in DataStage Basic. This Job is a very old one (more than 10 years old). This Job reads out a binary-coded complex-flat file into csv-files. I suppose, that at the time when the job was designed, the complex-flat-file stage was not yet existing in DataStage. But now I don't want to re-write a new job using the complex-flat-file stage, because the file seems to be really complex and time-consuming.
As far as I know, it had made never problems so far, when upgrading DataStage to a newer version within SERVER EDITION, but now it does when for the first time migrating to PARALLEL EDITION.

The problem starts at the very beginning, reading header information but returning only broken strange looking characters.

My questions are:

• - Is DataStage Basic compatible between DS V7.5 SERVER Edition and DS V8.5 PARALLEL Edition?
  - Due to the strange looking characters: Could it be a character set problem and is there a way to fix it?
  - Besides the DS-Upgrade: Could it be a problem, that the operating system and hardware/server also changed, named from AIX to SUSE-Linux?

Thanks in advance

[ray.wurlod]

Does your server job still work? If so, keep it. Server jobs can be run even though you have Enterprise Edition.

[Kerem2222]

Yes, the old job still works on the old currently still active AIX-server with DS V7.5 server edition.

• - on the new server the job can be compiled with no error
  - anyway I exported the DS-Job from the old AIX-server and imported it to the new Linux-server a several times. But it doesn' t run correctly
  - I also re-checked and re-transfered the binary testfile a several times from the old AIX-server to the new Linux-server, but although everything seems to be fine, the file can't be read by the ds-job
  - under "Job Properties" there is a new window-tab under V8.5 Parallel Edition called "NLS" which I don't have under V7.5 Server Edition on the old server. How can I find out, what settings I have to set to read-out my binary-file correctly?

UPDATE: when just trying to build a new job with the complex-flat-file stage I saw, that on the old server I have EBCDIC available, but on the new server NLS instead. Is there a way to get EBCDIC or which NLS-setting is according to that?

Thank you

[sjfearnside]

I remember in the past when we switched from Linux to AIX that we had problems with some of the applications. If I remember correctly it had to do with the difference in endianess. Check into this to see if you are having this condition. There is plenty of information on the internet about the difference in endianess.

In big endian, you store the most significant byte in the smallest address. In little endian, you store the least significant byte in the smallest address.

[Kerem2222]

Hi sjfearnside,

thank you for your answer, I just read a litlle about "byte-Order" / "Endianness" in the internet. As far as I understand it, this could just be a problem when transfering data from one hardware to another using different endianness. In my case the transfered binary file was tested and so has for sure no errors. So I think this is the wrong track.

Greetings from Cologne, Germany

[asorrell]

It sounds like your old system did not have NLS (National Language Support) enabled and your new system does. Unless you are fortunate enough to receive data that is 100% clean text, that can have lots of side effects.

Before NLS is enabled, DataStage in many regards ignores characters it doesn't recognize and just passes them through. After NLS is enabled, DataStage automatically applies a character map and it will issue warnings that the character doesn't fit in the current character map. Depending on the situation, it may map that character to an alternate character.

This is a fairly big topic, it is documented in the IBM InfoSphere DataStage and QualityStage Globalization Guide.

With that said, I'm not certain that NLS is the actual root of your problem. As far as I know, NLS does not apply to binary (raw) data.

In your first post - you said that the job was written in DataStage BASIC. Did you really mean that it incorporates BASIC code? Or is it just a Server job using standard stages?

If it is really using BASIC, please post the code. If not, please put in more details about how it actually was written. Specifically, the stage that reads the data and its settings.

[asorrell]

Ok - my German isn't that great, but I got the basics of what your are doing (pun intended).

It turns out NLS may be your problem after all.

I did some research and found this in the BASIC guide under NLS:

In NLS mode all data in held in the InfoSphere DataStage internal character set. In all InfoSphere DataStage I/O operations, data is converted automatically by applying the map specified for a file or a device. One character can be more than one byte long and can occupy zero or more positions on the screen.

I'm not certain that setting NLS map to "NONE" will resolve this, but its the only thing I can think of at this point. Try that and let us know if it works.

There are several of our overseas experts that are more familiar with NLS issues, maybe they can assist now that the actual problem is a bit clearer.

[Kerem2222]

Good Morning asorrell,

unfortunately setting "Map name to use with stage" in the register NLS to "NONE" was not the solution. I get still that broken strange looking letters showed when I look at "view data".

Regards

[ray.wurlod]

Sometimes the View Data utility is the problem. Can you try looking at the file using some other utility such as a UNIX command?

[Kerem2222]

On the new Linux-Server, we are running a C-written job, which checks the binary file, before starting with DataStage. We recompiled that job on Linux (because it was running on AIX before) and the result is clear, the binary file has now errors.

[Kerem2222]

I am quite sure it is a coding problem. As far as I know, the binary InputFile is coded in EBCDIC.

I want to contact my DataStage-admin and ask him to install EBCDIC code-mapping.
What precisely should I tell my admin to do?
Where could my admin get the EBCDIC coding-maps from?

Thanks

[asorrell]

EBCIDIC is an eight bit character language-independent coding set used by ancient IBM equipment. I'm not certain if this is really the problem, but there are code sets called CCSID50 and UTF-EBCIDIC that are EBCIDIC variants - check for maps under those names.

[Kerem2222]

Hello,

here is a brief update about the issue discussed:
- the problem is not solved yet
- the DS-Admin loaded all 37 EBCDIC-codepages available on the Linux-server, but "brute-force" trial-and-error all of them brought no final result
- one of the admin of the BS2000 Siemens mainframe (where the inputfiles are generated) checked the sources and told us, that the files are coded in EBCDIC.DF.03 (CCSN:EDF03IRV), which is not one of the 37 mentioned before
- checking the inputfile in hexadecimal by the command "od -ax" on the new Linux-server and on the old AIX-server returns differnet results
- the inputfile on the new Linux-server is definitely not corrupt, because when transfering it back to cologne it works on the old AIX-server

Questions:
- The test-result of "od- ax" seems to me, that there is a problem even on Linux itself, meaning before starting DataStage? Could this be? What can I do?
- What hazards are possible, if I ask the mainframe guys to change the EBCDIC-codepage used to one of the available on the new Linux-server?
- As the current used EBCDIC-codepages makes no problems on the old AIX-server: How can I check the settings on AIX and DataStage?
=> It is really curious to see the job running on the old DataStage 7.5 server edition, but not on the new further developed DataStage 8.5. I can believe that certain EBCDIC-codepages are not supported any more. There has to be an easy solution!?

Thanks to everybody from Cologne

[priyadarshikunal]

can you compare the locale identified as LANG, or LC_XXXX (where X can be any character such as LC_ALL) on both systems.

You can check it by running env (in linux not sure about AIX) along with that you should check it in dsenv.

[Kerem2222]

AIX:

Code: Select all

uadecgn1013:/home1/ardent/shell/rc $ set | grep -i lang
LANG=en_US

Linux:

Code: Select all

se1089@z1tl0124:~> env | grep -i lang
LANG=C
se1089@z1tl0124:~> env | grep -i lc_
LC_ALL=C

How can I change the setting on Linux to "en_US"?