Elusive Missing Data Fields

Archive of postings to DataStageUsers@Oliver.com. This forum intended only as a reference and cannot be posted to.

Moderators: chulett, rschirm

Locked
admin
Posts: 8720
Joined: Sun Jan 12, 2003 11:26 pm

Elusive Missing Data Fields

Post by admin »

We have a job that reads a simple, fixed-length, sequential file.
When we view the file in a text editor, data shows up in a certain field. BUT - when we view the file inside the sequential file stage in the designer, SOMETIMES the data in that same field does not appear (and it's also not processed, so we know that datastage is not finding data in the field). The field is defined as character length 6. To make matters even more interesting, it is a very random occurrence, even on the field in question.
The input field has been examined for patterns, hidden characters, control characters, etc., and nothing unusual was found.
Does anybody have a clue what the problem is??

John & Lindi



*****E-Mail Confidentiality Notice*****
This message (including any attachments) contains information intended for a specific individual(s) and purpose that may be privileged, confidential or otherwise protected from disclosure pursuant to applicable law. Any inappropriate use, distribution or copying of the message is strictly prohibited and may subject you to criminal or civil penalty. If you have received this transmission in error, please reply to the sender indicating this error and delete the transmission from your system immediately.


_______________________________________________
datastage-users mailing list
datastage-users@oliver.com
http://www.oliver.com/mailman/listinfo/datastage-users
<b>PLEASE READ</b>
Do not contact admin unless you have technical support or account questions. Do not send email or Private Messages about discussion topics to ADMIN. Contact the webmaster concerning abusive or offensive posts.
admin
Posts: 8720
Joined: Sun Jan 12, 2003 11:26 pm

Elusive Missing Data Fields

Post by admin »

I've had 2 problems that sound similar.... so not sure if they're
related or not:

1) I've had times where my field sizes are all set up correctly, but
my DISPLAY sizes are not. Although i don't think that should have impact
on processing, it's actually caused my programs to NOT work correctly and
mess up my data.
2) I've had data on my output file, but when i go to "View" it
within DataStage, it shows as blank. From within the "View", i click on
the "Display" button, see that everything is checked, "OK" out of that, and
then my data suddenly appears.

Seem like some little "fluky" bugs in the programs. Hope that helps.
:> We're using version 6.1 here.


Eric
----------
I guess one of the funniest memories of my grandfather was the time i
was at his house and that tied up man with the gag in his mouth came
hopping out of the closet and started yelling that HE was really my
grandfather and the other guy was an imposter and to run for help. Who WAS
that guy?! Oh well... we never saw HIM again.





"Lindi Reichel"
To:
Sent by: cc:
datastage-users-bounces Subject: Elusive Missing Data Fields
@oliver.com


03/03/2004 11:03 AM
Please respond to
"DataStage Users
Discussion List"






We have a job that reads a simple, fixed-length, sequential file.
When we view the file in a text editor, data shows up in a certain field.
BUT - when we view the file inside the sequential file stage in the
designer, SOMETIMES the data in that same field does not appear (and it's
also not processed, so we know that datastage is not finding data in the
field). The field is defined as character length 6. To make matters even
more interesting, it is a very random occurrence, even on the field in
question.
The input field has been examined for patterns, hidden characters,
control characters, etc., and nothing unusual was found.
Does anybody have a clue what the problem is??

John & Lindi



*****E-Mail Confidentiality Notice*****
This message (including any attachments) contains information intended for
a specific individual(s) and purpose that may be privileged, confidential
or otherwise protected from disclosure pursuant to applicable law. Any
inappropriate use, distribution or copying of the message is strictly
prohibited and may subject you to criminal or civil penalty. If you have
received this transmission in error, please reply to the sender indicating
this error and delete the transmission from your system immediately.


_______________________________________________
datastage-users mailing list
datastage-users@oliver.com
http://www.oliver.com/mailman/listinfo/datastage-users






_______________________________________________
datastage-users mailing list
datastage-users@oliver.com
http://www.oliver.com/mailman/listinfo/datastage-users
<b>PLEASE READ</b>
Do not contact admin unless you have technical support or account questions. Do not send email or Private Messages about discussion topics to ADMIN. Contact the webmaster concerning abusive or offensive posts.
admin
Posts: 8720
Joined: Sun Jan 12, 2003 11:26 pm

Elusive Missing Data Fields

Post by admin »

Thanks for the ideas, Eric. We tried them both - I wish I could tell you that one of them solved my problem!
Your suggestions did get us looking around more in the field setups, though, and after three hours of getting nowhere, we decided to change the job to define each entire record as one huge field. When we did this..... all of the records were BLANK in the designer view!! That led me to believe that somewhere in the records there must have been some problem with nulls or spaces or some sort of incomplete data, so I went into the "columns" tab on the sequential stage and changed the "Incomplete columns" value from "replace" to "retain".
Voila - all the data showed up.

Thanks again for your help!
Lindi

*****E-Mail Confidentiality Notice*****
This message (including any attachments) contains information intended for a specific individual(s) and purpose that may be privileged, confidential or otherwise protected from disclosure pursuant to applicable law. Any inappropriate use, distribution or copying of the message is strictly prohibited and may subject you to criminal or civil penalty. If you have received this transmission in error, please reply to the sender indicating this error and delete the transmission from your system immediately.


>>> Eric.Roufs@stpaul.com 03/03/04 01:11PM >>>

I've had 2 problems that sound similar.... so not sure if they're
related or not:

1) I've had times where my field sizes are all set up correctly, but
my DISPLAY sizes are not. Although i don't think that should have impact
on processing, it's actually caused my programs to NOT work correctly and
mess up my data.
2) I've had data on my output file, but when i go to "View" it
within DataStage, it shows as blank. From within the "View", i click on
the "Display" button, see that everything is checked, "OK" out of that, and
then my data suddenly appears.

Seem like some little "fluky" bugs in the programs. Hope that helps.
:> We're using version 6.1 here.


Eric
----------
I guess one of the funniest memories of my grandfather was the time i
was at his house and that tied up man with the gag in his mouth came
hopping out of the closet and started yelling that HE was really my
grandfather and the other guy was an imposter and to run for help. Who WAS
that guy?! Oh well... we never saw HIM again.





"Lindi Reichel"
To:
Sent by: cc:
datastage-users-bounces Subject: Elusive Missing Data Fields
@oliver.com


03/03/2004 11:03 AM
Please respond to
"DataStage Users
Discussion List"






We have a job that reads a simple, fixed-length, sequential file.
When we view the file in a text editor, data shows up in a certain field.
BUT - when we view the file inside the sequential file stage in the
designer, SOMETIMES the data in that same field does not appear (and it's
also not processed, so we know that datastage is not finding data in the
field). The field is defined as character length 6. To make matters even
more interesting, it is a very random occurrence, even on the field in
question.
The input field has been examined for patterns, hidden characters,
control characters, etc., and nothing unusual was found.
Does anybody have a clue what the problem is??

John & Lindi



*****E-Mail Confidentiality Notice*****
This message (including any attachments) contains information intended for
a specific individual(s) and purpose that may be privileged, confidential
or otherwise protected from disclosure pursuant to applicable law. Any
inappropriate use, distribution or copying of the message is strictly
prohibited and may subject you to criminal or civil penalty. If you have
received this transmission in error, please reply to the sender indicating
this error and delete the transmission from your system immediately.


_______________________________________________
datastage-users mailing list
datastage-users@oliver.com
http://www.oliver.com/mailman/listinfo/datastage-users






_______________________________________________
datastage-users mailing list
datastage-users@oliver.com
http://www.oliver.com/mailman/listinfo/datastage-users
_______________________________________________
datastage-users mailing list
datastage-users@oliver.com
http://www.oliver.com/mailman/listinfo/datastage-users
<b>PLEASE READ</b>
Do not contact admin unless you have technical support or account questions. Do not send email or Private Messages about discussion topics to ADMIN. Contact the webmaster concerning abusive or offensive posts.
admin
Posts: 8720
Joined: Sun Jan 12, 2003 11:26 pm

Elusive Missing Data Fields

Post by admin »

Hi Lindi,

the "VIEW DATA" is a help visualation tool. The real relevant part to look
at is the output. It is known, that with VIEW DATA you sometimes don't see
things which are in reality present, we know this and we respect this.

(1) So use an editor for to check you input and your output data, if then
something is missing: continue examination.

(2) Read the fields of sequential file preferaly all as Character and change
them later into dates, numbers or whatever you want. By this you have more
control.

(3) Check if decimal delimiter is set correctly. A number 12 scale 5 means
that you have 12 numbers 8 before decimal delimiter and 5 after, but the
total display size has to be 14 one additional for the decimal delimiter and
one for the prefix (+/-).

(4) Another source of trouble is field seperator (if you don't have fix
length), do field separators reappear within fields, if yes field can't be
assigned correctly.

(5) Characters with strange behaviour are not only between ASCII(0) and
ASCII(31) with some high values the VIEW DATA has also problems.

(6) Control the line end characters are their, what you want UNIX-Style or
DOS-Style are different, strange is sometimes the double-Style if you have
done a wrong filetransfer.

(7) Variable number of fields in an delimited file or varible record length
may also result into a problem.

IN ANY CASE: If you know, what additional specifications your input file
has, you are able to solve the problem with DataStage.

So go on with further investigation and let us know what you found out.

Wolfgang



-----Original Message-----
From: Lindi Reichel [mailto:lreichel@psu.edu]
Sent: Wednesday, March 03, 2004 9:30 PM
To: datastage-users@oliver.com
Subject: Re: Elusive Missing Data Fields


Thanks for the ideas, Eric. We tried them both - I wish I could tell you
that one of them solved my problem!
Your suggestions did get us looking around more in the field setups,
though, and after three hours of getting nowhere, we decided to change the
job to define each entire record as one huge field. When we did this.....
all of the records were BLANK in the designer view!! That led me to
believe that somewhere in the records there must have been some problem with
nulls or spaces or some sort of incomplete data, so I went into the
"columns" tab on the sequential stage and changed the "Incomplete columns"
value from "replace" to "retain".
Voila - all the data showed up.

Thanks again for your help!
Lindi

*****E-Mail Confidentiality Notice*****
This message (including any attachments) contains information intended for a
specific individual(s) and purpose that may be privileged, confidential or
otherwise protected from disclosure pursuant to applicable law. Any
inappropriate use, distribution or copying of the message is strictly
prohibited and may subject you to criminal or civil penalty. If you have
received this transmission in error, please reply to the sender indicating
this error and delete the transmission from your system immediately.


>>> Eric.Roufs@stpaul.com 03/03/04 01:11PM >>>

I've had 2 problems that sound similar.... so not sure if they're
related or not:

1) I've had times where my field sizes are all set up correctly, but
my DISPLAY sizes are not. Although i don't think that should have impact
on processing, it's actually caused my programs to NOT work correctly and
mess up my data.
2) I've had data on my output file, but when i go to "View" it
within DataStage, it shows as blank. From within the "View", i click on
the "Display" button, see that everything is checked, "OK" out of that, and
then my data suddenly appears.

Seem like some little "fluky" bugs in the programs. Hope that helps.
:> We're using version 6.1 here.


Eric
----------
I guess one of the funniest memories of my grandfather was the time i
was at his house and that tied up man with the gag in his mouth came
hopping out of the closet and started yelling that HE was really my
grandfather and the other guy was an imposter and to run for help. Who WAS
that guy?! Oh well... we never saw HIM again.






"Lindi Reichel"

To:

Sent by: cc:

datastage-users-bounces Subject: Elusive
Missing Data Fields
@oliver.com





03/03/2004 11:03 AM

Please respond to

"DataStage Users

Discussion List"









We have a job that reads a simple, fixed-length, sequential file.
When we view the file in a text editor, data shows up in a certain field.
BUT - when we view the file inside the sequential file stage in the
designer, SOMETIMES the data in that same field does not appear (and it's
also not processed, so we know that datastage is not finding data in the
field). The field is defined as character length 6. To make matters even
more interesting, it is a very random occurrence, even on the field in
question.
The input field has been examined for patterns, hidden characters,
control characters, etc., and nothing unusual was found.
Does anybody have a clue what the problem is??

John & Lindi



*****E-Mail Confidentiality Notice*****
This message (including any attachments) contains information intended for
a specific individual(s) and purpose that may be privileged, confidential
or otherwise protected from disclosure pursuant to applicable law. Any
inappropriate use, distribution or copying of the message is strictly
prohibited and may subject you to criminal or civil penalty. If you have
received this transmission in error, please reply to the sender indicating
this error and delete the transmission from your system immediately.


_______________________________________________
datastage-users mailing list
datastage-users@oliver.com
http://www.oliver.com/mailman/listinfo/datastage-users






_______________________________________________
datastage-users mailing list
datastage-users@oliver.com
http://www.oliver.com/mailman/listinfo/datastage-users
_______________________________________________
datastage-users mailing list
datastage-users@oliver.com
http://www.oliver.com/mailman/listinfo/datastage-users
_______________________________________________
datastage-users mailing list
datastage-users@oliver.com
http://www.oliver.com/mailman/listinfo/datastage-users
<b>PLEASE READ</b>
Do not contact admin unless you have technical support or account questions. Do not send email or Private Messages about discussion topics to ADMIN. Contact the webmaster concerning abusive or offensive posts.
admin
Posts: 8720
Joined: Sun Jan 12, 2003 11:26 pm

Elusive Missing Data Fields

Post by admin »

Wow - thanks for the checklist!
It turns out in our case that every time a field value was lost, it was a value of length 5 in a field of length 6. While we couldn't see what the extra character was in those fields (and we tried everything!!), the problem obviously revolved around that issue. (Although EVERY field of length 5 was not affected - only some were lost...)
By changing the "incomplete columns" value from "replace" to retain", we were able to see (an pass on)our data.
Thanks again for your help!
Lindi

*****E-Mail Confidentiality Notice*****
This message (including any attachments) contains information intended for a specific individual(s) and purpose that may be privileged, confidential or otherwise protected from disclosure pursuant to applicable law. Any inappropriate use, distribution or copying of the message is strictly prohibited and may subject you to criminal or civil penalty. If you have received this transmission in error, please reply to the sender indicating this error and delete the transmission from your system immediately.


>>> wolfgang.huerter@gerling.de 03/04/04 01:55AM >>>
Hi Lindi,

the "VIEW DATA" is a help visualation tool. The real relevant part to look
at is the output. It is known, that with VIEW DATA you sometimes don't see
things which are in reality present, we know this and we respect this.

(1) So use an editor for to check you input and your output data, if then
something is missing: continue examination.

(2) Read the fields of sequential file preferaly all as Character and change
them later into dates, numbers or whatever you want. By this you have more
control.

(3) Check if decimal delimiter is set correctly. A number 12 scale 5 means
that you have 12 numbers 8 before decimal delimiter and 5 after, but the
total display size has to be 14 one additional for the decimal delimiter and
one for the prefix (+/-).

(4) Another source of trouble is field seperator (if you don't have fix
length), do field separators reappear within fields, if yes field can't be
assigned correctly.

(5) Characters with strange behaviour are not only between ASCII(0) and
ASCII(31) with some high values the VIEW DATA has also problems.

(6) Control the line end characters are their, what you want UNIX-Style or
DOS-Style are different, strange is sometimes the double-Style if you have
done a wrong filetransfer.

(7) Variable number of fields in an delimited file or varible record length
may also result into a problem.

IN ANY CASE: If you know, what additional specifications your input file
has, you are able to solve the problem with DataStage.

So go on with further investigation and let us know what you found out.

Wolfgang



-----Original Message-----
From: Lindi Reichel [mailto:lreichel@psu.edu]
Sent: Wednesday, March 03, 2004 9:30 PM
To: datastage-users@oliver.com
Subject: Re: Elusive Missing Data Fields


Thanks for the ideas, Eric. We tried them both - I wish I could tell you
that one of them solved my problem!
Your suggestions did get us looking around more in the field setups,
though, and after three hours of getting nowhere, we decided to change the
job to define each entire record as one huge field. When we did this.....
all of the records were BLANK in the designer view!! That led me to
believe that somewhere in the records there must have been some problem with
nulls or spaces or some sort of incomplete data, so I went into the
"columns" tab on the sequential stage and changed the "Incomplete columns"
value from "replace" to "retain".
Voila - all the data showed up.

Thanks again for your help!
Lindi

*****E-Mail Confidentiality Notice*****
This message (including any attachments) contains information intended for a
specific individual(s) and purpose that may be privileged, confidential or
otherwise protected from disclosure pursuant to applicable law. Any
inappropriate use, distribution or copying of the message is strictly
prohibited and may subject you to criminal or civil penalty. If you have
received this transmission in error, please reply to the sender indicating
this error and delete the transmission from your system immediately.


>>> Eric.Roufs@stpaul.com 03/03/04 01:11PM >>>

I've had 2 problems that sound similar.... so not sure if they're
related or not:

1) I've had times where my field sizes are all set up correctly, but
my DISPLAY sizes are not. Although i don't think that should have impact
on processing, it's actually caused my programs to NOT work correctly and
mess up my data.
2) I've had data on my output file, but when i go to "View" it
within DataStage, it shows as blank. From within the "View", i click on
the "Display" button, see that everything is checked, "OK" out of that, and
then my data suddenly appears.

Seem like some little "fluky" bugs in the programs. Hope that helps.
:> We're using version 6.1 here.


Eric
----------
I guess one of the funniest memories of my grandfather was the time i
was at his house and that tied up man with the gag in his mouth came
hopping out of the closet and started yelling that HE was really my
grandfather and the other guy was an imposter and to run for help. Who WAS
that guy?! Oh well... we never saw HIM again.






"Lindi Reichel"

To:

Sent by: cc:

datastage-users-bounces Subject: Elusive
Missing Data Fields
@oliver.com





03/03/2004 11:03 AM

Please respond to

"DataStage Users

Discussion List"









We have a job that reads a simple, fixed-length, sequential file.
When we view the file in a text editor, data shows up in a certain field.
BUT - when we view the file inside the sequential file stage in the
designer, SOMETIMES the data in that same field does not appear (and it's
also not processed, so we know that datastage is not finding data in the
field). The field is defined as character length 6. To make matters even
more interesting, it is a very random occurrence, even on the field in
question.
The input field has been examined for patterns, hidden characters,
control characters, etc., and nothing unusual was found.
Does anybody have a clue what the problem is??

John & Lindi



*****E-Mail Confidentiality Notice*****
This message (including any attachments) contains information intended for
a specific individual(s) and purpose that may be privileged, confidential
or otherwise protected from disclosure pursuant to applicable law. Any
inappropriate use, distribution or copying of the message is strictly
prohibited and may subject you to criminal or civil penalty. If you have
received this transmission in error, please reply to the sender indicating
this error and delete the transmission from your system immediately.


_______________________________________________
datastage-users mailing list
datastage-users@oliver.com
http://www.oliver.com/mailman/listinfo/datastage-users






_______________________________________________
datastage-users mailing list
datastage-users@oliver.com
http://www.oliver.com/mailman/listinfo/datastage-users
_______________________________________________
datastage-users mailing list
datastage-users@oliver.com
http://www.oliver.com/mailman/listinfo/datastage-users
_______________________________________________
datastage-users mailing list
datastage-users@oliver.com
http://www.oliver.com/mailman/listinfo/datastage-users
_______________________________________________
datastage-users mailing list
datastage-users@oliver.com
http://www.oliver.com/mailman/listinfo/datastage-users
<b>PLEASE READ</b>
Do not contact admin unless you have technical support or account questions. Do not send email or Private Messages about discussion topics to ADMIN. Contact the webmaster concerning abusive or offensive posts.
Locked