Convert Issue - Non printable characters ÿ and ›

Post questions here relative to DataStage Enterprise/PX Edition for such areas as Parallel job design, Parallel datasets, BuildOps, Wrappers, etc.

Moderators: chulett, rschirm, roy

Post Reply
chulett
Charter Member
Charter Member
Posts: 43085
Joined: Tue Nov 12, 2002 4:34 pm
Location: Denver, CO

Post by chulett »

How did you determine their values?

I'd also suggest you review the syntax for the convert function. You need two spaces in your new.list or only the first character will be replaced. Right now the second character will be removed from the target string.
-craig

"You can never have too many knives" -- Logan Nine Fingers
sandhya.budhi
Premium Member
Premium Member
Posts: 18
Joined: Wed Nov 15, 2017 10:50 am

Post by sandhya.budhi »

I referred the values from the Extended ASCII table.

https://www.ascii-code.com/

I have noticed the values were different in some websites. Is there any way I can validate with any other reference?
Thanks,
Sandhya
UCDI
Premium Member
Premium Member
Posts: 383
Joined: Mon Mar 21, 2016 2:00 pm

Post by UCDI »

validate it with a Visual basic transformer, just convert each byte of the string to its numeric format with | between.
chulett
Charter Member
Charter Member
Posts: 43085
Joined: Tue Nov 12, 2002 4:34 pm
Location: Denver, CO

Post by chulett »

Was worried about that... it's nothing you should be doing visually. If you can, use an "octal dump" on the server to see what those values actually are:

Decimal dump
od -d <filename>

Hex dump
od -x <filename>
-craig

"You can never have too many knives" -- Logan Nine Fingers
ray.wurlod
Participant
Posts: 54607
Joined: Wed Oct 23, 2002 10:52 pm
Location: Sydney, Australia
Contact:

Post by ray.wurlod »

UCDI wrote:validate it with a Visual basic transformer
You've posted on a number of occasions about a Visual basic transformer. What is this? I currently have projects in versions 8.0.1 through 11.7.0.1 and can find no reference to a stage of this name.
IBM Software Services Group
Any contribution to this forum is my own opinion and does not necessarily reflect any position that IBM may hold.
UCDI
Premium Member
Premium Member
Posts: 383
Joined: Mon Mar 21, 2016 2:00 pm

Post by UCDI »

Heh... about that...

Look in your repository under stage types. There are a number of stages in there that are not in the palette. Its under Parallel/Processing/BasicTransformer

Be careful with it. I know for sure that it cannot handle timestamps with microseconds, it loses the extended data. I don't know what else it can damage in this way... I generally pipe only a key & the offending data into it, invoke the routine, and rejoin it back, just for extra safety. I don't know why they won't clean this up or roll the routine calling capability into the standard transform.

Ive used this I think 3, maybe 4 times total in as many years. It feels like a questionable tool, but its too powerful to ignore for a small handful of issues. Remember also that I have been blocked from writing server jobs via a policy, so this may be a work around for oddball string processing that could have been done server side. /shrug. The VB transformer is the compromise I worked out with our powers that be between C (they disliked this a lot) and server jobs (requires too many hoops to be allowed into jobs) for doing the occasional hands-on string processing / cleanup / reformatting/ etc type work.

** I apparently have a hangup about 'visual' and basic. The word visual is only in my own mind, and not anywhere in datastage. Sorry about that.
ray.wurlod
Participant
Posts: 54607
Joined: Wed Oct 23, 2002 10:52 pm
Location: Sydney, Australia
Contact:

Post by ray.wurlod »

Very good. The BASIC Transformer stage was included in the product to make the transition from server jobs to parallel jobs a bit easier. I use it fairly regularly. I use server jobs a LOT. I use server components in server Shared Containers in parallel jobs.

Since the BASIC Transformer stage deals only with strings, it can handle timestamps with microseconds quite happily, as strings of 26 characters.
IBM Software Services Group
Any contribution to this forum is my own opinion and does not necessarily reflect any position that IBM may hold.
rameshrr3
Premium Member
Premium Member
Posts: 609
Joined: Mon May 10, 2004 3:32 am
Location: BRENTWOOD, TN

Post by rameshrr3 »

Pity that Server Jobs and Basic transformer are nowadays frequently considered 'illicit' workarounds in many places, despite their unique and powerful capabilities, and despite the fact that basic transformer is capable of running on multiple nodes on an SMP system.
Post Reply