DSXchange: DataStage and IBM Websphere Data Integration Forum
View next topic
View previous topic
Add To Favorites
Author Message
goutam



Group memberships:
Premium Members

Joined: 26 Jul 2007
Posts: 109

Points: 1057

Post Posted: Mon Feb 11, 2019 1:23 am Reply with quote    Back to top    

DataStage® Release: 11x
Job Type: Parallel
OS: Unix
HI All,

I have an input column which holds arithmetic expressions e.g. [(20*5)+9], [(30/5)+4]. I need to evaluate them and send the final value to target system. Any way to achieve this ?

_________________
Goutam Sahoo
qt_ky



Group memberships:
Premium Members

Joined: 03 Aug 2011
Posts: 2862
Location: USA
Points: 21759

Post Posted: Tue Feb 12, 2019 1:17 am Reply with quote    Back to top    

Seems like an unusual requirement...

The first thing that comes to mind is to take the expression and put in into a UNIX Korn shell for evaluation inside of the $(( )) syntax:

Code:
$ echo $((4+5))
9

$ echo $(((20*5)+9))
109

$ echo $(((30/5)+4))
10

$ echo $((7 / 0))
ksh: 7 / 0: 0403-056 Cannot divide by zero.

$ echo $((4 m 3))
ksh: 4 m 3: 0403-057 Syntax error

_________________
Choose a job you love, and you will never have to work a day in your life. - Confucius
Rate this response:  
Not yet rated
chulett

Premium Poster


since January 2006

Group memberships:
Premium Members, Inner Circle, Server to Parallel Transition Group

Joined: 12 Nov 2002
Posts: 42985
Location: Denver, CO
Points: 221730

Post Posted: Tue Feb 12, 2019 2:45 am Reply with quote    Back to top    

Or perhaps "bc" at the command line? Or something using C++?

https://stackoverflow.com/questions/3422673/how-to-evaluate-a-math-expression-given-in-string-form

In any case... not fun. Leverage the work of others.

_________________
-craig

Help I'm steppin' into the twilight zone, place is a madhouse, feels like being cold
My beacon's been moved under moon and star, where am I to go now that I've gone too far?
Rate this response:  
Not yet rated
qt_ky



Group memberships:
Premium Members

Joined: 03 Aug 2011
Posts: 2862
Location: USA
Points: 21759

Post Posted: Tue Feb 12, 2019 8:49 am Reply with quote    Back to top    

I ran a quick test file with expressions through an External Filter stage with 'bc' as the filter command and got correctly evaluated outputs.

_________________
Choose a job you love, and you will never have to work a day in your life. - Confucius
Rate this response:  
Not yet rated
chulett

Premium Poster


since January 2006

Group memberships:
Premium Members, Inner Circle, Server to Parallel Transition Group

Joined: 12 Nov 2002
Posts: 42985
Location: Denver, CO
Points: 221730

Post Posted: Tue Feb 12, 2019 5:52 pm Reply with quote    Back to top    

Sweet.

Question for Goutam, what kind of processing volume are we talking about here?

_________________
-craig

Help I'm steppin' into the twilight zone, place is a madhouse, feels like being cold
My beacon's been moved under moon and star, where am I to go now that I've gone too far?
Rate this response:  
Not yet rated
goutam



Group memberships:
Premium Members

Joined: 26 Jul 2007
Posts: 109

Points: 1057

Post Posted: Wed Feb 13, 2019 12:16 am Reply with quote    Back to top    

HI Chulett, Thanks for your response. Actaully we are getting 30k records on a daily basis and there are 20 fields like this which we need to evaluate..

_________________
Goutam Sahoo
Rate this response:  
Not yet rated
goutam



Group memberships:
Premium Members

Joined: 26 Jul 2007
Posts: 109

Points: 1057

Post Posted: Wed Feb 13, 2019 12:24 am Reply with quote    Back to top    

Thanks a lot, qt_ky. If i need to evaluate multiple columns having arithmetic expressions then do i need to use that many no of external filter stages?

_________________
Goutam Sahoo
Rate this response:  
Not yet rated
qt_ky



Group memberships:
Premium Members

Joined: 03 Aug 2011
Posts: 2862
Location: USA
Points: 21759

Post Posted: Wed Feb 13, 2019 1:31 am Reply with quote    Back to top    

That I don't know. Sounds like a custom routine may be a better fit.

_________________
Choose a job you love, and you will never have to work a day in your life. - Confucius
Rate this response:  
Not yet rated
JRodriguez



Group memberships:
Premium Members

Joined: 19 Nov 2005
Posts: 420
Location: New York City
Points: 4707

Post Posted: Wed Feb 13, 2019 1:43 am Reply with quote    Back to top    

Play around to force the evaluation of the string input as arithmetic by using the AsInteger, AsDouble, AsFloat functions. Just pass the input column to one of these and that might do the trick... You might need to enforce the result by multiplying by a constant factor like below....just play around


AsFloat(input arithmetic) * 1


I wonder if in your site the use of Server jobs is allowed? In the old server job any string arithmetic expression will be evaluated as arithmetic plain and simple. You would only need the input column as a derivation to the target column.

Let us know how it goes

_________________
Julio Rodriguez
ETL Developer by choice

"Sure we have lots of reasons for being rude - But no excuses
Rate this response:  
Not yet rated
UCDI



Group memberships:
Premium Members

Joined: 21 Mar 2016
Posts: 373

Points: 3853

Post Posted: Wed Feb 13, 2019 4:38 am Reply with quote    Back to top    

are they all of the same format, like A operator B operator C?
Rate this response:  
Not yet rated
goutam



Group memberships:
Premium Members

Joined: 26 Jul 2007
Posts: 109

Points: 1057

Post Posted: Wed Feb 13, 2019 10:48 pm Reply with quote    Back to top    

No..it varies..Somewhere we have one operator and somewhere it's more than one..max no of operator is 5.

_________________
Goutam Sahoo
Rate this response:  
Not yet rated
ray.wurlod

Premium Poster
Participant

Group memberships:
Premium Members, Inner Circle, Australia Usergroup, Server to Parallel Transition Group

Joined: 23 Oct 2002
Posts: 54524
Location: Sydney, Australia
Points: 295662

Post Posted: Thu Feb 14, 2019 4:13 pm Reply with quote    Back to top    

Somewhat unrelated. Recently I was installing 11.7 IIS with Enterprise Search, and the delivered Linux server did not have bc installed. That actually caused the installation to fail. Checking ...

_________________
IBM Software Services Group
Any contribution to this forum is my own opinion and does not necessarily reflect any position that IBM may hold.
Rate this response:  
Not yet rated
chulett

Premium Poster


since January 2006

Group memberships:
Premium Members, Inner Circle, Server to Parallel Transition Group

Joined: 12 Nov 2002
Posts: 42985
Location: Denver, CO
Points: 221730

Post Posted: Fri Feb 15, 2019 2:52 am Reply with quote    Back to top    

Well, haven't put much thought into this to be honest but you may want to take a shot at seeing how shelling out to use "bc" on each row will work for you. 30K isn't really that large of a volume but the context switching or whatever overhead is added by leveraging something on the command line may or may not be acceptable to your load timings. But seems to me should be the easiest to test. If the overhead is too much then look into something more tightly coupled into the parallel framework, something custom in C++ for example.

What IS your source, by the way? You've not said. Flat file? If so then maybe you could work out a "filter" that would do that for you.

_________________
-craig

Help I'm steppin' into the twilight zone, place is a madhouse, feels like being cold
My beacon's been moved under moon and star, where am I to go now that I've gone too far?
Rate this response:  
Not yet rated
goutam



Group memberships:
Premium Members

Joined: 26 Jul 2007
Posts: 109

Points: 1057

Post Posted: Thu Feb 21, 2019 3:18 am Reply with quote    Back to top    

HI Rodriguez,

I tried with functions like AsInteger(), AsFloat() and AsDouble(). Nothing is working as they expect numeric string But we have arithmetic operators in the input string given to above functions.

_________________
Goutam Sahoo
Rate this response:  
Not yet rated
Display posts from previous:       

Add To Favorites
View next topic
View previous topic
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum



Powered by phpBB © 2001, 2002 phpBB Group
Theme & Graphics by Daz :: Portal by Smartor
All times are GMT - 6 Hours