DSXchange: DataStage and IBM Websphere Data Integration Forum
View next topic
View previous topic
Add To Favorites
Author Message
telenet_bi



Group memberships:
Premium Members

Joined: 23 Jul 2008
Posts: 33
Location: Mechelen, Belgium
Points: 308

Post Posted: Thu Sep 25, 2008 2:08 am Reply with quote    Back to top    

DataStage® Release: 7x
Job Type: Parallel
OS: Unix
Additional info: how to optimize memory-usage of sort
hi,


our production server is heavily loaded on CPU-usage and on IO, but we see still a lot of free memory. Since a lot of sort-stages are used I want to take a look at optimizing memory-usage in these.

Untill now the option for Restrict Memory Usage is not used anywhere. I understood that this limits the process to 20MB of memory(per node). I would like to get better settings for this per sort-stage, but I'm a bit in the dark about the effects.

Some questions:
- if we increase this for a stage, but no memory is available will the job then abort, Or will it just start using swap-space? (anyway if both memory and swap are full there's definately problems).

- If we use varchar-fields the scratch-disk usage during sorts handles these as chars(so a varchar(100) takes 100 bytes for every record, no matter what the real length is). Is this also the way we would need to think in memory-usage?

-what would be the approach which gets the most result the fastest:
* increase the setting for all large sorts to 100MB
* take out some jobs and get them to work completely in memory
(probably here the first remark will be that this depends on my environment and type of jobs, but I still would like to know what you would do within your environment)
ray.wurlod

Premium Poster
Participant

Group memberships:
Premium Members, Inner Circle, Australia Usergroup, Server to Parallel Transition Group

Joined: 23 Oct 2002
Posts: 54225
Location: Sydney, Australia
Points: 294087

Post Posted: Thu Sep 25, 2008 5:06 am Reply with quote    Back to top    

Memory usage is described in the manuals. First place it uses is the scratch disk specified in the configuration file. That is, it does not go to your swap device(s). There are unbounded and bo ...

_________________
RXP Services Ltd
Melbourne | Canberra | Sydney | Hong Kong | Hobart | Brisbane
currently hiring: Canberra, Sydney and Melbourne
Rate this response:  
Not yet rated
telenet_bi



Group memberships:
Premium Members

Joined: 23 Jul 2008
Posts: 33
Location: Mechelen, Belgium
Points: 308

Post Posted: Thu Sep 25, 2008 6:05 am Reply with quote    Back to top    

ray,

for the temp space used in sort-space I find in the manual:
Quote:

The stage uses temporary disk space when performing a sort. It looks in
the following locations, in the following order, for this temporary space.
1. Scratch disks in the disk pool sort (you can create these pools in the
configuration file).
2. Scratch disks in the default disk pool (scratch disks are included here
by default).
3. The directory specified by the TMPDIR environment variable.
4. The directory /tmp.


This doesn't actually say anything about memory, but what happens when memory alone is not sufficient.
The parameter I want to use tells datastage to use more memory, I was worried what would happen if it is then not available.
Rate this response:  
ray.wurlod

Premium Poster
Participant

Group memberships:
Premium Members, Inner Circle, Australia Usergroup, Server to Parallel Transition Group

Joined: 23 Oct 2002
Posts: 54225
Location: Sydney, Australia
Points: 294087

Post Posted: Thu Sep 25, 2008 6:24 am Reply with quote    Back to top    

The same happens as described for temporary space. You just get more - or differently sized - temporary sort files on disk.

_________________
RXP Services Ltd
Melbourne | Canberra | Sydney | Hong Kong | Hobart | Brisbane
currently hiring: Canberra, Sydney and Melbourne
Rate this response:  
Not yet rated
rohit_mca2003
Participant



Joined: 08 Oct 2008
Posts: 37

Points: 449

Post Posted: Mon Jan 15, 2018 4:46 am Reply with quote    Back to top    

Try using variable APT_OLD_BOUNDED_LENGTH and set to 'True'.

_________________
Rohit
Rate this response:  
Not yet rated
chulett

Premium Poster


since January 2006

Group memberships:
Premium Members, Inner Circle, Server to Parallel Transition Group

Joined: 12 Nov 2002
Posts: 42585
Location: Denver, CO
Points: 219186

Post Posted: Mon Jan 15, 2018 9:02 am Reply with quote    Back to top    

Idea Public Service Announcement

These stupid spam posts are bringing lots of random posts up to the surface, including this 10 year old one. Not saying there's no value to adding comments or suggestions to them but do keep that in mind, please.

Hopefully someone with the power will get them cleaned up... soon.

_________________
-craig

And I'm hovering like a fly, waiting for the windshield on the freeway...
Rate this response:  
Not yet rated
Display posts from previous:       

Add To Favorites
View next topic
View previous topic
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum



Powered by phpBB © 2001, 2002 phpBB Group
Theme & Graphics by Daz :: Portal by Smartor
All times are GMT - 6 Hours