DSXchange: DataStage and IBM Websphere Data Integration Forum
View next topic
View previous topic
Add To Favorites
Author Message
chandankambli
Participant



Joined: 11 Jun 2006
Posts: 14

Points: 128

Post Posted: Sat Jan 06, 2007 11:50 pm Reply with quote    Back to top    

DataStage® Release: 7x
Job Type: Server
OS: Unix
Dear Experts:

What is a typical size of DataWarehouse and Data Marts?

_________________
Thanks experts.
datastage_learner
ray.wurlod

Premium Poster
Participant

Group memberships:
Premium Members, Inner Circle, Australia Usergroup, Server to Parallel Transition Group

Joined: 23 Oct 2002
Posts: 54072
Location: Sydney, Australia
Points: 293282

Post Posted: Sun Jan 07, 2007 12:07 am Reply with quote    Back to top    

Welcome aboard. Your question is in the "how long is a piece of string?" category. The only valid answer is something like "big enough, and growing over time". I have worked with sta ...

_________________
RXP Services Ltd
Melbourne | Canberra | Sydney | Hong Kong | Hobart | Brisbane
currently hiring: Canberra, Sydney and Melbourne
Rate this response:  
kumar_s

Premium Poster


since February 2006

Group memberships:
Premium Members, Heartland Usergroup

Joined: 16 Jun 2005
Posts: 5240

Points: 26349

Post Posted: Sun Jan 07, 2007 1:06 am Reply with quote    Back to top    

Hi,
And btw, pls avoid interview question in this forum.

_________________
Impossible doesn't mean 'it is not possible' actually means... 'NOBODY HAS DONE IT SO FAR'
Rate this response:  
DSguru2B

Premium Poster


since February 2006

Group memberships:
Premium Members, Heartland Usergroup

Joined: 09 Feb 2005
Posts: 6854
Location: Houston, TX
Points: 35675

Post Posted: Sun Jan 07, 2007 2:40 am Reply with quote    Back to top    

Welcome Aboard
Answer to such questions can be found by a simple google. Try doing some research.

_________________
Creativity is allowing yourself to make mistakes. Art is knowing which ones to keep.
Rate this response:  
chandankambli
Participant



Joined: 11 Jun 2006
Posts: 14

Points: 128

Post Posted: Sun Jan 07, 2007 12:47 pm Reply with quote    Back to top    

First of all, thanks for replying my question and welcoming me to this forum.
kumar_s wrote:
Hi,
And btw, pls avoid interview question in this forum.
I will certainly keep in mind you comment.

But, pls. try to understand the meaning behind the question. You are working for a co. say retail industry as a DWH/ETL Architect you are planning for building a DWH. You know certain idea on the total data size in question. But real question is how to expand or minimize the size and scope of DWH/DM’s. So to begin with how much size you allocate for the underlying hardware etc.

Pls. do not get me wrong. I am certainly not trying to be disrespectful. I do agree, though answers to most of the problems can be found on google.com. But I feel google.com is not the solution all the issues in this world. Moreover, professional discussions, mentoring and interactions can ONLY happen in forums like dsxchange.com. Google is only for static search. Do you agree experts?

Now, coming back to the question say Wal-mart their data grows massively every year. But their DWH Architects must have started/initiate somewhere by keeping some matrix in mind. I was wondering if, is there is any way to calculate/understand that?

_________________
Thanks experts.
datastage_learner
Rate this response:  
ray.wurlod

Premium Poster
Participant

Group memberships:
Premium Members, Inner Circle, Australia Usergroup, Server to Parallel Transition Group

Joined: 23 Oct 2002
Posts: 54072
Location: Sydney, Australia
Points: 293282

Post Posted: Sun Jan 07, 2007 2:36 pm Reply with quote    Back to top    

DH Architects are, in my experience, governed by consultation with Business Analysts who prepare target-from-source mapping specifications which, along with row counts from source, allows the size of ...

_________________
RXP Services Ltd
Melbourne | Canberra | Sydney | Hong Kong | Hobart | Brisbane
currently hiring: Canberra, Sydney and Melbourne
Rate this response:  
chandankambli
Participant



Joined: 11 Jun 2006
Posts: 14

Points: 128

Post Posted: Sun Jan 07, 2007 3:38 pm Reply with quote    Back to top    

ray.wurlod wrote:
DH Architects are, in my experience, governed by consultation with Business Analysts who prepare target-from-source mapping specifications which, along with row counts from source, allows the size of ...


Ray, a perfect understanding of the question and a perfect response that's what I can appreciate.

_________________
Thanks experts.
datastage_learner
Rate this response:  
kumar_s

Premium Poster


since February 2006

Group memberships:
Premium Members, Heartland Usergroup

Joined: 16 Jun 2005
Posts: 5240

Points: 26349

Post Posted: Sun Jan 07, 2007 4:35 pm Reply with quote    Back to top    

Planning has its own weightage. As mentioned, its purely depends on your source data and the business needs. The business requirements will be in such a way that, the DM1 will required only 5% of the operational data. Ray has mentioned, how even the 5% grows to 1000%.
Calculate the capacity of individual datamart, excluding the confirmed facts, you ll get the DWH size. (If you are on bottom up.)

_________________
Impossible doesn't mean 'it is not possible' actually means... 'NOBODY HAS DONE IT SO FAR'
Rate this response:  
ray.wurlod

Premium Poster
Participant

Group memberships:
Premium Members, Inner Circle, Australia Usergroup, Server to Parallel Transition Group

Joined: 23 Oct 2002
Posts: 54072
Location: Sydney, Australia
Points: 293282

Post Posted: Sun Jan 07, 2007 4:53 pm Reply with quote    Back to top    

... of course, I meant DWH Architects, not DH Architects! Embarassed

_________________
RXP Services Ltd
Melbourne | Canberra | Sydney | Hong Kong | Hobart | Brisbane
currently hiring: Canberra, Sydney and Melbourne
Rate this response:  
chulett

Premium Poster


since January 2006

Group memberships:
Premium Members, Inner Circle, Server to Parallel Transition Group

Joined: 12 Nov 2002
Posts: 42277
Location: Denver, CO
Points: 217088

Post Posted: Sun Jan 07, 2007 6:34 pm Reply with quote    Back to top    

chandankambli wrote:
Ray, a perfect understanding of the question and a perfect response that's what I can appreciate.

So say we all. Nothing like shooting for the moon in your first thread here, however. Wink

I would think you would need to start with the perfect question before you could see perfect understanding and receive a perfect answer... and that only in a perfect world. Crying or Very sad

_________________
-craig

Watch out where the huskies go and don't you eat that yellow snow
Rate this response:  
narasimha


since February 2010

Group memberships:
Premium Members, Heartland Usergroup

Joined: 22 Oct 2004
Posts: 1236
Location: Staten Island, NY
Points: 6560

Post Posted: Sun Jan 07, 2007 9:34 pm Reply with quote    Back to top    

Like everybody stated above there is no typical size for a Datawarehouse.
It depends on your intended business needs.

You can determine the size, growth of your DWH with the help of the Business Analysts, Domain experts who are supposed to have knowledge of how much of data you get on a daily, monthly and yearly basis.

It is not a easy task. You need to have a complete knowledge of your Data Model.
You can do it yourself or take the help of your DBA's to estimate the size of individual tables.
You can calculate the Average Row length of your tables, multiply it with your estimated data to get an estimate of the space required for that table.

Your estimates may not match the actual numbers all the time.
It is better to ask for more space before hand rather than getting caught with space issues later.
(With the cost of disk space decreasing day by day, the fight for disk space is not a major issue in most of the companies nowadays)

_________________
Narasimha Kade

Finding answers is simple, all you need to do is come up with the correct questions.
Rate this response:  
ray.wurlod

Premium Poster
Participant

Group memberships:
Premium Members, Inner Circle, Australia Usergroup, Server to Parallel Transition Group

Joined: 23 Oct 2002
Posts: 54072
Location: Sydney, Australia
Points: 293282

Post Posted: Sun Jan 07, 2007 9:50 pm Reply with quote    Back to top    

Once you've done all those calculations, add 42% as a safety margin. Document it as a safety margin. When challenged, mutter something about the "ultimate answer (to life, the universe, and everything)".

_________________
RXP Services Ltd
Melbourne | Canberra | Sydney | Hong Kong | Hobart | Brisbane
currently hiring: Canberra, Sydney and Melbourne
Rate this response:  
kumar_s

Premium Poster


since February 2006

Group memberships:
Premium Members, Heartland Usergroup

Joined: 16 Jun 2005
Posts: 5240

Points: 26349

Post Posted: Mon Jan 08, 2007 12:07 am Reply with quote    Back to top    

42% With any specific formula you arrived this number Question

_________________
Impossible doesn't mean 'it is not possible' actually means... 'NOBODY HAS DONE IT SO FAR'
Rate this response:  
chulett

Premium Poster


since January 2006

Group memberships:
Premium Members, Inner Circle, Server to Parallel Transition Group

Joined: 12 Nov 2002
Posts: 42277
Location: Denver, CO
Points: 217088

Post Posted: Mon Jan 08, 2007 12:36 am Reply with quote    Back to top    

The reference to 'Life, the Universe and Everything' was supposed to clue you into a Hitchhiker's Guide to the Galaxy reference - and 42 was a pivotal number from the series and was, in fact as Ray notes, the answer to that very question. Wink

_________________
-craig

Watch out where the huskies go and don't you eat that yellow snow
Rate this response:  
kumar_s

Premium Poster


since February 2006

Group memberships:
Premium Members, Heartland Usergroup

Joined: 16 Jun 2005
Posts: 5240

Points: 26349

Post Posted: Mon Jan 08, 2007 1:03 am Reply with quote    Back to top    

So that was an ultimate answer to the amateur question.

_________________
Impossible doesn't mean 'it is not possible' actually means... 'NOBODY HAS DONE IT SO FAR'
Rate this response:  
Display posts from previous:       

Add To Favorites
View next topic
View previous topic
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum



Powered by phpBB © 2001, 2002 phpBB Group
Theme & Graphics by Daz :: Portal by Smartor
All times are GMT - 6 Hours