DSX export progress intermittently very slow to start

qt_ky · Post by **qt_ky** » Wed Dec 23, 2015 12:30 pm

Sometimes DSX export progress is very slow, sometimes there is no delay.

When slow, it will appear to hang for several minutes on the "Export progress" window, which says, "Getting list of dependent items......" then all of a sudden there is rapid movement and in a matter of seconds it will finish.

The slowness is not related to project size, as it can happen even in a new project with less than 10 jobs.

Any ideas what may cause such a long hang time at this point, and how to resolve it?

kumar_s · Post by **kumar_s** » Wed Dec 23, 2015 2:47 pm

If this happens from various client machine, Could be possible that the temp folders are full or almost full.
Try clearing those, and along with it, do the clean up of COMO and PH folders.
Its also worth doing a reindex of the project. Make sure you follow the steps in doing it.

ArndW · Post by **ArndW** » Thu Dec 24, 2015 3:51 am

I've noticed the long time it takes to track down dependencies on some projects as well, but have never encountered that issue in a small project. Can you uncheck the dependencies and see if that is the only cause of the slowdown?

qt_ky · Post by **qt_ky** » Tue Dec 29, 2015 12:14 pm

Thanks for all the suggestions. I do have a script that clears these various things out every week, from every project.

/tmp is only 5% used.

&PH& contains only 20 files.

&COMO& contains only 3 files.

reindex the project - it is a brand new project having only 10 objects imported into it. I have not tried to reindex it, as I just created it.

With "Include dependent items" checked, there is a delay of just over 16 minutes on "Export progress" window saying "Getting list of dependent items......".

With "Include dependent items" unchecked, there is no delay, and the total export time is under 10 seconds.

So it does appear that checking "Include dependent items" is the only cause of the slowdown. Still, I would like to find the root cause and the resolution. I have opened a PMR about it as well.

ray.wurlod · Post by **ray.wurlod** » Fri Jan 01, 2016 1:12 am

The root cause is getting the list of dependent items. There is no smart way to do this because there is no a priori knowledge of the items you have selected for export. Therefore, for each of those items, the exporter has to search (relevant object types) in the entire project repository.

qt_ky · Post by **qt_ky** » Fri Jan 01, 2016 10:33 am

That makes sense. However, the project is new and contains only 7 jobs. Since the project contains very few objects that would have to be searched, I am guessing that the actual behavior is that it is inefficiently searching all projects across the server. Support is having me turn on server tracing at the project level and 2 types of client tracing. I will update this topic with any findings.

ray.wurlod · Post by **ray.wurlod** » Fri Jan 01, 2016 11:00 pm

The search is only ever within the project from which the export is being performed. Objects in other projects cannot participate in dependency relationships with objects in the current project.

IBM Analytics Champion 2009 - 2020 · Post by **asorrell** » Sat Jan 02, 2016 12:01 pm

ray.wurlod wrote:The search is only ever within the project from which the export is being performed.

Actually Ray, due to a bug, that may not be true if they are using DataStage 11.3.1+ and Oracle. IBM did a less than spectacular job of implementing the XMETA database on Oracle. As one of the very first Oracle XMETA 11.3 installs, we saw consistent extremely slow exports with long start-up times.

Our DBA's monitored some of the exports and realized that there were some large complete table scans going on in XMETA. Due to a bug, it was scanning the entire XMETA database (all projects). We reported it to IBM as a PMR, but they said it would be a while before it could be fixed. Our DBA then put some indexes in place to help mitigate the problem. Unfortunately, I'm no longer at that company and I don't remember which tables / columns needed the indexes.

So - are you using 11.3.1+ with Oracle?

ray.wurlod · Post by **ray.wurlod** » Sat Jan 02, 2016 3:14 pm

Not at the moment. Currently on 8.5 with DB2.

Just before the break I installed an 11.3.1 system (with FP2) using SQL Server as the repository database. But there's nothing there to export yet.

qt_ky · Post by **qt_ky** » Mon Jan 04, 2016 8:27 am

I am on 11.3.1.2 using the bundled DB2 as the repository. Perhaps it is the same bug regardless of the database. We'll see.

PaulVL · Post by **PaulVL** » Mon Jan 04, 2016 9:22 am

Out of curiosity...

I'm wondering if the following command is also slow:

$DSHOME/bin/dsadmin -listenv yourproject;

qt_ky · Post by **qt_ky** » Tue Jan 05, 2016 8:41 am

The first run of that command took about 15 seconds to respond. Subsequent commands took about 3 to 4 seconds each.

I found the same behavior and timings on another server, different project.

Not sure this is related, but the WAS startup times are also quite slow on these same servers, with slow meaning around 17-18 minutes instead of the usual 3-5 minutes.

PaulVL · Post by **PaulVL** » Tue Jan 05, 2016 9:32 am

When doing the exports, are you using credentials "-user -pass -domain, etc..." or just a straight extract using no LDAP lookup via WAS?

qt_ky · Post by **qt_ky** » Tue Jan 05, 2016 11:47 am

These exports are done via Designer's export menu (no command line). All server tiers are on the same computer. OS is AIX. User registry is local OS authenticated (no LDAP). I've ruled out my client tier. The export is slow from multiple clients.

PaulVL · Post by **PaulVL** » Tue Jan 05, 2016 12:27 pm

AIX hosts are set up on LPARs on the physical machine. Ask your HW guys if the box is pegged during your slow period.

Memory consumption on the host, network IO, etc...

You should also ask them if the other LPARs on that machine are also getting hammered.

When I was out near Boston, one of our AIX machines was cabled up to the 10Gb network cards, but internally to AIX , the LPAR was configured to use 1Gb network speed to the actual card via the internal LAN. Might want to ask your HW guys that too.

You should also try to run an isx export via the command line just to see that speed.