Connecting to Hadoop from FileConnector

Post questions here relative to DataStage Enterprise/PX Edition for such areas as Parallel job design, Parallel datasets, BuildOps, Wrappers, etc.

Moderators: chulett, rschirm, roy

Post Reply
rkashyap
Premium Member
Premium Member
Posts: 532
Joined: Fri Dec 02, 2011 12:02 pm
Location: Richmond VA

Connecting to Hadoop from FileConnector

Post by rkashyap »

DataStage 11.501; Patches applied: Datadirect JDBC Driver for Hive(JR56156) and Hive Connector(JR56097).

We are trying to use File Connector stage for connecting to Cloudera Hadoop server. Connection parameters are:

Code: Select all

 FileSystem: HttpFS
 Use Custom URL: No
 Use SSL: Yes
 Use Ketab: Yes
 Use Kerberos: Yes
 Host: <Cloudera Server>
 PortNo: 14000
 Service principal: 
 User name: <username@domain>
 Keytab:<Path to keytab file>
Cloudera server is setup with an edge node and we are connecting with edge node as host. As per Hadoop admin's recommendation "HttpFS with SSL" is being used. Truststore has been setup. Keytab is provided by Hadoop Admin and we have installed Kerberos on DS server with krb5.conf file copied from Hadoop server to DataStage server.

File connector job fails with "File_Connector_1: com.ascential.e2.common.CC_Exception: java.lang.Exception: java.lang.RuntimeException: java.lang.RuntimeException: javax.security.auth.login.LoginException: Login Failure: all modules ignored at com.ibm.iis.cc.filesystem.FileSystem.initialize(FileSystem.java:540) ....

Appreciate if someone can advise on what could be the issue.
Timato
Participant
Posts: 24
Joined: Tue Sep 30, 2014 10:51 pm

Post by Timato »

The 'Login Error' seems to point more or less to your authentication parameters (be it the keytab or username or something different altogether).
Have you tried perhaps putting in your FQDN for your domain?
Test out the connectivity via command line on your engine?
Confirm that you can connect via WebHDFS to rule out any networking issues?
rkashyap
Premium Member
Premium Member
Posts: 532
Joined: Fri Dec 02, 2011 12:02 pm
Location: Richmond VA

Post by rkashyap »

Thank you. Failure was due to convergence of two issues:
1. Bad Keytab. Cloudera admin recreated the keytab file for this.

2. Kerberos ticket was generated using knit from "/usr/sbin/". Ticket had to be regenerated using kinit from IBM's JDK:

Code: Select all

$ kdestroy
$ /opt/IBM/InformationServer/jdk/jre/bin/kinit -k -t <keytab_filename> <principal>
We have also updated the "path" to keep IBM's JDK library above the ones from RHEL.
ray.wurlod
Participant
Posts: 54607
Joined: Wed Oct 23, 2002 10:52 pm
Location: Sydney, Australia
Contact:

Post by ray.wurlod »

Thank you for posting the solution.
IBM Software Services Group
Any contribution to this forum is my own opinion and does not necessarily reflect any position that IBM may hold.
Post Reply