（Kerberos済みの）WebHDFSをHAProxyをつかってHAにする

http://qiita.com/saka1_p/items/3634ba70f9ecd74b0860
https://www.haproxy.com/doc/aloha/7.0/haproxy/healthchecks.html

Node1がHAProxyサーバ
Node2がNameNode1
Node3がNameNode2

１）HAProxyをインストール
[root@node1 ~]# yum install -y haproxy
[root@node1 ~]# cp -p /etc/haproxy/haproxy.cfg /etc/haproxy/haproxy.cfg.orig
[root@node1 ~]# vim /etc/haproxy/haproxy.cfg
...
#---------------------------------------------------------------------
# main frontend which proxys to the backends
#---------------------------------------------------------------------
frontend  main *:50070
    default_backend             app

#---------------------------------------------------------------------
# round robin balancing between the various backends
#---------------------------------------------------------------------
backend app
    balance     roundrobin
    option      httpchk GET /webhdfs/v1/?op=CHECKACCESS
    http-check expect rstatus ([23][0-9][0-9]|401)
    server  node2 node2.localdomain:50070 check
    server  node3 node3.localdomain:50070 check

２）KDCサーバ上で（もしくはkadmin -p admin/adminなど）でHAProxy用のSPNEGO Keytabを作成
kadmin.local -q "addprinc -randkey HTTP/node1.localdomain@HO-UBU02"

[root@node1 ~]# mv /etc/security/keytabs/spnego.service.keytab /etc/security/keytabs/spnego.service.keytab.old

注意：ktaddはKvnoをインクリメントする模様
[root@node1 ~]# kadmin -p ambari/admin -q "ktadd -k /etc/security/keytabs/spnego.service.keytab HTTP/node1.localdomain@HO-UBU02"
Authenticating as principal ambari/admin with password.
Password for ambari/admin@HO-UBU02:
Entry for principal HTTP/node1.localdomain@HO-UBU02 with kvno 5, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/etc/security/keytabs/spnego.service.keytab.
Entry for principal HTTP/node1.localdomain@HO-UBU02 with kvno 5, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/etc/security/keytabs/spnego.service.keytab.
Entry for principal HTTP/node1.localdomain@HO-UBU02 with kvno 5, encryption type des3-cbc-sha1 added to keytab WRFILE:/etc/security/keytabs/spnego.service.keytab.
Entry for principal HTTP/node1.localdomain@HO-UBU02 with kvno 5, encryption type arcfour-hmac added to keytab WRFILE:/etc/security/keytabs/spnego.service.keytab.

確認：
[root@node1 ~]# klist -kte /etc/security/keytabs/spnego.service.keytab
Keytab name: FILE:/etc/security/keytabs/spnego.service.keytab
KVNO Timestamp         Principal
---- ----------------- --------------------------------------------------------
   5 03/22/17 08:38:43 HTTP/node1.localdomain@HO-UBU02 (aes256-cts-hmac-sha1-96)
   5 03/22/17 08:38:43 HTTP/node1.localdomain@HO-UBU02 (aes128-cts-hmac-sha1-96)
   5 03/22/17 08:38:43 HTTP/node1.localdomain@HO-UBU02 (des3-cbc-sha1)
   5 03/22/17 08:38:43 HTTP/node1.localdomain@HO-UBU02 (arcfour-hmac)

各NameNodeへ送信：
[root@node1 ~]# scp /etc/security/keytabs/spnego.service.keytab node2.localdomain:/tmp/node1.spnego.service.keytab
spnego.service.keytab                                                               100%  306     0.3KB/s   00:00
[root@node1 ~]# scp /etc/security/keytabs/spnego.service.keytab node3.localdomain:/tmp/node1.spnego.service.keytab
spnego.service.keytab                                                               100%  306     0.3KB/s   00:00

３）両方のNameNodeでキータブをマージ：
まず、SPNEGOキータブファイルの場所を確認
[root@node2 ~]# grep 'dfs.web.authentication.kerberos.keytab' -A1 /etc/hadoop/conf/hdfs-site.xml
      <name>dfs.web.authentication.kerberos.keytab</name>
      <value>/etc/security/keytabs/spnego.service.keytab</value>
[root@node2 ~]# mv /etc/security/keytabs/spnego.service.keytab /etc/security/keytabs/spnego.service.keytab.orig

二つのキータブをKtutilでマージ：
[root@node2 ~]# ktutil
ktutil:  rkt /etc/security/keytabs/spnego.service.keytab.orig
ktutil:  rkt /tmp/node1.spnego.service.keytab
ktutil:  wkt /etc/security/keytabs/spnego.service.keytab
ktutil:  quit

確認：
[root@node2 ~]# klist -kte /etc/security/keytabs/spnego.service.keytab
Keytab name: FILE:/etc/security/keytabs/spnego.service.keytab
KVNO Timestamp         Principal
---- ----------------- --------------------------------------------------------
   2 03/22/17 08:45:02 HTTP/node2.localdomain@HO-UBU02 (aes256-cts-hmac-sha1-96)
   2 03/22/17 08:45:02 HTTP/node2.localdomain@HO-UBU02 (aes128-cts-hmac-sha1-96)
   2 03/22/17 08:45:02 HTTP/node2.localdomain@HO-UBU02 (des3-cbc-sha1)
   2 03/22/17 08:45:02 HTTP/node2.localdomain@HO-UBU02 (arcfour-hmac)
   5 03/22/17 08:45:02 HTTP/node1.localdomain@HO-UBU02 (aes256-cts-hmac-sha1-96)
   5 03/22/17 08:45:02 HTTP/node1.localdomain@HO-UBU02 (aes128-cts-hmac-sha1-96)
   5 03/22/17 08:45:02 HTTP/node1.localdomain@HO-UBU02 (des3-cbc-sha1)
   5 03/22/17 08:45:02 HTTP/node1.localdomain@HO-UBU02 (arcfour-hmac)

ファイルパーミッションを直す：
[root@node2 ~]# chown root:hadoop /etc/security/keytabs/spnego.service.keytab
[root@node2 ~]# chmod 440 /etc/security/keytabs/spnego.service.keytab

４）上記のステップをNode3でも実行
TODO: KVNOは同じ必要がある？ "Specified version of key is not available"

５）Ambariからdfs.web.authentication.kerberos.principalを"*"に変更する
そしてHDFSを再起動
このままだと、"*"がAmbari Alertのkinitに使われてしまうので、全ての/usr/lib/python2.6/site-packages/ambari_agent/alerts/base_alert.pyを下記に変更（またはすべてのAlert JSONを更新）：
    if 'kerberos_principal' in uri_structure:
      kerberos_principal = uri_structure['kerberos_principal']
      if kerberos_principal == "*":
        kerberos_principal = 'HTTP/node1.localdomain@HO-UBU02'

Ambariのデータベスにログイン（psql -Uambari ambari）し、UPDATEステートメントを実行：
select label, alert_source from alert_definition where alert_source like '%{hdfs-site/dfs.web.authentication.kerberos.principal}%';

update alert_definition set alert_source = replace(alert_source, '{hdfs-site/dfs.web.authentication.kerberos.principal}', '{hdfs-site/dfs.namenode.kerberos.internal.spnego.principal}') where alert_source like '%{hdfs-site/dfs.web.authentication.kerberos.principal}%' and component_name in ('NAMENODE', 'JOURNALNODE', 'DATANODE');

Ambari Server上で下記のコマンドを実行：
cd /var/lib/ambari-server/resources/common-services/HDFS/2.1.0.2.0/package/alerts
sed -i_$(date +"%Y%m%d%H%M%S").bak 's/dfs.web.authentication.kerberos.principal/dfs.namenode.kerberos.internal.spnego.principal/' *.py
ambari-server restart

Ambari UIから、AlertをDisable/Enableする必要あり

６）テスト
[root@node1 ~]# curl --negotiate -u : -X GET 'http://node3.localdomain:50070/webhdfs/v1/?op=CHECKACCESS'
[root@node1 ~]# curl --negotiate -u : -X GET 'http://node2.localdomain:50070/webhdfs/v1/?op=CHECKACCESS'
{"RemoteException":{"exception":"StandbyException","javaClassName":"org.apache.hadoop.ipc.StandbyException","message":"Operation category READ is not supported in state standby"}}[root@node1 ~]#
[root@node1 ~]# curl -s -I --negotiate -u : 'http://node1.localdomain:50070/webhdfs/v1/?op=CHECKACCESS' | grep ^HTTP
HTTP/1.1 401 Authentication required
HTTP/1.1 200 OK

HTTP/1.1 403 GSSException: Failure unspecified at GSS-API level (Mechanism level: Checksum failed)がでたら、dfs.web.authentication.kerberos.principalをチェック