2017年3月25日土曜日

Sandbox HDP 2.5.0でHadoop (HDFS, YARN, MR2) をSSL化する


備考:
事前にDockerをインストールする必要があります。
すでにDocker版のSandboxコンテナが作成済みの場合は、HTTPSに使われるポートをなんとかして開ける必要があります。

1)Sandboxをホストする側(Ubuntuとか)で、スクリプトをダウンロードしてソース
source ./start_hdp.sh

2)もしSandboxをまだインストールしていない場合、Docker版Sandboxをインストール
f_docker_sandbox_install

3)作業用フォルダを作成
mkdir ssl_setup; cd ssl_setup

f_ssl_self_signed_cert "/C=AU/ST=QLD/O=Hortonworks/CN=*.hortonworks.com" "server"

ls -ltr
total 28
-rw-r--r-- 1 root root 1679 Mar 25 08:35 server.key
-rw-r--r-- 1 root root  968 Mar 25 08:35 server.csr
-rw-r--r-- 1 root root 1131 Mar 25 08:35 server.crt
-rw-r--r-- 1 root root 2450 Mar 25 08:35 server.p12
-rw-r--r-- 1 root root 2147 Mar 25 08:35 server.keystore.jks
-rw-r--r-- 1 root root  857 Mar 25 08:35 server.truststore.jks
-rw-r--r-- 1 root root  857 Mar 25 08:35 client.truststore.jks

5)Sandboxに証明書をコピー
ssh root@sandbox.hortonworks.com -t "mkdir -p ${g_SERVER_KEY_LOCATION%/}"

scp ./*.jks root@sandbox.hortonworks.com:${g_SERVER_KEY_LOCATION%/}/

ssh root@sandbox.hortonworks.com -t "chmod 755 $g_SERVER_KEY_LOCATION
chown root:hadoop ${g_SERVER_KEY_LOCATION%/}/*.jks
chmod 440 ${g_SERVER_KEY_LOCATION%/}/$g_KEYSTORE_FILE
chmod 440 ${g_SERVER_KEY_LOCATION%/}/$g_TRUSTSTORE_FILE
chmod 444 ${g_SERVER_KEY_LOCATION%/}/$g_CLIENT_TRUSTSTORE_FILE"


core-site.xml
hadoop.ssl.require.client.cert=false
hadoop.ssl.hostname.verifier=DEFAULT
hadoop.ssl.keystores.factory.class=org.apache.hadoop.security.ssl.FileBasedKeyStoresFactory
hadoop.ssl.server.conf=ssl-server.xml
hadoop.ssl.client.conf=ssl-client.xml
hdfs-site.xml
dfs.http.policy=HTTPS_ONLY    # HTTP_AND_HTTPS?
dfs.client.https.need-auth=false # trueの方がいい?
dfs.datanode.https.address=0.0.0.0:50475
dfs.namenode.https-address=sandbox.hortonworks.com:50470    # 0.0.0.0だとダメ?
mapred-site.xml
mapreduce.jobhistory.http.policy=HTTPS_ONLY
mapreduce.jobhistory.webapp.https.address=0.0.0.0:19889
yarn-site.xml
yarn.http.policy=HTTPS_ONLY
yarn.log.server.url=https://sandbox.hortonworks.com:19889/jobhistory/logs
yarn.resourcemanager.webapp.https.address=sandbox.hortonworks.com:8090
yarn.nodemanager.webapp.https.address=0.0.0.0:8044
ssl-server.xml
ssl.server.keystore.password=hadoop
ssl.server.keystore.location=/etc/hadoop/conf/secure/server.keystore.jks
ssl.server.keystore.type=jks
ssl.server.keystore.keypassword=hadoop
ssl.server.truststore.location=/etc/hadoop/conf/secure/server.truststore.jks
ssl.server.truststore.password=changeit
ssl.server.truststore.type=jks
ssl-client.xml
#ssl.client.keystore.location=/etc/security/clientKeys/keystore.jks # 必要ない?
ssl.client.truststore.location=/etc/hadoop/conf/secure/client.truststore.jks
ssl.client.truststore.password=changeit
tez-site.xml
tez.runtime.shuffle.ssl.enable=true
tez.runtime.shuffle.keep-alive.enabled=true

補足:おそらくKnoxからHDFSUIを使うには、NameNodeの証明書(server.crt)をKnoxのTrustStore(cacerts)に入れる必要がある。
 Knoxの証明書をExportしNameNodeのTrustStoreに入れる必要もある?

2017年3月23日木曜日

(Kerberos済みの)WebHDFSをHAProxyをつかってHAにする

http://qiita.com/saka1_p/items/3634ba70f9ecd74b0860
https://www.haproxy.com/doc/aloha/7.0/haproxy/healthchecks.html

Node1がHAProxyサーバ
Node2がNameNode1
Node3がNameNode2

1)HAProxyをインストール
[root@node1 ~]# yum install -y haproxy
[root@node1 ~]# cp -p /etc/haproxy/haproxy.cfg /etc/haproxy/haproxy.cfg.orig
[root@node1 ~]# vim /etc/haproxy/haproxy.cfg
...
#---------------------------------------------------------------------
# main frontend which proxys to the backends
#---------------------------------------------------------------------
frontend  main *:50070
    default_backend             app

#---------------------------------------------------------------------
# round robin balancing between the various backends
#---------------------------------------------------------------------
backend app
    balance     roundrobin
    option      httpchk GET /webhdfs/v1/?op=CHECKACCESS
    http-check expect rstatus ([23][0-9][0-9]|401)
    server  node2 node2.localdomain:50070 check
    server  node3 node3.localdomain:50070 check

2)KDCサーバ上で(もしくはkadmin -p admin/adminなど)でHAProxy用のSPNEGO Keytabを作成
kadmin.local -q "addprinc -randkey HTTP/node1.localdomain@HO-UBU02"

[root@node1 ~]# mv /etc/security/keytabs/spnego.service.keytab /etc/security/keytabs/spnego.service.keytab.old

注意:ktaddはKvnoをインクリメントする模様
[root@node1 ~]# kadmin -p ambari/admin -q "ktadd -k /etc/security/keytabs/spnego.service.keytab HTTP/node1.localdomain@HO-UBU02"
Authenticating as principal ambari/admin with password.
Password for ambari/admin@HO-UBU02:
Entry for principal HTTP/node1.localdomain@HO-UBU02 with kvno 5, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/etc/security/keytabs/spnego.service.keytab.
Entry for principal HTTP/node1.localdomain@HO-UBU02 with kvno 5, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/etc/security/keytabs/spnego.service.keytab.
Entry for principal HTTP/node1.localdomain@HO-UBU02 with kvno 5, encryption type des3-cbc-sha1 added to keytab WRFILE:/etc/security/keytabs/spnego.service.keytab.
Entry for principal HTTP/node1.localdomain@HO-UBU02 with kvno 5, encryption type arcfour-hmac added to keytab WRFILE:/etc/security/keytabs/spnego.service.keytab.

確認:
[root@node1 ~]# klist -kte /etc/security/keytabs/spnego.service.keytab
Keytab name: FILE:/etc/security/keytabs/spnego.service.keytab
KVNO Timestamp         Principal
---- ----------------- --------------------------------------------------------
   5 03/22/17 08:38:43 HTTP/node1.localdomain@HO-UBU02 (aes256-cts-hmac-sha1-96)
   5 03/22/17 08:38:43 HTTP/node1.localdomain@HO-UBU02 (aes128-cts-hmac-sha1-96)
   5 03/22/17 08:38:43 HTTP/node1.localdomain@HO-UBU02 (des3-cbc-sha1)
   5 03/22/17 08:38:43 HTTP/node1.localdomain@HO-UBU02 (arcfour-hmac)

各NameNodeへ送信:
[root@node1 ~]# scp /etc/security/keytabs/spnego.service.keytab node2.localdomain:/tmp/node1.spnego.service.keytab
spnego.service.keytab                                                               100%  306     0.3KB/s   00:00
[root@node1 ~]# scp /etc/security/keytabs/spnego.service.keytab node3.localdomain:/tmp/node1.spnego.service.keytab
spnego.service.keytab                                                               100%  306     0.3KB/s   00:00

3)両方のNameNodeでキータブをマージ:
まず、SPNEGOキータブファイルの場所を確認
[root@node2 ~]# grep 'dfs.web.authentication.kerberos.keytab' -A1 /etc/hadoop/conf/hdfs-site.xml
      <name>dfs.web.authentication.kerberos.keytab</name>
      <value>/etc/security/keytabs/spnego.service.keytab</value>
[root@node2 ~]# mv /etc/security/keytabs/spnego.service.keytab /etc/security/keytabs/spnego.service.keytab.orig

二つのキータブをKtutilでマージ:
[root@node2 ~]# ktutil
ktutil:  rkt /etc/security/keytabs/spnego.service.keytab.orig
ktutil:  rkt /tmp/node1.spnego.service.keytab
ktutil:  wkt /etc/security/keytabs/spnego.service.keytab
ktutil:  quit

確認:
[root@node2 ~]# klist -kte /etc/security/keytabs/spnego.service.keytab
Keytab name: FILE:/etc/security/keytabs/spnego.service.keytab
KVNO Timestamp         Principal
---- ----------------- --------------------------------------------------------
   2 03/22/17 08:45:02 HTTP/node2.localdomain@HO-UBU02 (aes256-cts-hmac-sha1-96)
   2 03/22/17 08:45:02 HTTP/node2.localdomain@HO-UBU02 (aes128-cts-hmac-sha1-96)
   2 03/22/17 08:45:02 HTTP/node2.localdomain@HO-UBU02 (des3-cbc-sha1)
   2 03/22/17 08:45:02 HTTP/node2.localdomain@HO-UBU02 (arcfour-hmac)
   5 03/22/17 08:45:02 HTTP/node1.localdomain@HO-UBU02 (aes256-cts-hmac-sha1-96)
   5 03/22/17 08:45:02 HTTP/node1.localdomain@HO-UBU02 (aes128-cts-hmac-sha1-96)
   5 03/22/17 08:45:02 HTTP/node1.localdomain@HO-UBU02 (des3-cbc-sha1)
   5 03/22/17 08:45:02 HTTP/node1.localdomain@HO-UBU02 (arcfour-hmac)

ファイルパーミッションを直す:
[root@node2 ~]# chown root:hadoop /etc/security/keytabs/spnego.service.keytab
[root@node2 ~]# chmod 440 /etc/security/keytabs/spnego.service.keytab

4)上記のステップをNode3でも実行
TODO: KVNOは同じ必要がある? "Specified version of key is not available"

5)Ambariからdfs.web.authentication.kerberos.principalを"*"に変更する
そしてHDFSを再起動
このままだと、"*"がAmbari Alertのkinitに使われてしまうので、全ての/usr/lib/python2.6/site-packages/ambari_agent/alerts/base_alert.pyを下記に変更(またはすべてのAlert JSONを更新):
    if 'kerberos_principal' in uri_structure:
      kerberos_principal = uri_structure['kerberos_principal']
      if kerberos_principal == "*":
        kerberos_principal = 'HTTP/node1.localdomain@HO-UBU02'

Ambariのデータベスにログイン(psql -Uambari ambari)し、UPDATEステートメントを実行:
select label, alert_source from alert_definition where alert_source like '%{hdfs-site/dfs.web.authentication.kerberos.principal}%';

update alert_definition set alert_source = replace(alert_source, '{hdfs-site/dfs.web.authentication.kerberos.principal}', '{hdfs-site/dfs.namenode.kerberos.internal.spnego.principal}') where alert_source like '%{hdfs-site/dfs.web.authentication.kerberos.principal}%' and component_name in ('NAMENODE', 'JOURNALNODE', 'DATANODE');

Ambari Server上で下記のコマンドを実行:
cd /var/lib/ambari-server/resources/common-services/HDFS/2.1.0.2.0/package/alerts
sed -i_$(date +"%Y%m%d%H%M%S").bak 's/dfs.web.authentication.kerberos.principal/dfs.namenode.kerberos.internal.spnego.principal/' *.py
ambari-server restart

Ambari UIから、AlertをDisable/Enableする必要あり

6)テスト
[root@node1 ~]# curl --negotiate -u : -X GET 'http://node3.localdomain:50070/webhdfs/v1/?op=CHECKACCESS'
[root@node1 ~]# curl --negotiate -u : -X GET 'http://node2.localdomain:50070/webhdfs/v1/?op=CHECKACCESS'
{"RemoteException":{"exception":"StandbyException","javaClassName":"org.apache.hadoop.ipc.StandbyException","message":"Operation category READ is not supported in state standby"}}[root@node1 ~]#
[root@node1 ~]# curl -s -I --negotiate -u : 'http://node1.localdomain:50070/webhdfs/v1/?op=CHECKACCESS' | grep ^HTTP
HTTP/1.1 401 Authentication required
HTTP/1.1 200 OK

HTTP/1.1 403 GSSException: Failure unspecified at GSS-API level (Mechanism level: Checksum failed)がでたら、dfs.web.authentication.kerberos.principalをチェック




2017年3月21日火曜日

Ambari 2.4.2でTLSv1.2を使用する => TLSv1.2"のみ”は不可(AMBARI-17666 )

備考:https://issues.apache.org/jira/browse/AMBARI-18910 のためAmbari2.4.2以前のバージョンではできないと思われる。

1)まずAmbariのSSLを有効にする
http://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.5.3/bk_security/content/set_up_ssl_for_ambari.html
http://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.5.3/bk_security/content/_set_up_truststore_for_ambari_server.html

設定用のスクリプトをダウンロード
[root@sandbox ~]# curl https://raw.githubusercontent.com/hajimeo/samples/master/bash/start_hdp.sh -O
[root@sandbox ~]# source ./start_hdp.sh

証明書を作成
[root@sandbox ~]# cd /etc/ambari-server/conf
[root@sandbox conf]# f_ssl_self_signed_cert "/C=AU/ST=QLD/O=Hortonworks/CN=sandbox.hortonworks.com" "server"

[root@sandbox conf]# ambari-server setup-security
Using python  /usr/bin/python
Security setup options...
===========================================================================
Choose one of the following options:
  [1] Enable HTTPS for Ambari server.
  [2] Encrypt passwords stored in ambari.properties file.
  [3] Setup Ambari kerberos JAAS configuration.
  [4] Setup truststore.
  [5] Import certificate to truststore.
===========================================================================
Enter choice, (1-5): 1
Do you want to configure HTTPS [y/n] (y)?
SSL port [8443] ? 8080     # NOTE: Sandboxの場合はKnoxとポートがかぶるので注意
Enter path to Certificate: /etc/ambari-server/conf/srever.crt
Enter path to Private Key: /etc/ambari-server/conf/srever.key
Please enter password for Private Key:
WARNING: Common Name in Certificate: sandbox.hortonworks.com does not match the server FQDN: ho-ubu03.openstacklocal
WARNING: Unable to validate Certificate hostname
Importing and saving Certificate...done.
Ambari server URL changed. To make use of the Tez View in Ambari please update the property tez.tez-ui.history-url.base in tez-site
Adjusting ambari-server permissions and ownership...

[root@sandbox conf]# ambari-server setup-security
Using python  /usr/bin/python
Security setup options...
===========================================================================
Choose one of the following options:
  [1] Enable HTTPS for Ambari server.
  [2] Encrypt passwords stored in ambari.properties file.
  [3] Setup Ambari kerberos JAAS configuration.
  [4] Setup truststore.
  [5] Import certificate to truststore.
===========================================================================
Enter choice, (1-5): 4
Do you want to configure a truststore [y/n] (y)?
TrustStore type [jks/jceks/pkcs12] (jks):jks
Path to TrustStore file :/etc/ambari-server/conf/ambari-srever.jks
Password for TrustStore: (changeit)
Re-enter password: (changeit)
Ambari Server 'setup-security' completed successfully.

[root@sandbox conf]# keytool -import -file ./server.crt -alias ambari-server -noprompt -storepass changeit -keypass hadoop -keystore /etc/ambari-server/conf/ambari-srever.jks


[root@sandbox conf]# ambari-server restart
この時点でAmbariにSSL(HTTPS)でアクセスできるか確認

2)ambari.propertiesを編集して、またrestart
security.server.disabled.protocols=TLSv1.1|SSLv2Hello|SSLv3
https://issues.apache.org/jira/browse/AMBARI-17666 のため、TLSv1はdisableしない
TODO: security.server.disabled.ciphersは?

3)確認する:
root@ho-ubu03:~# ./test_ciphers.sh sandbox.hortonworks.com 8080 2>/dev/null
Obtaining cipher list from sandbox.hortonworks.com:8080 with OpenSSL 1.0.1f 6 Jan 2014.
ECDHE-RSA-AES256-GCM-SHA384     YES
ECDHE-RSA-AES256-SHA384 YES
ECDHE-RSA-AES256-SHA    YES
DHE-RSA-AES256-GCM-SHA384       YES
DHE-RSA-AES256-SHA256   YES
DHE-RSA-AES256-SHA      YES
AES256-GCM-SHA384       YES
AES256-SHA256   YES
AES256-SHA      YES
ECDHE-RSA-DES-CBC3-SHA  YES
EDH-RSA-DES-CBC3-SHA    YES
DES-CBC3-SHA    YES
ECDHE-RSA-AES128-GCM-SHA256     YES
ECDHE-RSA-AES128-SHA256 YES
ECDHE-RSA-AES128-SHA    YES
DHE-RSA-AES128-GCM-SHA256       YES
DHE-RSA-AES128-SHA256   YES
DHE-RSA-AES128-SHA      YES
AES128-GCM-SHA256       YES
AES128-SHA256   YES
AES128-SHA      YES
root@ho-ubu03:~# echo -n | openssl s_client -connect sandbox.hortonworks.com:8080 -tls1_2 

参考1:設定可能な値
factory.setIncludeProtocols(new String[] {"SSLv2Hello","SSLv3","TLSv1","TLSv1.1","TLSv1.2"});

参考2:TLSv1.2のCipherリストを表示には
[root@sandbox ~]# openssl ciphers -v | grep TLSv1.2 | wc -l
28

参考3:AmbariのJetty: /usr/lib/ambari-server/jetty-server-8.1.19.v20160209.jar

参考4:https://www.eclipse.org/jetty/documentation/9.4.x/configuring-ssl.html
TLS v1.2: The protocol which should be used wherever possible. All CBC based ciphers are supported since Java 7, the new GCM modes are supported since Java 8.
でも、ambari.propertiesのjava.homeを変更すると他のサービスのJAVA_HOMEも変わってしまう。
そこで、/usr/sbin/ambari_server_main.pyの"java_exe = get_java_exe_path()"(280行目あたり)を変更すると、どうやらJavaの実行ファイルを指定できる模様。

参考5:JVMが使えるCipherを表示する
https://confluence.atlassian.com/stashkb/list-ciphers-used-by-jvm-679609085.html
[root@sandbox ~]# wget https://confluence.atlassian.com/stashkb/files/679609085/679772359/1/1414093373406/Ciphers.java
[root@sandbox ~]# grep 'java.home' /etc/ambari-server/conf/ambari.properties
java.home=/usr/jdk64/jdk1.7.0_67
[root@sandbox ~]# /usr/jdk64/jdk1.7.0_67/bin/javac Ciphers.java
[root@sandbox ~]# /usr/jdk64/jdk1.7.0_67/bin/java Ciphers > jdk1.7_ciphers.out



TODO: 別の手段?
grep jdk.tls.disabledAlgorithms /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.101-3.b13.el6_8.x86_64/jre/lib/security/java.security
#   jdk.tls.disabledAlgorithms=MD5, SSLv3, DSA, RSA keySize < 2048
jdk.tls.disabledAlgorithms=SSLv3, RC4, MD5withRSA, DH keySize < 768

2017年2月24日金曜日

HostCleanupでうまくパッケージが消せない(アンインストールできない)

基本情報:

https://cwiki.apache.org/confluence/display/AMBARI/Host+Cleanup+for+Ambari+and+Stack

HostCleanup.pyは下記のResultファイルを作ります

[root@node4 ~]# ls -ltr /var/lib/ambari-agent/data/*.result
-rw-r--r-- 1 root root   855 Feb 24 02:20 /var/lib/ambari-agent/data/hostcheck.result
-rw-r--r-- 1 root root   911 Feb 24 02:20 /var/lib/ambari-agent/data/hostcheck_custom_actions.result
-rw-r--r-- 1 root root 21052 Feb 24 03:46 /var/lib/ambari-agent/data/hostcleanup.result

hostcheckfile

[root@node4 ~]# cat /var/lib/ambari-agent/data/hostcheck.result
[metadata]
created = 2017-02-24 02:20:05.757245

[users]
usr_list = hive,zookeeper,ambari-qa,tez,hdfs,yarn,hcat,mapred
usr_homedir_list = /home/hive,/home/zookeeper,/home/ambari-qa,/home/tez,/home/hdfs,/home/yarn,/home/hcat,/home/mapred

[alternatives]
symlink_list =
target_list =

[directories]
dir_list = /etc/hadoop,/etc/hive,/etc/zookeeper,/etc/hive-hcatalog,/etc/tez,/etc/hive-webhcat,/etc/pig,/var/run/hadoop,/var/run/hive,/var/run/zookeeper,/var/run/hive-hcatalog,/var/run/webhcat,/var/run/hadoop-yarn,/var/run/hadoop-mapreduce,/var/log/hadoop,/var/log/hive,/var/log/zookeeper,/var/log/hive-hcatalog,/var/log/hadoop-yarn,/var/log/hadoop-mapreduce,/var/lib/hive,/var/lib/hadoop-hdfs,/var/lib/hadoop-yarn,/var/lib/hadoop-mapreduce,/tmp/hadoop-hdfs,/hadoop/zookeeper,/hadoop/hdfs,/hadoop/yarn,/usr/hdp/current

[processes]
proc_list = 15757,17427


hostcheckfileca

[root@node4 ~]# cat /var/lib/ambari-agent/data/hostcheck_custom_actions.result
[metadata]
created = 2017-02-24 02:20:26.754150

[packages]
pkg_list = hadoop_2_5_3_0_37-hdfs.x86_64,zookeeper_2_5_3_0_37.noarch,hdp-select.noarch,hadoop_2_5_3_0_37-yarn.x86_64,hadoop_2_5_3_0_37-libhdfs.x86_64,atlas-metadata_2_5_3_0_37-hive-plugin.noarch,ranger_2_5_3_0_37-yarn-plugin.x86_64,hive_2_5_3_0_37-hcatalog.noarch,pig_2_5_3_0_37.noarch,spark_2_5_3_0_37-yarn-shuffle.noarch,hive2_2_5_3_0_37.noarch,ranger_2_5_3_0_37-hdfs-plugin.x86_64,zookeeper_2_5_3_0_37-server.noarch,tez_hive2_2_5_3_0_37.noarch,hive_2_5_3_0_37.noarch,hadoop_2_5_3_0_37.x86_64,hive_2_5_3_0_37-webhcat.noarch,spark2_2_5_3_0_37-yarn-shuffle.noarch,tez_2_5_3_0_37.noarch,hadoop_2_5_3_0_37-mapreduce.x86_64,hadoop_2_5_3_0_37-client.x86_64,datafu_2_5_3_0_37.noarch,hive2_2_5_3_0_37-jdbc.noarch,hive_2_5_3_0_37-jdbc.noarch,ranger_2_5_3_0_37-hive-plugin.x86_64,bigtop-jsvc.x86_64

[repositories]
repo_list = HDP-2.5,Updates-ambari-2.4.2.0

Resultファイルを消してみると

[root@node4 ~]# rm /var/lib/ambari-agent/data/hostcheck_custom_actions.result
rm: remove regular file `/var/lib/ambari-agent/data/hostcheck_custom_actions.result'? y
[root@node4 ~]# python /usr/lib/python2.6/site-packages/ambari_agent/HostCleanup.py --skip=users
Host Check results not found. There is no /var/lib/ambari-agent/data/hostcheck_custom_actions.result. Do you want to run host checks [ /n] (y)

INFO:HostCleanup:Executing command: /var/lib/ambari-agent/ambari-sudo.sh /var/lib/ambari-agent/cache/custom_actions/scripts/check_host.py ACTIONEXECUTE /tmp/tmpSni9Le /var/lib/ambari-agent/cache/custom_actions /tmp/tmpRX36M4 INFO /var/lib/ambari-agent/tmp

TODO: Host Checksとは?

[root@node4 ~]# cat /tmp/tmpSni9Le
{"commandParams": {"check_execute_list": "*BEFORE_CLEANUP_HOST_CHECKS*"}}

[root@node4 ~]# cat /tmp/tmpRX36M4{"transparentHugePage": {"message": "", "exit_code": 0}, "last_agent_env_check": {"transparentHugePage": "", "hostHealth": {"agentTimeStampAtReporting": 1487908261777, "activeJavaProcs": [], "liveServices": [{"status": "Unhealthy", "name": "ntpd", "desc": "ntpd: unrecognized service\n"}]}, "reverseLookup": true, "alternatives": [], "umask": "18", "firewallName": "iptables", "stackFoldersAndFiles": [], "existingUsers": [{"status": "Available", "name": "hive", "homeDir": "/home/hive"}, {"status": "Available", "name": "zookeeper", "homeDir": "/home/zookeeper"}, {"status": "Available", "name": "ambari-qa", "homeDir": "/home/ambari-qa"}, {"status": "Available", "name": "tez", "homeDir": "/home/tez"}, {"status": "Available", "name": "hdfs", "homeDir": "/home/hdfs"}, {"status": "Available", "name": "yarn", "homeDir": "/home/yarn"}, {"status": "Available", "name": "hcat", "homeDir": "/home/hcat"}, {"status": "Available", "name": "mapred", "homeDir": "/home/mapred"}], "firewallRunning": true}, "installed_packages": [], "existing_repos": ["Updates-ambari-2.4.2.0"]}

注意:
1. HDFS dfs.datanode.data.dir dfs.namenode.name.dir dfs.journalnode.edits.dir
2. 再インストール時に、/etc/hadoop/<version>/0 が作成されなかった
3. /etc/hadoop/conf が /usr/hdp/current/hadoop-client/conf ヘシムリンクされなかった

2017年2月15日水曜日

HDP 2.5 SandboxのZeppelinをKnoxのDemo LDAPと連携してみる

https://community.hortonworks.com/articles/76938/zeppelin-ldap-authentication-with-openldap.html

前のHDP 2.5.3のKnox Demo Ldapを使ってみるを参考に、Advanced zeppelin-envからshiro_ini_contentを下記に変更 ([users]は全てコメントアウトしてある):

[main]
ldapRealm = org.apache.shiro.realm.ldap.JndiLdapRealm
ldapRealm.userDnTemplate = uid={0},ou=people,dc=hadoop,dc=apache,dc=org
ldapRealm.contextFactory.url = ldap://sandbox.hortonworks.com:33389
ldapRealm.contextFactory.authenticationMechanism = SIMPLE
sessionManager = org.apache.shiro.web.session.mgt.DefaultWebSessionManager
securityManager.sessionManager = $sessionManager
securityManager.sessionManager.globalSessionTimeout = 86400000
shiro.loginUrl = /api/login

[urls]
/** = authc

あとは再起動すればOK。
HDP 2.5.3, 2.6.0だと設定が変わった模様
http://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.6.0/bk_zeppelin-component-guide/content/config-secure-prod.html
https://zeppelin.apache.org/docs/0.6.0/security/shiroauthentication.html

2017年2月12日日曜日

Ambari AgentをPythonからスタートしてみる


備考:インポートしているのは最低限。例えばPIDFILEとかは設定していません。 (でも必要なし?)

A. シェルの"export"で:
[root@node9 ~]# export PYTHON=/usr/bin/python2.6
[root@node9 ~]# export PYTHONPATH=/usr/lib/python2.6/site-packages:/usr/lib/python2.6/site-packages/ambari_commons
[root@node9 ~]# cd /usr/lib/python2.6/site-packages/ambari_agent/
[root@node9 ambari_agent]# python
import sys, os

B. Pythonの"sys.path"で
#cd /var/lib/ambari-agent
python
import sys, os
sys.path.append("/usr/lib/python2.6/site-packages")
sys.path.append("/usr/lib/python2.6/site-packages/ambari_agent")
sys.path.append("/usr/lib/ambari-agent")
# TODO: 下記も必要?
sys.path.append("/usr/lib/python2.6/site-packages/ambari_commons")
sys.path.append("/usr/lib/ambari-agent/lib")
sys.path.append("/usr/lib/ambari-agent/lib/resource_management/libraries/functions")
sys.path.append("/var/lib/ambari-agent/tmp")

共通
from ambari_agent.AmbariConfig import AmbariConfig
from ambari_agent.Controller import Controller
config = AmbariConfig()
config.read(AmbariConfig.getConfigFile())


オペレーションのコンフィグをロードしてみる
# load config for /var/lib/ambari-agent/cache/stacks/HDP/2.2/services/HBaseRest./package/scripts/params.py
#sys.path.append("/var/lib/ambari-agent/cache/stacks/HDP/2.2/services/HBaseRest./package/scripts")
import json
from resource_management.libraries.script.config_dictionary import ConfigDictionary, UnknownConfiguration
from resource_management.libraries import Script

command_data_file = "/var/lib/ambari-agent/data/command-1016.json"
with open(command_data_file) as f:
     pass
     Script.config = ConfigDictionary(json.load(f))

config = Script.get_config()
config['configurations'].keys()
[u'spark-defaults', u'ranger-knox-plugin-properties', u'ranger-hdfs-audit', u'zeppelin-config', u'ranger-hdfs-policymgr-ssl', u'pig-env', u'anonymization-rules', u'ranger-knox-audit', u'ranger-kafka-plugin-properties', u'slider-env', u'usersyn...


HBaseのコンフィグを使ってみる(失敗)
sys.path.append("/var/lib/ambari-agent/cache/common-services/HBASE/0.96.0.2.0/package/scripts")
from hbase import hbase
hbase(name='client')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/lib/python2.6/site-packages/ambari_commons/os_family_impl.py", line 89, in thunk
    return fn(*args, **kwargs)
  File "/var/lib/ambari-agent/cache/common-services/HBASE/0.96.0.2.0/package/scripts/hbase.py", line 47, in hbase
    import params
  File "/var/lib/ambari-agent/cache/common-services/HBASE/0.96.0.2.0/package/scripts/params.py", line 26, in <module>
    from params_linux import *
  File "/var/lib/ambari-agent/cache/common-services/HBASE/0.96.0.2.0/package/scripts/params_linux.py", line 20, in <module>
    import status_params
  File "/var/lib/ambari-agent/cache/common-services/HBASE/0.96.0.2.0/package/scripts/status_params.py", line 46, in <module>
    pid_dir = config['configurations']['hbase-env']['hbase_pid_dir']
TypeError: 'NoneType' object is unsubscriptable
>>>

HBaseのスタート時に使われるPath
['/var/lib/ambari-agent/cache/stacks/HDP/2.2/services/HBaseRest./package/scripts', '/usr/lib/python2.6/site-packages', '/var/lib/ambari-agent', '/usr/lib64/python26.zip', '/usr/lib64/python2.6', '/usr/lib64/python2.6/plat-linux2', '/usr/lib64/python2.6/lib-tk', '/usr/lib64/python2.6/lib-old', '/usr/lib64/python2.6/lib-dynload', '/usr/lib64/python2.6/site-packages', '/usr/lib64/python2.6/site-packages/gtk-2.0', '/usr/lib/python2.6/site-packages/setuptools-0.6c11-py2.6.egg-info', '/usr/lib/python2.6/site-packages/setuptools-0.6c11-py2.6.egg-info', '/usr/lib/python2.6/site-packages/setuptools-0.6c11-py2.6.egg-info']

HBaseのスタート時に使われるArgv
['/var/lib/ambari-agent/cache/stacks/HDP/2.2/services/HBaseRest./package/scripts/client.py', 'RESTART', '/var/lib/ambari-agent/data/command-1016.json', '/var/lib/ambari-agent/cache/stacks/HDP/2.2/services/HBaseRest./package', '/var/lib/ambari-agent/data/structured-out-1016.json', 'INFO', '/var/lib/ambari-agent/tmp']

上記の出力方法例:
/var/lib/ambari-agent/cache/stacks/HDP/2.2/services/HBaseRest./package/scripts/params.py", line 4
    print(str(sys.path), file=sys.stderr)
                             ^

参考:https://cwiki.apache.org/confluence/display/AMBARI/Defining+a+Custom+Stack+and+Services


Misc.:
export PYTHON=/usr/bin/python2.6
export PYTHONPATH=/usr/lib/python2.6/site-packages:/usr/lib/python2.6/site-packages/ambari_commons
#export PYTHONPATH=/usr/lib/python2.6/site-packages:/usr/lib/python2.6/site-packages/ambari_commons:/usr/lib/ambari-agent/lib/resource_management/libraries/functions:/var/lib/ambari-agent/tmp
export AMBARI_PASSPHRASE=DEV
PATH=/usr/sbin:/sbin:/usr/lib/ambari-server/*:/usr/lib64/qt-3.3/bin:/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin:/root/bin:/var/lib/ambari-agent /usr/bin/python2 -mtrace --trace /usr/lib/python2.6/site-packages/ambari_agent/AmbariAgent.py stop


備忘録:もしNumber of Under-Replicated Blocksが減らない場合チェックすること

NameNodeログ
INFO  BlockStateChange (BlockManager.java:computeReplicationWorkForBlocks(1527)) - BLOCK* neededReplications = 2030564, pendingReplications = 1497.
INFO  blockmanagement.BlockManager (BlockManager.java:computeReplicationWorkForBlocks(1534)) - Blocks chosen but could not be replicated = 20; of which 0 have no target, 20 have no source, 0 are UC, 0 are abandoned, 0 already have enough replicas.



最終手段的に、再度Replicateしてみる
https://community.hortonworks.com/articles/4427/fix-under-replicated-blocks-in-hdfs-manually.html

su - <$hdfs_user>

hdfs fsck / | grep 'Under replicated' | awk -F':' '{print $1}' >> /tmp/under_replicated_files 

for hdfsfile in `cat /tmp/under_replicated_files`; do echo "Fixing $hdfsfile :" ;  hadoop fs -setrep 3 $hdfsfile; done