Hadoop and DevSecOps notes & tips: 6月 2017

2017年6月21日水曜日

Sandbox HDP 2.5 のZeppelinをKnoxのKeystoreでSSL

慣れないとKeystore/Truststoreを設定するのは面倒なので、Knoxのを使いまわします。

https://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.6.0/bk_zeppelin-component-guide/content/config-ssl-zepp.html

Advanced zeppelin-config
zeppelin.ssl = true
zeppelin.ssl.key.manager.password = knox
zeppelin.ssl.keystore.password = knox
zeppelin.ssl.keystore.path = /etc/security/serverKeys/keystore.jks

cp /usr/hdp/current/knox-server/data/security/keystores/gateway.jks /etc/security/serverKeys/keystore.jks
chmod a+r /etc/security/serverKeys/keystore.jks

2017年6月19日月曜日

ノート：Hadoop Configuration.java

private Document parse(DocumentBuilder builder, URL url)
throws IOException, SAXException {
if (!quietmode) {
LOG.debug("parsing URL " + url);
}
if (url == null) {
return null;
}
return parse(builder, url.openStream(), url.toString());
}

とあるので、
Configuration conf = new Configuration();
conf.setQuietMode(false);

static{
//print deprecation warning if hadoop-site.xml is found in classpath
ClassLoader cL = Thread.currentThread().getContextClassLoader();
if (cL == null) {
cL = Configuration.class.getClassLoader();
}
if(cL.getResource("hadoop-site.xml")!=null) {
LOG.warn("DEPRECATED: hadoop-site.xml found in the classpath. " +
"Usage of hadoop-site.xml is deprecated. Instead use core-site.xml, "
+ "mapred-site.xml and hdfs-site.xml to override properties of " +
"core-default.xml, mapred-default.xml and hdfs-default.xml " +
"respectively");
}
addDefaultResource("core-default.xml");
addDefaultResource("core-site.xml");
}

とあるので、
ClassLoader cl = Thread.currentThread().getContextClassLoader();
System.out.println(Thread.currentThread().getName()+" core-site.xml path = "+cl.getResource("core-site.xml"));

2017年6月15日木曜日

Ambariはどうやってコンポーネントのステータスを確認しているのか

/var/lib/ambari-agent/cache/common-services/HDFS/2.1.0.2.0/package/scripts/hdfs_namenode.py

from resource_management.libraries.functions.check_process_status import check_process_status

elif action == "status":
import status_params
check_process_status(status_params.namenode_pid_file)

Ambari Server側：
[root@node16 resources]# find . -type d -name package
./common-services/AMBARI_METRICS/0.1.0/package
./common-services/ATLAS/0.1.0.2.3/package
./common-services/HDFS/2.1.0.2.0/package
./common-services/HIVE/0.12.0.2.0/package
./common-services/HAWQ/2.0.0/package
./common-services/RANGER/0.4.0/package
./common-services/YARN/2.1.0.2.0/package
./common-services/SLIDER/0.60.0.2.2/package
./common-services/KNOX/0.5.0.2.2/package
./common-services/AMBARI_INFRA/0.1.0/package
./common-services/PIG/0.12.0.2.0/package
./common-services/TEZ/0.4.0.2.1/package
./common-services/OOZIE/4.0.0.2.0/package
./common-services/ZOOKEEPER/3.4.5/package
./common-services/GANGLIA/3.5.0/package
./common-services/RANGER_KMS/0.5.0.2.3/package
./common-services/KERBEROS/1.10.3-10/package
./common-services/SQOOP/1.4.4.2.0/package
./common-services/FLUME/1.4.0.2.0/package
./common-services/ACCUMULO/1.6.1.2.2.0/package
./common-services/MAHOUT/1.0.0.2.3/package
./common-services/PXF/3.0.0/package
./common-services/ZEPPELIN/0.6.0.2.5/package
./common-services/SPARK2/2.0.0/package
./common-services/KAFKA/0.8.1/package
./common-services/LOGSEARCH/0.5.0/package
./common-services/FALCON/0.5.0.2.1/package
./common-services/DRUID/0.9.2/package
./common-services/SPARK/1.2.1/package
./common-services/HBASE/0.96.0.2.0/package
./common-services/STORM/0.9.1/package
./stacks/HDP/2.3.ECS/services/ECS/package
./stacks/HDP/2.3.GlusterFS/services/GLUSTERFS/package
./stacks/HDP/2.1/services/SMARTSENSE/package
./stacks/HDP/2.0.6.GlusterFS/services/YARN/package
./stacks/HDP/2.0.6.GlusterFS/services/GLUSTERFS/package
./stacks/HDP/2.1.GlusterFS/services/YARN/package
./stacks/HDP/2.1.GlusterFS/services/GLUSTERFS/package
./stacks/HDP/2.1.GlusterFS/services/TEZ/package
./stacks/HDP/2.1.GlusterFS/services/FALCON/package
./stacks/HDP/2.1.GlusterFS/services/STORM/package

check_process_statusはKillコマンドで確認している
sudo.kill(pid, 0)

Smart SenseはPIDを確認していない
/var/lib/ambari-server/resources/stacks/HDP/2.1/services/SMARTSENSE/package/scripts/activity.py
def status(self, env):
import params

cmd = '/usr/sbin/hst activity status '
if self.component is 'explorer':
cmd = '/usr/sbin/hst activity-explorer status '
elif self.component is 'analyzer':
cmd = '/usr/sbin/hst activity-analyzer status '

2017年6月14日水曜日

HDP 2.5 SandboxにあえてHadoop KMSを設定する

参考：
https://www.ibm.com/support/knowledgecenter/en/SSWTQQ_2.0.0/install/t_trd_hadoopencryption.html
https://hadoop.apache.org/docs/current3/hadoop-kms/index.html

インストール？からスタートまで
su - kms
cp /usr/hdp/2.5.0.0-1245/hadoop/mapreduce.tar.gz ./
tar xvf mapreduce.tar.gz
cd ./hadoop/sbin/
./kms.sh start

コンフィグロケーション
[kms@sandbox sbin]$ ls -l ~/hadoop/etc/hadoop
total 156
-rw-r--r-- 1 kms hadoop 4436 Aug 26 2016 capacity-scheduler.xml
-rw-r--r-- 1 kms hadoop 1335 Aug 26 2016 configuration.xsl
-rw-r--r-- 1 kms hadoop 318 Aug 26 2016 container-executor.cfg
-rw-r--r-- 1 kms hadoop 894 Apr 12 11:58 core-site.xml
-rw-r--r-- 1 kms hadoop 3979 Aug 26 2016 hadoop-env.cmd
-rw-r--r-- 1 kms hadoop 4529 Aug 26 2016 hadoop-env.sh
-rw-r--r-- 1 kms hadoop 2598 Aug 26 2016 hadoop-metrics2.properties
-rw-r--r-- 1 kms hadoop 2490 Aug 26 2016 hadoop-metrics.properties
-rw-r--r-- 1 kms hadoop 9683 Aug 26 2016 hadoop-policy.xml
-rw-r--r-- 1 kms hadoop 775 Aug 26 2016 hdfs-site.xml
-rw-r--r-- 1 kms hadoop 1449 Aug 26 2016 httpfs-env.sh
-rw-r--r-- 1 kms hadoop 1657 Aug 26 2016 httpfs-log4j.properties
-rw-r--r-- 1 kms hadoop 21 Aug 26 2016 httpfs-signature.secret
-rw-r--r-- 1 kms hadoop 620 Aug 26 2016 httpfs-site.xml
-rw-r--r-- 1 kms hadoop 3518 Aug 26 2016 kms-acls.xml
-rw-r--r-- 1 kms hadoop 1527 Aug 26 2016 kms-env.sh
-rw-r--r-- 1 kms hadoop 1631 Aug 26 2016 kms-log4j.properties
-rw-r--r-- 1 kms hadoop 5511 Aug 26 2016 kms-site.xml
-rw-r--r-- 1 kms hadoop 12302 Aug 26 2016 log4j.properties
-rw-r--r-- 1 kms hadoop 951 Aug 26 2016 mapred-env.cmd
-rw-r--r-- 1 kms hadoop 1383 Aug 26 2016 mapred-env.sh
-rw-r--r-- 1 kms hadoop 4113 Aug 26 2016 mapred-queues.xml.template
-rw-r--r-- 1 kms hadoop 758 Aug 26 2016 mapred-site.xml.template
-rw-r--r-- 1 kms hadoop 10 Aug 26 2016 slaves
-rw-r--r-- 1 kms hadoop 2316 Aug 26 2016 ssl-client.xml.example
-rw-r--r-- 1 kms hadoop 2697 Aug 26 2016 ssl-server.xml.example
-rw-r--r-- 1 kms hadoop 2250 Aug 26 2016 yarn-env.cmd
-rw-r--r-- 1 kms hadoop 4567 Aug 26 2016 yarn-env.sh
-rw-r--r-- 1 kms hadoop 690 Aug 26 2016 yarn-site.xml

ポート番号は16000と16001ぽい
[kms@sandbox sbin]$ ps auxwww | grep kms
kms 27734 2.1 1.1 6513948 190760 pts/1 Sl 10:14 0:08 /usr/lib/jvm/java/bin/java -Djava.util.logging.config.file=/var/lib/ranger/kms/hadoop/share/hadoop/kms/tomcat/conf/logging.properties -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager -Dkms.home.dir=/var/lib/ranger/kms/hadoop -Dkms.config.dir=/var/lib/ranger/kms/hadoop/etc/hadoop -Dkms.log.dir=/var/lib/ranger/kms/hadoop/logs -Dkms.temp.dir=/var/lib/ranger/kms/hadoop/temp -Dkms.admin.port=16001 -Dkms.http.port=16000 -Dkms.max.threads=1000 -Dkms.ssl.keystore.file=/var/lib/ranger/kms/.keystore -Djava.library.path=/var/lib/ranger/kms/hadoop/libexec/../lib/native/ -Djava.endorsed.dirs=/var/lib/ranger/kms/hadoop/share/hadoop/kms/tomcat/endorsed -classpath /var/lib/ranger/kms/hadoop/share/hadoop/kms/tomcat/bin/bootstrap.jar -Dcatalina.base=/var/lib/ranger/kms/hadoop/share/hadoop/kms/tomcat -Dcatalina.home=/var/lib/ranger/kms/hadoop/share/hadoop/kms/tomcat -Djava.io.tmpdir=/var/lib/ranger/kms/hadoop/share/hadoop/kms/tomcat/temp org.apache.catalina.startup.Bootstrap start
[kms@sandbox sbin]$ lsof -p 27734
-bash: lsof: command not found
[kms@sandbox sbin]$ netstat -aopen | grep 27734
(Not all processes could be identified, non-owned process info
will not be shown, you would have to be root to see it all.)
tcp 0 0 0.0.0.0:16000 0.0.0.0:* LISTEN 1006 4759355 27734/java off (0.00/0/0)
tcp 0 0 127.0.0.1:16001 0.0.0.0:* LISTEN 1006 5741705 27734/java off (0.00/0/0)
unix 2 [ ] STREAM CONNECTED 4759353 27734/java
[kms@sandbox sbin]$ grep -Ew '16000|16001' ~/hadoop/etc/hadoop/*
/var/lib/ranger/kms/hadoop/etc/hadoop/kms-env.sh:# export KMS_HTTP_PORT=16000

SandboxにRanger KMSを入れてしまったので、面倒なのでポートを9292（と、勝手に9293になる）に変更
変更しない場合はhadoop.security.key.provider.path (kms-site.xml?)とdfs.encryption.key.provider.uri (hdfs-site.xml)を変更する必要あり。
[kms@sandbox sbin]$ ./kms.sh start
setting KMS_HTTP_PORT=9292
Using CATALINA_BASE: /var/lib/ranger/kms/hadoop/share/hadoop/kms/tomcat
Using CATALINA_HOME: /var/lib/ranger/kms/hadoop/share/hadoop/kms/tomcat
Using CATALINA_TMPDIR: /var/lib/ranger/kms/hadoop/share/hadoop/kms/tomcat/temp
Using JRE_HOME: /usr/lib/jvm/java
Using CLASSPATH: /var/lib/ranger/kms/hadoop/share/hadoop/kms/tomcat/bin/bootstrap.jar
Using CATALINA_PID: /tmp/kms.pid

Keystoreの場所とデフォルトのパスワード
<property>
<name>hadoop.kms.key.provider.uri</name>
<value>jceks://file@/${user.home}/kms.keystore</value>
<description>
URI of the backing KeyProvider for the KMS.
</description>
</property>

<property>
<name>hadoop.security.keystore.JavaKeyStoreProvider.password</name>
<value>none</value>
<description>
If using the JavaKeyStoreProvider, the password for the keystore file.
</description>
</property>

テスト（リストと作成）
[root@sandbox sbin]# su - ambari-qa
[ambari-qa@sandbox ~]$ hadoop key list
Listing keys for KeyProvider: KMSClientProvider[http://sandbox.hortonworks.com:9292/kms/v1/]
[ambari-qa@sandbox ~]$ hadoop key create ambariqa-key
ambariqa-key has been successfully created with options Options{cipher='AES/CTR/NoPadding', bitLength=128, description='null', attributes=null}.
KMSClientProvider[http://sandbox.hortonworks.com:9292/kms/v1/] has been updated.
[ambari-qa@sandbox ~]$ hadoop key list
Listing keys for KeyProvider: KMSClientProvider[http://sandbox.hortonworks.com:9292/kms/v1/]
ambariqa-key

Keystoreを見てみる
[root@sandbox sbin]# su - kms
[kms@sandbox ~]$ ls -ltr
total 200636
-rw-r--r-- 1 kms hadoop 205440821 Jun 14 10:12 mapreduce.tar.gz
drwxr-xr-x 11 kms hadoop 4096 Jun 14 10:14 hadoop
-rwx------ 1 kms hadoop 1140 Jun 14 10:33 kms.keystore
[kms@sandbox ~]$ file kms.keystore
kms.keystore: Java JCE KeyStore
[kms@sandbox ~]$ keytool -list -keystore ./kms.keystore
keytool error: java.io.IOException: Invalid keystore format
[kms@sandbox ~]$ keytool -list -keystore ./kms.keystore -storetype jceks
Enter keystore password:

***************** WARNING WARNING WARNING *****************
* The integrity of the information stored in your keystore *
* has NOT been verified! In order to verify its integrity, *
* you must provide your keystore password. *
***************** WARNING WARNING WARNING *****************

Keystore type: JCEKS
Keystore provider: SunJCE

Your keystore contains 2 entries

ambariqa-key@0, Jun 14, 2017, SecretKeyEntry,
ambariqa-key, Jun 14, 2017, SecretKeyEntry,
[kms@sandbox ~]$ keytool -list -keystore ./kms.keystore -storetype jceks -storepass none

Keystore type: JCEKS
Keystore provider: SunJCE

Your keystore contains 2 entries

ambariqa-key@0, Jun 14, 2017, SecretKeyEntry,
ambariqa-key, Jun 14, 2017, SecretKeyEntry,
[kms@sandbox ~]$

自分のホームディレクトリをエンクリプトしてみる
[root@sandbox ~]# su - hdfs
[hdfs@sandbox ~]$ hdfs crypto -createZone -keyName ambariqa-key -path /user/hajime
Added encryption zone /user/hajime
[hdfs@sandbox ~]$ hdfs dfs -chown hajime:hadoop /user/hajime

[hdfs@sandbox ~]$

確認してみる
[root@sandbox ~]# su - hajime
[hajime@sandbox ~]$ hdfs dfs -put /tmp/words.txt
[hajime@sandbox ~]$ hdfs dfs -cat /user/hajime/words.txt

a
[root@sandbox ~]# su - hdfs
[hdfs@sandbox ~]$ hdfs dfs -cat /.reserved/raw/user/hajime/words.txt

�[hdfs@sandbox ~]$

パスワードを変更してみる
keytool -storepasswd -keystore ./kms.keystore -storetype jceks -storepass 'none'
と、起動しなくなる
Stacktrace:
---------------------------------------------------
java.io.IOException: Keystore was tampered with, or password was incorrect
at com.sun.crypto.provider.JceKeyStore.engineLoad(JceKeyStore.java:865)
at java.security.KeyStore.load(KeyStore.java:1226)
at org.apache.hadoop.crypto.key.JavaKeyStoreProvider.loadFromPath(JavaKeyStoreProvider.java:305)

at org.apache.hadoop.crypto.key.JavaKeyStoreProvider.tryLoadFromPath(JavaKeyStoreProvider.java:213)

新しいパスワードで再起動：
HADOOP_KEYSTORE_PASSWORD=testtest ~/hadoop/sbin/kms.sh start

備考：

HDP付属のApache KMSは古いバージョンの模様。

CDHなどでトライすると（CDHのHadoopも古いですが）”hadoop.security.keystore.java-keystore-provider.password-file”を使用する必要あり。

そして、未解決のバグあり。

https://issues.apache.org/jira/browse/HADOOP-12777

$HADOOP_PREFIX/share/hadoop/kms/tomcat/webapps/kms/WEB-INF/classes/

export CLASSPATH=$CLASSPATH:/usr/local/hadoop/etc/hadoop/kms.keystore.password

2017年6月13日火曜日

Ambari2.2.2.xに新しい（ダミー）サービスを追加してみる (add custom service)

https://cwiki.apache.org/confluence/display/AMBARI/Custom+Services
https://cwiki.apache.org/confluence/display/AMBARI/Defining+a+Custom+Stack+and+Services
https://cwiki.apache.org/confluence/display/AMBARI/Defining+a+Custom+Service (Ambari 2.4.0以上っぽい参照：https://issues.apache.org/jira/browse/AMBARI-14854)

https://developer.ibm.com/hadoop/2015/10/30/adding-a-custom-service-to-ibm-open-platform/
Ambari 2.2.2.xだと下記のファイルを作成する (TODO: -configs.j2必要？):

1. /var/lib/ambari-server/resources/stacks/HDP/2.2/services/${NEW_SERVICE}/metainfo.xml

サービスに使用されるサービス名、コンポーネント名やファイル名を記述する

2. /var/lib/ambari-server/resources/stacks/HDP/2.2/services/${NEW_SERVICE}/package/scripts/params.py
TODO:Ambariのオペレーションで使われる

3. /var/lib/ambari-server/resources/stacks/HDP/2.2/services/${NEW_SERVICE}/package/scripts/(metainfo commandScript file name .py)

コンポーネントの開始・停止などのコマンドを記述する

4. /var/lib/ambari-server/resources/stacks/HDP/2.2/services/${NEW_SERVICE}/configuration/${NEW_SERVICE}-config.xml

Ambariに表示されるプロパティ名とそのデフォルトの値を記述する