参考：
https://www.ibm.com/support/knowledgecenter/en/SSWTQQ_2.0.0/install/t_trd_hadoopencryption.html
https://hadoop.apache.org/docs/current3/hadoop-kms/index.html
インストール？からスタートまで
su - kms
cp /usr/hdp/2.5.0.0-1245/hadoop/mapreduce.tar.gz ./
tar xvf mapreduce.tar.gz
cd ./hadoop/sbin/
./kms.sh start
コンフィグロケーション
[kms@sandbox sbin]$ ls -l ~/hadoop/etc/hadoop
total 156
-rw-r--r-- 1 kms hadoop 4436 Aug 26 2016 capacity-scheduler.xml
-rw-r--r-- 1 kms hadoop 1335 Aug 26 2016 configuration.xsl
-rw-r--r-- 1 kms hadoop 318 Aug 26 2016 container-executor.cfg
-rw-r--r-- 1 kms hadoop 894 Apr 12 11:58 core-site.xml
-rw-r--r-- 1 kms hadoop 3979 Aug 26 2016 hadoop-env.cmd
-rw-r--r-- 1 kms hadoop 4529 Aug 26 2016 hadoop-env.sh
-rw-r--r-- 1 kms hadoop 2598 Aug 26 2016 hadoop-metrics2.properties
-rw-r--r-- 1 kms hadoop 2490 Aug 26 2016 hadoop-metrics.properties
-rw-r--r-- 1 kms hadoop 9683 Aug 26 2016 hadoop-policy.xml
-rw-r--r-- 1 kms hadoop 775 Aug 26 2016 hdfs-site.xml
-rw-r--r-- 1 kms hadoop 1449 Aug 26 2016 httpfs-env.sh
-rw-r--r-- 1 kms hadoop 1657 Aug 26 2016 httpfs-log4j.properties
-rw-r--r-- 1 kms hadoop 21 Aug 26 2016 httpfs-signature.secret
-rw-r--r-- 1 kms hadoop 620 Aug 26 2016 httpfs-site.xml
-rw-r--r-- 1 kms hadoop 3518 Aug 26 2016 kms-acls.xml
-rw-r--r-- 1 kms hadoop 1527 Aug 26 2016 kms-env.sh
-rw-r--r-- 1 kms hadoop 1631 Aug 26 2016 kms-log4j.properties
-rw-r--r-- 1 kms hadoop 5511 Aug 26 2016 kms-site.xml
-rw-r--r-- 1 kms hadoop 12302 Aug 26 2016 log4j.properties
-rw-r--r-- 1 kms hadoop 951 Aug 26 2016 mapred-env.cmd
-rw-r--r-- 1 kms hadoop 1383 Aug 26 2016 mapred-env.sh
-rw-r--r-- 1 kms hadoop 4113 Aug 26 2016 mapred-queues.xml.template
-rw-r--r-- 1 kms hadoop 758 Aug 26 2016 mapred-site.xml.template
-rw-r--r-- 1 kms hadoop 10 Aug 26 2016 slaves
-rw-r--r-- 1 kms hadoop 2316 Aug 26 2016 ssl-client.xml.example
-rw-r--r-- 1 kms hadoop 2697 Aug 26 2016 ssl-server.xml.example
-rw-r--r-- 1 kms hadoop 2250 Aug 26 2016 yarn-env.cmd
-rw-r--r-- 1 kms hadoop 4567 Aug 26 2016 yarn-env.sh
-rw-r--r-- 1 kms hadoop 690 Aug 26 2016 yarn-site.xml
ポート番号は16000と16001ぽい
[kms@sandbox sbin]$ ps auxwww | grep kms
kms 27734 2.1 1.1 6513948 190760 pts/1 Sl 10:14 0:08 /usr/lib/jvm/java/bin/java -Djava.util.logging.config.file=/var/lib/ranger/kms/hadoop/share/hadoop/kms/tomcat/conf/logging.properties -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager -Dkms.home.dir=/var/lib/ranger/kms/hadoop -Dkms.config.dir=/var/lib/ranger/kms/hadoop/etc/hadoop -Dkms.log.dir=/var/lib/ranger/kms/hadoop/logs -Dkms.temp.dir=/var/lib/ranger/kms/hadoop/temp -Dkms.admin.port=16001 -Dkms.http.port=16000 -Dkms.max.threads=1000 -Dkms.ssl.keystore.file=/var/lib/ranger/kms/.keystore -Djava.library.path=/var/lib/ranger/kms/hadoop/libexec/../lib/native/ -Djava.endorsed.dirs=/var/lib/ranger/kms/hadoop/share/hadoop/kms/tomcat/endorsed -classpath /var/lib/ranger/kms/hadoop/share/hadoop/kms/tomcat/bin/bootstrap.jar -Dcatalina.base=/var/lib/ranger/kms/hadoop/share/hadoop/kms/tomcat -Dcatalina.home=/var/lib/ranger/kms/hadoop/share/hadoop/kms/tomcat -Djava.io.tmpdir=/var/lib/ranger/kms/hadoop/share/hadoop/kms/tomcat/temp org.apache.catalina.startup.Bootstrap start
[kms@sandbox sbin]$ lsof -p 27734
-bash: lsof: command not found
[kms@sandbox sbin]$ netstat -aopen | grep 27734
(Not all processes could be identified, non-owned process info
will not be shown, you would have to be root to see it all.)
tcp 0 0 0.0.0.0:16000 0.0.0.0:* LISTEN 1006 4759355 27734/java off (0.00/0/0)
tcp 0 0 127.0.0.1:16001 0.0.0.0:* LISTEN 1006 5741705 27734/java off (0.00/0/0)
unix 2 [ ] STREAM CONNECTED 4759353 27734/java
[kms@sandbox sbin]$ grep -Ew '16000|16001' ~/hadoop/etc/hadoop/*
/var/lib/ranger/kms/hadoop/etc/hadoop/kms-env.sh:# export KMS_HTTP_PORT=16000
SandboxにRanger KMSを入れてしまったので、面倒なのでポートを9292（と、勝手に9293になる）に変更
変更しない場合はhadoop.security.key.provider.path (kms-site.xml?)とdfs.encryption.key.provider.uri (hdfs-site.xml)を変更する必要あり。
[kms@sandbox sbin]$ ./kms.sh start
setting KMS_HTTP_PORT=9292
Using CATALINA_BASE: /var/lib/ranger/kms/hadoop/share/hadoop/kms/tomcat
Using CATALINA_HOME: /var/lib/ranger/kms/hadoop/share/hadoop/kms/tomcat
Using CATALINA_TMPDIR: /var/lib/ranger/kms/hadoop/share/hadoop/kms/tomcat/temp
Using JRE_HOME: /usr/lib/jvm/java
Using CLASSPATH: /var/lib/ranger/kms/hadoop/share/hadoop/kms/tomcat/bin/bootstrap.jar
Using CATALINA_PID: /tmp/kms.pid
Keystoreの場所とデフォルトのパスワード
<property>
<name>hadoop.kms.key.provider.uri</name>
<value>jceks://file@/${user.home}/kms.keystore</value>
<description>
URI of the backing KeyProvider for the KMS.
</description>
</property>
<property>
<name>hadoop.security.keystore.JavaKeyStoreProvider.password</name>
<value>none</value>
<description>
If using the JavaKeyStoreProvider, the password for the keystore file.
</description>
</property>
テスト（リストと作成）
[root@sandbox sbin]# su - ambari-qa
[ambari-qa@sandbox ~]$ hadoop key list
Listing keys for KeyProvider: KMSClientProvider[http://sandbox.hortonworks.com:9292/kms/v1/]
[ambari-qa@sandbox ~]$ hadoop key create ambariqa-key
ambariqa-key has been successfully created with options Options{cipher='AES/CTR/NoPadding', bitLength=128, description='null', attributes=null}.
KMSClientProvider[http://sandbox.hortonworks.com:9292/kms/v1/] has been updated.
[ambari-qa@sandbox ~]$ hadoop key list
Listing keys for KeyProvider: KMSClientProvider[http://sandbox.hortonworks.com:9292/kms/v1/]
ambariqa-key
Keystoreを見てみる
[root@sandbox sbin]# su - kms
[kms@sandbox ~]$ ls -ltr
total 200636
-rw-r--r-- 1 kms hadoop 205440821 Jun 14 10:12 mapreduce.tar.gz
drwxr-xr-x 11 kms hadoop 4096 Jun 14 10:14 hadoop
-rwx------ 1 kms hadoop 1140 Jun 14 10:33 kms.keystore
[kms@sandbox ~]$ file kms.keystore
kms.keystore: Java JCE KeyStore
[kms@sandbox ~]$ keytool -list -keystore ./kms.keystore
keytool error: java.io.IOException: Invalid keystore format
[kms@sandbox ~]$ keytool -list -keystore ./kms.keystore -storetype jceks
Enter keystore password:
***************** WARNING WARNING WARNING *****************
* The integrity of the information stored in your keystore *
* has NOT been verified! In order to verify its integrity, *
* you must provide your keystore password. *
***************** WARNING WARNING WARNING *****************
Keystore type: JCEKS
Keystore provider: SunJCE
Your keystore contains 2 entries
ambariqa-key@0, Jun 14, 2017, SecretKeyEntry,
ambariqa-key, Jun 14, 2017, SecretKeyEntry,
[kms@sandbox ~]$ keytool -list -keystore ./kms.keystore -storetype jceks -storepass none
Keystore type: JCEKS
Keystore provider: SunJCE
Your keystore contains 2 entries
ambariqa-key@0, Jun 14, 2017, SecretKeyEntry,
ambariqa-key, Jun 14, 2017, SecretKeyEntry,
[kms@sandbox ~]$
自分のホームディレクトリをエンクリプトしてみる
[root@sandbox ~]# su - hdfs
[hdfs@sandbox ~]$ hdfs crypto -createZone -keyName ambariqa-key -path /user/hajime
Added encryption zone /user/hajime
[hdfs@sandbox ~]$ hdfs dfs -chown hajime:hadoop /user/hajime
[hdfs@sandbox ~]$
確認してみる
[root@sandbox ~]# su - hajime
[hajime@sandbox ~]$ hdfs dfs -put /tmp/words.txt
[hajime@sandbox ~]$ hdfs dfs -cat /user/hajime/words.txt
a
[root@sandbox ~]# su - hdfs
[hdfs@sandbox ~]$ hdfs dfs -cat /.reserved/raw/user/hajime/words.txt
�[hdfs@sandbox ~]$
パスワードを変更してみる
keytool -storepasswd -keystore ./kms.keystore -storetype jceks -storepass 'none'
と、起動しなくなる
Stacktrace:
---------------------------------------------------
java.io.IOException: Keystore was tampered with, or password was incorrect
at com.sun.crypto.provider.JceKeyStore.engineLoad(JceKeyStore.java:865)
at java.security.KeyStore.load(KeyStore.java:1226)
at org.apache.hadoop.crypto.key.JavaKeyStoreProvider.loadFromPath(JavaKeyStoreProvider.java:305)
at org.apache.hadoop.crypto.key.JavaKeyStoreProvider.tryLoadFromPath(JavaKeyStoreProvider.java:213)
新しいパスワードで再起動：
HADOOP_KEYSTORE_PASSWORD=testtest ~/hadoop/sbin/kms.sh start
https://www.ibm.com/support/knowledgecenter/en/SSWTQQ_2.0.0/install/t_trd_hadoopencryption.html
https://hadoop.apache.org/docs/current3/hadoop-kms/index.html
インストール？からスタートまで
su - kms
cp /usr/hdp/2.5.0.0-1245/hadoop/mapreduce.tar.gz ./
tar xvf mapreduce.tar.gz
cd ./hadoop/sbin/
./kms.sh start
コンフィグロケーション
[kms@sandbox sbin]$ ls -l ~/hadoop/etc/hadoop
total 156
-rw-r--r-- 1 kms hadoop 4436 Aug 26 2016 capacity-scheduler.xml
-rw-r--r-- 1 kms hadoop 1335 Aug 26 2016 configuration.xsl
-rw-r--r-- 1 kms hadoop 318 Aug 26 2016 container-executor.cfg
-rw-r--r-- 1 kms hadoop 894 Apr 12 11:58 core-site.xml
-rw-r--r-- 1 kms hadoop 3979 Aug 26 2016 hadoop-env.cmd
-rw-r--r-- 1 kms hadoop 4529 Aug 26 2016 hadoop-env.sh
-rw-r--r-- 1 kms hadoop 2598 Aug 26 2016 hadoop-metrics2.properties
-rw-r--r-- 1 kms hadoop 2490 Aug 26 2016 hadoop-metrics.properties
-rw-r--r-- 1 kms hadoop 9683 Aug 26 2016 hadoop-policy.xml
-rw-r--r-- 1 kms hadoop 775 Aug 26 2016 hdfs-site.xml
-rw-r--r-- 1 kms hadoop 1449 Aug 26 2016 httpfs-env.sh
-rw-r--r-- 1 kms hadoop 1657 Aug 26 2016 httpfs-log4j.properties
-rw-r--r-- 1 kms hadoop 21 Aug 26 2016 httpfs-signature.secret
-rw-r--r-- 1 kms hadoop 620 Aug 26 2016 httpfs-site.xml
-rw-r--r-- 1 kms hadoop 3518 Aug 26 2016 kms-acls.xml
-rw-r--r-- 1 kms hadoop 1527 Aug 26 2016 kms-env.sh
-rw-r--r-- 1 kms hadoop 1631 Aug 26 2016 kms-log4j.properties
-rw-r--r-- 1 kms hadoop 5511 Aug 26 2016 kms-site.xml
-rw-r--r-- 1 kms hadoop 12302 Aug 26 2016 log4j.properties
-rw-r--r-- 1 kms hadoop 951 Aug 26 2016 mapred-env.cmd
-rw-r--r-- 1 kms hadoop 1383 Aug 26 2016 mapred-env.sh
-rw-r--r-- 1 kms hadoop 4113 Aug 26 2016 mapred-queues.xml.template
-rw-r--r-- 1 kms hadoop 758 Aug 26 2016 mapred-site.xml.template
-rw-r--r-- 1 kms hadoop 10 Aug 26 2016 slaves
-rw-r--r-- 1 kms hadoop 2316 Aug 26 2016 ssl-client.xml.example
-rw-r--r-- 1 kms hadoop 2697 Aug 26 2016 ssl-server.xml.example
-rw-r--r-- 1 kms hadoop 2250 Aug 26 2016 yarn-env.cmd
-rw-r--r-- 1 kms hadoop 4567 Aug 26 2016 yarn-env.sh
-rw-r--r-- 1 kms hadoop 690 Aug 26 2016 yarn-site.xml
ポート番号は16000と16001ぽい
[kms@sandbox sbin]$ ps auxwww | grep kms
kms 27734 2.1 1.1 6513948 190760 pts/1 Sl 10:14 0:08 /usr/lib/jvm/java/bin/java -Djava.util.logging.config.file=/var/lib/ranger/kms/hadoop/share/hadoop/kms/tomcat/conf/logging.properties -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager -Dkms.home.dir=/var/lib/ranger/kms/hadoop -Dkms.config.dir=/var/lib/ranger/kms/hadoop/etc/hadoop -Dkms.log.dir=/var/lib/ranger/kms/hadoop/logs -Dkms.temp.dir=/var/lib/ranger/kms/hadoop/temp -Dkms.admin.port=16001 -Dkms.http.port=16000 -Dkms.max.threads=1000 -Dkms.ssl.keystore.file=/var/lib/ranger/kms/.keystore -Djava.library.path=/var/lib/ranger/kms/hadoop/libexec/../lib/native/ -Djava.endorsed.dirs=/var/lib/ranger/kms/hadoop/share/hadoop/kms/tomcat/endorsed -classpath /var/lib/ranger/kms/hadoop/share/hadoop/kms/tomcat/bin/bootstrap.jar -Dcatalina.base=/var/lib/ranger/kms/hadoop/share/hadoop/kms/tomcat -Dcatalina.home=/var/lib/ranger/kms/hadoop/share/hadoop/kms/tomcat -Djava.io.tmpdir=/var/lib/ranger/kms/hadoop/share/hadoop/kms/tomcat/temp org.apache.catalina.startup.Bootstrap start
[kms@sandbox sbin]$ lsof -p 27734
-bash: lsof: command not found
[kms@sandbox sbin]$ netstat -aopen | grep 27734
(Not all processes could be identified, non-owned process info
will not be shown, you would have to be root to see it all.)
tcp 0 0 0.0.0.0:16000 0.0.0.0:* LISTEN 1006 4759355 27734/java off (0.00/0/0)
tcp 0 0 127.0.0.1:16001 0.0.0.0:* LISTEN 1006 5741705 27734/java off (0.00/0/0)
unix 2 [ ] STREAM CONNECTED 4759353 27734/java
[kms@sandbox sbin]$ grep -Ew '16000|16001' ~/hadoop/etc/hadoop/*
/var/lib/ranger/kms/hadoop/etc/hadoop/kms-env.sh:# export KMS_HTTP_PORT=16000
SandboxにRanger KMSを入れてしまったので、面倒なのでポートを9292（と、勝手に9293になる）に変更
変更しない場合はhadoop.security.key.provider.path (kms-site.xml?)とdfs.encryption.key.provider.uri (hdfs-site.xml)を変更する必要あり。
[kms@sandbox sbin]$ ./kms.sh start
setting KMS_HTTP_PORT=9292
Using CATALINA_BASE: /var/lib/ranger/kms/hadoop/share/hadoop/kms/tomcat
Using CATALINA_HOME: /var/lib/ranger/kms/hadoop/share/hadoop/kms/tomcat
Using CATALINA_TMPDIR: /var/lib/ranger/kms/hadoop/share/hadoop/kms/tomcat/temp
Using JRE_HOME: /usr/lib/jvm/java
Using CLASSPATH: /var/lib/ranger/kms/hadoop/share/hadoop/kms/tomcat/bin/bootstrap.jar
Using CATALINA_PID: /tmp/kms.pid
Keystoreの場所とデフォルトのパスワード
<property>
<name>hadoop.kms.key.provider.uri</name>
<value>jceks://file@/${user.home}/kms.keystore</value>
<description>
URI of the backing KeyProvider for the KMS.
</description>
</property>
<property>
<name>hadoop.security.keystore.JavaKeyStoreProvider.password</name>
<value>none</value>
<description>
If using the JavaKeyStoreProvider, the password for the keystore file.
</description>
</property>
テスト（リストと作成）
[root@sandbox sbin]# su - ambari-qa
[ambari-qa@sandbox ~]$ hadoop key list
Listing keys for KeyProvider: KMSClientProvider[http://sandbox.hortonworks.com:9292/kms/v1/]
[ambari-qa@sandbox ~]$ hadoop key create ambariqa-key
ambariqa-key has been successfully created with options Options{cipher='AES/CTR/NoPadding', bitLength=128, description='null', attributes=null}.
KMSClientProvider[http://sandbox.hortonworks.com:9292/kms/v1/] has been updated.
[ambari-qa@sandbox ~]$ hadoop key list
Listing keys for KeyProvider: KMSClientProvider[http://sandbox.hortonworks.com:9292/kms/v1/]
ambariqa-key
Keystoreを見てみる
[root@sandbox sbin]# su - kms
[kms@sandbox ~]$ ls -ltr
total 200636
-rw-r--r-- 1 kms hadoop 205440821 Jun 14 10:12 mapreduce.tar.gz
drwxr-xr-x 11 kms hadoop 4096 Jun 14 10:14 hadoop
-rwx------ 1 kms hadoop 1140 Jun 14 10:33 kms.keystore
[kms@sandbox ~]$ file kms.keystore
kms.keystore: Java JCE KeyStore
[kms@sandbox ~]$ keytool -list -keystore ./kms.keystore
keytool error: java.io.IOException: Invalid keystore format
[kms@sandbox ~]$ keytool -list -keystore ./kms.keystore -storetype jceks
Enter keystore password:
***************** WARNING WARNING WARNING *****************
* The integrity of the information stored in your keystore *
* has NOT been verified! In order to verify its integrity, *
* you must provide your keystore password. *
***************** WARNING WARNING WARNING *****************
Keystore type: JCEKS
Keystore provider: SunJCE
Your keystore contains 2 entries
ambariqa-key@0, Jun 14, 2017, SecretKeyEntry,
ambariqa-key, Jun 14, 2017, SecretKeyEntry,
[kms@sandbox ~]$ keytool -list -keystore ./kms.keystore -storetype jceks -storepass none
Keystore type: JCEKS
Keystore provider: SunJCE
Your keystore contains 2 entries
ambariqa-key@0, Jun 14, 2017, SecretKeyEntry,
ambariqa-key, Jun 14, 2017, SecretKeyEntry,
[kms@sandbox ~]$
自分のホームディレクトリをエンクリプトしてみる
[root@sandbox ~]# su - hdfs
[hdfs@sandbox ~]$ hdfs crypto -createZone -keyName ambariqa-key -path /user/hajime
Added encryption zone /user/hajime
[hdfs@sandbox ~]$ hdfs dfs -chown hajime:hadoop /user/hajime
[hdfs@sandbox ~]$
確認してみる
[root@sandbox ~]# su - hajime
[hajime@sandbox ~]$ hdfs dfs -put /tmp/words.txt
[hajime@sandbox ~]$ hdfs dfs -cat /user/hajime/words.txt
a
[root@sandbox ~]# su - hdfs
[hdfs@sandbox ~]$ hdfs dfs -cat /.reserved/raw/user/hajime/words.txt
�[hdfs@sandbox ~]$
パスワードを変更してみる
keytool -storepasswd -keystore ./kms.keystore -storetype jceks -storepass 'none'
と、起動しなくなる
Stacktrace:
---------------------------------------------------
java.io.IOException: Keystore was tampered with, or password was incorrect
at com.sun.crypto.provider.JceKeyStore.engineLoad(JceKeyStore.java:865)
at java.security.KeyStore.load(KeyStore.java:1226)
at org.apache.hadoop.crypto.key.JavaKeyStoreProvider.loadFromPath(JavaKeyStoreProvider.java:305)
at org.apache.hadoop.crypto.key.JavaKeyStoreProvider.tryLoadFromPath(JavaKeyStoreProvider.java:213)
新しいパスワードで再起動：
HADOOP_KEYSTORE_PASSWORD=testtest ~/hadoop/sbin/kms.sh start
備考：
HDP付属のApache KMSは古いバージョンの模様。
CDHなどでトライすると（CDHのHadoopも古いですが）”hadoop.security.keystore.java-keystore-provider.password-file”を使用する必要あり。
そして、未解決のバグあり。
$HADOOP_PREFIX/share/hadoop/kms/tomcat/webapps/kms/WEB-INF/classes/
or
export CLASSPATH=$CLASSPATH:/usr/local/hadoop/etc/hadoop/kms.keystore.password
0 件のコメント:
コメントを投稿