Hadoop and DevSecOps notes & tips: 11月 2017

2017年11月30日木曜日

HDP 2.6 Sandboxで Knox Demo LDAPをつかってHadoop Group Mappingを設定する

参考：https://docs.hortonworks.com/HDPDocuments/Ambari-2.6.0.0/bk_ambari-security/content/setting_up_hadoop_group_mappping_for_ldap_ad.html

1. 必要であれば、Ambariから/etc/knox/conf/users.ldifにユーザやグループを追加
Knox => Configs => Advanced users-ldif

2. Knox DEMO LDAPを開始
コマンドで開始したい場合は：
sudo -u knox -i /usr/hdp/current/knox-server/bin/ldap.sh start
または
sudo -u knox -i java -jar /usr/hdp/current/knox-server/bin/ldap.jar /usr/hdp/current/knox-server/conf &

確認：
yum install -y openldap-clients
ldapsearch -H 'ldap://sandbox-hdp.hortonworks.com:33389/' -x -D 'uid=admin,ou=people,dc=hadoop,dc=apache,dc=org' -w admin-password '(objectclass=person)' uid
ldapsearch -H 'ldap://sandbox-hdp.hortonworks.com:33389/' -x -D 'uid=admin,ou=people,dc=hadoop,dc=apache,dc=org' -w admin-password '(objectclass=groupOfNames)' member cn
ldapsearch -H 'ldap://sandbox-hdp.hortonworks.com:33389/' -x -D 'uid=admin,ou=people,dc=hadoop,dc=apache,dc=org' -w admin-password -b 'dc=hadoop,dc=apache,dc=org' '(&(objectclass=person)(uid=sam))'

3. Ambariから、下記のプロパティをHDFS => Configs => Custom core-site に追加
hadoop.security.group.mapping=org.apache.hadoop.security.LdapGroupsMapping
hadoop.security.group.mapping.ldap.bind.user=uid=admin,ou=people,dc=hadoop,dc=apache,dc=org
hadoop.security.group.mapping.ldap.url=ldap://sandbox-hdp.hortonworks.com:33389/dc=hadoop,dc=apache,dc=org
#hadoop.security.group.mapping.ldap.base=
hadoop.security.group.mapping.ldap.search.filter.user=(&(objectclass=person)(uid={0}))
hadoop.security.group.mapping.ldap.search.filter.group=(objectclass=groupofnames)
hadoop.security.group.mapping.ldap.search.attr.member=member
hadoop.security.group.mapping.ldap.search.attr.group.name=cn
# PASSWORDタイプで
hadoop.security.group.mapping.ldap.bind.password=admin-password

コンポジットの場合:
hadoop.security.group.mapping=org.apache.hadoop.security.CompositeGroupsMapping
hadoop.security.group.mapping.providers=shell4services,ldap-demo4users
hadoop.security.group.mapping.provider.shell4services=org.apache.hadoop.security.ShellBasedUnixGroupsMapping
hadoop.security.group.mapping.provider.ldap-demo4users=org.apache.hadoop.security.LdapGroupsMapping
hadoop.security.group.mapping.provider.ldap-demo4users.ldap.url=ldap://sandbox-hdp.hortonworks.com:33389/dc=hadoop,dc=apache,dc=org
hadoop.security.group.mapping.provider.ldap-demo4users.ldap.bind.user=uid=admin,ou=people,dc=hadoop,dc=apache,dc=org
#hadoop.security.group.mapping.provider.ldap-demo4users.ldap.base=
hadoop.security.group.mapping.provider.ldap-demo4users.ldap.search.filter.user=(&(objectclass=person)(uid={0}))
hadoop.security.group.mapping.provider.ldap-demo4users.ldap.search.filter.group=(objectclass=groupofnames)
hadoop.security.group.mapping.provider.ldap-demo4users.ldap.search.attr.member=member
hadoop.security.group.mapping.provider.ldap-demo4users.ldap.search.attr.group.name=cn

# PASSWORDタイプで
hadoop.security.group.mapping.provider.ldap-demo4users.ldap.bind.password=admin-password

4. HDFS, YARN, MapReduce2を再起動

再起動するので下記のコマンドは必要ないはず

sudo -u hdfs -i hdfs dfsadmin -refreshUserToGroupsMappings

sudo -u yarn -i yarn rmadmin -refreshUserToGroupsMappings

5. 確認：うまく行っていたら、下記コマンドに何か出てくるはずです

sudo -u hdfs -i hdfs groups sam

Ambari: failed with database inconsistency errors

Ambariでたまに、下記のようなエラーが出ますが、何が原因（Inconsistent）？

ERROR - Required config(s): ranger-hbase-security,ranger-hbase-policymgr-ssl,ranger-hbase-audit,ranger-hbase-plugin-properties is(are) not available for service HBASE with service config version 12 in cluster TEST

DatabaseConsistencyCheckHelper.java

# エラーの出るところ
serviceConfigsFromStack.removeAll(serviceConfigsFromDB);
if (!serviceConfigsFromStack.isEmpty()) {
LOG.error("Required config(s): {} is(are) not available for service {} with service config version {} in cluster {}",
StringUtils.join(serviceConfigsFromStack, ","), serviceName, Integer.toString(serviceVersion), clusterName);
errorAvailable = true;
}

# チェックに使われるクエリー
String GET_SERVICES_WITH_CONFIGS_QUERY = "

SELECT  c.cluster_name,
  cs.service_name,
  cc.type_name,
  sc.versionFROM clusterservices cs
  JOIN serviceconfig sc ON cs.service_name = sc.service_name AND cs.cluster_id = sc.cluster_id  JOIN serviceconfigmapping scm ON sc.service_config_id = scm.service_config_id  JOIN clusterconfig cc ON scm.config_id = cc.config_id AND sc.cluster_id = cc.cluster_id  JOIN clusters c ON cc.cluster_id = c.cluster_id AND sc.stack_id = c.desired_stack_idWHERE sc.group_id IS NULL      AND sc.service_config_id = (SELECT max(service_config_id)
                                  FROM serviceconfig sc2
                                  WHERE sc2.service_name = sc.service_name AND sc2.cluster_id = sc.cluster_id)
GROUP BY c.cluster_name, cs.service_name, cc.type_name, sc.version

";

# Ambariログ
07 Feb 2017 14:48:36,889 INFO [main] ClusterImpl:352 - Service config types loaded: {PIG=[pig-properties, pig-env, pig-log4j], KAFKA=[ranger-kafka-policymgr-ssl, kafka-log4j, kafka-env, kafka-broker, ranger-kafka-security, ranger-kafka-plugin-properties, ranger-kafka-audit], LOGSEARCH=[logsearch-service_logs-solrconfig, logfeeder-log4j, logsearch-admin-json, logsearch-env, logfeeder-env, logsearch-audit_logs-solrconfig, logfeeder-properties, logsearch-properties, logsearch-log4j], RANGER_KMS=[kms-properties, ranger-kms-security, ranger-kms-site, kms-site, kms-env, dbks-site, ranger-kms-audit, ranger-kms-policymgr-ssl, kms-log4j], MAPREDUCE2=[mapred-site, mapred-env], SLIDER=[slider-log4j, slider-env, slider-client], HIVE=[webhcat-env, ranger-hive-plugin-properties, hive-exec-log4j, ranger-hive-policymgr-ssl, hive-env, webhcat-site, hive-log4j, ranger-hive-audit, hive-site, webhcat-log4j, hiveserver2-site, hcat-env, ranger-hive-security], TEZ=[tez-env, tez-site], HBASE=[ranger-hbase-security, hbase-policy, hbase-env, hbase-log4j, hbase-site, ranger-hbase-policymgr-ssl, ranger-hbase-audit, ranger-hbase-plugin-properties], RANGER=[admin-properties, ranger-admin-site, usersync-properties, ranger-site, ranger-env, ranger-ugsync-site], OOZIE=[oozie-log4j, oozie-env, oozie-site], FLUME=[flume-env, flume-conf], MAHOUT=[mahout-log4j, mahout-env], HDFS=[ssl-server, hdfs-log4j, ranger-hdfs-audit, ranger-hdfs-plugin-properties, ssl-client, hdfs-site, ranger-hdfs-policymgr-ssl, hadoop-policy, ranger-hdfs-security, hadoop-env, core-site], AMBARI_METRICS=[ams-ssl-client, ams-ssl-server, ams-hbase-log4j, ams-hbase-policy, ams-hbase-security-site, ams-grafana-env, ams-hbase-env, ams-env, ams-log4j, ams-grafana-ini, ams-site, ams-hbase-site], SPARK=[spark-thrift-sparkconf, spark-log4j-properties, spark-defaults, spark-javaopts-properties, spark-metrics-properties, spark-hive-site-override, spark-env], SMARTSENSE=[hst-log4j, hst-server-conf, activity-zeppelin-shiro, activity-log4j, activity-zeppelin-site, anonymization-rules, activity-zeppelin-env, activity-zeppelin-interpreter, activity-env, activity-conf, hst-agent-conf], AMBARI_INFRA=[infra-solr-client-log4j, infra-solr-env, infra-solr-xml, infra-solr-log4j], YARN=[ranger-yarn-policymgr-ssl, yarn-site, ranger-yarn-audit, ranger-yarn-security, yarn-env, ranger-yarn-plugin-properties, capacity-scheduler, yarn-log4j], FALCON=[falcon-runtime.properties, falcon-log4j, falcon-client.properties, falcon-startup.properties, falcon-env], SQOOP=[sqoop-site, sqoop-env], ATLAS=[atlas-log4j, atlas-env, application-properties], ZOOKEEPER=[zoo.cfg, zookeeper-log4j, zookeeper-env], STORM=[ranger-storm-plugin-properties, storm-site, ranger-storm-audit, storm-cluster-log4j, storm-worker-log4j, ranger-storm-policymgr-ssl, ranger-storm-security, storm-env], GANGLIA=[ganglia-env], KNOX=[knoxsso-topology, ranger-knox-security, users-ldif, knox-env, ranger-knox-plugin-properties, gateway-log4j, gateway-site, ranger-knox-policymgr-ssl, ranger-knox-audit, topology, admin-topology, ldap-log4j], KERBEROS=[kerberos-env, krb5-conf], ACCUMULO=[accumulo-env, accumulo-log4j, client, accumulo-site]}

ちなみに、Ambari2.4.2ではJavaコマンドがチェックに使われます。
java -cp '/etc/ambari-server/conf:/usr/lib/ambari-server/*:/usr/share/java/postgresql-jdbc.jar' org.apache.ambari.server.checks.DatabaseConsistencyChecker
Ambari2.2.xだと、CheckDatabaseHelper.

直すのに使うSQL:

SELECT  service_name,
  max(service_config_id) AS service_config_id,
  max(version)           AS versionFROM serviceconfig
WHERE service_name IN ('HBASE') AND version IN (12)
GROUP BY service_name;

HDP 2.6.1 , Ambari 2.5.1.0 (Sandbox) で Knox SSOを試す

https://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.6.2/bk_security/content/setting_up_knox_sso_for_ambari.html

Knox SSOに必要なもの

- Ambari ServerはLDAPの設定が必要 (ambari-server setup-ldap)
- そしてユーザがインポートされている必要あり(ambari-server sync-ldap)
- KnoxとAmbari Serverは同じドメインである必要あり

- Hostnameが"{somehost}.{someorganisation}.{someTLD}"である必要あり
だからsandbox.hortonworks.comはOK、でも、node1.localdomainはNG

1. Knox設定

knoxsso.redirect.whitelist.regexを変更する必要あり
^https?:\/\/(sandbox-hdp\.hortonworks\.com|172\.18\.0\.2|172\.26\.74\.244|localhost|127\.0\.0\.1|0:0:0:0:0:0:0:1|::1):[0-9].*$

knoxsso.cookie.secure.onlyはfalseでないとうまくいかないかも？

Knoxの証明書をエクスポートしておく（ambari-server setup-sso中に必要）

keytool -export -alias gateway-identity -rfc -file ./gateway.crt -keystore /usr/hdp/current/knox-server/data/security/keystores/gateway.jks

echo -n | openssl s_client -connect sandbox-hdp.hortonworks.com:8443 -showcerts 2>&1 | sed -ne '/-BEGIN CERTIFICATE-/,/-END CERTIFICATE-/p' | tee ./gateway.crt

念のためCNを確認：
openssl x509 -noout -subject -in ./gateway.crt
subject= /C=US/ST=Test/L=Test/O=Hadoop/OU=Test/CN=sandbox-hdp.hortonworks.com

ここで、Ambariの"admin"（LOCAL）ユーザをどうするかを決めます。

普通にインポートすると同じ名前なのでEXTERNALユーザになります、が、そうするとあとでSSOを止めた時や、Demo LDAPが止まっている場合に面倒です。

そこで、Knox => Advanced users-ldif から、別のAdmin候補を作ります。

# entry for sample user adminldap
dn: uid=adminldap,ou=people,dc=hadoop,dc=apache,dc=org
objectclass:top
objectclass:person
objectclass:organizationalPerson
objectclass:inetOrgPerson
cn: adminldap
sn: adminldap
uid: adminldap
userPassword:admin-password

Knox Demo LDAPをAmbariから再起動（Stop/Start）たまに止まったように見えて、止まってない場合あり。
さらに、
#sudo -u hdfs kinit -kt /etc/security/keytabs/hdfs.headless.keytab hdfs-sandbox
sudo -u hdfs hdfs dfs -mkdir /user/adminldap
useradd adminldap

Ambari LDAP設定

Knox Demo LDAPを開始してから、

_LDAP_SERVER="sandbox.hortonworks.com:33389"

ambari-server setup-ldap --ldap-url=${_LDAP_SERVER} --ldap-user-class=person --ldap-user-attr=uid --ldap-group-class=groupofnames --ldap-ssl=false --ldap-secondary-url="" --ldap-referral="" --ldap-group-attr=cn --ldap-member-attr=member --ldap-dn=dn --ldap-base-dn=dc=hadoop,dc=apache,dc=org --ldap-bind-anonym=false --ldap-manager-dn=uid=admin,ou=people,dc=hadoop,dc=apache,dc=org --ldap-manager-password=admin-password --ldap-sync-username-collisions-behavior=skip --ldap-save-settings

* "skip"にしないと、"admin"がLDAPユーザの"admin"に換えられてしまいます。（それでもよければSkipでも可）

さらに、下記のプロパティを追加する必要あり
echo "authentication.ldap.pagination.enabled=false" >> /etc/ambari-server/conf/ambari.properties
で、Ambari Serverの再起動
ambari-server restart

Ambari sync-ldap

ambari-server sync-ldap --ldap-sync-admin-name=admin --ldap-sync-admin-password=admin --all

"--ldap-sync-"と言いながら、AmbariローカルユーザのAdminです。

LDAPユーザでログインできるかテスト。
また、Admin候補を作った場合は、Adminにしておく。

"ambari-server setup-sso"

Provider URL = https://sandbox-hdp.hortonworks.com:8443/gateway/knoxsso/api/v1/websso

証明書を貼り付けする場合は、最初と最後の行はペーストしない。すると：
/var/log/ambari-server/ambari-server.log:08 Nov 2017 09:34:33,680 ERROR [ambari-client-thread-1470] Configuration:5133 - Unable to parse public certificate file. JWT auth will be disabled.

ambari-server setup-sso後は下記のようになります。
[root@sandbox ~]# grep -iw jwt /etc/ambari-server/conf/*
/etc/ambari-server/conf/ambari.properties:authentication.jwt.enabled=true
/etc/ambari-server/conf/ambari.properties:authentication.jwt.providerUrl=https://sandbox.hortonworks.com:8443/gateway/knoxsso/api/v1/websso
/etc/ambari-server/conf/ambari.properties:authentication.jwt.publicKey=/etc/ambari-server/conf/jwt-cert.pem

ambari-server restart

2017年11月1日水曜日

HDP 2.6.1 SandboxのデータベースをPostgreSQLからMySQLに変更して見る

うまくいくかはわかりませんが、挑戦です。

参考：https://docs.hortonworks.com/HDPDocuments/Ambari-2.5.2.0/bk_ambari-administration/content/using_ambari_with_mysql.html

MySQL Connectorがあるか確認

[root@sandbox ambari-server]# lsof -nPp `cat /var/run/ambari-server/ambari-server.pid` | grep mysql

java 20058 root 160r REG 252,1 819803 1052266 /usr/share/java/mysql-connector-java-5.1.17.jar

[root@sandbox ambari-server]# zipgrep 'Bundle-Version' /usr/share/java/mysql-connector-java.jar

META-INF/MANIFEST.MF:Bundle-Version: 5.1.17

[root@sandbox ambari-server]# rm /usr/share/java/mysql-connector-java.jar

rm: remove symbolic link `/usr/share/java/mysql-connector-java.jar'? y

[root@sandbox ambari-server]# ln -s /usr/share/java/mysql-connector-java-5.1.37.jar /usr/share/java/mysql-connector-java.jar

Ambari Serverを停止する

[root@sandbox ~]# ambari-server stop
...

DatabaseをMySQLに変更

[root@sandbox ~]# ambari-server setup
Using python /usr/bin/python
Setup ambari-server
Checking SELinux...
SELinux status is 'disabled'
Customize user account for ambari-server daemon [y/n] (n)?
Adjusting ambari-server permissions and ownership...
Checking firewall status...
WARNING: iptables is running. Confirm the necessary Ambari ports are accessible. Refer to the Ambari documentation for more details on ports.
OK to continue [y/n] (y)?
Checking JDK...
Do you want to change Oracle JDK [y/n] (n)?
Completing setup...
Configuring database...
Enter advanced database configuration [y/n] (n)? y
Configuring database...
==============================================================================
Choose one of the following options:
[1] - PostgreSQL (Embedded)
[2] - Oracle
[3] - MySQL / MariaDB
[4] - PostgreSQL
[5] - Microsoft SQL Server (Tech Preview)
[6] - SQL Anywhere
[7] - BDB
==============================================================================
Enter choice (1): 3
Hostname (localhost):
Port (3306):
Database name (ambari):
Username (ambari):
Enter Database Password (bigdata):
Configuring ambari database...
Configuring remote database connection properties...
WARNING: Before starting Ambari Server, you must run the following DDL against the database to create the schema: /var/lib/ambari-server/resources/Ambari-DDL-MySQL-CREATE.sql
Proceed with configuring remote database connection properties [y/n] (y)?
Extracting system views...
............
Adjusting ambari-server permissions and ownership...
Ambari Server 'setup' completed successfully.

TODO: PostgreSQLとMySQLではテーブル名の大文字小文字が違う？(ClusterHostMapping)

[root@sandbox ~]# vim /etc/my.cnf

...

lower_case_table_names = 1

[root@sandbox ~]# service mysqld restart

Stopping mysqld: [ OK ]

Starting mysqld: [ OK ]

[root@sandbox ~]#

追記：lower_case_table_namesは他のサービスに影響があるかも（Hive,Oozie)

AmbariデータベースとAmbariユーザを作る

[root@sandbox ~]# mysql -u root -phadoop
...
mysql>
CREATE DATABASE ambari;
CREATE USER 'ambari'@'%' IDENTIFIED BY 'bigdata';
GRANT ALL PRIVILEGES ON *.* TO 'ambari'@'%';
CREATE USER 'ambari'@'localhost' IDENTIFIED BY 'bigdata';
GRANT ALL PRIVILEGES ON *.* TO 'ambari'@'localhost';
CREATE USER 'ambari'@'sandbox.hortonworks.com' IDENTIFIED BY 'bigdata';
GRANT ALL PRIVILEGES ON *.* TO 'ambari'@'sandbox.hortonworks.com';
FLUSH PRIVILEGES;

デフォルトのスキーマをロードする

[root@sandbox ~]# mysql -u ambari -pbigdata ambari < /var/lib/ambari-server/resources/Ambari-DDL-MySQL-CREATE.sql

興味のないテーブルを除いて、ダンプを作る

[root@sandbox ~]# pg_dump -a --column-inserts -T alert_history -T host_role_command -T execution_command -T request -T role_success_criteria -T stage -T requestresourcefilter -T requestoperationlevel -T upgrade_item -T upgrade_item -T servicecomponent_history -T upgrade -T topology_logical_task -T topology_host_task -T topology_host_request -T topology_host_request -T topology_host_request -T topology_host_request -T topology_request -Uambari ambari -f ambari_data.sql

加工する

[root@sandbox ~]# grep -oE '^INSERT INTO [^ ]+' ambari_data.sql | sort | uniq | sed 's/INSERT INTO/TRUNCATE/g' | sed 's/$/;/g' > ambari_delete.sql

[root@sandbox ~]# sed -i '1s/^/SET FOREIGN_KEY_CHECKS=0;\n/' ambari_delete.sql

[root@sandbox ~]# grep -vw '^SET' ambari_data.sql > ambari_data_no_SET.sql

[root@sandbox ~]# sed 's/(key, value)/(`key`, `value`)/g' ambari_data_no_SET.sql > ambari_data_no_SET_no_keywords.sql

[root@sandbox ~]# sed -i '1s/^/SET FOREIGN_KEY_CHECKS=0;\n/' ambari_data_no_SET_no_keywords.sql

削除してからインサート

[root@sandbox ~]# mysql -u ambari -pbigdata ambari < ambari_delete.sql

[root@sandbox ~]# mysql -u ambari -pbigdata ambari < ambari_data_no_SET_no_keywords.sql

Ambari Serverを開始する

[root@sandbox ~]# ambari-server start
...
[root@sandbox ~]# service postgresql stop

Stopping postgresql service: [ OK ]

問題：

01 Nov 2017 09:42:17,606 WARN [C3P0PooledConnectionPoolManager[identityToken->1bqrg1u9rx02fyjjc9tfe|5d708ef6]-HelperThread-#1] StatementUtils:223 - Statement close FAILED.

com.mysql.jdbc.exceptions.jdbc4.MySQLSyntaxErrorException: You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near 'OPTION SQL_SELECT_LIMIT=DEFAULT' at line 1

at sun.reflect.GeneratedConstructorAccessor209.newInstance(Unknown Source)

at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)

at java.lang.reflect.Constructor.newInstance(Constructor.java:423)

at com.mysql.jdbc.Util.handleNewInstance(Util.java:411)

at com.mysql.jdbc.Util.getInstance(Util.java:386)

at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:1052)

at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3597)

at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3529)

at com.mysql.jdbc.MysqlIO.sendCommand(MysqlIO.java:1990)

at com.mysql.jdbc.MysqlIO.sqlQueryDirect(MysqlIO.java:2151)

at com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2619)

...

JDBC DriverがMySQLサーバと合わないとこの問題が出る模様。
JDBC Driverをアップデートすることで止まりました。

登録: 投稿 (Atom)