Hadoop and DevSecOps notes & tips

2017年6月21日水曜日

Sandbox HDP 2.5 のZeppelinをKnoxのKeystoreでSSL

慣れないとKeystore/Truststoreを設定するのは面倒なので、Knoxのを使いまわします。

https://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.6.0/bk_zeppelin-component-guide/content/config-ssl-zepp.html

Advanced zeppelin-config
zeppelin.ssl = true
zeppelin.ssl.key.manager.password = knox
zeppelin.ssl.keystore.password = knox
zeppelin.ssl.keystore.path = /etc/security/serverKeys/keystore.jks

cp /usr/hdp/current/knox-server/data/security/keystores/gateway.jks /etc/security/serverKeys/keystore.jks
chmod a+r /etc/security/serverKeys/keystore.jks

2017年6月19日月曜日

ノート：Hadoop Configuration.java

private Document parse(DocumentBuilder builder, URL url)
throws IOException, SAXException {
if (!quietmode) {
LOG.debug("parsing URL " + url);
}
if (url == null) {
return null;
}
return parse(builder, url.openStream(), url.toString());
}

とあるので、
Configuration conf = new Configuration();
conf.setQuietMode(false);

static{
//print deprecation warning if hadoop-site.xml is found in classpath
ClassLoader cL = Thread.currentThread().getContextClassLoader();
if (cL == null) {
cL = Configuration.class.getClassLoader();
}
if(cL.getResource("hadoop-site.xml")!=null) {
LOG.warn("DEPRECATED: hadoop-site.xml found in the classpath. " +
"Usage of hadoop-site.xml is deprecated. Instead use core-site.xml, "
+ "mapred-site.xml and hdfs-site.xml to override properties of " +
"core-default.xml, mapred-default.xml and hdfs-default.xml " +
"respectively");
}
addDefaultResource("core-default.xml");
addDefaultResource("core-site.xml");
}

とあるので、
ClassLoader cl = Thread.currentThread().getContextClassLoader();
System.out.println(Thread.currentThread().getName()+" core-site.xml path = "+cl.getResource("core-site.xml"));

2017年6月15日木曜日

Ambariはどうやってコンポーネントのステータスを確認しているのか

/var/lib/ambari-agent/cache/common-services/HDFS/2.1.0.2.0/package/scripts/hdfs_namenode.py

from resource_management.libraries.functions.check_process_status import check_process_status

elif action == "status":
import status_params
check_process_status(status_params.namenode_pid_file)

Ambari Server側：
[root@node16 resources]# find . -type d -name package
./common-services/AMBARI_METRICS/0.1.0/package
./common-services/ATLAS/0.1.0.2.3/package
./common-services/HDFS/2.1.0.2.0/package
./common-services/HIVE/0.12.0.2.0/package
./common-services/HAWQ/2.0.0/package
./common-services/RANGER/0.4.0/package
./common-services/YARN/2.1.0.2.0/package
./common-services/SLIDER/0.60.0.2.2/package
./common-services/KNOX/0.5.0.2.2/package
./common-services/AMBARI_INFRA/0.1.0/package
./common-services/PIG/0.12.0.2.0/package
./common-services/TEZ/0.4.0.2.1/package
./common-services/OOZIE/4.0.0.2.0/package
./common-services/ZOOKEEPER/3.4.5/package
./common-services/GANGLIA/3.5.0/package
./common-services/RANGER_KMS/0.5.0.2.3/package
./common-services/KERBEROS/1.10.3-10/package
./common-services/SQOOP/1.4.4.2.0/package
./common-services/FLUME/1.4.0.2.0/package
./common-services/ACCUMULO/1.6.1.2.2.0/package
./common-services/MAHOUT/1.0.0.2.3/package
./common-services/PXF/3.0.0/package
./common-services/ZEPPELIN/0.6.0.2.5/package
./common-services/SPARK2/2.0.0/package
./common-services/KAFKA/0.8.1/package
./common-services/LOGSEARCH/0.5.0/package
./common-services/FALCON/0.5.0.2.1/package
./common-services/DRUID/0.9.2/package
./common-services/SPARK/1.2.1/package
./common-services/HBASE/0.96.0.2.0/package
./common-services/STORM/0.9.1/package
./stacks/HDP/2.3.ECS/services/ECS/package
./stacks/HDP/2.3.GlusterFS/services/GLUSTERFS/package
./stacks/HDP/2.1/services/SMARTSENSE/package
./stacks/HDP/2.0.6.GlusterFS/services/YARN/package
./stacks/HDP/2.0.6.GlusterFS/services/GLUSTERFS/package
./stacks/HDP/2.1.GlusterFS/services/YARN/package
./stacks/HDP/2.1.GlusterFS/services/GLUSTERFS/package
./stacks/HDP/2.1.GlusterFS/services/TEZ/package
./stacks/HDP/2.1.GlusterFS/services/FALCON/package
./stacks/HDP/2.1.GlusterFS/services/STORM/package

check_process_statusはKillコマンドで確認している
sudo.kill(pid, 0)

Smart SenseはPIDを確認していない
/var/lib/ambari-server/resources/stacks/HDP/2.1/services/SMARTSENSE/package/scripts/activity.py
def status(self, env):
import params

cmd = '/usr/sbin/hst activity status '
if self.component is 'explorer':
cmd = '/usr/sbin/hst activity-explorer status '
elif self.component is 'analyzer':
cmd = '/usr/sbin/hst activity-analyzer status '

2017年6月14日水曜日

HDP 2.5 SandboxにあえてHadoop KMSを設定する

参考：
https://www.ibm.com/support/knowledgecenter/en/SSWTQQ_2.0.0/install/t_trd_hadoopencryption.html
https://hadoop.apache.org/docs/current3/hadoop-kms/index.html

インストール？からスタートまで
su - kms
cp /usr/hdp/2.5.0.0-1245/hadoop/mapreduce.tar.gz ./
tar xvf mapreduce.tar.gz
cd ./hadoop/sbin/
./kms.sh start

コンフィグロケーション
[kms@sandbox sbin]$ ls -l ~/hadoop/etc/hadoop
total 156
-rw-r--r-- 1 kms hadoop 4436 Aug 26 2016 capacity-scheduler.xml
-rw-r--r-- 1 kms hadoop 1335 Aug 26 2016 configuration.xsl
-rw-r--r-- 1 kms hadoop 318 Aug 26 2016 container-executor.cfg
-rw-r--r-- 1 kms hadoop 894 Apr 12 11:58 core-site.xml
-rw-r--r-- 1 kms hadoop 3979 Aug 26 2016 hadoop-env.cmd
-rw-r--r-- 1 kms hadoop 4529 Aug 26 2016 hadoop-env.sh
-rw-r--r-- 1 kms hadoop 2598 Aug 26 2016 hadoop-metrics2.properties
-rw-r--r-- 1 kms hadoop 2490 Aug 26 2016 hadoop-metrics.properties
-rw-r--r-- 1 kms hadoop 9683 Aug 26 2016 hadoop-policy.xml
-rw-r--r-- 1 kms hadoop 775 Aug 26 2016 hdfs-site.xml
-rw-r--r-- 1 kms hadoop 1449 Aug 26 2016 httpfs-env.sh
-rw-r--r-- 1 kms hadoop 1657 Aug 26 2016 httpfs-log4j.properties
-rw-r--r-- 1 kms hadoop 21 Aug 26 2016 httpfs-signature.secret
-rw-r--r-- 1 kms hadoop 620 Aug 26 2016 httpfs-site.xml
-rw-r--r-- 1 kms hadoop 3518 Aug 26 2016 kms-acls.xml
-rw-r--r-- 1 kms hadoop 1527 Aug 26 2016 kms-env.sh
-rw-r--r-- 1 kms hadoop 1631 Aug 26 2016 kms-log4j.properties
-rw-r--r-- 1 kms hadoop 5511 Aug 26 2016 kms-site.xml
-rw-r--r-- 1 kms hadoop 12302 Aug 26 2016 log4j.properties
-rw-r--r-- 1 kms hadoop 951 Aug 26 2016 mapred-env.cmd
-rw-r--r-- 1 kms hadoop 1383 Aug 26 2016 mapred-env.sh
-rw-r--r-- 1 kms hadoop 4113 Aug 26 2016 mapred-queues.xml.template
-rw-r--r-- 1 kms hadoop 758 Aug 26 2016 mapred-site.xml.template
-rw-r--r-- 1 kms hadoop 10 Aug 26 2016 slaves
-rw-r--r-- 1 kms hadoop 2316 Aug 26 2016 ssl-client.xml.example
-rw-r--r-- 1 kms hadoop 2697 Aug 26 2016 ssl-server.xml.example
-rw-r--r-- 1 kms hadoop 2250 Aug 26 2016 yarn-env.cmd
-rw-r--r-- 1 kms hadoop 4567 Aug 26 2016 yarn-env.sh
-rw-r--r-- 1 kms hadoop 690 Aug 26 2016 yarn-site.xml

ポート番号は16000と16001ぽい
[kms@sandbox sbin]$ ps auxwww | grep kms
kms 27734 2.1 1.1 6513948 190760 pts/1 Sl 10:14 0:08 /usr/lib/jvm/java/bin/java -Djava.util.logging.config.file=/var/lib/ranger/kms/hadoop/share/hadoop/kms/tomcat/conf/logging.properties -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager -Dkms.home.dir=/var/lib/ranger/kms/hadoop -Dkms.config.dir=/var/lib/ranger/kms/hadoop/etc/hadoop -Dkms.log.dir=/var/lib/ranger/kms/hadoop/logs -Dkms.temp.dir=/var/lib/ranger/kms/hadoop/temp -Dkms.admin.port=16001 -Dkms.http.port=16000 -Dkms.max.threads=1000 -Dkms.ssl.keystore.file=/var/lib/ranger/kms/.keystore -Djava.library.path=/var/lib/ranger/kms/hadoop/libexec/../lib/native/ -Djava.endorsed.dirs=/var/lib/ranger/kms/hadoop/share/hadoop/kms/tomcat/endorsed -classpath /var/lib/ranger/kms/hadoop/share/hadoop/kms/tomcat/bin/bootstrap.jar -Dcatalina.base=/var/lib/ranger/kms/hadoop/share/hadoop/kms/tomcat -Dcatalina.home=/var/lib/ranger/kms/hadoop/share/hadoop/kms/tomcat -Djava.io.tmpdir=/var/lib/ranger/kms/hadoop/share/hadoop/kms/tomcat/temp org.apache.catalina.startup.Bootstrap start
[kms@sandbox sbin]$ lsof -p 27734
-bash: lsof: command not found
[kms@sandbox sbin]$ netstat -aopen | grep 27734
(Not all processes could be identified, non-owned process info
will not be shown, you would have to be root to see it all.)
tcp 0 0 0.0.0.0:16000 0.0.0.0:* LISTEN 1006 4759355 27734/java off (0.00/0/0)
tcp 0 0 127.0.0.1:16001 0.0.0.0:* LISTEN 1006 5741705 27734/java off (0.00/0/0)
unix 2 [ ] STREAM CONNECTED 4759353 27734/java
[kms@sandbox sbin]$ grep -Ew '16000|16001' ~/hadoop/etc/hadoop/*
/var/lib/ranger/kms/hadoop/etc/hadoop/kms-env.sh:# export KMS_HTTP_PORT=16000

SandboxにRanger KMSを入れてしまったので、面倒なのでポートを9292（と、勝手に9293になる）に変更
変更しない場合はhadoop.security.key.provider.path (kms-site.xml?)とdfs.encryption.key.provider.uri (hdfs-site.xml)を変更する必要あり。
[kms@sandbox sbin]$ ./kms.sh start
setting KMS_HTTP_PORT=9292
Using CATALINA_BASE: /var/lib/ranger/kms/hadoop/share/hadoop/kms/tomcat
Using CATALINA_HOME: /var/lib/ranger/kms/hadoop/share/hadoop/kms/tomcat
Using CATALINA_TMPDIR: /var/lib/ranger/kms/hadoop/share/hadoop/kms/tomcat/temp
Using JRE_HOME: /usr/lib/jvm/java
Using CLASSPATH: /var/lib/ranger/kms/hadoop/share/hadoop/kms/tomcat/bin/bootstrap.jar
Using CATALINA_PID: /tmp/kms.pid

Keystoreの場所とデフォルトのパスワード
<property>
<name>hadoop.kms.key.provider.uri</name>
<value>jceks://file@/${user.home}/kms.keystore</value>
<description>
URI of the backing KeyProvider for the KMS.
</description>
</property>

<property>
<name>hadoop.security.keystore.JavaKeyStoreProvider.password</name>
<value>none</value>
<description>
If using the JavaKeyStoreProvider, the password for the keystore file.
</description>
</property>

テスト（リストと作成）
[root@sandbox sbin]# su - ambari-qa
[ambari-qa@sandbox ~]$ hadoop key list
Listing keys for KeyProvider: KMSClientProvider[http://sandbox.hortonworks.com:9292/kms/v1/]
[ambari-qa@sandbox ~]$ hadoop key create ambariqa-key
ambariqa-key has been successfully created with options Options{cipher='AES/CTR/NoPadding', bitLength=128, description='null', attributes=null}.
KMSClientProvider[http://sandbox.hortonworks.com:9292/kms/v1/] has been updated.
[ambari-qa@sandbox ~]$ hadoop key list
Listing keys for KeyProvider: KMSClientProvider[http://sandbox.hortonworks.com:9292/kms/v1/]
ambariqa-key

Keystoreを見てみる
[root@sandbox sbin]# su - kms
[kms@sandbox ~]$ ls -ltr
total 200636
-rw-r--r-- 1 kms hadoop 205440821 Jun 14 10:12 mapreduce.tar.gz
drwxr-xr-x 11 kms hadoop 4096 Jun 14 10:14 hadoop
-rwx------ 1 kms hadoop 1140 Jun 14 10:33 kms.keystore
[kms@sandbox ~]$ file kms.keystore
kms.keystore: Java JCE KeyStore
[kms@sandbox ~]$ keytool -list -keystore ./kms.keystore
keytool error: java.io.IOException: Invalid keystore format
[kms@sandbox ~]$ keytool -list -keystore ./kms.keystore -storetype jceks
Enter keystore password:

***************** WARNING WARNING WARNING *****************
* The integrity of the information stored in your keystore *
* has NOT been verified! In order to verify its integrity, *
* you must provide your keystore password. *
***************** WARNING WARNING WARNING *****************

Keystore type: JCEKS
Keystore provider: SunJCE

Your keystore contains 2 entries

ambariqa-key@0, Jun 14, 2017, SecretKeyEntry,
ambariqa-key, Jun 14, 2017, SecretKeyEntry,
[kms@sandbox ~]$ keytool -list -keystore ./kms.keystore -storetype jceks -storepass none

Keystore type: JCEKS
Keystore provider: SunJCE

Your keystore contains 2 entries

ambariqa-key@0, Jun 14, 2017, SecretKeyEntry,
ambariqa-key, Jun 14, 2017, SecretKeyEntry,
[kms@sandbox ~]$

自分のホームディレクトリをエンクリプトしてみる
[root@sandbox ~]# su - hdfs
[hdfs@sandbox ~]$ hdfs crypto -createZone -keyName ambariqa-key -path /user/hajime
Added encryption zone /user/hajime
[hdfs@sandbox ~]$ hdfs dfs -chown hajime:hadoop /user/hajime

[hdfs@sandbox ~]$

確認してみる
[root@sandbox ~]# su - hajime
[hajime@sandbox ~]$ hdfs dfs -put /tmp/words.txt
[hajime@sandbox ~]$ hdfs dfs -cat /user/hajime/words.txt

a
[root@sandbox ~]# su - hdfs
[hdfs@sandbox ~]$ hdfs dfs -cat /.reserved/raw/user/hajime/words.txt

�[hdfs@sandbox ~]$

パスワードを変更してみる
keytool -storepasswd -keystore ./kms.keystore -storetype jceks -storepass 'none'
と、起動しなくなる
Stacktrace:
---------------------------------------------------
java.io.IOException: Keystore was tampered with, or password was incorrect
at com.sun.crypto.provider.JceKeyStore.engineLoad(JceKeyStore.java:865)
at java.security.KeyStore.load(KeyStore.java:1226)
at org.apache.hadoop.crypto.key.JavaKeyStoreProvider.loadFromPath(JavaKeyStoreProvider.java:305)

at org.apache.hadoop.crypto.key.JavaKeyStoreProvider.tryLoadFromPath(JavaKeyStoreProvider.java:213)

新しいパスワードで再起動：
HADOOP_KEYSTORE_PASSWORD=testtest ~/hadoop/sbin/kms.sh start

備考：

HDP付属のApache KMSは古いバージョンの模様。

CDHなどでトライすると（CDHのHadoopも古いですが）”hadoop.security.keystore.java-keystore-provider.password-file”を使用する必要あり。

そして、未解決のバグあり。

https://issues.apache.org/jira/browse/HADOOP-12777

$HADOOP_PREFIX/share/hadoop/kms/tomcat/webapps/kms/WEB-INF/classes/

export CLASSPATH=$CLASSPATH:/usr/local/hadoop/etc/hadoop/kms.keystore.password

2017年6月13日火曜日

Ambari2.2.2.xに新しい（ダミー）サービスを追加してみる (add custom service)

https://cwiki.apache.org/confluence/display/AMBARI/Custom+Services
https://cwiki.apache.org/confluence/display/AMBARI/Defining+a+Custom+Stack+and+Services
https://cwiki.apache.org/confluence/display/AMBARI/Defining+a+Custom+Service (Ambari 2.4.0以上っぽい参照：https://issues.apache.org/jira/browse/AMBARI-14854)

https://developer.ibm.com/hadoop/2015/10/30/adding-a-custom-service-to-ibm-open-platform/
Ambari 2.2.2.xだと下記のファイルを作成する (TODO: -configs.j2必要？):

1. /var/lib/ambari-server/resources/stacks/HDP/2.2/services/${NEW_SERVICE}/metainfo.xml

サービスに使用されるサービス名、コンポーネント名やファイル名を記述する

2. /var/lib/ambari-server/resources/stacks/HDP/2.2/services/${NEW_SERVICE}/package/scripts/params.py
TODO:Ambariのオペレーションで使われる

3. /var/lib/ambari-server/resources/stacks/HDP/2.2/services/${NEW_SERVICE}/package/scripts/(metainfo commandScript file name .py)

コンポーネントの開始・停止などのコマンドを記述する

4. /var/lib/ambari-server/resources/stacks/HDP/2.2/services/${NEW_SERVICE}/configuration/${NEW_SERVICE}-config.xml

Ambariに表示されるプロパティ名とそのデフォルトの値を記述する

2017年4月25日火曜日

HDP 2.5.3のKnox Demo LDAPをAmbariと連携して見る

参考：https://hadoop-and-hdp.blogspot.com.au/2017/01/hdp-253knox-demo-ldapbeeline.html
http://docs.hortonworks.com/HDPDocuments/Ambari-2.4.2.0/bk_ambari-security/content/configuring_ambari_for_ldap_or_active_directory_authentication.html

[root@node1 ~]# ambari-server setup-ldap
Using python /usr/bin/python
Setting up LDAP properties...
Primary URL* {host:port} : node1.localdomain:33389
Secondary URL {host:port} :
Use SSL* [true/false] (false):
User object class* (posixAccount): person
User name attribute* (uid):
Group object class* (posixGroup): groupofnames
Group name attribute* (cn):
Group member attribute* (memberUid): member
Distinguished name attribute* (dn):
Base DN* : dc=hadoop,dc=apache,dc=org
Referral method [follow/ignore] : ignore
Bind anonymously* [true/false] (false):
Manager DN* : uid=admin,ou=people,dc=hadoop,dc=apache,dc=org
Enter Manager Password* :
Re-enter password:
====================
Review Settings
====================
authentication.ldap.managerDn: uid=admin,ou=people,dc=hadoop,dc=apache,dc=org
authentication.ldap.managerPassword: *****
Save settings [y/n] (y)?
Saving...done
Ambari Server 'setup-ldap' completed successfully.

[root@node1 ~]# ambari-server restart

[root@node1 ambari-server]# grep ldap /etc/ambari-server/conf/ambari.properties
ambari.ldap.isConfigured=true
authentication.ldap.baseDn=dc=hadoop,dc=apache,dc=org
authentication.ldap.bindAnonymously=false
authentication.ldap.dnAttribute=dn
authentication.ldap.groupMembershipAttr=member
authentication.ldap.groupNamingAttr=cn
authentication.ldap.groupObjectClass=groupofnames
authentication.ldap.managerDn=uid=admin,ou=people,dc=hadoop,dc=apache,dc=org
authentication.ldap.managerPassword=/etc/ambari-server/conf/ldap-password.dat
authentication.ldap.pagination.enabled=false
authentication.ldap.primaryUrl=node1.localdomain:33389
authentication.ldap.referral=ignore
authentication.ldap.useSSL=false
authentication.ldap.userObjectClass=person
authentication.ldap.usernameAttribute=uid
client.security=ldap

/etc/ambari-server/conf/log4j.properties
#LDAP debug (そんなに有効ではない）
log4j.logger.org.apache.ambari.server.security=DEBUG

AmbariServerログに、下記のエラーが出たらBase DNを疑ってみる。
25 Apr 2017 07:23:15,290 FATAL [pool-16-thread-1] AbstractRequestControlDirContextProcessor:186 - No matching response control found for paged results - looking for 'class javax.naming.ldap.PagedResultsResponseControl

それでもだめなら、下記をambari.propertiesに追加してみる
authentication.ldap.pagination.enabled=false

補足１: 'admin'ユーザがかぶるので、パスワードがadmin-passwordになります。
再度、ローカルユーザ'admin'を作ると、Ambariが正しくユーザリストを表示しなくなります。（DBはユーザネームがユニークでないのに、Ambari側はユーニークなのを期待している模様）
そこで、Admin以外のユーザだけを追加する：
ambari-server sync-ldap --users <(grep -E '^uid' /etc/knox/conf/users.ldif | grep -vw admin | awk '{print $2}' | tr '\n' ',') --verbose

補足２：ユーザをデータベースを消すのはよくなさそうなので、APIで：
for u in `grep -E '^uid' /etc/knox/conf/users.ldif | grep -vw admin | awk '{print $2}'`; do curl -i -u admin:admin -H 'X-Requested-By: ambari' -X DELETE http://localhost:8080/api/v1/users/$u; done

2017年4月20日木曜日

HDP-2.6で追加されたMERGEを使って見る

https://cwiki.apache.org/confluence/display/Hive/LanguageManual+DML#LanguageManualDML-Merge
https://hortonworks.com/blog/apache-hive-moving-beyond-analytics-offload-with-sql-merge/
https://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.6.0/bk_data-access/content/new-feature-insert-values-update-delete.html

ACIDが必要 (AcidOutputFormat)

CREATE TABLE `ts_test2`(
`id` int,
`ts` timestamp)
CLUSTERED BY (id) INTO 4 BUCKETS
STORED AS ORC
TBLPROPERTIES (
'transactional'='true');

すべて上書き：

MERGE INTO ts_test2 AS T
USING ( select * from ts_test) AS S
ON T.id = S.id
when matched then update set ts = S.ts
when not matched then insert values (S.id, S.ts);

なかったら追加：
MERGE INTO ts_test2 AS T
USING ( select * from ts_test) AS S
ON T.id = S.id
when not matched then insert values (S.id, S.ts);

出力例：

0: jdbc:hive2://node7.localdomain:10000> select * from ts_test;
Getting log thread is interrupted, since query is done!
+-------------+------------------------+--+
| ts_test.id | ts_test.ts |
+-------------+------------------------+--+
| 1 | NULL |
| 2 | 1970-01-01 00:00:00.0 |
+-------------+------------------------+--+
2 rows selected (0.23 seconds)
0: jdbc:hive2://node7.localdomain:10000> select * from ts_test2;
Getting log thread is interrupted, since query is done!
+--------------+------------------------+--+
| ts_test2.id | ts_test2.ts |
+--------------+------------------------+--+
| 1 | 1973-10-20 00:00:00.0 |
+--------------+------------------------+--+
1 row selected (0.187 seconds)
0: jdbc:hive2://node7.localdomain:10000> MERGE INTO ts_test2 AS T
0: jdbc:hive2://node7.localdomain:10000> USING ( select * from ts_test) AS S
0: jdbc:hive2://node7.localdomain:10000> ON T.id = S.id
0: jdbc:hive2://node7.localdomain:10000> when matched then update set ts = S.ts
0: jdbc:hive2://node7.localdomain:10000> when not matched then insert values (S.id, S.ts);
INFO : Session is already open
INFO : Dag name: MERGE INTO ts_test2 AS T
USING ( sel...S.ts)(Stage-3)
INFO : Setting tez.task.scale.memory.reserve-fraction to 0.30000001192092896
INFO : Status: Running (Executing on YARN cluster with App id application_1492638963323_0002)

--------------------------------------------------------------------------------
VERTICES STATUS TOTAL COMPLETED RUNNING PENDING FAILED KILLED
--------------------------------------------------------------------------------
Map 1 .......... SUCCEEDED 1 1 0 0 0 0
Map 5 .......... SUCCEEDED 4 4 0 0 0 0
Reducer 2 ...... SUCCEEDED 4 4 0 0 0 0
Reducer 3 ...... SUCCEEDED 1 1 0 0 0 0
Reducer 4 ...... SUCCEEDED 4 4 0 0 0 0
--------------------------------------------------------------------------------
VERTICES: 05/05 [==========================>>] 100% ELAPSED TIME: 14.64 s
--------------------------------------------------------------------------------
Getting log thread is interrupted, since query is done!
INFO : Loading data to table default.ts_test2 from hdfs://nnha-c6-ubu02/apps/hive/warehouse/ts_test2/.hive-staging_hive_2017-04-20_12-07-04_535_7291759981953472305-1/-ext-10002
INFO : Loading data to table default.merge_tmp_table from hdfs://nnha-c6-ubu02/tmp/hive/hive/31e5df2a-7dab-498c-85c0-0810484896f4/_tmp_space.db/merge_tmp_table/.hive-staging_hive_2017-04-20_12-07-04_535_7291759981953472305-1/-ext-10004
INFO : Loading data to table default.ts_test2 from hdfs://nnha-c6-ubu02/apps/hive/warehouse/ts_test2/.hive-staging_hive_2017-04-20_12-07-04_535_7291759981953472305-1/-ext-10000
INFO : Table default.ts_test2 stats: [numFiles=3, numRows=0, totalSize=1855, rawDataSize=0]
INFO : Table default.merge_tmp_table stats: [numFiles=0, totalSize=0]
INFO : Table default.ts_test2 stats: [numFiles=3, numRows=0, totalSize=1855, rawDataSize=0]
No rows affected (16.536 seconds)
0: jdbc:hive2://node7.localdomain:10000> select * from ts_test2;
Getting log thread is interrupted, since query is done!
+--------------+------------------------+--+
| ts_test2.id | ts_test2.ts |
+--------------+------------------------+--+
| 1 | NULL |
| 2 | 1970-01-01 00:00:00.0 |
+--------------+------------------------+--+
0: jdbc:hive2://node7.localdomain:10000> MERGE INTO ts_test2 AS T
0: jdbc:hive2://node7.localdomain:10000> USING ( select * from ts_test) AS S
0: jdbc:hive2://node7.localdomain:10000> ON T.id = S.id
0: jdbc:hive2://node7.localdomain:10000> when not matched then insert values (S.id, S.ts);
INFO : Session is already open
INFO : Dag name: MERGE INTO ts_test2 AS T
USING ( sel...S.ts)(Stage-1)
INFO : Setting tez.task.scale.memory.reserve-fraction to 0.30000001192092896
INFO : Status: Running (Executing on YARN cluster with App id application_1492638963323_0002)

--------------------------------------------------------------------------------
VERTICES STATUS TOTAL COMPLETED RUNNING PENDING FAILED KILLED
--------------------------------------------------------------------------------
Map 1 .......... SUCCEEDED 1 1 0 0 0 0
Map 3 .......... SUCCEEDED 4 4 0 0 0 0
Reducer 2 ...... SUCCEEDED 4 4 0 0 0 0
--------------------------------------------------------------------------------
VERTICES: 03/03 [==========================>>] 100% ELAPSED TIME: 8.57 s
--------------------------------------------------------------------------------
Getting log thread is interrupted, since query is done!
INFO : Loading data to table default.ts_test2 from hdfs://nnha-c6-ubu02/apps/hive/warehouse/ts_test2/.hive-staging_hive_2017-04-20_12-11-24_801_8984288262712289131-1/-ext-10000
INFO : Table default.ts_test2 stats: [numFiles=2, numRows=0, totalSize=1271, rawDataSize=0]
No rows affected (10.344 seconds)
0: jdbc:hive2://node7.localdomain:10000> select * from ts_test2;
Getting log thread is interrupted, since query is done!
+--------------+------------------------+--+
| ts_test2.id | ts_test2.ts |
+--------------+------------------------+--+
| 1 | 1973-10-20 00:00:00.0 |
| 2 | 1970-01-01 00:00:00.0 |
+--------------+------------------------+--+
2 rows selected (0.485 seconds)

0: jdbc:hive2://node7.localdomain:10000>

エラーになる例：
なぜかエイリアスは使えない
when matched then update set T.ts = S.ts
ワイルドカードはダメ
when not matched then insert values (S.*);

登録: 投稿 (Atom)