HDP2.5 SandboxでOozie Example/sampleを試す

https://oozie.apache.org/docs/4.3.0/DG_Examples.html

１）Ooize Exampleがインストールされているかチェック
yum list installed | grep oozie
hue-oozie.x86_64                        2.6.1.2.5.0.0-1245.el6       @HDP-2.5
oozie_2_5_0_0_1245.noarch               4.2.0.2.5.0.0-1245.el6       @HDP-2.5
oozie_2_5_0_0_1245-client.noarch        4.2.0.2.5.0.0-1245.el6       @HDP-2.5

rpm -Vv oozie_2_5_0_0_1245-client | grep example
.........  d /usr/hdp/2.5.0.0-1245/oozie/doc/oozie-examples.tar.gz

1.2）上記のように削除されている場合、/etc/yum.confを変更し、再インストール
sed -i.bak 's/^tsflags=nodocs/#tsflags=nodocs/g' /etc/yum.conf

yum reinstall oozie_*-client
# 確認 tar tvf /usr/hdp/current/oozie-client/doc/oozie-examples.tar.gz

1.3）yum.confをもどす
mv /etc/yum.conf.bak /etc/yum.conf

1.4）HDFSにExampleをプッシュ
su - <test user>    #例えば oozie
tar xvf /usr/hdp/current/oozie-client/doc/oozie-examples.tar.gz
hdfs dfs -put examples examples
hdfs dfs -chmod -R 777 examples/*-data

２）Hiveアクションを実行してみる
# ダミーデータを作る
[ -s ~/examples/input-data/table/int.txt ] || for i in {1..10}; do echo $i >> ~/examples/input-data/table/int.txt; done
hdfs dfs -put -f ~/examples/input-data/table/int.txt examples/input-data/table/int.txt

# 実行してみる

cd ~/examples/apps/hive

ls -l
total 20
-rw-r--r-- 1 oozie hadoop 1000 Aug 26 03:50 job.properties
-rw-r--r-- 1 oozie hadoop  110 Aug 26 03:50 README
-rw-r--r-- 1 oozie hadoop  966 Aug 26 03:50 script.q
-rw-r--r-- 1 oozie hadoop 2003 Aug 26 03:50 workflow.xml
-rw-r--r-- 1 oozie hadoop 2398 Aug 26 03:50 workflow.xml.security

cat job.properties
nameNode=hdfs://sandbox.hortonworks.com:8020
jobTracker=sandbox.hortonworks.com:8050 # check yarn.resourcemanager.address
queueName=test
examplesRoot=examples
oozie.use.system.libpath=true
oozie.wf.application.path=${nameNode}/user/${user.name}/${examplesRoot}/apps/hive

oozie job -config ./job.properties -run
Job ID : 0000000-170111083341216-oozie-oozi-W

oozie job -info 0000000-170111083341216-oozie-oozi-W -verbose

# 確認
hdfs dfs -ls -R examples/output-data
drwxrwxrwx   - oozie hdfs          0 2017-01-11 09:34 examples/output-data/hive
-rwxrwxrwx   1 oozie hdfs         21 2017-01-11 09:34 examples/output-data/hive/000000_0


３）Hive2アクションを実行してみる
# 事前にダミーデータを作っておく

cd ~/examples/apps/hive2

ls -l
total 24
-rw-r--r-- 1 oozie hadoop 1046 Aug 26 03:50 job.properties
-rw-r--r-- 1 oozie hadoop 1087 Aug 26 03:50 job.properties.security
-rw-r--r-- 1 oozie hadoop  681 Aug 26 03:50 README
-rw-r--r-- 1 oozie hadoop  966 Aug 26 03:50 script.q
-rw-r--r-- 1 oozie hadoop 2073 Aug 26 03:50 workflow.xml
-rw-r--r-- 1 oozie hadoop 2481 Aug 26 03:50 workflow.xml.security

cat job.properties

nameNode=hdfs://sandbox.hortonworks.com:8020

jobTracker=sandbox.hortonworks.com:8050    # If yarn HA, 8032

queueName=test

jdbcURL=jdbc:hive2://sandbox.hortonworks.com:10000/default

examplesRoot=examples

oozie.use.system.libpath=true

oozie.wf.application.path=${nameNode}/user/${user.name}/${examplesRoot}/apps/hive2

oozie job -config ./job.properties -run

job: 0000004-170111083341216-oozie-oozi-W

oozie job -info 0000004-170111083341216-oozie-oozi-W -verbose

3.1）tez.queue.nameを変更するために、workflow.xmlにargumentタグを追加

            <param>OUTPUT=/user/${wf:user()}/${examplesRoot}/output-data/hive2</param>

            <argument>-hiveconf</argument>

            <argument>tez.queue.name=${queueName}</argument>

        </hive2>

追記 HDP2.6.0 Sandbox: 

もしKerberosが適用されていてかつHTTPモードの場合、Hive2 Actionはjob.properties.securityの変更が必要:
nameNode=hdfs://sandbox.hortonworks.com:8020
jobTracker=sandbox.hortonworks.com:8032 #2.6は8032の模様
queueName=default
jdbcURL=jdbc:hive2://sandbox.hortonworks.com:2181/;serviceDiscoveryMode=zooKeeper;zooKeeperNamespace=hiveserver2
#jdbcURL=jdbc:hive2://sandbox.hortonworks.com:10001/default;transportMode=http;httpPath=cliservice
#jdbcURL=jdbc:hive2://sandbox.hortonworks.com:10000/default
jdbcPrincipal=hive/_HOST@EXAMPLE.COM
examplesRoot=examples
oozie.use.system.libpath=true
oozie.wf.application.path=${nameNode}/user/${user.name}/${examplesRoot}/apps/hive2/workflow.xml.security

configファイルは拡張子が.propertiesか.xmlでないとダメなので
mv ./job.properties.security ./job.security.properties
oozie job -config ./job.security.properties -run -verbose

Hive Actionはworkflow.xml.securityの変更が必要:
<workflow-app xmlns="uri:oozie:workflow:0.5" name="hive-wf">
    <credentials>
      <credential name='hcatauth' type='hcat'>
         <property>
            <name>hcat.metastore.uri</name>
            <value>thrift://sandbox.horonworks.com:9083</value>
         </property>
         <property>
             <name>hcat.metastore.principal</name>
             <value>hive/_HOST@EXAMPLE.COM</value>
         </property>

hdfs dfs -put -f workflow.xml.security examples/apps/hive/workflow.xml