Hive SymlinkTextInputFormatを使ってみる

https://issues.apache.org/jira/browse/HIVE-1272

まず、ダミーテーブル達を作ります

curl -O https://raw.githubusercontent.com/hajimeo/samples/master/bash/hive_dummies.sh
bash -x ./hive_dummies.sh

"sample_07"をベースにSymlinkTextテーブルを作ってみます。

CREATE TABLE `sample_07_sym`(
  `code` string,
  `description` string,
  `total_emp` int,
  `salary` int)
ROW FORMAT DELIMITED
  FIELDS TERMINATED BY '\t'
STORED AS INPUTFORMAT
  'org.apache.hadoop.hive.ql.io.SymlinkTextInputFormat'
OUTPUTFORMAT
  'org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat';

[admin@node1 ~]$ cat link.txt
/apps/hive/warehouse/*/sample_07/*
[admin@node1 ~]$ hdfs dfs -put -f link.txt /apps/hive/warehouse/dummies.db/sample_07_sym/

hive> set hive.fetch.task.conversion.threshold;
hive.fetch.task.conversion.threshold=1073741824
hive> select * from sample_07_sym limit 1;
OK
00-0000 All Occupations 134354250       40690
Time taken: 0.048 seconds, Fetched: 1 row(s)