Sandbox (HDP 2.6.3)でYarn Scheduler Load Simulator (SLS)を実行してみる

Ref: https://hadoop.apache.org/docs/r2.4.1/hadoop-sls/SchedulerLoadSimulator.html

su - yarn    # yarnになるのに特に理由なし
export HADOOP_ROOT=/home/yarn/hadoop

tar -xf /usr/hdp/2.6.3.0-235/hadoop/mapreduce.tar.gz
cat $HADOOP_ROOT/share/hadoop/tools/sls/sample-conf/sls-runner.xml   #Refと場所が違う
cp -r /etc/hadoop/conf/ $HADOOP_ROOT/etc/hadoop/
cp $HADOOP_ROOT/share/hadoop/tools/sls/sample-conf/sls-runner.xml $HADOOP_ROOT/etc/hadoop/sls-runner.xml
vim $HADOOP_ROOT/etc/hadoop/sls-runner.xml   # デフォルトでOK?

cd $HADOOP_ROOT/share/hadoop/tools/sls # or cd ./hadoop/share/hadoop/tools/sls
bash -x bin/slsrun.sh --input-rumen=sample-data/2jobs2min-rumen-jh.json --output-dir=sample-output


TODO: NPEがでる

18/03/06 00:49:43 INFO capacity.CapacityScheduler: Added node a2116.smile.com:3 clusterResource: <memory:40960, vCores:40>
Exception in thread "main" java.lang.RuntimeException: java.lang.NullPointerException
        at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:134)
        at org.apache.hadoop.yarn.sls.SLSRunner.startAMFromRumenTraces(SLSRunner.java:398)
        at org.apache.hadoop.yarn.sls.SLSRunner.startAM(SLSRunner.java:250)
        at org.apache.hadoop.yarn.sls.SLSRunner.start(SLSRunner.java:145)
        at org.apache.hadoop.yarn.sls.SLSRunner.main(SLSRunner.java:528)
Caused by: java.lang.NullPointerException
        at java.util.concurrent.ConcurrentHashMap.get(ConcurrentHashMap.java:936)
        at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:126)
        ... 4 more


http://sandbox-hdp.hortonworks.com:10001/simulate

java.lang.NullPointerException
        at org.apache.hadoop.yarn.sls.web.SLSWebApp.printPageSimulate(SLSWebApp.java:219)
        at org.apache.hadoop.yarn.sls.web.SLSWebApp.access$100(SLSWebApp.java:54)
        at org.apache.hadoop.yarn.sls.web.SLSWebApp$1.handle(SLSWebApp.java:131)
        at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
        at org.mortbay.jetty.Server.handle(Server.java:326)
        at org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542)
        at org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:928)
        at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:549)
        at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212)
        at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)
        at org.mortbay.jetty.bio.SocketConnector$Connection.run(SocketConnector.java:228)
        at org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)