yarn capacitiy 스케줄러 설정

yarn의 capacity-scheduler.xml을 설정하여

큐를 나누어보겠습니다.

 

hadoop이 설치되어있음을 가정합니다.

조언과 충고 감사합니다.

 

https://developer-woong.tistory.com/36

hadoop 설치 (정리모음버전)

 

1. yarn config 수정

(* default 큐를 75%, haetlusr 큐를 25%로 설정)

action server: mast02

user: hadoop

pwd: /opt/apps/hadoop-3.2.1/etc/hadoop

cmd:

vi capacity-scheduler.xml

<configuration>
    <property>
        <name>yarn.scheduler.capacity.root.haetlusr.maximum-application-lifetime</name>
        <value>-1</value>
    </property>
    <property>
        <name>yarn.resourcemanager.nodes.exclude-path</name>
        <value>/opt/apps/hadoop-3.2.1/etc/hadoop/yarn_work_exclude_list</value>
    </property>
    <property>
        <name>yarn.scheduler.capacity.maximum-am-resource-percent</name>
        <value>0.1</value>
    </property>
    <property>
        <name>yarn.scheduler.capacity.root.haetlusr.state</name>
        <value>RUNNING</value>
    </property>
    <property>
        <name>yarn.scheduler.capacity.root.queues</name>
        <value>default,haetlusr</value>
    </property>
    <property>
        <name>yarn.scheduler.capacity.root.default.acl_submit_applications</name>
        <value>*</value>
    </property>
    <property>
        <name>yarn.scheduler.capacity.root.default.acl_administer_queue</name>
        <value>*</value>
    </property>
    <property>
        <name>yarn.scheduler.capacity.root.default.acl_application_max_priority</name>
        <value>*</value>
    </property>
    <property>
        <name>yarn.scheduler.capacity.root.default.maximum-application-lifetime</name>
        <value>-1</value>
    </property>
    <property>
        <name>yarn.scheduler.capacity.maximum-applications</name>
        <value>10000</value>
    </property>
    <property>
        <name>yarn.scheduler.capacity.queue-mappings-override.enable</name>
        <value>false</value>
    </property>
    <property>
        <name>yarn.scheduler.capacity.root.haetlusr.capacity</name>
        <value>25</value>
    </property>
    <property>
        <name>yarn.scheduler.capacity.root.haetlusr.maximum-capacity</name>
        <value>90</value>
    </property>
    <property>
        <name>yarn.scheduler.capacity.root.default.state</name>
        <value>RUNNING</value>
    </property>
    <property>
        <name>yarn.scheduler.capacity.rack-locality-additional-delay</name>
        <value>-1</value>
    </property>
    <property>
        <name>yarn.resourcemanager.nodes.include-path</name>
        <value>/opt/apps/hadoop-3.2.1/etc/hadoop/yarn_work_include_list</value>
    </property>
    <property>
        <name>yarn.scheduler.capacity.root.haetlusr.default-application-lifetime</name>
        <value>-1</value>
    </property>
    <property>
        <name>yarn.scheduler.capacity.root.haetlusr.acl_submit_applications</name>
        <value>*</value>
    </property>
    <property>
        <name>yarn.scheduler.capacity.per-node-heartbeat.maximum-offswitch-assignments</name>
        <value>3</value>
    </property>
    <property>
        <name>yarn.scheduler.capacity.resource-calculator</name>
        <value>org.apache.hadoop.yarn.util.resource.DefaultResourceCalculator</value>
    </property>
    <property>
        <name>yarn.scheduler.capacity.root.default.capacity</name>
        <value>75</value>
    </property>
    <property>
        <name>yarn.scheduler.capacity.root.default.default-application-lifetime</name>
        <value>-1</value>
    </property>
    <property>
        <name>yarn.scheduler.capacity.root.default.user-limit-factor</name>
        <value>1</value>
    </property>
    <property>
        <name>yarn.scheduler.capacity.root.default.maximum-capacity</name>
        <value>100</value>
    </property>
    <property>
        <name>yarn.scheduler.capacity.node-locality-delay</name>
        <value>40</value>
    </property>
    <property>
        <name>yarn.scheduler.capacity.queue-mappings</name>
        <value/>
    </property>
    <property>
        <name>yarn.scheduler.capacity.application.fail-fast</name>
        <value>false</value>
    </property>
    <property>
        <name>yarn.scheduler.capacity.root.haetlusr.user-limit-factor</name>
        <value>1</value>
    </property>
    <property>
        <name>yarn.scheduler.capacity.root.haetlusr.acl_administer_queue</name>
        <value>*</value>
    </property>
    <property>
        <name>yarn.scheduler.capacity.root.haetlusr.acl_application_max_prioity</name>
        <value>*</value>
    </property>
</configuration>

 

2. 설정 파일 재 배포 및 yarn restart

action server: mast02

user: hadoop

pwd: /opt/apps/hadoop-3.2.1/etc/hadoop

cmd:

# hadoop 클러스터 서버들에게 배포 
scp -P 22002 ./capacity-scheduler.xml hadoop@mast01:/opt/apps/hadoop-3.2.1/etc/hadoop
scp -P 22002 ./capacity-scheduler.xml hadoop@mast03:/opt/apps/hadoop-3.2.1/etc/hadoop
scp -P 22002 ./capacity-scheduler.xml hadoop@work01:/opt/apps/hadoop-3.2.1/etc/hadoop
scp -P 22002 ./capacity-scheduler.xml hadoop@work02:/opt/apps/hadoop-3.2.1/etc/hadoop
scp -P 22002 ./capacity-scheduler.xml hadoop@work03:/opt/apps/hadoop-3.2.1/etc/hadoop

# timeline server가 실행중이라면 먼저 중지해주세요.
/opt/apps/hadoop-3.2.1/sbin/yarn-daemon.sh stop timelineserver

# mast02서버 실행(yarn이 동작하는 서버)
/opt/apps/hadoop-3.2.1/sbin/stop-yarn
/opt/apps/hadoop-3.2.1/sbin/start-yarn

 

결과확인

mast02:38088 웹 UI좌측 Scheduler클릭,

-> default, haetlusr로 큐가 나누어져있고,

마우스 오버 시 할당된 퍼센트가 나옵니다