Caused by: java.lang.ClassNotFoundException: org.codehaus.jackson.jaxrs.JacksonJsonProvider

ranger와 HDFS 플러그인을 연동 중, 

enable-hdfs-plugin.sh 실행 후 hdfs namenode를 재기동하였을 때,

에러와 함께 namenode가 기동되지않는 현상이었습니다.

전체 로그

java.lang.NoClassDefFoundError: org/codehaus/jackson/jaxrs/JacksonJsonProvider
        at org.apache.ranger.plugin.util.RangerRESTClient.buildClient(RangerRESTClient.java:226)
        at org.apache.ranger.plugin.util.RangerRESTClient.getClient(RangerRESTClient.java:193)
        at org.apache.ranger.plugin.util.RangerRESTClient.get(RangerRESTClient.java:473)
        at org.apache.ranger.admin.client.RangerAdminRESTClient.getRangerRolesDownloadResponse(RangerAdminRESTClient.java:1340)
        at org.apache.ranger.admin.client.RangerAdminRESTClient.getRolesIfUpdatedWithCred(RangerAdminRESTClient.java:1202)
        at org.apache.ranger.admin.client.RangerAdminRESTClient.getRolesIfUpdated(RangerAdminRESTClient.java:167)
        at org.apache.ranger.plugin.util.RangerRolesProvider.loadUserGroupRolesFromAdmin(RangerRolesProvider.java:183)
        at org.apache.ranger.plugin.util.RangerRolesProvider.loadUserGroupRoles(RangerRolesProvider.java:123)
        at org.apache.ranger.plugin.util.PolicyRefresher.loadRoles(PolicyRefresher.java:495)
        at org.apache.ranger.plugin.util.PolicyRefresher.startRefresher(PolicyRefresher.java:144)
        at org.apache.ranger.plugin.service.RangerBasePlugin.init(RangerBasePlugin.java:245)
        at org.apache.ranger.authorization.hadoop.RangerHdfsAuthorizer.start(RangerHdfsAuthorizer.java:113)
        at org.apache.ranger.authorization.hadoop.RangerHdfsAuthorizer.start(RangerHdfsAuthorizer.java:86)
        at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.startCommonServices(FSNamesystem.java:1321)
        at org.apache.hadoop.hdfs.server.namenode.NameNode.startCommonServices(NameNode.java:877)
        at org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:798)
        at org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:1027)
        at org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:1002)
        at org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1776)
        at org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1841)
Caused by: java.lang.ClassNotFoundException: org.codehaus.jackson.jaxrs.JacksonJsonProvider
        at java.lang.ClassLoader.findClass(ClassLoader.java:530)
        at org.apache.ranger.plugin.classloader.RangerPluginClassLoader$MyClassLoader.findClass(RangerPluginClassLoader.java:316)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
        at org.apache.ranger.plugin.classloader.RangerPluginClassLoader.loadClass(RangerPluginClassLoader.java:160)
        ... 20 more

 

 

저의 환경은 ranger 2.4.0, hadoop 3.3.5버전이었습니다.

hadoop 3.3.5버전 부터 jackson 종속성이 업그레이드되었고, jackson-jaxrs jar를 포함한 모든 jackson 1.x 패키지를 제거했다고 나와있습니다. 이에 플러그인 활성화 후  jackson 1.x를 로드할 수 없기 때문에 hadoop namenode가 기동되지 않았습니다.

ranger 2.4.0버전을 사용하기위해 수동으로 github의 소스를 받아 tags/release-ranger-2.4.0 태그로 빌드를 하였습니다.

위 문제에 대한 커밋은 아직 해당 태그에는 반영되어있지않아. 2개의 파일을 수동으로 수정 후 재빌드하여 해당 문제를 해결하였습니다.

vi {RANGER_SOURCE_HOME}/distro/src/main/assembly/hdfs-agent.xml
     84         <fileMode>644</fileMode>
     85         <includes>
     86           <include>commons-lang:commons-lang</include>
                  <!-- 해당 줄을 추가해줍니다. -->
     87           <include>org.codehaus.jackson:jackson-jaxrs:jar:${codehaus.jackson.version}</include>
     88           <include>org.eclipse.jetty:jetty-client:jar:${jetty-client.version}</include>
     89           <include>org.apache.httpcomponents:httpmime:jar:${httpcomponents.httpmime.version}</include>
     90           <include>org.apache.httpcomponents:httpclient:jar:${httpcomponents.httpclient.version}</include>
     91           <include>org.apache.httpcomponents:httpcore:jar:${httpcomponents.httpcore.version}</include>


vi ./distro/src/main/assembly/plugin-yarn.xml
     54         <fileMode>644</fileMode>
     55         <includes>
     56           <include>commons-lang:commons-lang:jar:${commons.lang.version}</include>
                  <!-- 해당 줄을 추가해줍니다. --> 
     57           <include>org.codehaus.jackson:jackson-jaxrs:jar:${codehaus.jackson.version}</include>
     58           <include>org.eclipse.jetty:jetty-client:jar:${jetty-client.version}</include>
     59           <include>org.apache.httpcomponents:httpmime:jar:${httpcomponents.httpmime.version}</include>
     60           <include>org.apache.httpcomponents:httpclient:jar:${httpcomponents.httpclient.version}</include>
     61           <include>org.apache.httpcomponents:httpcore:jar:${httpcomponents.httpcore.version}</include>
     62           <include>org.noggit:noggit:jar:${noggit.version}</include>

 

 

해당 문제에 대한 자료:

https://issues.apache.org/jira/browse/RANGER-4327

[RANGER-4327] Make ranger2.4 hdfs plugin compatible with hadoop 3.3.5 + - ASF JIRA

 

ranger 빌드:

https://developer-woong.tistory.com/95

ranger) Apache Ranger 2.4.0 빌드

 

감사합니다.