YARN incompatibility with 2.3.GlusterFS stack and ambari 2.2.1.0 #134
Comments
|
Hi, I missed some properties from https://github.com/gluster/glusterfs-hadoop/wiki/Configuring-Ambari which solved the problem with Yarn. However, Glusterfs Hadoop plugin with Ambari needs NAMENODE component to be chosen to function. The problem is that NAMENODE belongs to HDFS service. It is not allowed to have components from 2 file system services. Is there any workaround ? |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Hi,
We are trying to install Hadoop cluster using 2.3.GlusterFS stack through Ambari 2.2.1.0.
All provisioning is done with Ambari blueprints like :
Blueprint
{ "configurations" : [ { "core-site": { "hadoop.proxyuser.hcat.groups" : "*", "hadoop.proxyuser.hcat.hosts" : "*", "hadoop.proxyuser.hue.groups" : "*", "hadoop.proxyuser.hue.hosts" : "*", "hadoop.security.authentication" : "simple", "fs.AbstractFileSystem.glusterfs.impl" : "org.apache.hadoop.fs.local.GlusterFs", "fs.trash.interval" : "360", "fs.glusterfs.impl" : "org.apache.hadoop.fs.glusterfs.GlusterFileSystem", "hadoop.security.authorization" : "false", "net.topology.script.file.name" : "/etc/hadoop/conf/topology_script.py" } }, { "ams-hbase-env" : { "regionserver_xmn_size" : "384m", "hbase_regionserver_heapsize" : "1024m", "hbase_master_heapsize" : "1024m", "hbase_master_xmn_size" : "384m" } }, { "ams-env" : { "metrics_collector_heapsize" : "1024m" } }, { "hadoop-env" : { "dtnode_heapsize" : "1024m", "namenode_heapsize" : "2048m", "namenode_opt_maxnewsize" : "384m", "namenode_opt_newsize" : "384m", "namenode_host" : "master01", "snamenode_host" : "master01", "glusterfs_user" : "root", "hdfs_user" : "hdfs", "hdfs_log_dir_prefix" : "/var/log/hadoop" } }, { "hdfs-site" : { "dfs.datanode.data.dir" : "/hadoop/hdfs/data", "dfs.datanode.balance.bandwidthPerSec" : "12500000", "dfs.datanode.max.transfer.threads": "4096", "dfs.datanode.failed.volumes.tolerated" : "1", "dfs.replication" : "true" } }, { "spark-defaults" : { "spark.executor.instances" : "2", "spark.executor.memory" : "7808m", "spark.driver.memory" : "3712m", "spark.yarn.am.memory" : "3712m", "spark.yarn.executor.memoryOverhead" : "384", "spark.yarn.driver.memoryOverhead" : "384", "spark.yarn.am.memoryOverhead" : "384" } }, { "mapred-site" : { "mapreduce.map.java.opts" : "-Xmx1638m", "mapreduce.map.memory.mb" : "2048", "mapreduce.reduce.java.opts" : "-Xmx1638m", "mapreduce.reduce.memory.mb" : "2048", "mapreduce.task.io.sort.mb" : "768", "yarn.app.mapreduce.am.command-opts" : "-Xmx1638m -Dhdp.version=${hdp.version}", "yarn.app.mapreduce.am.resource.mb" : "2048" } }, { "tez-site" : { "tez.am.resource.memory.mb" : "2048", "tez.task.resource.memory.mb" : "2048" } }, { "spark-defaults" : { "spark.executor.instances" : "1", "spark.executor.memory" : "3712m", "spark.driver.memory" : "1664m", "spark.yarn.am.memory" : "1664m", "spark.yarn.executor.memoryOverhead" : "384", "spark.yarn.driver.memoryOverhead" : "384", "spark.yarn.am.memoryOverhead" : "384" } }, { "storm-site" : { "logviewer.port" : "8005" } }, { "oozie-site" : { "oozie.service.ProxyUserService.proxyuser.hue.groups" : "*", "oozie.service.ProxyUserService.proxyuser.hue.hosts" : "*" } }, { "webhcat-site" : { "webhcat.proxyuser.hue.groups" : "*", "webhcat.proxyuser.hue.hosts" : "*" } }, { "hive-site" : { "hive.tez.container.size" : "-1", "hive.tez.java.opts": "-1", "fs.file.impl.disable.cache" : "true", "fs.hdfs.impl.disable.cache" : "true", "javax.jdo.option.ConnectionPassword" : "my-pw" } } ], "host_groups" : [ { "name" : "slavenode_simple", "configurations" : [ ], "components" : [ { "name" : "ZOOKEEPER_CLIENT" }, { "name" : "OOZIE_CLIENT" }, { "name" : "HIVE_CLIENT" }, { "name" : "GLUSTERFS_CLIENT" }, { "name" : "YARN_CLIENT" }, { "name" : "TEZ_CLIENT" }, { "name" : "SPARK_CLIENT" }, { "name" : "NODEMANAGER" }, { "name" : "METRICS_MONITOR" } ], "cardinality" : "2" }, { "name" : "masternode_1", "configurations" : [ ], "components" : [ { "name" : "NODEMANAGER" }, { "name" : "SPARK_CLIENT" }, { "name" : "YARN_CLIENT" }, { "name" : "GLUSTERFS_CLIENT" }, { "name" : "METRICS_MONITOR" }, { "name" : "TEZ_CLIENT" }, { "name" : "ZOOKEEPER_CLIENT" }, { "name" : "ZOOKEEPER_SERVER" }, { "name" : "AMBARI_SERVER" }, { "name" : "SPARK_JOBHISTORYSERVER" }, { "name" : "APP_TIMELINE_SERVER" }, { "name" : "METRICS_COLLECTOR" }, { "name" : "RESOURCEMANAGER" }, { "name" : "WEBHCAT_SERVER" }, { "name" : "OOZIE_SERVER" }, { "name" : "HIVE_METASTORE" }, { "name" : "HIVE_SERVER" } ], "cardinality" : "1" } ], "Blueprints" : { "stack_name" : "HDP", "stack_version" : "2.3.GlusterFS" } }Template used for blueprint to be provided to :
{ "blueprint": "cluster_blueprint", "default_password": "my-pw", "host_groups": [ { "name" : "masternode_1", "hosts" : [ { "fqdn": "master01.domain.com" } ] } ] }Here is the full stack :
23 Mar 2016 11:05:51,256 INFO [qtp-ambari-client-23] AmbariManagementControllerImpl:1471 - Applying configuration with tag 'INITIAL' to cluster 'test-cluster2' for configuration type zookeeper-env 23 Mar 2016 11:05:51,280 INFO [qtp-ambari-client-23] AmbariManagementControllerImpl:1471 - Applying configuration with tag 'INITIAL' to cluster 'test-cluster2' for configuration type zookeeper-log4j 23 Mar 2016 11:05:51,323 INFO [qtp-ambari-client-23] ClusterConfigurationRequest:358 - Sending cluster config update request for service = YARN 23 Mar 2016 11:05:51,324 INFO [qtp-ambari-client-23] AmbariManagementControllerImpl:1353 - Received a updateCluster request, clusterId=null, clusterName=test-cluster2, securityType=null, request={ clusterName=test-cluster2, clusterId=null, provisioningState=null, securityType=null, stackVersion=null, desired_scv=null, hosts=[] } 23 Mar 2016 11:05:51,324 INFO [qtp-ambari-client-23] AmbariManagementControllerImpl:1471 - Applying configuration with tag 'INITIAL' to cluster 'test-cluster2' for configuration type hdfs-site 23 Mar 2016 11:05:51,351 INFO [qtp-ambari-client-23] AmbariManagementControllerImpl:1471 - Applying configuration with tag 'INITIAL' to cluster 'test-cluster2' for configuration type capacity-scheduler 23 Mar 2016 11:05:51,378 INFO [qtp-ambari-client-23] AmbariManagementControllerImpl:1471 - Applying configuration with tag 'INITIAL' to cluster 'test-cluster2' for configuration type yarn-env 23 Mar 2016 11:05:51,411 INFO [qtp-ambari-client-23] AmbariManagementControllerImpl:1471 - Applying configuration with tag 'INITIAL' to cluster 'test-cluster2' for configuration type yarn-site 23 Mar 2016 11:05:51,490 INFO [qtp-ambari-client-23] AmbariManagementControllerImpl:1471 - Applying configuration with tag 'INITIAL' to cluster 'test-cluster2' for configuration type yarn-log4j 23 Mar 2016 11:05:51,567 ERROR [qtp-ambari-client-23] ClusterImpl:2635 - Updating configs for multiple services by a single API request isn't supported, config version not created 23 Mar 2016 11:05:51,572 ERROR [qtp-ambari-client-23] BaseManagementHandler:66 - Caught a runtime exception while attempting to create a resource: Failed to set configurations on cluster: org.apache.ambari.server.AmbariException: Updating configs for multiple services by a single API request isn't supported java.lang.RuntimeException: Failed to set configurations on cluster: org.apache.ambari.server.AmbariException: Updating configs for multiple services by a single API request isn't supported at org.apache.ambari.server.topology.AmbariContext.setConfigurationOnCluster(AmbariContext.java:390) at org.apache.ambari.server.topology.ClusterConfigurationRequest.setConfigurationsOnCluster(ClusterConfigurationRequest.java:359) at org.apache.ambari.server.topology.ClusterConfigurationRequest.setConfigurationsOnCluster(ClusterConfigurationRequest.java:279) at org.apache.ambari.server.topology.ClusterConfigurationRequest.<init>(ClusterConfigurationRequest.java:78) at org.apache.ambari.server.topology.ClusterConfigurationRequest.<init>(ClusterConfigurationRequest.java:83) at org.apache.ambari.server.topology.TopologyManager.provisionCluster(TopologyManager.java:191) at org.apache.ambari.server.controller.internal.ClusterResourceProvider.processBlueprintCreate(ClusterResourceProvider.java:517) at org.apache.ambari.server.controller.internal.ClusterResourceProvider.createResources(ClusterResourceProvider.java:174) at org.apache.ambari.server.controller.internal.ClusterControllerImpl.createResources(ClusterControllerImpl.java:289) at org.apache.ambari.server.api.services.persistence.PersistenceManagerImpl.create(PersistenceManagerImpl.java:76) at org.apache.ambari.server.api.handlers.CreateHandler.persist(CreateHandler.java:36) at org.apache.ambari.server.api.handlers.BaseManagementHandler.handleRequest(BaseManagementHandler.java:72) at org.apache.ambari.server.api.services.BaseRequest.process(BaseRequest.java:135) at org.apache.ambari.server.api.services.BaseService.handleRequest(BaseService.java:106) at org.apache.ambari.server.api.services.BaseService.handleRequest(BaseService.java:75) at org.apache.ambari.server.api.services.ClusterService.createCluster(ClusterService.java:131) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:497) at com.sun.jersey.spi.container.JavaMethodInvokerFactory$1.invoke(JavaMethodInvokerFactory.java:60) at com.sun.jersey.server.impl.model.method.dispatch.AbstractResourceMethodDispatchProvider$ResponseOutInvoker._dispatch(AbstractResourceMethodDispatchProvider.java:205) at com.sun.jersey.server.impl.model.method.dispatch.ResourceJavaMethodDispatcher.dispatch(ResourceJavaMethodDispatcher.java:75) at com.sun.jersey.server.impl.uri.rules.HttpMethodRule.accept(HttpMethodRule.java:302) at com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147) at com.sun.jersey.server.impl.uri.rules.ResourceClassRule.accept(ResourceClassRule.java:108) at com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147) at com.sun.jersey.server.impl.uri.rules.RootResourceClassesRule.accept(RootResourceClassesRule.java:84) at com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1542) at com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1473) at com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1419) at com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1409) at com.sun.jersey.spi.container.servlet.WebComponent.service(WebComponent.java:409) at com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:540) at com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:715) at javax.servlet.http.HttpServlet.service(HttpServlet.java:770) at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:684) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1496) at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:330) at org.springframework.security.web.access.intercept.FilterSecurityInterceptor.invoke(FilterSecurityInterceptor.java:118) at org.springframework.security.web.access.intercept.FilterSecurityInterceptor.doFilter(FilterSecurityInterceptor.java:84) at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342) at org.springframework.security.web.access.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:113) at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342) at org.springframework.security.web.session.SessionManagementFilter.doFilter(SessionManagementFilter.java:103) at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342) at org.springframework.security.web.authentication.AnonymousAuthenticationFilter.doFilter(AnonymousAuthenticationFilter.java:113) at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342) at org.springframework.security.web.servletapi.SecurityContextHolderAwareRequestFilter.doFilter(SecurityContextHolderAwareRequestFilter.java:54) at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342) at org.springframework.security.web.savedrequest.RequestCacheAwareFilter.doFilter(RequestCacheAwareFilter.java:45) at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342) at org.apache.ambari.server.security.authorization.AmbariAuthorizationFilter.doFilter(AmbariAuthorizationFilter.java:196) at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342) at org.springframework.security.web.authentication.www.BasicAuthenticationFilter.doFilter(BasicAuthenticationFilter.java:201) at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342) at org.springframework.security.web.context.SecurityContextPersistenceFilter.doFilter(SecurityContextPersistenceFilter.java:87) at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342) at org.springframework.security.web.FilterChainProxy.doFilterInternal(FilterChainProxy.java:192) at org.springframework.security.web.FilterChainProxy.doFilter(FilterChainProxy.java:160) at org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:237) at org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:167) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1467) at org.apache.ambari.server.api.MethodOverrideFilter.doFilter(MethodOverrideFilter.java:72) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1467) at org.apache.ambari.server.api.AmbariPersistFilter.doFilter(AmbariPersistFilter.java:47) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1467) at org.apache.ambari.server.security.AbstractSecurityHeaderFilter.doFilter(AbstractSecurityHeaderFilter.java:109) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1467) at org.eclipse.jetty.servlets.UserAgentFilter.doFilter(UserAgentFilter.java:82) at org.eclipse.jetty.servlets.GzipFilter.doFilter(GzipFilter.java:294) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1467) at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:501) at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:137) at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:557) at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:231) at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1086) at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:429) at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:193) at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1020) at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:135) at org.apache.ambari.server.controller.AmbariHandlerList.processHandlers(AmbariHandlerList.java:216) at org.apache.ambari.server.controller.AmbariHandlerList.processHandlers(AmbariHandlerList.java:205) at org.apache.ambari.server.controller.AmbariHandlerList.handle(AmbariHandlerList.java:139) at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:116) at org.eclipse.jetty.server.Server.handle(Server.java:370) at org.eclipse.jetty.server.AbstractHttpConnection.handleRequest(AbstractHttpConnection.java:494) at org.eclipse.jetty.server.AbstractHttpConnection.content(AbstractHttpConnection.java:982) at org.eclipse.jetty.server.AbstractHttpConnection$RequestHandler.content(AbstractHttpConnection.java:1043) at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:865) at org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:240) at org.eclipse.jetty.server.AsyncHttpConnection.handle(AsyncHttpConnection.java:82) at org.eclipse.jetty.io.nio.SelectChannelEndPoint.handle(SelectChannelEndPoint.java:696) at org.eclipse.jetty.io.nio.SelectChannelEndPoint$1.run(SelectChannelEndPoint.java:53) at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608) at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.ambari.server.AmbariException: Updating configs for multiple services by a single API request isn't supported at org.apache.ambari.server.utils.RetryHelper.executeWithRetry(RetryHelper.java:110) at org.apache.ambari.server.topology.AmbariContext.setConfigurationOnCluster(AmbariContext.java:381) ... 96 more Caused by: java.lang.IllegalArgumentException: Updating configs for multiple services by a single API request isn't supported at org.apache.ambari.server.state.cluster.ClusterImpl.applyConfigs(ClusterImpl.java:2634) at org.apache.ambari.server.orm.AmbariJpaLocalTxnInterceptor.invoke(AmbariJpaLocalTxnInterceptor.java:68) at org.apache.ambari.server.state.cluster.ClusterImpl.addDesiredConfig(ClusterImpl.java:2172) at org.apache.ambari.server.controller.AmbariManagementControllerImpl.updateCluster(AmbariManagementControllerImpl.java:1485) at org.apache.ambari.server.controller.AmbariManagementControllerImpl.updateClusters(AmbariManagementControllerImpl.java:1337) at org.apache.ambari.server.topology.AmbariContext$5.call(AmbariContext.java:384) at org.apache.ambari.server.utils.RetryHelper.executeWithRetry(RetryHelper.java:95) ... 97 moreWhen we change GLUSTERFS_CLIENT to HDFS_CLIENT and stack to be 2.4 (HDFS), everything is fine.
Do you have any idea please ? Is there any incompatibility ?
Thanks
The text was updated successfully, but these errors were encountered: