ubuntu14.04下编译安装ambari-2.4.2.0

相关文章推荐
眼睛小的柚子 · 如何保持json序列化的顺序性？-腾讯云开发 ...· 3 月前 ·
唠叨的豌豆 · 在兴奋、质疑与期待中全面开花｜AI赋能药物研 ...· 1 年前 ·
伤情的匕首 · jQuery中parent与remove方法 ...· 1 年前 ·
温文尔雅的大白菜 · sqlserver运行超大sql文件_sql ...· 1 年前 ·
爱热闹的海豚 · install open mpi ...· 1 年前 ·
1、fs.inotify.max_user_watches = 8192 调整到524288
echo fs.inotify.max_user_watches=524288 >> /etc/sysctl.conf && sysctl -p
2、将每个进程可以打开的文件数目加大到65535，缺省为1024
ulimit -SHn 65535
sudo tee /etc/security/limits.d/90-nproc.conf <<-'EOF'
* hard nproc 64000
* soft nproc 64000
root hard nproc 65535
root soft nproc 65535
ulimit -a
系统依赖（编译环境）
sudo apt-get install build-essential cmake automake autoconf bison libboost-all-dev libtool check libmcrypt-dev libeditline-dev libssl-dev
离线安装包
1. httpserver
sh starthttpd.sh
#/usr/bin
cd /media/gordon/DataDisk/软件包
/usr/bin/python -m SimpleHTTPServer 8000 >/dev/null 2>&1 &
exit 0
2. 布置本地软件源
http://192.168.56.1:8000/ambari-HDP/

-rwxr-x--- 1 gitlab root  23667150  3月 31 16:26 grafana-2.6.0.linux-x64.tar.gz*

-rwxr-x--- 1 gitlab root 210185315  3月 31 17:15 hadoop-2.7.1.2.3.4.0-3347.tar.gz*

-rwxr-x--- 1 gitlab root 101551599  3月 31 16:26 hbase-1.1.2.2.3.4.0-3347.tar.gz*

-rwxr-x--- 1 gitlab root 251655846  3月 31 16:26 phoenix-4.4.0.2.3.4.0-3347.tar.gz*
java环境
vim /etc/profile
#set for java
export JAVA_HOME=/opt/jdk1.8.0_111
export PATH=$PATH:$JAVA_HOME/bin
export CLASSPATH=.:$CLASSPATH:$JAVA_HOME/lib
export _JAVA_OPTIONS="-Xmx2048m -XX:MaxPermSize=512m -Djava.awt.headless=true"
maven环境
vim /etc/profile
#set maven environment
M2_HOME=/opt/apache-maven-3.3.9/
export MAVEN_OPTS="-Xms256m -Xmx512m"
export PATH=$M2_HOME/bin:$PATH 
maven有2个位置可以选择配置M2_HOME/conf/setting.xml和~/.m2/setting.xml

前者是全局性的，后者是针对用户的。

查询资料是说用户配置优先级高于全局配置
设置maven-repo本地源
vim /opt/apache-maven-3.3.9/conf/settings.xml
  <localRepository>/opt/maven-repo</localRepository>
gord@gord:/opt$ mvn -v
Apache Maven 3.3.9 (bb52d8502b132ec0a5a3f4c09453c07478323dc5; 2015-11-11T00:41:47+08:00)
Maven home: /opt/apache-maven-3.3.9
Java version: 1.8.0_111, vendor: Oracle Corporation
Java home: /opt/jdk1.8.0_111/jre
Default locale: en_US, platform encoding: UTF-8
OS name: "linux", version: "3.13.0-24-generic", arch: "amd64", family: "unix"
Nodejs环境
今天是2017.04.01 https://nodejs.org/en/官网推荐v6.10.1版本  #实验证明这个版本安装ambari存在很大问题，切换到0.10.44版本的nodejs

可在淘宝下载各个版本：https://npm.taobao.org/mirrors/node
tar xfp node-v0.10.44-linux-x64.tar.gz -C /opt
vim /etc/profile
#set for nodejs
export NODE_HOME=/opt/node-v0.10.44-linux-x64
#export NODE_HOME=/opt/node-v4.4.4-linux-x64
#export NODE_HOME=/opt/node-v6.10.1-linux-x64
export PATH=$PATH:$NODE_HOME/bin
source /etc/profile
配置淘宝源
kylin@Ubuntu:~$ npm config set registry https://registry.npm.taobao.org  
kylin@Ubuntu:~$ npm info underscor  （如果上面配置正确这个命令会有字符串response）
gord@gord:~$ npm --registry https://registry.npm.taobao.org info underscore
修改源地址为官方源

npm config set registry https://registry.npmjs.org/
安装指定版本的brunch
npm root -g
cd ...
npm install -g brunch@1.7.20  #务必要装这个版本
brunch -V
1.7.20
如果存在问题，执行卸载brunch

npm remove -g brunc
git环境
root@gord:~# git --version
git version 1.9.1
ant环境
root@gord:~# apt-get install ant
root@gord:~# ant -version
Picked up _JAVA_OPTIONS: -Xmx2048m -XX:MaxPermSize=512m -Djava.awt.headless=true
Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=512m; support was removed in 8.0
Apache Ant(TM) version 1.9.3 compiled on April 8 2014
python环境
apt install python-pip
准备工作总结：
profile配置总结
#set for java
export JAVA_HOME=/opt/jdk1.8.0_111
export PATH=$PATH:$JAVA_HOME/bin
export CLASSPATH=.:$CLASSPATH:$JAVA_HOME/lib
export _JAVA_OPTIONS="-Xmx2048m -XX:MaxPermSize=512m -Djava.awt.headless=true"
#set for maven
M2_HOME=/opt/apache-maven-3.3.9/
export MAVEN_OPTS="-Xms256m -Xmx512m"
export PATH=$M2_HOME/bin:$PATH
#set for nodejs
export NODE_HOME=/opt/node-v0.10.44-linux-x64
#export NODE_HOME=/opt/node-v4.4.4-linux-x64
#export NODE_HOME=/opt/node-v6.10.1-linux-x64
export PATH=$PATH:$NODE_HOME/bin
至此编译ambari所依赖的环境配置完毕，并可作为通用的编译环境推广使用。

推广到其它编译环境中，注意nodejs的版本可能需要更换！

准备工作至关重要，关系到编译的成败和时间的消耗！
编译安装ambari
主要步骤：
整理deb包
https://cwiki.apache.org/confluence/display/AMBARI/Installation+Guide+for+Ambari+2.4.2
wget http://www.apache.org/dist/ambari/ambari-2.4.2/apache-ambari-2.4.2-src.tar.gz (use the suggested mirror from above)
tar xfvz apache-ambari-2.4.2-src.tar.gz
cd apache-ambari-2.4.2-src
mvn versions:set -DnewVersion=2.4.2.0.0
root@gord:/opt/apache-ambari-2.4.2-src# cd ambari-metrics
/opt/apache-ambari-2.4.2-src/ambari-metrics /opt/apache-ambari-2.4.2-src
mvn versions:set -DnewVersion=2.4.2.0.0
root@gord:/opt/apache-ambari-2.4.2-src/ambari-metrics# cd ..
/opt/apache-ambari-2.4.2-src
-B非交互

mvn -B clean install package jdeb:jdeb -DnewVersion=2.4.2.0.0 -DskipTests -Dpython.ver="python >= 2.6"

-X -e debug

mvn -X -e clean install package jdeb:jdeb -DnewVersion=2.4.2.0.0 -DskipTests -Dpython.ver="python >= 2.6"
如果存在手工下载软件包到指定位置的情况，上述命令去掉clean参数，防止被clean
编译成功显示
[INFO] ------------------------------------------------------------------------
[INFO] Reactor Summary:
[INFO] 
[INFO] Ambari Main ........................................ SUCCESS [  2.741 s]
[INFO] Apache Ambari Project POM .......................... SUCCESS [  0.155 s]
[INFO] Ambari Web ......................................... SUCCESS [01:00 min]
[INFO] Ambari Views ....................................... SUCCESS [  1.475 s]
[INFO] Ambari Admin View .................................. SUCCESS [ 19.390 s]
[INFO] ambari-metrics ..................................... SUCCESS [  0.770 s]
[INFO] Ambari Metrics Common .............................. SUCCESS [  0.732 s]
[INFO] Ambari Metrics Hadoop Sink ......................... SUCCESS [  4.208 s]
[INFO] Ambari Metrics Flume Sink .......................... SUCCESS [  2.511 s]
[INFO] Ambari Metrics Kafka Sink .......................... SUCCESS [  1.676 s]
[INFO] Ambari Metrics Storm Sink .......................... SUCCESS [  4.837 s]
[INFO] Ambari Metrics Storm Sink (Legacy) ................. SUCCESS [  3.420 s]
[INFO] Ambari Metrics Collector ........................... SUCCESS [ 23.296 s]
[INFO] Ambari Metrics Monitor ............................. SUCCESS [  1.944 s]
[INFO] Ambari Metrics Grafana ............................. SUCCESS [  2.544 s]
[INFO] Ambari Metrics Assembly ............................ SUCCESS [02:26 min]
[INFO] Ambari Server ...................................... SUCCESS [04:24 min]
[INFO] Ambari Functional Tests ............................ SUCCESS [  1.385 s]
[INFO] Ambari Agent ....................................... SUCCESS [ 31.964 s]
[INFO] Ambari Client ...................................... SUCCESS [  0.084 s]
[INFO] Ambari Python Client ............................... SUCCESS [  1.168 s]
[INFO] Ambari Groovy Client ............................... SUCCESS [  8.157 s]
[INFO] Ambari Shell ....................................... SUCCESS [  0.105 s]
[INFO] Ambari Python Shell ................................ SUCCESS [  1.422 s]
[INFO] Ambari Groovy Shell ................................ SUCCESS [  3.714 s]
[INFO] ambari-logsearch ................................... SUCCESS [  0.041 s]
[INFO] Ambari Logsearch Appender .......................... SUCCESS [ 31.627 s]
[INFO] Ambari Logsearch Solr Client ....................... SUCCESS [01:38 min]
[INFO] Ambari Logsearch Portal ............................ SUCCESS [06:10 min]
[INFO] Ambari Logsearch Log Feeder ........................ SUCCESS [06:03 min]
[INFO] Ambari Logsearch Assembly .......................... SUCCESS [  0.323 s]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 24:14 min
[INFO] Finished at: 2017-04-05T18:44:18+08:00
[INFO] Final Memory: 274M/850M
[INFO] ------------------------------------------------------------------------
整理deb包
root@gord:/opt/apache-ambari-2.4.2-src# find ./ -name "*.deb" |awk '{print "cp",$0,"/media/sf_DataDisk/软件包/ambari-repo/Compiled/"}'|sh
./target/ambari_2.4.2.0.0_all.deb
./ambari-metrics/target/ambari-metrics_2.4.2.0.0_all.deb
./ambari-metrics/ambari-metrics-assembly/target/ambari-metrics-assembly_2.4.2.0.0_all.deb
./ambari-agent/target/ambari-agent_2.4.2.0-0.deb
./ambari-shell/target/ambari-shell_2.4.2.0.0_all.deb
./ambari-shell/ambari-python-shell/target/ambari-python-shell_2.4.2.0-0.deb
./ambari-project/target/ambari-project_2.4.2.0.0_all.deb
./ambari-server/target/ambari-server_2.4.2.0-0-dist.deb
./ambari-client/target/ambari-client_2.4.2.0.0_all.deb
./ambari-client/python-client/target/python-client_2.4.2.0-0.deb
./ambari-client/groovy-client/target/groovy-client_2.4.2.0.0_all.deb
服务端安装
##gdebi ambari-server*   #This should also pull in postgres packages as well.
dpkg -i ambari-server_2.4.2.0-0-dist.deb
apt-get -f install
设置与启动

ambari-server setup		#默认即可
ambari-server start

如果遇到报错：/usr/sbin/ambari-server: line 33: buildNumber: unbound variable

打开ambari-server脚本，把${buildNumber}这行换成 HASH="${VERSION}"
相关的命令有以下：
ambari-server status
ambari-server stop
客户端安装
dpkg -i ambari-agent*

apt-get -f install
编辑ambari-agent配置文件，设置之前安装过Ambari-server 的主机名：
vi /etc/ambari-agent/conf/ambari-agent.ini
[server]
hostname=<your.ambari.server.hostname>
url_port=8440
secured_url_port=8441
ambari-agent start

如果遇到报错：/usr/sbin/ambari-agent: line 23: buildNumber: unbound variable

打开ambari-agent脚本，把${buildNumber}这行换成 HASH="${VERSION}"
相关的命令有以下：
ambari-agent status # 检查agent服务运行状态
ambari-agent stop   # 停止agent服务
http://<ambari-server-host>:8080.
Log in with username admin and password admin
1、Caused by: org.vafer.jdeb.PackagingException: “/opt/apache-ambari-2.4.2-src/ambari-metrics/ambari-metrics-grafana/src/main/package/deb/control” is not a valid ‘control’ directory)

/opt/apache-ambari-2.4.2-src/ambari-metrics/ambari-metrics-grafana# vim pom.xml
      <plugin>
         <groupId>org.vafer</groupId>
         <artifactId>jdeb</artifactId>
         <version>1.0.1</version>
         <executions>
           <execution>
             <!--Stub execution on direct plugin call - workaround for ambari deb build process-->
             <id>stub-execution</id>
             <phase>none</phase>
             <goals>
               <goal>jdeb</goal>
             </goals>
           </execution>
         </executions>
         <configuration>
           <skip>true</skip>
           <attach>false</attach>
           <submodules>false</submodules>
           <controlDir>${project.basedir}/../src/main/package/deb/control</controlDir>
         </configuration>
       </plugin>
2、如果中间断掉，建议去掉clean，继续编译。
3、安装Nodejs 6.x.x版本不能编译，建议卸载，安装0.10.44。

[INFO] Ambari Web ......................................... FAILURE [ 13.325 s]

[ERROR] Failed to execute goal org.codehaus.mojo:exec-maven-plugin:1.2.1:exec (compile-brunch) on project ambari-web: Command execution failed. Process exited with an error: 1 (Exit value: 1) -> [Help 1]

版本检查：
root@gord:/opt/node-v6.10.1-linux-x64/lib/node_modules# brunch --version
2.10.9
root@gord:/opt/node-v6.10.1-linux-x64/lib/node_modules# npm --version
3.10.10
root@gord:/opt/node-v6.10.1-linux-x64/lib/node_modules# node --version
v6.10.1
改成安装node 0.10.44，brunch1.7.20后正常
4、编译过程中出现版本不一致问题，请参考http://blog.csdn.net/royma_1990/article/details/51749981。
5、下载慢问题处理

修改如下配置文件：

cat ambari-metrics/pom.xml
   <!--TODO change to HDP URL-->
    <hbase.tar>http://192.168.56.1:8000/ambari-HDP/hbase-1.1.2.2.3.4.0-3347.tar.gz</hbase.tar>
    <hbase.folder>hbase-1.1.2.2.3.4.0-3347</hbase.folder>
    <hadoop.tar>http://192.168.56.1:8000/ambari-HDP/hadoop-2.7.1.2.3.4.0-3347.tar.gz</hadoop.tar>
    <hadoop.folder>hadoop-2.7.1.2.3.4.0-3347</hadoop.folder>
    <hbase.winpkg.zip>https://msibuilds.blob.core.windows.net/hdp/2.x/2.2.4.2/2/hbase-0.98.4.2.2.4.2-0002-hadoop2.winpkg.zip</hbase.winpkg.zip>
    <hbase.winpkg.folder>hbase-0.98.4.2.2.4.2-0002-hadoop2</hbase.winpkg.folder>
    <hadoop.winpkg.zip>https://msibuilds.blob.core.windows.net/hdp/2.x/2.2.4.2/2/hadoop-2.6.0.2.2.4.2-0002.winpkg.zip</hadoop.winpkg.zip>
    <hadoop.winpkg.folder>hadoop-2.6.0.2.2.4.2-0002</hadoop.winpkg.folder>
    <grafana.folder>grafana-2.6.0</grafana.folder>
    <grafana.tar>http://192.168.56.1:8000/ambari-HDP/grafana-2.6.0.linux-x64.tar.gz</grafana.tar>
    <phoenix.tar>http://192.168.56.1:8000/ambari-HDP/phoenix-4.4.0.2.3.4.0-3347.tar.gz</phoenix.tar>
    <phoenix.folder>phoenix-4.4.0.2.3.4.0-3347</phoenix.folder>
6、警告：（可忽略）

[INFO] npm WARN deprecated graceful-fs@2.0.3: graceful-fs v3.0.0 and before will fail on node releases >= v7.0. Please update to graceful-fs@^4.0.0 as soon as possible. Use 'npm ls graceful-fs' to find it in the tree.
root@gord:/opt/apache-ambari-2.4.2-src/ambari-web# npm ls graceful-fs
Ambari@2.4.0 /opt/apache-ambari-2.4.2-src/ambari-web
├─┬ karma@0.11.14
│ └── graceful-fs@2.0.3 
├─┬ karma-coverage@0.2.7
│ └─┬ dateformat@1.0.12
│   └─┬ meow@3.7.0
│     └─┬ read-pkg-up@1.0.1
│       └─┬ read-pkg@1.1.0
│         ├─┬ load-json-file@1.1.0
│         │ └── graceful-fs@4.1.11 
│         └─┬ path-type@1.1.0
│           └── graceful-fs@4.1.11 
├─┬ mocha-phantomjs@3.1.6
│ └─┬ mocha@1.13.0
│   └─┬ glob@3.2.3
│     └── graceful-fs@2.0.3 
└─┬ phantomjs@1.9.20
  └─┬ fs-extra@0.26.7
    └── graceful-fs@4.1.11 
7、找不到org.apache.storm:storm-core:jar:1.1.0-SNAPSHOT

[INFO] Ambari Metrics Storm Sink .......................... FAILURE [  4.111 s]

[ERROR] Failed to execute goal on project ambari-metrics-storm-sink: Could not resolve dependencies for project org.apache.ambari:ambari-metrics-storm-sink:jar:2.4.2.0.0: Could not find artifact org.apache.storm:storm-core:jar:1.1.0-SNAPSHOT in apache-hadoop (http://repo.hortonworks.com/content/repositories/releases/org/apache/) -> [Help 1]

vim ambari-metrics/ambari-metrics-storm-sink/pom.xml
  <properties>
    <!--storm.version>1.1.0-SNAPSHOT</storm.version-->
    <storm.version>1.1.0</storm.version>
  </properties>
8、ambari-funtest

[ERROR] Failed to create debian package /home/kylin/apache-ambari-2.4.2-src/ambari-funtest/target/ambari-funtest_2.4.2.0.0_all.deb

org.vafer.jdeb.PackagingException: “/home/kylin/apache-ambari-2.4.2-src/ambari-funtest/src/main/package/deb/control” is not a valid ‘control’ directory)

解决办法：在上面添加一个元素

/opt/apache-ambari-2.4.2-src/ambari-funtest$ vi pom.xml
    <plugins>
      <plugin>
         <groupId>org.vafer</groupId>
         <artifactId>jdeb</artifactId>
         <version>1.0.1</version>
         <executions>
           <execution>
             <!--Stub execution on direct plugin call - workaround for ambari deb build process-->
             <id>stub-execution</id>
             <phase>none</phase>
             <goals>
               <goal>jdeb</goal>
             </goals>
           </execution>
         </executions>
         <configuration>
           <skip>true</skip>
           <attach>false</attach>
           <submodules>false</submodules>
           <controlDir>${project.basedir}/../src/main/package/deb/control</controlDir>
         </configuration>
       </plugin>
    </plugins>
    <resources>
      <resource>
10、[ERROR] Failed to execute goal org.codehaus.mojo:xml-maven-plugin:1.0:transform (default) on project ambari-server: Failed to transform input file /opt/apache-ambari-2.4.2-src/ambari-server/target/findbugs/findbugsXml.html: java.net.UnknownHostException: www.w3.org: Unknown host www.w3.org -> [Help 1]
解决办法：重新执行

mvn -B install package jdeb:jdeb -DnewVersion=2.4.2.0.0 -DskipTests -Dpython.ver="python >= 2.6" -Drat.skip=true
11、findbugsXml.html

[ERROR] Failed to execute goal org.codehaus.mojo:xml-maven-plugin:1.0:transform (default) on project ambari-server: Failed to transform input file /opt/apache-ambari-2.4.2-src/ambari-server/target/findbugs/findbugsXml.html: The entity name must immediately follow the ‘&’ in the entity reference. -> [Help 1]

解决办法1： 重新执行

mvn -B install package jdeb:jdeb -DnewVersion=2.4.2.0.0 -DskipTests -Dpython.ver="python >= 2.6" -Drat.skip=true

解决办法2: 如果网络情况不好，这一步就很难过去。那只好修改pom.xml文件，将findbugs内容注释掉：

/opt/apache-ambari-2.4.2-src/ambari-server$ vim pom.xml
      <plugin>
        <groupId>org.codehaus.mojo</groupId>
        <artifactId>findbugs-maven-plugin</artifactId>
        <version>3.0.3</version>
        <configuration>
          <failOnError>false</failOnError>
          <threshold>Low</threshold>
          <findbugsXmlOutputDirectory>${project.basedir}/target/findbugs</findbugsXmlOutputDirectory>
        </configuration>
        <executions>
          <execution>
            <phase>verify</phase>
            <goals>
              <goal>check</goal>
            </goals>
          </execution>
        </executions>
      </plugin>
      <plugin>
        <groupId>org.codehaus.mojo</groupId>
        <artifactId>xml-maven-plugin</artifactId>
        <version>1.0</version>
        <executions>
           <execution>
            <phase>verify</phase>
            <goals>
              <goal>transform</goal>
            </goals>
          </execution>
        </executions>
12、[ERROR] Failed to execute goal org.vafer:jdeb:1.0.1:jdeb (default-cli) on project ambari-logsearch: Failed to create debian package /opt/apache-ambari-2.4.2-src/ambari-logsearch/target/ambari-logsearch_2.4.2.0.0_all.deb: "/opt/apache-ambari-2.4.2-src/ambari-logsearch/src/main/package/deb/control" is not a valid 'control' directory) -> [Help 1]
解决办法：在上面添加一个元素
/opt/apache-ambari-2.4.2-src/ambari-logsearch$ vim pom.xml
       <plugin>
         <groupId>org.vafer</groupId>
         <artifactId>jdeb</artifactId>
         <version>1.0.1</version>
         <executions>
           <execution>
             <!--Stub execution on direct plugin call - workaround for ambari deb build process-->
             <id>stub-execution</id>
             <phase>none</phase>
             <goals>
               <goal>jdeb</goal>
             </goals>
           </execution>
         </executions>
         <configuration>
           <skip>true</skip>
           <attach>false</attach>
           <submodules>false</submodules>
           <controlDir>${project.basedir}/../src/main/package/deb/control</controlDir>
         </configuration>
       </plugin>
    </plugins>
  </build>
参考资源：
https://github.com/apache/ambari

https://cwiki.apache.org/confluence/display/AMBARI/Installation+Guide+for+Ambari+2.4.2

http://blog.csdn.net/chengyuqiang/article/details/53788351

http://blog.csdn.net/royma_1990/article/details/51749981
@firecontainer

April 6, 2017 4:53 PM
    本文系作者原创，转载请注明出处。如您阅读的是转载，请最好再看下原文，原文随时会更新和勘误的。
@Gordon_chang 

1997年毕业于北京联合大学，先后在中国万网，新媒传信，亚信等公司工作，现在在一家创业型公司担任云计算与大数据运维方面的 PM & Engineer。 专注于以下四个领域：
分布式存储
分布式数据库
重点通过技术架构与性能优化（底层）实现基于私有云的大数据平台能力