侧边栏壁纸
博主头像
SRE实践 博主等级

行动起来,活在当下

  • 累计撰写 22 篇文章
  • 累计创建 2 个标签
  • 累计收到 0 条评论

目 录CONTENT

文章目录

Ambari2.7.3 和HDP3.1.0搭建Hadoop集群

佚名
2025-02-28 / 0 评论 / 0 点赞 / 4 阅读 / 0 字
温馨提示:
部分素材来自网络,若不小心影响到您的利益,请联系我们删除。

一、环境及软件准备

1、集群规划

  hdp01/10.1.1.11 hdp02/10.1.1.12 hdp03/10.1.1.13 hdp04/10.1.1.14 hdp05/10.1.1.15
ambari server
ambari agent
namenode
datanode
resourcemanager
journalnode
zookeeper

01和02主要做管理机,03-05做数据数据节点,01-03做zookeeper集群。
主机环境为Ubuntu server18.04,主机间已经建立互信。

2、下载ambari安装包

https://docs.hortonworks.com/HDPDocuments/Ambari-2.7.3.0/bk_ambari-installation/content/ambari_repositories.html
找到对应的版本下载即可。

 
image.png

ps需要下载repo file,ambari.list

 

3、下载hdp安装包

https://docs.hortonworks.com/HDPDocuments/Ambari-2.7.3.0/bk_ambari-installation/content/hdp_31_repositories.html

 
image.png

ps需要下载repo file,hdp.gpl.list,hdp.list.

 

4、安装一台lnmp备用

选择hadoop01

https://lnmp.org/install.html

wget -c http://soft.vpser.net/lnmp/lnmp1.5.tar.gz && tar zxf lnmp1.5.tar.gz && cd lnmp1.5 && ./install.sh lnmp

选择安装mysql5.7版本用于支持ambari
开启远程访问mysql。

 mysql -u root -p
GRANT ALL PRIVILEGES ON *.* TO 'root'@'%' IDENTIFIED BY 'root'  ;
flush privileges;

5、复制下载文件

放到lnmp的默认目录下。

mkdir /home/wwwroot/default/ambari
mkdir /home/wwwroot/default/ambari/HDP-UTILS
cd /home/wwwroot/default/ambari

复制下载文件到相关目录并解压。

tar -zxvf ambari-2.7.3.0-ubuntu18.tar.gz tar -zxvf HDP-GPL-3.1.0.0-ubuntu18-gpl.tar.gz tar -zxvf HDP-3.1.0.0-ubuntu18-deb.tar.gz tar -zxvf HDP-UTILS-1.1.0.22-ubuntu18.tar.gz -C HDP-UTILS

开启nginx目录浏览
nginx.conf http { }加入

        autoindex on;   #开启nginx目录浏览功能
    autoindex_exact_size off;   #文件大小从KB开始显示

    autoindex_localtime on;</pre>
<div class="image-package">
    <div class="image-container">
        <div class="image-container-fill">
            &nbsp;
        </div>
        <div class="image-view" data-width="768" data-height="258">
            <img src="https://img.sresz.cn/upload/2020/04/20200412010951158662499160646.png" alt="" data-original-src="//upload-images.jianshu.io/upload_images/1409498-4cf6838c9a980986.png" data-original-width="768" data-original-height="258" data-original-format="image/png" data-original-filesize="10865" style="border: 0px; max-width: 800px; height: auto;"/>
        </div>
    </div>
    <div class="image-caption">
        image.png
    </div>
</div>
<h4 style="font-size: 14px; margin: 10px 0px;">
    6、配置本地源
</h4>
<p style="margin: 10px auto; line-height: 1.5;">
    cd /etc/apt/sources.list.d<br/>ambari.list
</p>
<pre class="hljs ruby" style="margin-top: 0px; margin-bottom: 0px; white-space: pre-wrap; overflow-wrap: break-word;">#VERSION_NUMBER=2.7.3.0-139#json.url = http://public-repo-1.hortonworks.com/HDP/hdp_urlinfo.jsondeb http://192.168.0.151/ambari/ambari/ubuntu18/2.7.3.0-139/ Ambari main</pre>
<p style="margin: 10px auto; line-height: 1.5;">
    hdp.list
</p>
<pre class="hljs ruby" style="margin-top: 0px; margin-bottom: 0px; white-space: pre-wrap; overflow-wrap: break-word;">#VERSION_NUMBER=3.1.0.0-78deb http://192.168.0.151/ambari/HDP/ubuntu18/3.1.0.0-78/ HDP main

deb http://192.168.0.151/ambari/HDP-UTILS/HDP-UTILS/ubuntu18/1.1.0.22/ HDP-UTILS main



hdp.gpl.list


#VERSION_NUMBER=3.1.0.0-78deb http://192.168.0.151/ambari/HDP-GPL/ubuntu18/3.1.0.0-78/ HDP-GPL main


更新list


sudo apt-key adv --recv-keys --keyserver keyserver.ubuntu.com B9733A7A07513CADsudo apt-get updateapt-cache showpkg ambari-serverapt-cache showpkg ambari-agentapt-cache showpkg ambari-metrics-assembly


复制到各台主机,并在各台主机更新list。


scp /etc/apt/sources.list.d/ambari.list 192.168.0.152:/etc/apt/sources.list.d/
scp /etc/apt/sources.list.d/ambari.list 192.168.0.153:/etc/apt/sources.list.d/
scp /etc/apt/sources.list.d/ambari.list 192.168.0.154:/etc/apt/sources.list.d/
scp /etc/apt/sources.list.d/ambari.list 192.168.0.155:/etc/apt/sources.list.d/

scp /etc/apt/sources.list.d/hdp.list 192.168.0.152:/etc/apt/sources.list.d/
scp /etc/apt/sources.list.d/hdp.list 192.168.0.153:/etc/apt/sources.list.d/
scp /etc/apt/sources.list.d/hdp.list 192.168.0.154:/etc/apt/sources.list.d/
scp /etc/apt/sources.list.d/hdp.list 192.168.0.155:/etc/apt/sources.list.d/

scp /etc/apt/sources.list.d/hdp.gpl.list 192.168.0.152:/etc/apt/sources.list.d/
scp /etc/apt/sources.list.d/hdp.gpl.list 192.168.0.153:/etc/apt/sources.list.d/
scp /etc/apt/sources.list.d/hdp.gpl.list 192.168.0.154:/etc/apt/sources.list.d/
scp /etc/apt/sources.list.d/hdp.gpl.list 192.168.0.155:/etc/apt/sources.list.d/



二、安装hdp集群



1、安装ambari &hadoop01


sudo apt-get install ambari-server

Success. You can now start the database server using: /usr/lib/postgresql/10/bin/pg_ctl -D /var/lib/postgresql/10/main -l logfile start

Ver Cluster Port Status Owner Data directory Log file10 main 5432 down postgres /var/lib/postgresql/10/main /var/log/postgresql/postgresql-10-main.log
update-alternatives: using /usr/share/postgresql/10/man/man1/postmaster.1.gz to provide /usr/share/man/man1/postmaster.1.gz (postmaster.1.gz) in auto mode
Setting up postgresql (10+190) ...
Setting up ambari-server (2.7.3.0-139) ...
Processing triggers for systemd (237-3ubuntu10.9) ...
Processing triggers for ureadahead (0.100.0-20) ...
Processing triggers for libc-bin (2.27-3ubuntu1) ...



2、配置ambari


root@hadoop01:/home/wwwroot/default/ambari# sudo ambari-server setupUsing python  /usr/bin/python
Setup ambari-server
Checking SELinux...WARNING: Could not run /usr/sbin/sestatus: OK
Customize user account for ambari-server daemon [y/n] (n)? y
Enter user account for ambari-server daemon (root):
Adjusting ambari-server permissions and ownership...
Checking firewall status...
Checking JDK...
[1] Oracle JDK 1.8 + Java Cryptography Extension (JCE) Policy Files 8[2] Custom JDK

Enter choice (1):



配置java路径即可
/usr/lib/jvm/java-8-openjdk-amd64



配置mysql


Minimum JDK version is 8 for Ambari. Skipping to setup different JDK for Ambari Server.
Checking GPL software agreement...
GPL License for LZO: https://www.gnu.org/licenses/old-licenses/gpl-2.0.en.html
Enable Ambari Server to download and install GPL Licensed LZO packages [y/n] (n)?
Completing setup...
Configuring database...
Enter advanced database configuration [y/n] (n)? y
Configuring database...

Choose one of the following options:[1] - PostgreSQL (Embedded)
[2] - Oracle
[3] - MySQL / MariaDB
[4] - PostgreSQL
[5] - Microsoft SQL Server (Tech Preview)
[6] - SQL Anywhere
[7] - BDB

Enter choice (1): 3



配置jdbc,需要上传jdbc.bin.jar
路径如下。
/var/lib/ambari-server/resources/mysql-connector-java-5.1.47-bin.jar


Configuring ambari database...
Enter full path to custom jdbc driver:
Configuring ambari database...
Enter full path to custom jdbc driver: /var/lib/ambari-server/resources/mysql-connector-java-5.1.47-bin.jar
Copying /var/lib/ambari-server/resources/mysql-connector-java-5.1.47-bin.jar to /usr/share/java
Configuring remote database connection properties...
WARNING: Before starting Ambari Server, you must run the following DDL directly from the database shell to create the schema: /var/lib/ambari-server/resources/Ambari-DDL-MySQL-CREATE.sql
Proceed with configuring remote database connection properties [y/n] (y)?
Extracting system views...
....ambari-admin-2.7.3.0.139.jar

Ambari repo file contains latest json url http://public-repo-1.hortonworks.com/HDP/hdp_urlinfo.json, updating stacks repoinfos with it...Adjusting ambari-server permissions and ownership...
Ambari Server 'setup' completed successfully.



三、配置ambari



1、导入mysql库


mysql -u root -p
mysql> create database ambari;
mysql> use ambari;
mysql> source /var/lib/ambari-server/resources/Ambari-DDL-MySQL-CREATE.sql;


2、启动ambari


root@hadoop01:/home/wwwroot/default/ambari#sudo ambari-server start
Using python /usr/bin/python
Starting ambari-server
Ambari Server running with administrator privileges.
Organizing resource files at /var/lib/ambari-server/resources...
Ambari database consistency check started...
Server PID at: /var/run/ambari-server/ambari-server.pid
Server out at: /var/log/ambari-server/ambari-server.out
Server log at: /var/log/ambari-server/ambari-server.log
Waiting for server start..................................................
Server started listening on 8080DB configs consistency check: no errors and warnings were found.
Ambari Server 'start' completed successfully.


四、安装hadoop集群



http://192.168.0.151:8080
用户名密码、admin/admin












image.png













image.png













image.png


选择本地库。
输入目录。
http://192.168.0.151/ambari/HDP/ubuntu18/3.1.0.0-78
http://192.168.0.151/ambari/HDP-UTILS/HDP-UTILS/ubuntu18/1.1.0.22/














image.png




cat ~/.ssh/id_rsa
获取ssh密钥。













image.png













image.png




其他主机出现错误,跟agent没安装上有关。
在其他主机执行
apt install ambari-agent -y













image.png


Ranger、Ranger KMS选了安装失败。取消可以成功。











image.png












image.png














0

评论区