集群安装 吴脂娟 2021-12-03 [TOC]

1. CDH集群安装

1.1. 环境准备

  1. 准备6台虚拟机，8核200G，2T硬盘，修改主机名 cdh01的IP地址为：10.10.0.49 cdh02的IP地址为：10.10.0.48 cdh03的IP地址为：10.10.0.47 cdh04的IP地址为：10.10.0.51 cdh05的IP地址为：10.10.0.52 cdh06的IP地址为：10.10.0.53
  2. 关闭所有节点的selinux,修改sysctl.conf
    sed -i '/SELINUX/s/enforcing/disabled/' /etc/selinux/config
echo "vm.swappiness = 10" >> /etc/sysctl.conf
sysctl -p
echo never > /sys/kernel/mm/transparent_hugepage/enabled
echo never > /sys/kernel/mm/transparent_hugepage/defrag
echo "echo never > /sys/kernel/mm/transparent_hugepage/enabled" >> /etc/rc.local
echo "echo never > /sys/kernel/mm/transparent_hugepage/defrag" >> /etc/rc.local
# 重启后才生效
  3. 设置所有节点的hosts文件
    echo "10.10.0.49 cdh01">> /etc/hosts
echo "10.10.0.48 cdh02">> /etc/hosts
echo "10.10.0.47 cdh03">> /etc/hosts
echo "10.10.0.51 cdh04">> /etc/hosts
echo "10.10.0.52 cdh05">> /etc/hosts
echo "10.10.0.53 cdh06">> /etc/hosts
  4. 关闭所有节点的防火墙和清空防火墙规则链
    systemctl status firewalld
systemctl stop firewalld
systemctl disable firewalld
systemctl status firewalld
iptables -F
  5. 重启服务器 reboot
  6. 设置所有节点的时区一致及时钟同步
  7. 所有节点：
    timedatectl set-timezone Asia/Shanghai
yum install -y ntp
  8. 主节点cdh01: ```bash vi /etc/ntp.conf

    2. time

    server 127.127.1.0 iburst local clock restrict 10.10.0.1 mask 255.255.255.0 nomodify notrap server 0.asia.pool.ntp.org server 1.asia.pool.ntp.org server 2.asia.pool.ntp.org server 3.asia.pool.ntp.org server 127.127.1.0 Fudge 127.127.1.0 stratum 10

systemctl start ntpd systemctl enable ntpd systemctl status ntpd ntpq -p

* 其他节点：
```bash
systemctl start ntpd
systemctl enable ntpd
/usr/sbin/ntpdate cdh01
crontab -e
00 00 * * * /usr/sbin/ntpdate cdh01
  1. 主机安装插件httpd并设置开机启动
    yum install -y httpd
systemctl start httpd
systemctl enable httpd
  2. 主机设置免密登入其他节点
    cd /root
ssh-keygen -t rsa
ssh-copy-id cdh01
ssh-copy-id cdh02
ssh-copy-id cdh03
ssh-copy-id cdh04
ssh-copy-id cdh05
ssh-copy-id cdh06
  3. 所有节点安装相关插件
    yum -y install chkconfig python bind-utils psmisc libxslt zlib sqlite cyrus-sasl-plain cyrus-sasl-gssapi fuse portmap fuse-libs redhat-lsb screen telnet lrzsz krb5-workstation krb5-libs python-pip psycopg2
  4. 扩容
    sudo fuser -km /home # 终止所有使用/home文件系统的进程,不要在/home目录下执行
sudo umount /home # 卸载
sudo lvremove /dev/mapper/centos-home # 删除/home所在的逻辑卷lv,选择y
sudo lvextend -L +1.6T /dev/mapper/centos-root # 扩大根目录所在的逻辑卷，这里增大1.6T
sudo xfs_growfs /dev/mapper/centos-root # 扩大/文件系统
sudo lvcreate -L 300G -n/dev/mapper/centos-home # 重建/home文件系统所需要的逻辑卷
sudo mkfs.xfs  /dev/mapper/centos-home # 创建文件系统
sudo mount /dev/mapper/centos-home # 将新建的文件系统挂载到/home目录下

    2.1. CDH部署

  5. 下载安装包，并上传到各节点的root目录下
    • 主节点： jdk-8u181-linux-x64.tar.gz mysql-5.7.11-linux-glibc2.5-x86_64.tar.gz mysql-connector-java-5.1.47.jar cdh: CDH-6.3.2-1.cdh6.3.2.p0.1605554-el7.parcel CDH-6.3.2-1.cdh6.3.2.p0.1605554-el7.parcel.sha manifest.json clouderaManager: cloudera-manager-daemons-6.3.1-1466458.el7.x86_64.rpm cloudera-manager-server-6.3.1-1466458.el7.x86_64.rpm cloudera-manager-agent-6.3.1-1466458.el7.x86_64.rpm phoenix: PHOENIX-1.0.jar PHOENIX-5.0.0-cdh6.2.0.p0.1308267-el7.parcel PHOENIX-5.0.0-cdh6.2.0.p0.1308267-el7.parcel.sha manifest.json
    • 其他节点： jdk-8u181-linux-x64.tar.gz cloudera-manager-daemons-6.3.1-1466458.el7.x86_64.rpm cloudera-manager-agent-6.3.1-1466458.el7.x86_64.rpm
  6. 每个节点部署JDK【Java的路径必须放在/usr/java下】 ```bash cd /root mkdir /usr/java tar -xzvf jdk-8u181-linux-x64.tar.gz mv jdk1.8.0_181 /usr/java/

vi /etc/profile export JAVA_HOME=/usr/java/jdk1.8.0_181 export PATH=/usr/java/jdk1.8.0_181/bin:$PATH

source /etc/profile which java

3. 主节点cdh01安装mysql并部署MySQL jdbc jar：[mysql安装步骤详情](http://121.199.10.158:8107/c5p8d4he6gjkxm0oq1zrbslt2v37wify9aun/wuzhijuan/django_serve_recovery_data.html)
```bash
cd /root
mkdir -p /usr/share/java/
mv mysql-connector-java-5.1.47.jar /usr/share/java/mysql-connector-java.jar
# 创建相关新库
mysql -uroot -p
create database cmf DEFAULT CHARACTER SET utf8;
create database amon DEFAULT CHARACTER SET utf8;
grant all on cmf.* TO 'cmf'@'%' IDENTIFIED BY '123456';
grant all on amon.* TO 'amon'@'%' IDENTIFIED BY '123456';
flush privileges;
create database hive default character set utf8; 
grant all privileges on hive.* to 'hive'@'%' identified by '123456' with grant option;
grant all privileges on hive.* to 'hive'@'cdh001' identified by '123456' with grant option;
flush privileges; 
create database oozie default character set utf8; 
grant all privileges on oozie.* to 'oozie'@'%' identified by '123456' with grant option;
grant all privileges on oozie.* to 'oozie'@'cdh001' identified by '123456' with grant option;
flush privileges; 
create database hue default character set utf8; 
grant all privileges on hue.* to 'hue'@'%' identified by '123456' with grant option;
grant all privileges on hue.* to 'hue'@'cdh001' identified by '123456' with grant option;
flush privileges; 
create database am default character set utf8; 
grant all privileges on am.* to 'am'@'%' identified by '123456' with grant option;
grant all privileges on am.* to 'am'@'cdh001' identified by '123456' with grant option;
flush privileges;
create database rm default character set utf8; 
grant all privileges on rm.* to 'rm'@'%' identified by '123456' with grant option;
grant all privileges on rm.* to 'rm'@'cdh001' identified by '123456' with grant option;
flush privileges;
  1. 主节点cdh01部署离线parcel源
    mkdir -p /var/www/html/cdh/phoenix
mv /root/cdh/CDH-6.3.2-1.cdh6.3.2.p0.1605554-el7.parcel /var/www/html/cdh/
mv /root/cdh/CDH-6.3.2-1.cdh6.3.2.p0.1605554-el7.parcel.sha /var/www/html/cdh/
mv /root/cdh/manifest.json /var/www/html/cdh/
mv /root/phoenix/PHOENIX-1.0.jar /var/www/html/cdh/phoenix
mv /root/phoenix/PHOENIX-5.0.0-cdh6.2.0.p0.1308267-el7.parcel /var/www/html/cdh/phoenix
mv /root/phoenix/PHOENIX-5.0.0-cdh6.2.0.p0.1308267-el7.parcel.sha /var/www/html/cdh/phoenix
mv /root/phoenix/manifest.json /var/www/html/cdh/phoenix
  2. 主节点部署cm server与agent不下载依赖包 ```bash cd /root/clouderaManager rpm -ivh cloudera-manager-daemons-6.3.1-1466458.el7.x86_64.rpm --nodeps --force rpm -ivh cloudera-manager-server-6.3.1-1466458.el7.x86_64.rpm --nodeps --force rpm -ivh cloudera-manager-agent-6.3.1-1466458.el7.x86_64.rpm --nodeps --force sed -i "s/server_host=localhost/server_host=cdh01/g" /etc/cloudera-scm-agent/config.ini

vi /etc/cloudera-scm-server/db.properties com.cloudera.cmf.db.type=mysql com.cloudera.cmf.db.host=cdh01 com.cloudera.cmf.db.name=cmf com.cloudera.cmf.db.user=cmf com.cloudera.cmf.db.password=123456 com.cloudera.cmf.db.setupType=EXTERNAL

cp /var/www/html/cdh/phoenix/PHOENIX-1.0.jar /opt/cloudera/csd

6. 其他节点部署cm agent不下载依赖包
```bash
rpm -ivh cloudera-manager-daemons-6.3.1-1466458.el7.x86_64.rpm --nodeps --force
rpm -ivh cloudera-manager-agent-6.3.1-1466458.el7.x86_64.rpm --nodeps --force
sed -i "s/server_host=localhost/server_host=cdh01/g" /etc/cloudera-scm-agent/config.ini
  1. 开启cm server 与所有节点agent，并设置开机自启动 ```bash systemctl start cloudera-scm-server systemctl enable cloudera-scm-server systemctl status cloudera-scm-server

systemctl start cloudera-scm-agent systemctl enable cloudera-scm-agent systemctl status cloudera-scm-agent

8. [CDHweb界面操作](https://www.cnblogs.com/djlsunshine/p/13356168.html)
访问地址：http://10.10.0.49:7180
账号密码：admin/admin
9. [PHOENIXweb界面操作](https://blog.csdn.net/gdkyxy2013/article/details/107282614?spm=1001.2101.3001.6650.2&utm_medium=distribute.pc_relevant.none-task-blog-2%7Edefault%7EBlogCommendFromBaidu%7Edefault-2.highlightwordscore&depth_1-utm_source=distribute.pc_relevant.none-task-blog-2%7Edefault%7EBlogCommendFromBaidu%7Edefault-2.highlightwordscore)


# TigerGraph集群安装
1. 环境准备
参考CDH，准备3台虚拟机graph01,graph02,graph03
2. 安装jdk 参考cdh
3. 上传tigergraph-3.1.6-offline.tar到主节点graph01
4. 解压
```bash
cd /root
tar -xzf tigergraph-3.1.6-offline.tar
  1. 非交互式模式安装 ```bash cd /root/tigergraph-3.1.6-offline vi install_conf.json "NodeList": [ "m1: 10.10.0.61", "m2: 10.10.0.62", "m3: 10.10.0.63" ] "LoginConfig": {
     "SudoUser": "root",
 "[comment]": "choose login method: 'P' for SSH using password or 'K' for SSH using key file (e.g. ec2_key.pem)",
 "Method": "P",
 "P": "huaun123",
 "K": "/root/.ssh/id_rsa"
    }

./install.sh -n

6. 查看启动程序
```bash
su tigergraph # 切换用户
gadmin status # 查看服务状态
gadmin start infra #启动服务
gadmin stop # 停止服务
gadmin restart infra # 重启服务
  1. 登入web 访问地址：http://10.10.0.61:14240/ 账号密码：tigergraph/tigergraph

3. airflow定时任务组件安装

  1. 下载相关插件
    yum -y install gcc mysql-devel libmysqlclient-dev
pip3 install SQLAlchemy
pip3 install Flask-SQLAlchemy
pip3 install mysqlclient
pip3 install apache-airflow==1.10.10
  2. 修改配置文件
    cd /root
airflow help
cd /root/airflow
mkdir dags
vi airflow.cfg
default_timezone = system
executor = LocalExecutor
sql_alchemy_conn = mysql://root:admin@202106@10.10.0.16:3306/airflow
default_ui_timezone = system
  3. 初始化数据库
    airflow db init
  4. 创建登入用户admin
    airflow users create --username admin --firstname admin --lastname admin --role Admin --email  wuzj@huaun.com
  5. 启动程序
    cd /root/airflow
nohup airflow webserver >>webserver.log 2>&1 &
nohup airflow scheduler >>scheduler.log 2>&1 &
  6. 登入web： 访问地址：http://10.10.0.52:8080/home 账号密码：admin/admin

results matching ""

    No results matching ""