FastDFS 分布式集群搭建详解

分类:热门推荐日期:2024-03-11浏览:1评论:0

文章目录

        • 前言
        • 1、整体架构
        • 2、安装配置FastDFS集群
          • 2.1 配置tracker
          • 2.2 配置storage
          • 3、启动集群
          • 4、查看集群情况
          • 5、nginx配置
            • 5.1 配置storage的四台机器的nginx
            • 5.2 配置tracker的两台机器的nginx
            • 5.3 配置统一入口
              前言

              阅读本文章之前请先看上一篇单机版FastDFS安装配置详解，所以本文有些内容我就一笔带过，传送门：《FastDFS 环境搭建及使用详解》

              1、整体架构

              大佬超这次使用了7个服务器节点，其中2个跟踪器节点，4个存储节点，需要注意的是 nginx 有三层，第一层使用了一个 nginx 做统一入口进行负载均衡，第二层 tracker 有两个 nginx 做负载均衡，第三层 storage 有四个 nginx 用于访问FastDFS上的文件，服务器节点信息如下

              节点角色所属组nginx
              192.168.122.1nginx
              192.168.122.22trackernginx1
              192.168.122.23trackernginx2
              192.168.122.24storage1Group1nginx1
              192.168.122.25storage2Group1nginx2
              192.168.122.26storage3Group2nginx3
              192.168.122.27storage4Group2nginx4

              2、安装配置FastDFS集群

              每个节点先按照前面单机版的步骤安装FastDFS。然后进行集群配置，如下

              2.1 配置tracker

              192.168.122.22 和 192.168.122.23 节点

              vi /etc/fdfs/tracker.conf

              #启用配置文件，默认启用配置文件（默认值）
disabled=false 
#设置tracker的端口号，通常采用22122这个默认端口 （默认值）
port=22122 
#设置tracker的数据文件和日志目录
base_path=/opt/fastdfs/tracker
#设置http端口号，（默认值）
http.server_port=8080

              需要注意的是，我们文件上传默认会负载均衡，也就是tracker里如下默认配置

              # the method for selecting group to upload files
# 0: round robin
# 1: specify group
# 2: load balance, select the max free space group to upload file
store_lookup = 2
# which group to upload file
# when store_lookup set to 1, must set store_group to the group name
store_group = group2

              store_lookup，该值默认是2（负载均衡策略，磁盘空间优先），0（轮询策略），1（指定group进行上传下载操作，也就是store_group的值），只有指定为1时store_group才会起作用。

              2.2 配置storage

              配置另外四台 storage，只有 group_name 不一样

              192.168.122.24 和 192.168.122.25 节点

              vi /etc/fdfs/storage.conf

              #启用配置文件（默认值）
disabled=false
#组名，根据实际情况修改 (默认值)
group_name=group1
#设置storage的端口号，默认是23000，同一个组的storage端口号必须一致 (默认值)
port=23000
#设置storage数据文件和日志目录
base_path=/opt/fastdfs/storage
#存储路径个数，需要和store_path个数匹配 （默认值）
store_path_count=1  
#实际文件存储路径
store_path0=/opt/fastdfs/storage/files
#tracker 服务器的 IP地址和端口号
tracker_server=192.168.122.22:22122
tracker_server=192.168.122.23:22122
#设置 http 端口号  (默认值)
http.server_port=8888

              192.168.122.26 和 192.168.122.27 节点

              vi /etc/fdfs/storage.conf

              #启用配置文件（默认值）
disabled=false
#组名，根据实际情况修改 (默认值)
group_name=group2
#设置storage的端口号，默认是23000，同一个组的storage端口号必须一致 (默认值)
port=23000
#设置storage数据文件和日志目录
base_path=/opt/fastdfs/storage
#存储路径个数，需要和store_path个数匹配 （默认值）
store_path_count=1  
#实际文件存储路径
store_path0=/opt/fastdfs/storage/files
#tracker 服务器的 IP地址和端口号
tracker_server=192.168.122.22:22122
tracker_server=192.168.122.23:22122
#设置 http 端口号  (默认值)
http.server_port=8888

              3、启动集群

              192.168.122.22、192.168.122.23：

              #启动tracker
fdfs_trackerd /etc/fdfs/tracker.conf
#重新启动tracker
fdfs_trackerd /etc/fdfs/tracker.conf restart

              192.168.122.24、192.168.122.25、192.168.122.26、192.168.122.27：

              #启动storage
fdfs_storaged /etc/fdfs/storage.conf
#重新启动storage
fdfs_storaged /etc/fdfs/storage.conf restart

              4、查看集群情况
              fdfs_monitor /etc/fdfs/storage.conf

              结果如下

              [root@mongodb-1 logs]# fdfs_monitor /etc/fdfs/storage.conf
server_count=2, server_index=0
tracker server is 192.168.122.22:22122
group count: 2
Group 1:
group name = group1
disk total space = 51,175 MB
disk free space = 47,554 MB
trunk free space = 0 MB
storage server count = 2
active server count = 2
storage server port = 23000
storage HTTP port = 8888
store path count = 1
subdir count per path = 256
current write server index = 0
current trunk file id = 0
        Storage 1:
                id = 192.168.122.24
                ip_addr = 192.168.122.24  ACTIVE
                http domain = 
                version = 6.11.
                join time = 2024-01-29 18:12:37
                up time = 2024-01-29 18:12:37
                total storage = 51,175 MB
                free storage = 49,349 MB
                upload priority = 10
                store_path_count = 1
                subdir_count_per_path = 256
                storage_port = 23000
                storage_http_port = 8888
                current_write_path = 0
                source storage id = 192.168.122.25
                if_trunk_server = 0
                connection.alloc_count = 64
                connection.current_count = 1
                connection.max_count = 2
                total_upload_count = 0
                success_upload_count = 0
                total_append_count = 0
                success_append_count = 0
                total_modify_count = 0
                success_modify_count = 0
                total_truncate_count = 0
                success_truncate_count = 0
                total_set_meta_count = 0
                success_set_meta_count = 0
                total_delete_count = 0
                success_delete_count = 0
                total_download_count = 0
                success_download_count = 0
                total_get_meta_count = 0
                success_get_meta_count = 0
                total_create_link_count = 0
                success_create_link_count = 0
                total_delete_link_count = 0
                success_delete_link_count = 0
                total_upload_bytes = 0
                success_upload_bytes = 0
                total_append_bytes = 0
                success_append_bytes = 0
                total_modify_bytes = 0
                success_modify_bytes = 0
                stotal_download_bytes = 0
                success_download_bytes = 0
                total_sync_in_bytes = 0
                success_sync_in_bytes = 0
                total_sync_out_bytes = 0
                success_sync_out_bytes = 0
                total_file_open_count = 0
                success_file_open_count = 0
                total_file_read_count = 0
                success_file_read_count = 0
                total_file_write_count = 0
                success_file_write_count = 0
                last_heart_beat_time = 2024-01-30 10:09:11
                last_source_update = 1970-01-01 08:00:00
                last_sync_update = 1970-01-01 08:00:00
                last_synced_timestamp = 1970-01-01 08:00:00 
        Storage 2:
                id = 192.168.122.25
                ip_addr = 192.168.122.25  ACTIVE
                http domain = 
                version = 6.11.
                join time = 2024-01-29 18:12:36
                up time = 2024-01-29 18:12:36
                total storage = 51,175 MB
                free storage = 47,554 MB
                upload priority = 10
                store_path_count = 1
                subdir_count_per_path = 256
                storage_port = 23000
                storage_http_port = 8888
                current_write_path = 0
                source storage id = 
                if_trunk_server = 0
                connection.alloc_count = 64
                connection.current_count = 1
                connection.max_count = 2
                total_upload_count = 0
                success_upload_count = 0
                total_append_count = 0
                success_append_count = 0
                total_modify_count = 0
                success_modify_count = 0
                total_truncate_count = 0
                success_truncate_count = 0
                total_set_meta_count = 0
                success_set_meta_count = 0
                total_delete_count = 0
                success_delete_count = 0
                total_download_count = 0
                success_download_count = 0
                total_get_meta_count = 0
                success_get_meta_count = 0
                total_create_link_count = 0
                success_create_link_count = 0
                total_delete_link_count = 0
                success_delete_link_count = 0
                total_upload_bytes = 0
                success_upload_bytes = 0
                total_append_bytes = 0
                success_append_bytes = 0
                total_modify_bytes = 0
                success_modify_bytes = 0
                stotal_download_bytes = 0
                success_download_bytes = 0
                total_sync_in_bytes = 0
                success_sync_in_bytes = 0
                total_sync_out_bytes = 0
                success_sync_out_bytes = 0
                total_file_open_count = 0
                success_file_open_count = 0
                total_file_read_count = 0
                success_file_read_count = 0
                total_file_write_count = 0
                success_file_write_count = 0
                last_heart_beat_time = 2024-01-30 10:09:09
                last_source_update = 1970-01-01 08:00:00
                last_sync_update = 1970-01-01 08:00:00
                last_synced_timestamp = 1970-01-01 08:00:00 
Group 2:
group name = group2
disk total space = 51,175 MB
disk free space = 48,108 MB
trunk free space = 0 MB
storage server count = 2
active server count = 2
storage server port = 23000
storage HTTP port = 8888
store path count = 1
subdir count per path = 256
current write server index = 1
current trunk file id = 0
        Storage 1:
                id = 192.168.122.26
                ip_addr = 192.168.122.26  ACTIVE
                http domain = 
                version = 6.11.
                join time = 2024-01-29 18:12:33
                up time = 2024-01-29 18:12:33
                total storage = 51,175 MB
                free storage = 48,118 MB
                upload priority = 10
                store_path_count = 1
                subdir_count_per_path = 256
                storage_port = 23000
                storage_http_port = 8888
                current_write_path = 0
                source storage id = 
                if_trunk_server = 0
                connection.alloc_count = 64
                connection.current_count = 1
                connection.max_count = 4
                total_upload_count = 13
                success_upload_count = 13
                total_append_count = 0
                success_append_count = 0
                total_modify_count = 0
                success_modify_count = 0
                total_truncate_count = 0
                success_truncate_count = 0
                total_set_meta_count = 13
                success_set_meta_count = 13
                total_delete_count = 0
                success_delete_count = 0
                total_download_count = 0
                success_download_count = 0
                total_get_meta_count = 0
                success_get_meta_count = 0
                total_create_link_count = 0
                success_create_link_count = 0
                total_delete_link_count = 0
                success_delete_link_count = 0
                total_upload_bytes = 151530173
                success_upload_bytes = 151530173
                total_append_bytes = 0
                success_append_bytes = 0
                total_modify_bytes = 0
                success_modify_bytes = 0
                stotal_download_bytes = 0
                success_download_bytes = 0
                total_sync_in_bytes = 272774924
                success_sync_in_bytes = 272774924
                total_sync_out_bytes = 0
                success_sync_out_bytes = 0
                total_file_open_count = 61
                success_file_open_count = 61
                total_file_read_count = 0
                success_file_read_count = 0
                total_file_write_count = 1657
                success_file_write_count = 1657
                last_heart_beat_time = 2024-01-30 10:09:06
                last_source_update = 2024-01-30 10:01:12
                last_sync_update = 2024-01-30 10:02:35
                last_synced_timestamp = 2024-01-30 10:02:35 (0s delay)
        Storage 2:
                id = 192.168.122.27
                ip_addr = 192.168.122.27  ACTIVE
                http domain = 
                version = 6.11.
                join time = 2024-01-24 10:29:49
                up time = 2024-01-29 18:12:28
                total storage = 51,175 MB
                free storage = 48,108 MB
                upload priority = 10
                store_path_count = 1
                subdir_count_per_path = 256
                storage_port = 23000
                storage_http_port = 8888
                current_write_path = 0
                source storage id = 
                if_trunk_server = 0
                connection.alloc_count = 64
                connection.current_count = 1
                connection.max_count = 3
                total_upload_count = 24
                success_upload_count = 24
                total_append_count = 0
                success_append_count = 0
                total_modify_count = 0
                success_modify_count = 0
                total_truncate_count = 0
                success_truncate_count = 0
                total_set_meta_count = 24
                success_set_meta_count = 24
                total_delete_count = 0
                success_delete_count = 0
                total_download_count = 1
                success_download_count = 1
                total_get_meta_count = 0
                success_get_meta_count = 0
                total_create_link_count = 0
                success_create_link_count = 0
                total_delete_link_count = 0
                success_delete_link_count = 0
                total_upload_bytes = 272773872
                success_upload_bytes = 272773872
                total_append_bytes = 0
                success_append_bytes = 0
                total_modify_bytes = 0
                success_modify_bytes = 0
                stotal_download_bytes = 29
                success_download_bytes = 29
                total_sync_in_bytes = 151530717
                success_sync_in_bytes = 151530717
                total_sync_out_bytes = 0
                success_sync_out_bytes = 0
                total_file_open_count = 51
                success_file_open_count = 51
                total_file_read_count = 1
                success_file_read_count = 1
                total_file_write_count = 1646
                success_file_write_count = 1646
                last_heart_beat_time = 2024-01-30 10:09:00
                last_source_update = 2024-01-30 10:02:35
                last_sync_update = 2024-01-30 10:01:13
                last_synced_timestamp = 2024-01-30 10:01:12 (0s delay)

              5、nginx配置

              需要在每个节点都安装nginx，但是作用不是一样的，storage上的nginx是用于访问具体的文件，并且需要使用fastdfs模块，tracker上的nginx是用于负载均衡，不需要使用fastdfs模块，而最外层nginx则是统一入口，用于负载均衡到tracker上的nginx。

              5.1 配置storage的四台机器的nginx

              安装nginx所需依赖

              #安装nginx所需依赖
yum -y install pcre pcre-devel
yum -y install zlib zlib-devel
yum -y install openssl openssl-devel

              解压 nginx 和 fastdfs-nginx-module

              cd /usr/local/fastdfs
tar -zxvf nginx-1.19.10.tar.gz 
unzip fastdfs-nginx-module-master.zip

              解压后进入 nginx 目录编译安装 nginx，并添加 fastdfs-nginx-module，我这里为了和原有Nginx进行区分，新安装的Nginx取名为nginx_fdfs

              # 创建nginx安装目录
mkdir -p /usr/local/nginx_fdfs
cd /usr/local/fastdfs/nginx-1.19.10
# 配置，指定安装路径，添加模块
./configure --prefix=/usr/local/nginx_fdfs --add-module=/usr/local/fastdfs/fastdfs-nginx-module-master/src
# 编译
make
# 安装
make install

              访问配置

              把 fastdfs-nginx-module-master/src 目录下的 mod_fastdfs.conf 拷贝到 /etc/fdfs 目录下，这样才能正常启动 Nginx

              cp /usr/local/fastdfs/fastdfs-nginx-module-master/src/mod_fastdfs.conf  /etc/fdfs/

              在 /opt/fastdfs/ 目录下创建 nginx_mod 目录

              mkdir -p /opt/fastdfs/nginx_mod

              修改 mod_fastdfs.conf 配置文件

              vi /etc/fdfs/mod_fastdfs.conf

              #日志目录
base_path=/opt/fastdfs/nginx_mod  
#tracker服务器的IP地址以及端口号
tracker_server=192.168.122.22:22122
tracker_server=192.168.122.23:22122
#storage服务器的端口号(默认值)
storage_server_port=23000 
#组名
group_name = group1
#文件url中是否有 group 名
url_have_group_name = true 
#存储路径
store_path0=/opt/fastdfs/storage/files
#组数量
group_count = 2
[group1]
group_name=group1
storage_server_port=23000
store_path_count=1
store_path0=/opt/fastdfs/storage/files
[group2]
group_name=group2
storage_server_port=23000
store_path_count=1
store_path0=/opt/fastdfs/storage/files

              注：group_name根据实际组名变化，这里192.168.122.24和192.168.122.25为group1，192.168.122.26和192.168.122.27为group2。

              修改 nginx.conf 配置文件

              vi /usr/local/nginx_fdfs/conf/nginx.conf

               server {
        listen       80;
        server_name  localhost;
        location / {
            root   html;
            index  index.html index.htm;
        }
	    # 拦截请求路径中包含 /group[1-9]/M0[0-9] 的请求，用fastdfs的Nginx模块进行转发
        location ~ /group[1-9]/M0[0-9] {	
             ngx_fastdfs_module;  
        }
 }

              启动nginx

              # 测试
/usr/local/nginx_fdfs/sbin/nginx -c /usr/local/nginx_fdfs/conf/nginx.conf -t
# 启动
/usr/local/nginx_fdfs/sbin/nginx -c /usr/local/nginx_fdfs/conf/nginx.conf
# 停止
/usr/local/nginx_fdfs/sbin/nginx -c /usr/local/nginx_fdfs/conf/nginx.conf -s stop
# 重载
/usr/local/nginx_fdfs/sbin/nginx -c /usr/local/nginx_fdfs/conf/nginx.conf -s reload

              现在四个storage节点的图片都能进行访问，例如

              http://192.168.122.24/group2/M00/00/00/wKh6GmW6AWyAL2tYAAAJ3w30duA26.jpeg
http://192.168.122.25/group2/M00/00/00/wKh6GmW6AWyAL2tYAAAJ3w30duA26.jpeg
http://192.168.122.26/group2/M00/00/00/wKh6GmW6AWyAL2tYAAAJ3w30duA26.jpeg
http://192.168.122.27/group2/M00/00/00/wKh6GmW6AWyAL2tYAAAJ3w30duA26.jpeg

              5.2 配置tracker的两台机器的nginx

              192.168.122.22 和 192.168.122.23 节点

              安装nginx所需依赖

              #安装nginx所需依赖
yum -y install pcre pcre-devel
yum -y install zlib zlib-devel
yum -y install openssl openssl-devel

              解压 nginx

              cd /usr/local/fastdfs
tar -zxvf nginx-1.19.10.tar.gz

              解压后进入 nginx 目录编译安装 nginx，这里不需要 fastdfs-nginx-module

              # 创建nginx安装目录
mkdir -p /usr/local/nginx_fdfs
cd /usr/local/fastdfs/nginx-1.19.10
# 配置，指定安装路径，这里不需要添加模块
./configure --prefix=/usr/local/nginx_fdfs
# 编译
make
# 安装
make install

              修改 nginx.conf 配置文件

              vi /usr/local/nginx_fdfs/conf/nginx.conf

              upstream fastdfs_group_server{
    server 192.168.122.24:80;
    server 192.168.122.25:80;
    server 192.168.122.26:80;
    server 192.168.122.27:80;
}
server {
        listen       80;
        server_name  localhost;
        location / {
            root   html;
            index  index.html index.htm;
        }
	    # 拦截请求路径中包含 /group[1-9]/M0[0-9] 的请求
        location ~ /group[1-9]/M0[0-9] {	
             proxy_pass http://fastdfs_group_server;
        }
 }

              启动nginx

              # 测试
/usr/local/nginx_fdfs/sbin/nginx -c /usr/local/nginx_fdfs/conf/nginx.conf -t
# 启动
/usr/local/nginx_fdfs/sbin/nginx -c /usr/local/nginx_fdfs/conf/nginx.conf
# 停止
/usr/local/nginx_fdfs/sbin/nginx -c /usr/local/nginx_fdfs/conf/nginx.conf -s stop
# 重载
/usr/local/nginx_fdfs/sbin/nginx -c /usr/local/nginx_fdfs/conf/nginx.conf -s reload

              现在通过tracker上的nginx可以访问到四台storage的文件，例如

              http://192.168.122.22/group2/M00/00/00/wKh6GmW6AWyAL2tYAAAJ3w30duA26.jpeg
http://192.168.122.23/group2/M00/00/00/wKh6GmW6AWyAL2tYAAAJ3w30duA26.jpeg

              5.3 配置统一入口

              这里tracker有两个nginx，我们在最外层再增加一个nginx负载均衡，也就是我这里的192.168.122.1节点，配置如下

              upstream fastdfs_group_server{ server 192.168.122.22:80;
    server 192.168.122.23:80;
}
server { listen       80;
        server_name  localhost;
        location / { root   html;
            index  index.html index.htm;
        }
	    # 拦截请求路径中包含 /group[1-9]/M0[0-9] 的请求
        location ~ /group[1-9]/M0[0-9] { proxy_pass http://fastdfs_group_server;
        }
 }

              现在通过统一入口上的nginx可以访问到文件，例如

              http://192.168.122.1/group2/M00/00/00/wKh6GmW6AWyAL2tYAAAJ3w30duA26.jpeg

              至此，集群搭建完成，我辛苦了。

              熬夜写的，觉得有帮助来个赞吧！！！

              更多技术干货，请持续关注程序员大佬超。

              原创不易，转载请务必注明出处。

相关推荐

关于我们常见问题广告服务免责声明联系我们
Powered By Z-BlogPHP 1.7.3 Theme By 优美主题
Copyright Your WebSite.Some Rights Reserved. 浙ICP备2023018783号