本书覆盖了开发人员在落地分布式架构过程中使用RocketMQ 的主要技术点，包括RocketMQ 的 高性能通信渠道、生产消息、消费消息、存储消息、分布式事务消息、消息的可追踪性、消息的稳定性、消息的高并发、消息的高可用、消息的高性能和RocketMQ 5.x 的新特性。采用"是什么→怎么用→什么原理（源码分析）”的主线来讲解这些技术点。 本书的主要目标：①让读者在动手中学习，而不是"看书时好像全明白了，一动手却发现什么都不会”；②让读者掌握整个RocketMQ 生态的全栈技术和最佳实践，而不是只有RocketMQ 框架；③让读者从RocketMQ 体系化的视角熟悉RocketMQ 的核心原理，而不是零散且碎片化；④让读者成为消息中间件领域的技术专家或架构师，而不只是熟悉RocketMQ；⑤让读者具备自研消息中间件的能力，而不是仅停留在源码分析的层次，"授人以鱼，不如授人以渔”是本书最大的宗旨。 本书适合对分布式架构及支撑分布式架构落地的中间件感兴趣的技术开发人员。无论读者是否接触过 分布式消息中间件，只要具备一定的Java 开发基础，都能通过本书的学习快速掌握分布式架构中消息中间件的开发技能，并利用RocketMQ 消息中间件支撑分布式架构的落地。

曾就职于网易考拉、微医集团、连连支付、政采云等杭州一线独角兽公司，高级技术专家（P8）和业务负责人，具备跨境电商、跨境支付等业务领域0-1的架构设计和开发落地能力，擅长高并发、高可用、和高性能系统架构设计。

第1 篇 基础
第1 章 初识RocketMQ ·························.2
1.1 认识分布式架构 ····························.2
1.1.1 什么是分布式 ·······················.2
1.1.2 什么是架构 ··························.3
1.1.3 分布式架构的冰与火 ··············.5
1.2 认识RocketMQ ·····························.7
1.2.1 什么是RocketMQ ··················.7
1.2.2 RocketMQ 与分布式架构 ·······.12
1.3 搭建RocketMQ 环境 ····················.13
1.3.1 了解RocketMQ 相关的安装包 .13
1.3.2 搭建单Master 的单机环境 ·····.15
1.3.3 搭建多Master 的集群环境 ·····.15
1.3.4 搭建单Master 和单Slave 的集群环境 ·································.16
1.3.5 【实例】搭建Raft 集群环境 ··.18
1.4 RocketMQ 5.0 的新特性 ················.20
第2 章 实现通信渠道 ··························.22
2.1 认识通信渠道 ·····························.22
2.2 通信渠道的架构 ··························.25
2.2.1 认识Netty ·························.25
2.2.2 RocketMQ 通信渠道的架构 ····.29
2.3 使用Netty 实现通信渠道的原理 ······.33
2.3.1 实现通信渠道的通信协议 ······.33
2.3.2 实现客户端通信渠道 ············.38
2.3.3 实现服务端通信渠道 ············.41
2.4 使用客户端连接服务端 ·················.47
2.4.1 分析模块通信渠道的类型 ······.48
2.4.2 连接服务端 ························.50
2.5 【实例】修改通信渠道客户端和服务端的线程模型 ·······················.54
第3 章 消息路由信息的无状态存储和管理 ······································.61
3.1 了解消息路由信息 ·······················.61
3.2 Name Server 的架构 ·····················.68
3.2.1 认识无状态架构 ··················.68
3.2.2 认识Name Server 架构 ·········.69
3.2.3 认识地址服务架构 ···············.72
3.2.4 【实例】用地址服务动态更新客户端中的Name Server 节点的地址信息 ··························.74
3.3 “使用Name Server 存储和管理消息路由信息”的原理 ·······················.77
3.3.1 Name Server 的启动流程 ·······.77
3.3.2 注册Broker Server ···············.84
3.3.3 取消注册的业务场景和取消注册Broker Server ···············.89
3.3.4 存储和管理消息路由信息 ······.93
3.4 【实例】启动多个Name Server 节点，模拟故障以验证Name Server 节点的无状态性 ·································.101
第4 章 生产消息和消费消息 ···············.104
4.1 生产和消费消息的模式 ················.104
4.1.1 生产消息 ··························.104
4.1.2 消费消息 ··························.110
4.2 消费消息的类型 ·························.116
4.2.1 集群消息 ··························.116
4.2.2 广播消息 ··························.118
4.2.3 顺序消息 ··························.119
4.2.4 并发消息 ··························.125
4.2.5 延迟消息 ··························.126
4.2.6 事务消息 ··························.130
4.3 生产消息 ··································.134
4.3.1 生产者SDK 的3 种使用方式 ································.135
4.3.2 采用“同步”模式生产消息 ···.137
4.3.3 采用“异步”模式生产消息 ···.142
4.3.4 采用“最多发送一次”模式生产消息 ··························.148
4.4 消费消息 ··································.152
4.4.1 采用pull 模式消费消息 ········.153
4.4.2 采用push 模式消费消息 ·······.171
4.4.3 采用pop 模式消费消息 ········.183
4.5 采用“请求/应答”消息实现同步调用 ········································.197
4.5.1 “请求/应答”消息的架构 ····.197
4.5.2 分析“请求/应答”消息的过程 ································.198
4.6 【实例】生产者发送消息，消费者顺序地消费消息 ·························.212
4.6.1 验证“普通顺序消息” ········.212
4.6.2 验证“严格顺序消息” ········.217
第5 章 存储消息 ······························.219
5.1 认识存储引擎 ····························.219
5.1.1 什么是存储引擎 ·················.219
5.1.2 存储引擎的架构 ·················.221
5.2 认识存储模型 ····························.222
5.2.1 消息模型 ··························.222
5.2.2 文件模型 ··························.229
5.2.3 【实例】利用源码远程调试存储模型 ································.236
5.3 启动存储引擎 ····························.238
5.3.1 初始化Broker Server 和存储引擎 ································.238
5.3.2 启动Broker Server 和存储引擎 ································.247
5.3.3 【实例】动态修改存储引擎的配置参数 ··························.251
5.4 使用存储引擎处理“储存消息”的请求 ········································.253
5.4.1 接收并处理生产消息的请求 ································.254
5.4.2 存储消息 ··························.257
5.4.3 【实例】利用存储引擎实现批量地存储消息 ·······················.262
5.5 对比存储单条消息和批量消息的性能 ········································.263

第6 章 治理消息 ······························.268
6.1 如何治理消息 ····························.268
6.1.1 治理消息的目标 ·················.268
6.1.2 使用消息度量提供治理消息的依据 ································.276
6.1.3 使用消息管控实现治理消息的落地 ································.280
6.2 认识命令控制台和UI 控制台 ········.285
6.2.1 什么是命令控制台和UI控制台 ·····························.285
6.2.2 启动命令控制台和UI控制台 ·····························.289
6.2.3 使用对象池管理RocketMQDashboard 中通信渠道客户端的核心类 ·····························.292
6.3 使用命令控制台治理消息 ·············.298
6.3.1 执行治理消息的命令 ···········.299
6.3.2 【实例】使用命令控制台，完成
RocketMQ 集群的扩容 ·········.322
6.3.3 【实例】使用命令控制台，完成
RocketMQ 集群的缩容 ·········.325
6.3.4 【实例】使用命令控制台，动态增加Topic 的读/写消息队列的数量 ·················.332
6.4 使用UI 控制台治理消息 ··············.334
6.4.1 使用UI 控制台治理消息的原理 ································.335
6.4.2 【实例】使用UI 控制台手动地禁用消费者组 ····················.341
6.4.3 【实例】使用UI 控制台重置消费者进度 ·······················.343
第7 章 实现分布式事务 ·····················.345
7.1 什么是分布式事务 ······················.345
7.1.1 本地事务 ··························.345
7.1.2 分布式事务 ·······················.350
7.1.3 事务消息与数据的最终一致性 ·····························.354
7.2 使用“两阶段提交”模式处理事务消息 ········································.359
7.3 使用“定时回调”机制查询本地事务状态 ········································.370
7.4 【实例】架构师如何在电商项目中落地分布式事务 ·························.380
7.4.1 分析业务场景 ····················.380
7.4.2 分析业务复杂度 ·················.381
7.4.3 分析技术复杂度 ·················.385
7.4.4 落地代码 ··························.391
7.4.5 验证接入的结果 ·················.392
7.5 【实例】使用泛娱乐业务验证事务消息的故障转移机制 ···················.393
7.5.1 准备环境 ··························.394
7.5.2 架构设计 ··························.394
7.5.3 代码设计 ··························.395
7.5.4 验证故障转移 ····················.397
第8 章 实现消息的可追踪性 ···············.402
8.1 消息的可追踪性 ·························.402
8.1.1 认识消息的可追踪性 ···········.402
8.1.2 为什么需要消息的可追踪性 ···.404
8.1.3 RocketMQ 支持的追踪类型 ···.404
8.2 使用消息轨迹实现消息的可追踪性 ··································.405
8.2.1 认识消息轨迹 ····················.405
8.2.2 轨迹树的原理 ····················.412
8.2.3 【实例】开启消息轨迹，追踪生产消息、消费消息和结束事务消息的过程 ································.421
8.3 使用Hook 机制收集Trace 消息 ······.422
8.3.1 什么是Hook 机制 ···············.422
8.3.2 “使用Hook 机制的实现类收集Trace 消息”的原理 ······.425
8.4 使用“定时线程任务”异步存储Trace信息 ········································.428
8.4.1 为什么要异步存储 ··············.428
8.4.2 异步存储Trace 信息的过程 ···.431
8.4.3 【实例】断点调试“客户端异步存储Trace 信息”···············.441
8.5 使用“OpenTracing + Jaeger”实现消息的分布式链路追踪 ·························.444
8.5.1 认识OpenTracing ···············.444
8.5.2 认识Jaeger ·······················.446
8.5.3 “使用Jaeger 收集消息链路数据”的原理 ·····························.446
8.5.4 “使用OpenTracing 规范消息链路追踪”的原理 ····················.448
8.5.5 【实例】收集普通消息和事务消息，使用Jaeger 实现可视化 ········.452
8.6 【实例】自定义Hook 和Trace 信息的派发类 ·····································.456
8.6.1 设计 ································.457
8.6.2 关键代码分析 ····················.457
第9 章 保证消息的稳定性 ··················.461
9.1 什么是消息的稳定性 ···················.461
9.1.1 认识稳定性 ·······················.461
9.1.2 认识消息的稳定性 ··············.466

9.1.3 消息稳定性技术 ·················.467
9.2 实现RocketMQ 的实时监控 ··········.472
9.2.1 认识RocketMQ Exporter 及其相关组件 ··························.472
9.2.2 实时监控的逻辑架构 ···········.474
9.2.3 “使用RocketMQ Exporter 实现实时监控”的原理 ··············.477
9.2.4 搭建监控环境 ····················.486
9.2.5 【实例】基于Prometheus 和Alert Manager 的RocketMQ实时告警 ··························.494
9.3 实现RocketMQ 客户端的实时度量 ··································.503
9.3.1 认识Skywalking ·················.503
9.3.2 “使用Skywalking 实时度量RocketMQ 客户端”的原理 ···.504
9.3.3 搭建监控环境 ····················.509
9.3.4 【实例】修改源码动态设置一些延迟时间，触发基于Skywalking和钉钉的告警阈值 ··············.511
9.4 【实例】使用RocketMQ Dashboard实时监控RocketMQ ····················.515
9.4.1 设计 ································.515
9.4.2 修改RocketMQ Dashboard 的源码 ································.516
9.4.3 配置监控规则 ····················.517
9.4.4 初始化生产者和消费者服务 ··.517
9.4.5 验证监控告警 ····················.518
9.5 【实例】改造RocketMQ 以支持Nacos 集群管理 ··························.520
9.5.1 架构设计 ··························.520
9.5.2 落地代码 ··························.521
9.5.3 验证 ································.524
第4 篇 高并发、高可用和高性能
第10 章 消息的高并发 ······················.528
10.1 消息的高并发概述 ·····················.528
10.1.1 什么是高并发 ··················.528
10.1.2 为什么要实现消息的高并发 ·····························.529
10.2 使用“管理通信渠道”实现消息的高并发 ·····································.530
10.2.1 通信渠道与高并发 ············.530
10.2.2 管理生产者与Broker Server之间的通信渠道 ·················.532
10.2.3 管理消费者与Broker Server之间的通信渠道 ·················.533
10.3 使用“提高消费者处理消息的功能”实现消息的高并发 ······················.534
10.3.1 消费者处理消息的功能与高并发 ·····························.534
10.3.2 使用“扩容消费者客户端”来提高消费消息的并行性 ········.534
10.3.3 采用pop 模式来提高消费消息的速度 ································.535

10.3.4 【实例】实现“单进程单消费者的并发和批量消费消息” ·····························.536
10.3.5 【实例】实现“单进程多消费者实例并发生产和消费消息” ·····························.541
10.4 使用线程池实现消息的高并发 ······.550
10.4.1 为什么要使用线程池 ·········.550
10.4.2 使用线程池实现消息的高并发的原理 ································.551
10.4.3 【实例】使用RocketMQ 自带的线程实现生产和消费消息的并发性 ·····························.558
10.4.4 【实例】动态修改Consumer客户端的核心线程数 ···········.559
10.5 采用“异步”模式实现消息的高并发 ····································.562
10.5.1 使用“回调+异步响应对象”实现生产者的“异步”模式 ································.562
10.5.2 使用“异步命令事件处理器”实现Broker Server 的“异步”模式 ································.563
10.5.3 使用“流水线”技术实现存储引擎的“异步”模式 ···········.564
10.6 使用“负载均衡”实现消息的高并发 ·····································.566
10.6.1 什么是负载均衡 ···············.566
10.6.2 “使用选择器实现生产消息的负载均衡”的原理 ··············.576
10.6.3 “使用客户端负载均衡实现消费消息的均衡性”的原理 ········.577
10.6.4 “使用服务端负载均衡实现消费消息的均衡性”的原理 ········.578
10.6.5 【实例】动态调整消费者客户端的负载均衡策略 ·················.579
第11 章 消息的高可用 ·······················.583
11.1 什么是消息的高可用 ··················.583
11.1.1 生产消息的高可用 ············.584
11.1.2 存储消息的高可用 ············.585
11.1.3 消费消息的高可用 ············.585
11.2 实现生产消息的高可用——利用故障容错机制 ··································.587
11.2.1 什么是故障容错机制 ·········.587
11.2.2 重试生产消息失败的请求 ···.588
11.2.3 过滤故障的Broker Server ····.590
11.2.4 故障延迟 ························.591
11.2.5 【实例】在Spring Cloud Alibaba项目中，植入故障并验证故障
容错机制 ··························.594
11.3 实现消费消息的高可用——利用重试队列和死信队列 ·························.596
11.3.1 利用“重试队列”实现高可用的核心原理 ··························.597
11.3.2 利用“死信队列”实现高可用的核心原理 ··························.598
11.3.3 【实例】动态调整重试策略，并验证消费失败的高可用性 ···.599
11.3.4 【实例】在消息重试失败后，消费死信队列中的消息 ········.601

11.4 实现Broker Server 的高可用——利用主/从同步 ·································.602
11.4.1 什么是主/从同步 ···············.602
11.4.2 实现主/从节点之间的通信渠道 ································.605
11.4.3 实现主/从节点之间消息的同步 ································.610
11.4.4 【实例】模拟Broker Server主/从同步的业务场景 ··········.614
11.5 实现Broker Server 的高可用——利用多副本架构（Dledger） ···············.617
11.5.1 什么是多副本架构（DLedger） ·······················.617
11.5.2 “多副本架构的通信渠道”的原理 ································.618
11.5.3 “多副本架构的协议”的原理 ································.621
11.5.4 “多副本架构的手动选举”的原理 ································.627
11.5.5 “多副本架构的自动选举”的原理 ································.631
11.5.6 “多副本架构的消息存储”的原理 ································.634
11.5.7 “多副本架构中集群之间消息同步”的原理 ····················.638
11.5.8 【实例】在主节点出现故障后，多副本架构（DLedger）自动选举，并保持主/从消息一致性 ·······.645
11.6 【实例】模拟业务场景“从主/从架构升级为多副本架构”，并验证消息的高可用性 ··································.651
11.7 【实例】使用微服务来验证多副本的手动选举功能的高可用性 ·············.654
第12 章 消息的高性能 ······················.657
12.1 什么是消息的高性能 ·················.657
12.1.1 生产消息的高性能 ············.657
12.1.2 存储消息的高性能 ············.659
12.1.3 消费消息的高性能 ············.660
12.2 使用本地缓存实现高性能 ···········.661
12.2.1 缓存客户端通信渠道 ·········.661
12.2.2 缓存消息路由信息 ············.663
12.2.3 【实例】使用IDEA 调试“缓存客户端通信渠道” ······.665
12.3 提升存储消息的性能——利用CommitLog 文件顺序写 ···············.670
12.3.1 什么是CommitLog 文件 ·····.670
12.3.2 CommitLog 文件的存储格式 ································.672
12.3.3 利用“顺序写”存储消息的原理 ································.673
12.4 提高消费消息的速度——利用ConsumeQueue 文件实现 ··············.676
12.4.1 什么是ConsumeQueue文件 ································.677
12.4.2 构建ConsumeQueue 文件 ····.678
12.4.3 利用ConsumeQueue 文件提高消费消息的性能 ·················.683
12.5 提高查询消息的速度——利用索引文件实现 ··································.684
12.5.1 什么是索引文件 ···············.684
12.5.2 构建索引文件 ··················.685
12.5.3 利用索引文件提高查询消息的性能 ································.687
12.6 提升消费消息RPC 请求的性能——利用零拷贝实现 ·························.687
12.6.1 什么是零拷贝 ··················.688
12.6.2 使用Netty 的FileRegion 类实现文件的零拷贝 ····················.688
12.6.3 在Broker Server 的消息处理器中使用零拷贝 ·······················.691
12.7 提升Broker Server 写消息的性能——利用内存映射、Page Cache 及堆外
内存实现 ··································.692
12.7.1 什么是内存映射、Page Cache 和堆外内存 ··························.693
12.7.2 写消息高性能的逻辑架构 ···.695
12.7.3 “利用MappedFile 文件实现内存映射”的原理 ··············.696
12.7.4 “利用内存映射实现Page Cache”的原理 ·····························.696
12.7.5 “管理堆外内存和Page Cache”的原理 ·····························.697
12.8 清理过期的ConsumeQueue 文件和CommitLog 文件 ························.702
12.8.1 清理过期的ConsumeQueue文件 ································.702
12.8.2 清理过期的CommitLog文件 ································.704
12.8.3 【实例】利用Spring CloudAlibaba 项目演示清理过期的ConsumeQueue 文件和CommitLog文件 ································.709
第5 篇 应用
第13 章 RocketMQ 在分布式架构中的应用 ····································.712
13.1 RocketMQ 在Spring Boot 中的应用 ········································.712
13.1.1 为什么RocketMQ 要支持Spring Boot ·······················.712
13.1.2 利用Spring Boot 作为基础框架生产和消费消息 ·················.713
13.2 RocketMQ 在Spring Cloud Alibaba 中的应用 ·······································.719
13.2.1 为什么Spring Cloud Alibaba 要支持RocketMQ ··················.719
13.2.2 利用Spring Cloud Alibaba 作为基础框架生产和消费消息 ·····.719
13.2.3 Spring Cloud Alibaba 定制化RocketMQ 的原理 ···············.728
13.3 【实例】将命令式编程模式升级为函数式编程模式，完成消息的生产和消费 ········································.730
13.3.1 采用命令式编程模式生产和消费消息 ··························.730

13.3.2 将命令式编程模式升级为函数式编程模式 ·················.731
13.4 RocketMQ 在RocketMQ Connect 中的应用 ········································.732
13.4.1 什么是Message Connect ·····.733
13.4.2 什么是RocketMQ Connect ··.735
13.4.3 RocketMQ Connect 的核心架构 ································.736
13.4.4 “利用生产者和消费者来同步集群信息”的原理 ··············.739
13.4.5 “利用监听器完成服务发现”的原理 ································.743
13.4.6 “利用客户端负载均衡实现同步消息的高吞吐量”的原理 ·····.744
13.4.7 【实例】搭建RocketMQ Connect的本地和分布式环境 ···········.747
13.5 RocketMQ 在Event Bridge 中的应用 ········································.750
13.5.1 什么是Event Bridge ···········.750
13.5.2 Event Bridge 的核心架构 ·····.751
13.5.3 利用RocketMQ 实现EventBridge 的核心原理 ··············.756
13.6 RocketMQ 在多语言环境中的应用 ········································.757
13.6.1 RocketMQ 的多语言架构设计 ································.757
13.6.2 【实例】使用Go 语言客户端生产和消费消息 ·················.758
13.7 RocketMQ 在大数据中的应用 ······.760
13.7.1 RocketMQ Streams 的核心架构 ································.761
13.7.2 【实例】利用RocketMQ Streams处理微服务中的流数据 ········.765
第6 篇 新特性
章 分析RocketMQ 5.x 的新特性 ··.768
14.1 利用“对等部署模式”实现BrokerServer 的高可用 ·························.768
14.1.1 认识“对等部署模式” ······.768
14.1.2 实现“对等部署模式”的原理 ································.771
14.1.3 【实例】利用微服务来验证“对等部署模式”的高可用 ···.777
14.2 利用“Slave 代理Master 模式”实现Broker Server 的高可用 ················.778
14.2.1 认识“Slave 代理Master模式” ·····························.778
14.2.2 实现“Slave 代理Master模式”的原理 ····················.779
14.2.3 【实例】利用微服务来验证“Slave 代理Master 模式”的
高可用 ·····························.788
14.3 利用“RocketMQ Controller 模式”实现Broker Server 的高可用 ·········.790
14.3.1 认识“RocketMQ Controller模式” ·····························.791
14.3.2 实现“RocketMQ Controller模式”的原理 ····················.793
14.3.3 利用“RocketMQ Controller模式”完成集群管理 ···········.797
14.3.4 实现“RocketMQ Controller 模式中多副本状态机”的原理 ·····.806
14.3.5 【实例】验证“RocketMQController 模式”的
高可用性 ··························.810
14.4 利用“RocketMQ Proxy 模式”统一多语言客户端 ····························.814
14.4.1 认识“RocketMQ Proxy模式” ·····························.814
14.4.2 实现“RocketMQ Proxy 模式”的原理 ································.816
14.4.3 实现“代理多语言客户端请求”的原理 ···························.821