Skip to content

Commit c2d7b06

Browse files
hs-liuxhhs-liuxh
committed
add: pg_repack
1 parent cb3865c commit c2d7b06

4 files changed

Lines changed: 253 additions & 0 deletions

File tree

‎CN/modules/ROOT/pages/master/ecosystem_components/ecosystem_overview.adoc‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -35,6 +35,7 @@ IvorySQL 作为一款兼容 Oracle 且基于 PostgreSQL 的高级开源数据库
3535
| 22 | xref:master/ecosystem_components/pg_bulkload.adoc[pg_bulkload] | 3.1.23 | 为IvorySQL提供高速数据载入工具，可以跳过PG的共享缓存直接将数据导入表中 | 海量数据初始加载，历史数据归档，跨库迁移
3636
| 23 | xref:master/ecosystem_components/pg_bigm.adoc[pg_bigm] | 1.2 | 为 IvorySQL 提供二元分词全文检索能力，适配中日韩文本，快速实现模糊检索与相似度查询 | 中日韩文内容、商品、地址类文字搜索场景
3737
| 24 | xref:master/ecosystem_components/pg_profile.adoc[pg_profile] | 4.11 | 收集数据库资源密集型活动的统计，基于两次采样点的差量分析生成历史负载报告，帮助定位性能瓶颈 | 数据库性能分析
38+
| 25 | xref:master/ecosystem_components/pg_repack.adoc[pg_repack] | 1.5.3 | 在几乎不阻塞业务读写的情况下在线重整表和索引、消除存储膨胀，效果类似 VACUUM FULL/CLUSTER | 表与索引在线重建、存储空间回收
3839
|====
3940

4041
这些插件均经过 IvorySQL 团队的测试和适配，确保在 IvorySQL 环境下稳定运行。用户可以根据业务需求选择合适的插件，进一步提升数据库系统的能力和灵活性。

‎CN/modules/ROOT/pages/master/ecosystem_components/pg_repack.adoc‎

Lines changed: 126 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,126 @@
1+
2+
:sectnums:
3+
:sectnumlevels: 5
4+
5+
= pg_repack
6+
7+
== 概述
8+
pg_repack 是 PostgreSQL 的一个扩展，用于*在线重整表和索引、消除存储膨胀*。它能达到与 `VACUUM FULL`、`CLUSTER` 类似的效果（回收空间、按索引重排物理顺序），但*几乎不阻塞业务读写*。pg_repack 通过另建新表而非原地修改的方式重整表，整个过程中只在开始和结束的瞬间持有短暂的排他锁，其余时间表都可以正常增删改查。
9+
10+
IvorySQL 的 PG 模式和 Oracle 兼容模式都已经适配 pg_repack。需要注意的是，在 Oracle 兼容模式下，pg_repack 在编译前，需要手动修改扩展源码的 lib/pg_repack.control.in 文件，增加一个配置项 `pg_dialect = true` ，否则无法正常安装。
11+
12+
项目地址：<https://github.com/reorg/pg_repack>
13+
14+
官方文档：<https://reorg.github.io/pg_repack/>
15+
16+
开源协议：BSD-3-Clause License
17+
18+
== 原理
19+
`VACUUM FULL` / `CLUSTER` 在重写表时会对整张表持有 `ACCESS EXCLUSIVE` 锁，期间无法读写。pg_repack 通过“影子表 + 触发器 + 文件节点交换”的方式避免长时间锁表。
20+
21+
=== 影子表与变更捕获
22+
* 创建一张与原表结构相同的影子表（`repack.table_<oid>`），并把原表数据批量拷入；
23+
24+
* 在原表上安装行级触发器，把重整期间发生的 `INSERT` / `UPDATE` / `DELETE` 记录进日志表（`repack.log_<oid>`）；
25+
26+
* 数据拷贝完成后回放日志，使影子表追平原表的最新状态。
27+
28+
=== 交换与清理
29+
* 在系统目录层面*交换原表与影子表的物理文件节点*，这一步是瞬时的，只需短暂的排他锁；
30+
31+
* 删除触发器、日志表等临时对象，并对新表执行 `ANALYZE`。
32+
33+
[NOTE]
34+
====
35+
使用前提：目标表必须有*主键或非空唯一键*；操作过程中约需要*两倍表大小*的临时磁盘空间。
36+
====
37+
38+
== 安装启用
39+
环境中已经安装了 IvorySQL5 及以上版本。pg_repack 由两部分组成：服务端扩展（`pg_repack.so` 与 SQL 脚本）和客户端命令行工具 `pg_repack`。
40+
41+
=== 源码安装
42+
设置 PG_CONFIG 环境变量：
43+
[literal]
44+
----
45+
export PG_CONFIG=/usr/local/ivorysql/bin/pg_config
46+
----
47+
48+
拉取 pg_repack 源码：
49+
[literal]
50+
----
51+
git clone --branch ver_1.5.3 https://github.com/reorg/pg_repack.git
52+
----
53+
54+
IvorySQL 的 Oracle 兼容模式下安装 pg_repack，需要先修改 pg_repack 的 lib/pg_repack.control.in 文件，增加一个配置项 `pg_dialect = true`：
55+
[literal]
56+
----
57+
# pg_repack/lib/pg_repack.control.in
58+
pg_dialect = true
59+
----
60+
61+
执行编译和安装：
62+
[literal]
63+
----
64+
cd pg_repack
65+
sudo --preserve-env=PG_CONFIG make
66+
sudo --preserve-env=PG_CONFIG make install
67+
----
68+
69+
=== 创建扩展
70+
[literal]
71+
----
72+
[ivorysql@localhost ivorysql]$ psql
73+
psql (18.0)
74+
Type "help" for help.
75+
76+
ivorysql=# CREATE EXTENSION pg_repack;
77+
CREATE EXTENSION
78+
----
79+
80+
[NOTE]
81+
====
82+
`CREATE EXTENSION pg_repack` 需要超级用户执行；客户端工具 `pg_repack` 默认也要求以超级用户身份连接运行。安装完成后，客户端工具位于 `/usr/local/ivorysql/bin/pg_repack`。
83+
====
84+
85+
== 使用流程
86+
pg_repack 通过命令行客户端驱动，连接参数与 psql 一致（`-h` 主机、`-p` 端口、`-U` 用户、`-d` 数据库，或使用 `PGHOST` / `PGPORT` / `PGUSER` 环境变量）。
87+
88+
=== 重整整个数据库中所有符合条件的表
89+
[literal]
90+
----
91+
pg_repack -d ivorysql
92+
----
93+
94+
=== 重整指定表
95+
推荐使用 schema.table 形式：
96+
[literal]
97+
----
98+
pg_repack -d ivorysql -t public.big_table
99+
----
100+
101+
=== 仅重建索引
102+
[literal]
103+
----
104+
pg_repack -d ivorysql -t big_table --only-indexes # 重建该表的全部索引
105+
pg_repack -d ivorysql -i public.big_table_idx # 重建单个索引
106+
----
107+
108+
=== 按指定列排序重建
109+
在线 CLUSTER 效果：
110+
[literal]
111+
----
112+
pg_repack -d ivorysql -t big_table -o "created_at DESC"
113+
----
114+
115+
=== 迁移表及其索引到另一个表空间
116+
[literal]
117+
----
118+
pg_repack -d ivorysql -t big_table -s fast_ssd_ts --moveidx
119+
----
120+
121+
=== 预演模式
122+
只显示将要执行的动作，不实际重整：
123+
[literal]
124+
----
125+
pg_repack -d ivorysql -t big_table --dry-run
126+
----

‎EN/modules/ROOT/pages/master/ecosystem_components/ecosystem_overview.adoc‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -36,6 +36,7 @@ IvorySQL, as an advanced open-source database compatible with Oracle and based o
3636
|*22*| xref:master/ecosystem_components/pg_bulkload.adoc[pg_bulkload] | 3.1.23 | Provides high-speed data loading tool for IvorySQL, which directly imports data into tables bypassing PostgreSQL shared buffers | initial loading of massive data, historical data archiving and cross-database migration
3737
|*23*| xref:master/ecosystem_components/pg_bigm.adoc[pg_bigm] | 1.2 | Equips IvorySQL with bigram full-text search capability, supporting Chinese, Japanese and Korean texts to implement fuzzy retrieval and similarity query efficiently | text search for articles, commodities and addresses in CJK languages
3838
|*24*| xref:master/ecosystem_components/pg_profile.adoc[pg_profile] | 4.11 | Collects statistics on resource-intensive database activities and generates historic workload reports based on delta analysis between two sample points, helping identify performance bottlenecks | Database performance analysis
39+
|*25*| xref:master/ecosystem_components/pg_repack.adoc[pg_repack] | 1.5.3 | Rebuilds tables and indexes online with minimal locking, eliminating storage bloat and achieving effects similar to VACUUM FULL/CLUSTER | Online table and index rebuild, storage bloat reclamation
3940
|====
4041

4142
These plugins have all been tested and adapted by the IvorySQL team to ensure stable operation in the IvorySQL environment. Users can select appropriate plugins based on business needs to further enhance the capabilities and flexibility of the database system.

‎EN/modules/ROOT/pages/master/ecosystem_components/pg_repack.adoc‎

Lines changed: 125 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,125 @@
1+
2+
:sectnums:
3+
:sectnumlevels: 5
4+
5+
= pg_repack
6+
7+
== Overview
8+
pg_repack is an extension for PostgreSQL used to *rebuild tables and indexes online and eliminate storage bloat*. It can achieve effects similar to `VACUUM FULL` and `CLUSTER` (reclaiming space, reordering physical layout by index), while *barely blocking business read/write operations*. pg_repack rebuilds a table by creating a new table rather than modifying it in place. Throughout the process, it only holds a brief exclusive lock at the very start and end; the rest of the time the table remains available for normal SELECT, INSERT, UPDATE, and DELETE operations.
9+
10+
Both the PG mode and Oracle compatibility mode of IvorySQL have been adapted for pg_repack. Note that in Oracle compatibility mode, before compiling pg_repack, you need to manually modify the lib/pg_repack.control.in file in the extension source code to add a configuration item `pg_dialect = true`, otherwise the installation will fail.
11+
12+
Github: <https://github.com/reorg/pg_repack>
13+
14+
Documentation: <https://reorg.github.io/pg_repack/>
15+
16+
License: BSD-3-Clause License
17+
18+
== Principle
19+
`VACUUM FULL` / `CLUSTER` hold an `ACCESS EXCLUSIVE` lock on the entire table while rewriting it, during which reads and writes are not possible. pg_repack avoids long-duration table locking by means of "shadow table + triggers + file node swap".
20+
21+
=== Shadow Table and Change Capture
22+
* Create a shadow table with the same structure as the original table (`repack.table_<oid>`), and bulk-copy the original table's data into it.
23+
24+
* Install row-level triggers on the original table to record the `INSERT` / `UPDATE` / `DELETE` operations that occur during the rebuild into a log table (`repack.log_<oid>`).
25+
26+
* After the data copy is complete, replay the log so that the shadow table catches up with the latest state of the original table.
27+
28+
=== Swap and Cleanup
29+
* At the system catalog level, *swap the physical file nodes of the original table and the shadow table*. This step is instantaneous and only requires a brief exclusive lock.
30+
31+
* Remove the triggers, log tables, and other temporary objects, and run `ANALYZE` on the new table.
32+
33+
[NOTE]
34+
====
35+
Prerequisites for use: the target table must have a *primary key or a non-null unique key*; the operation requires approximately *twice the table size* of temporary disk space during the process.
36+
====
37+
38+
== Installation and Enabling
39+
IvorySQL 5.0 or above is already installed in the environment. pg_repack consists of two parts: the server-side extension (`pg_repack.so` and SQL scripts) and the client-side command-line tool `pg_repack`.
40+
41+
=== Source Code Installation
42+
* Set the PG_CONFIG environment variable
43+
[literal]
44+
----
45+
export PG_CONFIG=/usr/local/ivorysql/bin/pg_config
46+
----
47+
48+
* Pull the pg_repack source code
49+
[literal]
50+
----
51+
git clone --branch ver_1.5.3 https://github.com/reorg/pg_repack.git
52+
----
53+
54+
* Compile and install pg_repack
55+
56+
To install pg_repack in IvorySQL's Oracle compatibility mode, you need to first modify the lib/pg_repack.control.in file of pg_repack to add the `pg_dialect` configuration item:
57+
[literal]
58+
----
59+
# pg_repack/lib/pg_repack.control.in
60+
pg_dialect = true
61+
----
62+
63+
Perform compilation and installation:
64+
[literal]
65+
----
66+
cd pg_repack
67+
sudo --preserve-env=PG_CONFIG make
68+
sudo --preserve-env=PG_CONFIG make install
69+
----
70+
71+
* Create the pg_repack extension
72+
[literal]
73+
----
74+
[ivorysql@localhost ivorysql]$ psql
75+
psql (18.0)
76+
Type "help" for help.
77+
78+
ivorysql=# CREATE EXTENSION pg_repack;
79+
CREATE EXTENSION
80+
----
81+
82+
[NOTE]
83+
====
84+
Note: `CREATE EXTENSION pg_repack` must be executed by a superuser; the client tool `pg_repack` also requires connecting and running as a superuser by default. After installation, the client tool is located at `/usr/local/ivorysql/bin/pg_repack`.
85+
====
86+
87+
== Usage
88+
pg_repack is driven through the command-line client. The connection parameters are the same as those of psql (`-h` host, `-p` port, `-U` user, `-d` database, or use the `PGHOST` / `PGPORT` / `PGUSER` environment variables).
89+
90+
* Rebuild all eligible tables in the entire database
91+
[literal]
92+
----
93+
pg_repack -d ivorysql
94+
----
95+
96+
* Rebuild a specified table (the schema.table form is recommended)
97+
[literal]
98+
----
99+
pg_repack -d ivorysql -t public.big_table
100+
----
101+
102+
* Rebuild indexes only
103+
[literal]
104+
----
105+
pg_repack -d ivorysql -t big_table --only-indexes # rebuild all indexes of this table
106+
pg_repack -d ivorysql -i public.big_table_idx # rebuild a single index
107+
----
108+
109+
* Rebuild ordered by a specified column (online CLUSTER effect)
110+
[literal]
111+
----
112+
pg_repack -d ivorysql -t big_table -o "created_at DESC"
113+
----
114+
115+
* Migrate a table and its indexes to another tablespace
116+
[literal]
117+
----
118+
pg_repack -d ivorysql -t big_table -s fast_ssd_ts --moveidx
119+
----
120+
121+
* Dry-run mode (only shows the actions that would be performed, without actually rebuilding)
122+
[literal]
123+
----
124+
pg_repack -d ivorysql -t big_table --dry-run
125+
----

0 commit comments

Comments
 (0)