使用gitbook+gitlab构建文档中心

文档中心

实现要求

在gitlab上创建一个project gitbook,用于指定需要通过gitbook构建的project文档,实现自动拉取gitlab上需要构建的project,并通过gitbook对每个project文档生成一本书。同时由于每个project单独生成一本书,无法支持所有文档的全文检索,需要引入ealsticsearch,将所有文档内容保存到elasticsearch中,并通过elasticsearch实现全文检索。

实现过程

通过shell脚本获取所有project对应的相关配置(git参考地址、生成文档名称、文档分类),并自动循环拉取gitlab中的指定文档,若文档有更新则自动通过gitbook build构建成一本书,并通过rsync同步到”文档分类路径”中指定的路径下,通过nginx发布web页面(避免build过程中导致页面无法访问)。同时,在构建每一本书的时候会自动按照文档分类将该书的链接添加到summary/README.md文件中,用于生成首页。

gitbook中创建一个config的配置文件,格式如下

1
2
3
# git项目路径                          项目分支        文档名称            文档分类路径
git@gitcode.xxx.com:<project1>.git master <document1_name> 项目文档
git@gitcode.xxx.com:<project2>.git master <document2_name> 接口文档

具体实现脚本

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
#!/bin/sh
DOCUMENT_CONFIG_DIR=gitbook
DOCUMENT_REPO=git@gitlab.leucs.com:server/document/gitbook.git

NUM=1
DOCUMENT_GIT_DIR=`pwd`
DOCUMENT_BUILD_DIR="/home/gitlab-runner/gitbook_build"
DOCUMENT_DIR="/home/gitlab-runner/gitbook"

index_init(){
> $DOCUMENT_GIT_DIR/summary/README.md
for i in `grep -v ^"#" $DOCUMENT_CONFIG_DIR/config|grep -v ^"$"|awk '{print $NF}'|uniq`
do
echo "# $i" >> $DOCUMENT_GIT_DIR/summary/README.md
echo "" >> $DOCUMENT_GIT_DIR/summary/README.md
done
}

books_build_sync(){
cd $DOCUMENT_GIT_DIR
for dir in `grep -v ^"#" $DOCUMENT_CONFIG_DIR/config|grep -v ^"$" |awk '{print $1}'|awk -F / '{print $NF}'|sed 's/.git//'`
do
cd $DOCUMENT_GIT_DIR
git_repo=`grep -w ${dir}.git $DOCUMENT_CONFIG_DIR/config | awk '{print $1}'`
branch=`grep -w ${dir}.git $DOCUMENT_CONFIG_DIR/config | awk '{print $2}'`
title=`grep -w ${dir}.git $DOCUMENT_CONFIG_DIR/config | awk '{print $3}'`
location=`grep -w ${dir}.git $DOCUMENT_CONFIG_DIR/config | awk '{print $4}'`
domain=`grep -w ${dir}.git $DOCUMENT_CONFIG_DIR/config |awk -F [@:] '{print $2}'`
project=`grep -w ${dir}.git $DOCUMENT_CONFIG_DIR/config |awk '{print $1}'|awk -F: '{print $2}'| sed 's/.git//'`

if [ ! -d "$dir" ];then
git clone -b $branch $git_repo
else
cd $dir && git pull origin
fi
###sh book_gen.sh
cat > $DOCUMENT_GIT_DIR/$dir/book.json <<EOF
{
"title": "$dir",
"description": "$dir",
"author": "yealink",
"output.name": "site",
"language": "zh-hans",
"gitbook": "3.2.3",
"root": ".",
"links": {
"sidebar": {
"Home": "http://docs.yealinkops.com"
}
},
"plugins": [
"-lunr",
"-highlight",
"-livereload",
"-search",
"search-plus",
"highlight-code",
"alerts",
"terminal",
"code",
"anchor-navigation-ex",
"expandable-chapters",
"edit-link",
"splitter"
],
"pluginsConfig": {
"terminal": {
"copyButtons": false,
"fade": true,
"style": "flat"
},
"anchor-navigation-ex": {
"showLevel": false
},
"code": {
"copyButtons": true
},
"fontSettings": {
"theme": "white",
"family": "msyh",
"size": 2
},
"edit-link": {
"base": "http://$domain/$project/edit/$branch",
"label": "编辑此页"
}
}
}
EOF


export BOOK${NUM}_GIT_DIR="$DOCUMENT_GIT_DIR/$dir"
export BOOK${NUM}_BUILD_DIR="$DOCUMENT_BUILD_DIR/$dir"
export BOOK${NUM}_DIR="$DOCUMENT_DIR/$dir"

if eval [ ! -d '$'"BOOK${NUM}_BUILD_DIR" ];then
eval mkdir -p '$'"BOOK${NUM}_BUILD_DIR"
fi
if eval [ ! -d '$'"BOOK${NUM}_DIR" ];then
eval mkdir -p '$'"BOOK${NUM}_DIR"
fi

# build book if found new files appear in 6 seconds
export change${NUM}=`eval find '$'"BOOK${NUM}_GIT_DIR" -type f -mmin -0.1 ! -name "SUMMARY.md" ! -name "FETCH_HEAD" ! -name book.json |wc -l`
if eval [ '$'"change${NUM}" -ne 0 ];then
eval cd '$'"BOOK${NUM}_GIT_DIR"
mv SUMMARY.md SUMMARY.md_bak > /dev/null 2>&1
/usr/bin/book sm

# delete catalog of node_module for SUMMARY.md
line1=`grep -n "\- Node Modules" SUMMARY.md |head -1 |awk -F: '{print $1}'`
line2=`grep -n node_modules SUMMARY.md |tail -1|awk -F: '{print $1}'`
if [ -n "$line1" -a -n "$line2" ];then
/usr/bin/sed -i "${line1},${line2}d" SUMMARY.md
fi

# add CHANGELOG
/bin/git log --graph -n 50 --pretty="[%cd] - <%an> %s" > CHANGELOG.md
sed -i '/* \[CHANGELOG](CHANGELOG.md)/d' SUMMARY.md
sed -i "2a * [CHANGELOG](CHANGELOG.md)" SUMMARY.md

# copy node_modules if it is not exist
if eval [ ! -d "node_modules" ];then
/bin/cp -rp ~/node_modules/ .
fi

# build book and sync to web for nginx
eval rm -rf '$'"BOOK${NUM}_BUILD_DIR"
echo -e "\033[33m==== Begin to build $dir[$title] ====\033[0m"
eval /usr/bin/gitbook build '$'"BOOK${NUM}_GIT_DIR" '$'"BOOK${NUM}_BUILD_DIR"

echo -e "\033[33m==== Begin to sync $dir[$title] ====\033[0m"
eval /usr/bin/rsync -avz '$'"BOOK${NUM}_BUILD_DIR/" '$'"BOOK${NUM}_DIR" > /dev/null 2>&1
else
echo -e "\033[32m==== Skip to build $dir[$title] ====\033[0m"
fi

# add catalog to index according to class for book
line_title=`grep -n "# $location" $DOCUMENT_GIT_DIR/summary/README.md |awk -F: '{print $1}'`
sed -i "${line_title}a - [$title]($dir/index.html)" $DOCUMENT_GIT_DIR/summary/README.md

NUM=`expr $NUM + 1`
done
}

summary_build(){
echo -e "\033[33m==== Begin to build summary ====\033[0m"
cd $DOCUMENT_GIT_DIR
/usr/bin/sed -i 's/.md/.html/' ./summary/README.md
/usr/bin/sed -i '$a # gitbook自动构建情况' summary/README.md
/usr/bin/sed -i '$a - [构建详情](http://jenkins-develop.yealinkops.com/job/gitbook/lastBuild/console)' summary/README.md

/usr/bin/gitbook build ./summary $DOCUMENT_DIR/summary/
sed -i 's/index.html/index.html" target="blank"/' $DOCUMENT_DIR/summary/index.html
ln -sf $DOCUMENT_DIR/summary/* $DOCUMENT_DIR/
}


index_init
books_build_sync
summary_build

全文检索支持

elasticsearch支持全文检索,通过将文档仓库中的所有markdown文件POST到elasticsearch中(URL路径、标题和内容),并定期将新增/更新的markdown文档POST到elasticsearch,最后通过调用elasticsearch的API进行检索,从而实现文档仓库的全文检索。

具体实现脚本

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
#! /usr/bin/env python
# -*- coding: utf-8 -*-
import base64
import os
import json
import sys
reload(sys)
sys.setdefaultencoding('utf8')
import argparse
import requests

es_host = "10.200.112.31:9200"
es_index = "test1"
es_index_type = "doc"

parser = argparse.ArgumentParser()
parser.add_argument("-f",dest="file")
args = parser.parse_args()

path = args.file
url=path.replace('/home/gitlab-runner/builds/gitbook/','http://docs.yealinkops.com/').replace('.md','.html')
basename=os.path.basename(path)
title=basename.replace('.md','')
id=base64.b64encode(url).replace('/','').replace('+','')

with open(path) as file:
lines = file.readlines()

content = ''
for line in lines:
content += line.strip()

jsondata = {
"title": title,
"url": url,
"content": content
}

data = json.dumps(jsondata, ensure_ascii=False, encoding='UTF-8')

headers = {}
headers['Content-Type'] = 'application/json;charset=UTF-8'
es_url = "http://{0}/{1}/{2}/{3}".format(es_host,es_index,es_index_type,id[37:])
response = requests.post(url=es_url, headers=headers, data=json.dumps(jsondata,ensure_ascii=False).encode('utf-8'))
print response.text

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
#!/bin/bash
IFS=`echo -en "\n\b"`
GITBOOK_DIR="/home/gitlab-runner/builds/gitbook"

DOC_DIR=`ls -l $GITBOOK_DIR|grep ^d|awk '{print $NF}'`
for dir in $DOC_DIR
do
#cd $GITBOOK_DIR/$dir
MARKDOWN_FILE=`find $GITBOOK_DIR/$dir -name "*md" ! -path "./node_modules/*" -mmin -5 -not -name "README.md" -not -name "CHANGELOG.md" -not -name "SUMMARY.md"`
if [ -n "$MARKDOWN_FILE" ];then
for file in $MARKDOWN_FILE
do
echo "======== $file ========"
python /root/markdown2es.py -f $file
sleep 1
done
fi
done

五、实现效果


六、LDAP支持

文档中心支持LDAP登陆认证,同时支持权限管理。
https://github.com/kvspb/nginx-auth-ldap

./configure --add-module=./modules/nginx-auth-ldap

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
http {
ldap_server operation {
auth_ldap_cache_enabled on;
auth_ldap_cache_expiration_time 10000;
auth_ldap_cache_size 1000;
url "ldap://192.168.1.20:389/OU=软件开发五组,OU=软件开发部,OU=VCS产品线,OU=亿联-用户,DC=yealink,DC=com?sAMAccountName?sub?(objectClass=person)";
binddn "CN=xxx,OU=Users,OU=Openstack,OU=运维管理,OU=研发中心,OU=亿联-用户,DC=yealink,DC=com";
binddn_passwd "password";
group_attribute uniquemember;
group_attribute_is_dn on;
satisfy any;
require valid_user;
}

ldap_server test {
auth_ldap_cache_enabled on;
auth_ldap_cache_expiration_time 10000;
auth_ldap_cache_size 1000;
url "ldap://192.168.1.20:389/OU=Openstack,OU=运维管理,OU=研发中心,OU=亿联-用户,DC=yealink,DC=com?sAMAccountName?sub?(objectClass=person)";
binddn "CN=xxx,OU=Users,OU=Openstack,OU=运维管理,OU=研发中心,OU=亿联-用户,DC=yealink,DC=com";
binddn_passwd "password";
group_attribute uniquemember;
group_attribute_is_dn on;
satisfy any;
require valid_user;
}

server {
listen 80;
server_name _;
root /var/www/gitbook;

location ^~ /document {
root /var/www/gitbook;
auth_ldap "LDAP";
auth_ldap_servers operation;
auth_ldap_servers test;
}

location ^~ /Cloud {
root /var/www/gitbook;
auth_ldap "LDAP";
auth_ldap_servers test;
}
}
}

七、前端首页开发

坚持原创技术分享,您的支持将鼓励我继续创作!
0%