Пример запуска coreos кластера с контейнерами docker, приложением, базой данных и прокси сервером
PS. Исходники с Dockerfile, можно взять здесь:
https://bitbucket.org/sysadm-ru/introduction_to_coreos
Они могу понадобиться, если захочется собрать собственные контейнеры или просто посмотреть примеры.
Для запуска примеров нужно:
- Установить virtualbox
- Установить vagrant
Vagrantfile и user-data
Скопировать файлы:
https://bitbucket.org/sysadm-ru/native-docker-clustering
$ cd ~
$ git clone https://bitbucket.org/sysadm-ru/native-docker-clustering
$ cd Native-Docker-Clustering
Сгенерировать ключ:
https://discovery.etcd.io/new?size=7
$ vi user-data
Заменить сгенерированным ключом.
discovery: https://discovery.etcd.io/89e341b6012e47d7e6654eea7b882418
$ vagrant box update
$ vagrant up
// Чтобы можно было по ssh ходить между узлами без пароля
$ ssh-add ~/.vagrant.d/insecure_private_key
$ vagrant status
Current machine states:
core-01 running (virtualbox)
core-02 running (virtualbox)
core-03 running (virtualbox)
core-04 running (virtualbox)
core-05 running (virtualbox)
core-06 running (virtualbox)
core-07 running (virtualbox)
$ vagrant ssh core-01
$ fleetctl list-machines
MACHINE IP METADATA
047ef507... 10.0.11.5 -
104a924a... 10.0.12.5 -
2ccc7711... 10.0.14.5 -
3c89f9a9... 10.0.15.5 -
8df586c8... 10.0.16.5 -
b9048ab8... 10.0.13.5 -
Базы данных
$ vi [email protected]
[Unit]
Description=Announce RethinkDB %i service
[Service]
EnvironmentFile=/etc/environment
ExecStart=/bin/sh -c "while true; do etcdctl set /services/rethinkdb/rethinkdb-%i ${COREOS_PUBLIC_IPV4} --ttl 60; sleep 45; done"
ExecStop=/usr/bin/etcdctl rm /services/rethinkdb/rethinkdb-%i
[X-Fleet]
X-Conflicts=rethinkdb-announce@*.service
$ vi [email protected]
[Unit]
Description=RethinkDB %i service
After=docker.service
BindsTo=rethinkdb-announce@%i.service
[Service]
EnvironmentFile=/etc/environment
TimeoutStartSec=0
ExecStartPre=-/usr/bin/docker kill rethinkdb-%i
ExecStartPre=-/usr/bin/docker rm rethinkdb-%i
ExecStartPre=-/usr/bin/mkdir -p /home/core/docker-volumes/rethinkdb
ExecStartPre=/usr/bin/docker pull marley/coreos-rethinkdb:latest
ExecStart=/bin/sh -c '/usr/bin/docker run --name rethinkdb-%i \
-p ${COREOS_PUBLIC_IPV4}:8080:8080 \
-p ${COREOS_PUBLIC_IPV4}:28015:28015 \
-p ${COREOS_PUBLIC_IPV4}:29015:29015 \
marley/coreos-rethinkdb:latest rethinkdb --bind all \
--canonical-address ${COREOS_PUBLIC_IPV4} \
$(/usr/bin/etcdctl ls /services/rethinkdb | \
xargs -I {} /usr/bin/etcdctl get {} | \
sed s/^/"--join "/ | sed s/$/":29015"/ | \
tr "\n" " ")'
ExecStop=/usr/bin/docker stop rethinkdb-%i
[X-Fleet]
X-ConditionMachineOf=rethinkdb-announce@%i.service
Что возвращается:
$ cat /etc/environment
COREOS_PUBLIC_IPV4=10.0.11.5
COREOS_PRIVATE_IPV4=10.0.11.5
$ echo $(/usr/bin/etcdctl ls /services/rethinkdb | \
> xargs -I {} /usr/bin/etcdctl get {} | \
> sed s/^/"--join "/ | sed s/$/":29015"/ | \
> tr "\n" " ")
--join 10.0.13.5:29015 --join 10.0.15.5:29015
$ fleetctl submit *
$ fleetctl list-unit-files
UNIT HASH DSTATE STATE TARGET
[email protected] 3f7611a inactive inactive -
[email protected] 96c6e09 inactive inactive -
$ fleetctl start rethinkdb@6 rethinkdb-announce@6
$ fleetctl start rethinkdb@7 rethinkdb-announce@7
$ fleetctl list-units
UNIT MACHINE ACTIVE SUB
[email protected] 09e7fca1.../10.0.13.5 active running
[email protected] 16d2848f.../10.0.15.5 active running
[email protected] 09e7fca1.../10.0.13.5 active running
[email protected] 16d2848f.../10.0.15.5 active running
$ curl 10.0.15.5:8080
Все ок. получил контент от сервера баз данных.
$ etcdctl ls --recursive
***
/services
/services/rethinkdb
/services/rethinkdb/rethinkdb-7
/services/rethinkdb/rethinkdb-6
Web Сервера
$ etcdctl get /services/rethinkdb/rethinkdb-6
10.0.13.5
$ etcdctl get /services/rethinkdb/rethinkdb-7
10.0.15.5
$ cd /tmp/
$ git clone --depth=1 https://github.com/sysadm-ru/Introduction_To_CoreOS
$ cd Introduction_To_CoreOS/Chapter5/todo-angular-express/
$ vi config.js
‘172.17.8.101’ меняю на ‘10.0.15.5’
10.0.15.5 - любой coreos хост с etcd, который предоставит информацию о подключении к базе. Разумеется, лучше потом какую-нибудь DNS запись для этого использовать.
$ docker build --rm -t marley/coreos-nodejs-web-app .
$ docker images
REPOSITORY TAG IMAGE ID CREATED SIZE
marley/coreos-app latest 54533dca8c65 8 minutes ago 736.1 MB
marley/coreos-rethinkdb latest ee254ccee514 8 weeks ago 181.8 MB
iojs 2.2 2a1868f3dfd8 20 months ago 703.8 MB
$ docker login
Ранее в веб интерфейсе создано репо.
$ docker push marley/coreos-nodejs-web-app
$ cd ~
$ vi [email protected]
[Unit]
Description=ToDo Service
Requires=docker.service
Requires=todo-sk@%i.service
After=docker.service
[Service]
EnvironmentFile=/etc/environment
User=core
Restart=always
TimeoutStartSec=0
ExecStartPre=-/usr/bin/docker kill %p-%i
ExecStartPre=-/usr/bin/docker rm %p-%i
ExecStartPre=/usr/bin/docker pull marley/coreos-nodejs-web-app
ExecStart=/usr/bin/docker run --name %p-%i \
-h %H \
-p ${COREOS_PUBLIC_IPV4}:3000:3000 \
-e INSTANCE=%p-%i \
marley/coreos-nodejs-web-app
ExecStop=-/usr/bin/docker kill %p-%i
ExecStop=-/usr/bin/docker rm %p-%i
[X-Fleet]
Conflicts=todo@*.service
$ vi [email protected]
[Unit]
Description=ToDo Sidekick
Requires=todo@%i.service
After=docker.service
After=todo@%i.service
BindsTo=todo@%i.service
[Service]
EnvironmentFile=/etc/environment
User=core
Restart=always
TimeoutStartSec=0
ExecStart=/bin/bash -c '\
while true; do \
port=$(docker inspect --format=\'\' todo-%i); \
curl -sf ${COREOS_PUBLIC_IPV4}:$port/ > /dev/null 2>&1; \
if [ $? -eq 0 ]; then \
etcdctl set /services/todo/todo-%i ${COREOS_PUBLIC_IPV4}:$port --ttl 10; \
else \
etcdctl rm /services/todo/todo-%i; \
fi; \
sleep 5; \
done'
ExecStop=/usr/bin/etcdctl rm /services/todo/todo-%i
[X-Fleet]
MachineOf=todo@%i.service
Следующая команда должна будет возвращать порт на котором работает вебсервер.
$ docker inspect --format="" todo-4
3000
$ fleetctl submit todo*
$ fleetctl list-unit-files
UNIT HASH DSTATE STATE TARGET
[email protected] 3f7611a inactive inactive -
[email protected] 3f7611a launched launched 09e7fca1.../10.0.13.5
[email protected] 3f7611a launched launched 16d2848f.../10.0.15.5
[email protected] 96c6e09 inactive inactive -
[email protected] 96c6e09 launched launched 09e7fca1.../10.0.13.5
[email protected] 96c6e09 launched launched 16d2848f.../10.0.15.5
[email protected] 64bb9b6 inactive inactive -
[email protected] 3dc7e5b inactive inactive -
$ fleetctl start todo@{3..5} todo-sk@{3..5}
$ fleetctl list-units
UNIT MACHINE ACTIVE SUB
[email protected] 09e7fca1.../10.0.13.5 active running
[email protected] 16d2848f.../10.0.15.5 active running
[email protected] 09e7fca1.../10.0.13.5 active running
[email protected] 16d2848f.../10.0.15.5 active running
[email protected] 72720a60.../10.0.17.5 active running
[email protected] 56b7dcad.../10.0.14.5 active running
[email protected] b420d775.../10.0.11.5 active running
[email protected] 72720a60.../10.0.17.5 active running
[email protected] 56b7dcad.../10.0.14.5 active running
[email protected] b420d775.../10.0.11.5 active running
$ curl 10.0.17.5:3000
Все ок. получил контент приложения от вебсервера.
$ etcdctl ls --recursive
***
/services
/services/rethinkdb
/services/rethinkdb/rethinkdb-6
/services/rethinkdb/rethinkdb-7
/services/todo
/services/todo/todo-3
/services/todo/todo-4
/services/todo/todo-5
На самом деле, с первого раза ничего не запустилось
Пришлось искать что это за виртуалка на которой располагается данный сервис.
Номер, виртуалки не совпадал.
// логи
$ fleetctl journal -f --lines=50 todo@3
$ fleetctl journal -f --lines=50 todo-sk@3
Пришлось не только перестартовывать, но и удалять конфиги, удалять docker images руками.
$ fleetctl stop todo@{3..5} todo-sk@{3..5}
$ fleetctl unload todo@{3..5} todo-sk@{3..5}
$ fleetctl destroy todo@{3..5} todo-sk@{3..5}
$ fleetctl destroy [email protected]
$ fleetctl destroy [email protected]
И далее повторять все с начала.
Proxy Nginx
$ cd /tmp/Introduction_To_CoreOS/Chapter5/nginx-proxy/
$ vi confd-watch
заменил
export HOST_IP=${HOST_IP:-172.17.8.101}
на
export HOST_IP=${HOST_IP:-10.0.15.5}
10.0.15.5 - любой coreos хост с etcd
$ docker build --rm -t marley/coreos-nginx .
$ docker images
REPOSITORY TAG IMAGE ID CREATED SIZE
marley/coreos-nginx latest 957faace8b64 11 seconds ago 159.3 MB
marley/coreos-app latest 54533dca8c65 41 minutes ago 736.1 MB
marley/coreos-rethinkdb latest ee254ccee514 8 weeks ago 181.8 MB
nginx 1.9.3 ea4b88a656c9 19 months ago 132.8 MB
iojs 2.2 2a1868f3dfd8 20 months ago 703.8 MB
$ docker login
Ранее в веб интерфейсе создано репо.
$ docker push marley/coreos-nginx
$ cd ~
$ vi nginx.service
[Unit]
Description=Nginx Proxy
Requires=docker.service
After=docker.service
After=etcd2.service
Requires=etcd2.service
[Service]
EnvironmentFile=/etc/environment
User=core
Restart=always
TimeoutStartSec=0
ExecStartPre=-/usr/bin/docker kill %p-%i
ExecStartPre=-/usr/bin/docker rm %p-%i
ExecStartPre=-/usr/bin/etcdctl mkdir /services/todo
ExecStartPre=-/usr/bin/docker pull marley/coreos-nginx
ExecStart=/usr/bin/docker run --name %p-%i \
-h %H \
-p ${COREOS_PUBLIC_IP}:80:80 \
marley/coreos-nginx
ExecStop=-/usr/bin/docker kill %p-%i
ExecStop=-/usr/bin/docker rm %p-%i
[X-Fleet]
Global=true
$ fleetctl submit nginx.service
$ fleetctl list-unit-files
UNIT HASH DSTATE STATE TARGET
nginx.service 111d636 inactive - global
[email protected] 3f7611a inactive inactive -
[email protected] 3f7611a launched launched 09e7fca1.../10.0.13.5
[email protected] 3f7611a launched launched 16d2848f.../10.0.15.5
[email protected] 96c6e09 inactive inactive -
[email protected] 96c6e09 launched launched 09e7fca1.../10.0.13.5
[email protected] 96c6e09 launched launched 16d2848f.../10.0.15.5
[email protected] e8b8fa1 inactive inactive -
[email protected] e8b8fa1 launched launched b420d775.../10.0.11.5
[email protected] e8b8fa1 launched launched 72720a60.../10.0.17.5
[email protected] e8b8fa1 launched launched 56b7dcad.../10.0.14.5
[email protected] b6473ba inactive inactive -
[email protected] b6473ba launched launched b420d775.../10.0.11.5
[email protected] b6473ba launched launched 72720a60.../10.0.17.5
[email protected] b6473ba launched launched 56b7dcad.../10.0.14.5
$ fleetctl start nginx.service
$ fleetctl list-units
UNIT MACHINE ACTIVE SUB
nginx.service 09e7fca1.../10.0.13.5 active running
nginx.service 16d2848f.../10.0.15.5 active running
nginx.service 56b7dcad.../10.0.14.5 active running
nginx.service 72720a60.../10.0.17.5 active running
nginx.service b420d775.../10.0.11.5 active running
nginx.service e5b75cfb.../10.0.12.5 active running
nginx.service f8083379.../10.0.16.5 active running
[email protected] 09e7fca1.../10.0.13.5 active running
[email protected] 16d2848f.../10.0.15.5 active running
[email protected] 09e7fca1.../10.0.13.5 active running
[email protected] 16d2848f.../10.0.15.5 active running
[email protected] b420d775.../10.0.11.5 active running
[email protected] 72720a60.../10.0.17.5 active running
[email protected] 56b7dcad.../10.0.14.5 active running
[email protected] b420d775.../10.0.11.5 active running
[email protected] 72720a60.../10.0.17.5 active running
[email protected] 56b7dcad.../10.0.14.5 active running
$ curl 10.0.17.5:80
Ок. Контент от вебсервера через proxy
