Coreos Small cluster > Пример запуска coreos кластера с контейнерами docker, приложением, базой данных и прокси сервером

По материалам из видео курса:

[O’Reilly Media / Infinite Skills] Introduction to CoreOS Training Video [2015, ENG]

Советы по улучшению, принимаются.


PS. Исходники с Dockerfile, можно взять здесь:

https://github.com/sysadm-ru/coreos-docker-examples/tree/master/01

Они могу понадобиться, если захочется собрать собственные контейнеры или просто посмотреть примеры.


Базы данных RethinkDB

$ vagrant ssh core-01


$ vi [email protected]



[Unit]
Description=Announce RethinkDB %i service

[Service]
EnvironmentFile=/etc/environment
ExecStart=/bin/sh -c "while true; do etcdctl set /services/rethinkdb/rethinkdb-%i ${COREOS_PUBLIC_IPV4} --ttl 60; sleep 45; done"
ExecStop=/usr/bin/etcdctl rm /services/rethinkdb/rethinkdb-%i

[X-Fleet]
X-Conflicts=rethinkdb-announce@*.service



$ vi [email protected]



[Unit]
Description=RethinkDB %i service
After=docker.service
BindsTo=rethinkdb-announce@%i.service

[Service]
EnvironmentFile=/etc/environment
TimeoutStartSec=0
ExecStartPre=-/usr/bin/docker kill rethinkdb-%i
ExecStartPre=-/usr/bin/docker rm rethinkdb-%i
ExecStartPre=-/usr/bin/mkdir -p /home/core/docker-volumes/rethinkdb
ExecStartPre=/usr/bin/docker pull marley/coreos-rethinkdb:latest
ExecStart=/bin/sh -c '/usr/bin/docker run --name rethinkdb-%i   \
 -p ${COREOS_PUBLIC_IPV4}:8080:8080                        \
 -p ${COREOS_PUBLIC_IPV4}:28015:28015                      \
 -p ${COREOS_PUBLIC_IPV4}:29015:29015                      \
 marley/coreos-rethinkdb:latest rethinkdb --bind all \
 --canonical-address ${COREOS_PUBLIC_IPV4}                 \
 $(/usr/bin/etcdctl ls /services/rethinkdb |               \
     xargs -I {} /usr/bin/etcdctl get {} |                 \
     sed s/^/"--join "/ | sed s/$/":29015"/ |              \
     tr "\n" " ")'

ExecStop=/usr/bin/docker stop rethinkdb-%i

[X-Fleet]
X-ConditionMachineOf=rethinkdb-announce@%i.service


Что возвращается:

$ cat /etc/environment
COREOS_PUBLIC_IPV4=172.17.8.101
COREOS_PRIVATE_IPV4=172.17.8.101



$ echo $(/usr/bin/etcdctl ls /services/rethinkdb |               \
          xargs -I {} /usr/bin/etcdctl get {} |                 \
          sed s/^/"--join "/ | sed s/$/":29015"/ |              \
          tr "\n" " ")
--join 172.17.8.107:29015 --join 172.17.8.101:29015



$ fleetctl submit *

$ fleetctl list-unit-files
UNIT				HASH	DSTATE		STATE		TARGET
[email protected]	3f7611a	inactive	inactive	-
[email protected]		96c6e09	inactive	inactive	-



$ fleetctl start rethinkdb@6 rethinkdb-announce@6
$ fleetctl start rethinkdb@7 rethinkdb-announce@7



$ fleetctl list-units
UNIT				MACHINE				ACTIVE	SUB
[email protected]	010edf2c.../172.17.8.107	active	running
[email protected]	0f1619f3.../172.17.8.101	active	running
[email protected]		010edf2c.../172.17.8.107	active	running
[email protected]		0f1619f3.../172.17.8.101	active	running



$ curl 172.17.8.107:8080

Все ок. получил контент от сервера баз данных.


coreos cluster example


$ etcdctl ls --recursive

***

/services
/services/rethinkdb
/services/rethinkdb/rethinkdb-7
/services/rethinkdb/rethinkdb-6


Web Сервера

$ etcdctl get /services/rethinkdb/rethinkdb-6
172.17.8.107

$ etcdctl get /services/rethinkdb/rethinkdb-7
172.17.8.101



$ cd ~



$ vi [email protected]



[Unit]
Description=ToDo Service

Requires=docker.service
Requires=todo-sk@%i.service
After=docker.service

[Service]
EnvironmentFile=/etc/environment
User=core

Restart=always
TimeoutStartSec=0
ExecStartPre=-/usr/bin/docker kill %p-%i
ExecStartPre=-/usr/bin/docker rm %p-%i
ExecStartPre=/usr/bin/docker pull marley/coreos-nodejs-web-app
ExecStart=/usr/bin/docker run --name %p-%i \
      -h %H \
      -p ${COREOS_PUBLIC_IPV4}:3000:3000 \
      -e INSTANCE=%p-%i \
      -e COREOS_PRIVATE_IPV4=${COREOS_PRIVATE_IPV4} \
      marley/coreos-nodejs-web-app
ExecStop=-/usr/bin/docker kill %p-%i
ExecStop=-/usr/bin/docker rm %p-%i

[X-Fleet]
Conflicts=todo@*.service


-e COREOS_PRIVATE_IPV4=${COREOS_PRIVATE_IPV4} - экспортирую переменную, чтобы она была доступна в контейнере.


$ vi [email protected]


Какая-то проблема с движком сайта. Он не хочет печатать параметры –format. Предлагаю смотреть исходник на github

https://github.com/sysadm-ru/coreos-docker-examples/blob/master/01/coreos-nodejs-web-app/todo-sk%40.service

[Unit]
Description=ToDo Sidekick
Requires=todo@%i.service

After=docker.service
After=todo@%i.service
BindsTo=todo@%i.service

[Service]
EnvironmentFile=/etc/environment
User=core
Restart=always
TimeoutStartSec=0
ExecStart=/bin/bash -c '\
while true; do \
 port=$(docker inspect --format=\'\' todo-%i); \
 curl -sf ${COREOS_PUBLIC_IPV4}:$port/ > /dev/null 2>&1; \
 if [ $? -eq 0 ]; then \
   etcdctl set /services/todo/todo-%i ${COREOS_PUBLIC_IPV4}:$port --ttl 10; \
 else \
   etcdctl rm /services/todo/todo-%i; \
 fi; \
 sleep 5; \
 done'

ExecStop=/usr/bin/etcdctl rm /services/todo/todo-%i

[X-Fleet]
MachineOf=todo@%i.service


// Что выполняет следующая команда?

$ docker inspect --format="" todo-3

// Сначала нужно переключиться на сервер, где стартован сервис

$ fleetctl ssh [email protected]

// Следующая команда должна будет возвращать порт на котором работает вебсервер.

$ docker inspect --format="" todo-3

3000


$ fleetctl submit todo*



$ fleetctl list-unit-files
UNIT				HASH	DSTATE		STATE		TARGET
[email protected]	3f7611a	inactive	inactive	-
[email protected]	3f7611a	launched	launched	010edf2c.../172.17.8.107
[email protected]	3f7611a	launched	launched	0f1619f3.../172.17.8.101
[email protected]		96c6e09	inactive	inactive	-
[email protected]		96c6e09	launched	launched	010edf2c.../172.17.8.107
[email protected]		96c6e09	launched	launched	0f1619f3.../172.17.8.101
[email protected]		e8b8fa1	inactive	inactive	-
[email protected]			094d679	inactive	inactive	-



$ fleetctl start todo@{3..5} todo-sk@{3..5}



$ fleetctl list-units
UNIT				MACHINE				ACTIVE	SUB
[email protected]	010edf2c.../172.17.8.107	active	running
[email protected]	0f1619f3.../172.17.8.101	active	running
[email protected]		010edf2c.../172.17.8.107	active	running
[email protected]		0f1619f3.../172.17.8.101	active	running
[email protected]		43425159.../172.17.8.103	active	running
[email protected]		2320be18.../172.17.8.106	active	running
[email protected]		6c66d6fb.../172.17.8.102	active	running
[email protected]			43425159.../172.17.8.103	active	running
[email protected]			2320be18.../172.17.8.106	active	running
[email protected]			6c66d6fb.../172.17.8.102	active	running



$ curl 172.17.8.103:3000

Все ок. получил контент приложения от вебсервера.


$ etcdctl ls --recursive

***

/services
/services/rethinkdb
/services/rethinkdb/rethinkdb-6
/services/rethinkdb/rethinkdb-7
/services/todo
/services/todo/todo-3
/services/todo/todo-4
/services/todo/todo-5



$ etcdctl get /services/todo/todo-3          
172.17.8.103:3000

$ etcdctl get /services/todo/todo-4          
172.17.8.106:3000

$ etcdctl get /services/todo/todo-5          
172.17.8.102:3000


coreos cluster example


На самом деле, с первого раза ничего не запустилось

// Переключиться на хост, где стартован сервис, можно следующей командой

$ fleetctl ssh [email protected]


// логи

$ fleetctl journal -f --lines=100 todo@3
$ fleetctl journal -f --lines=100 todo-sk@3


Пришлось не только перестартовывать, но и удалять конфиги, удалять docker images руками.


$ fleetctl stop todo@{3..5} todo-sk@{3..5}
$ fleetctl unload todo@{3..5} todo-sk@{3..5}
$ fleetctl destroy todo@{3..5} todo-sk@{3..5}
$ fleetctl destroy [email protected]
$ fleetctl destroy [email protected]

И далее повторять все с начала.


Proxy Nginx

$ cd ~



$ vi nginx.service



[Unit]
Description=Nginx Proxy

Requires=docker.service
After=docker.service
After=etcd2.service
Requires=etcd2.service

[Service]
EnvironmentFile=/etc/environment
User=core

Restart=always
TimeoutStartSec=0
ExecStartPre=-/usr/bin/docker kill %p-%i
ExecStartPre=-/usr/bin/docker rm %p-%i
ExecStartPre=-/usr/bin/etcdctl mkdir /services/todo
ExecStartPre=-/usr/bin/docker pull marley/coreos-nginx
ExecStart=/usr/bin/docker run --name %p-%i \
      -h %H \
      -p ${COREOS_PUBLIC_IP}:80:80 \
      -e COREOS_PRIVATE_IPV4=${COREOS_PRIVATE_IPV4} \
      marley/coreos-nginx
ExecStop=-/usr/bin/docker kill %p-%i
ExecStop=-/usr/bin/docker rm %p-%i

[X-Fleet]
Global=true



$ fleetctl submit nginx.service



$ fleetctl list-unit-files
UNIT				HASH	DSTATE		STATE		TARGET
nginx.service			ab2c0e2	inactive	-		global
[email protected]	3f7611a	inactive	inactive	-
[email protected]	3f7611a	launched	launched	010edf2c.../172.17.8.107
[email protected]	3f7611a	launched	launched	0f1619f3.../172.17.8.101
[email protected]		96c6e09	inactive	inactive	-
[email protected]		96c6e09	launched	launched	010edf2c.../172.17.8.107
[email protected]		96c6e09	launched	launched	0f1619f3.../172.17.8.101
[email protected]		e8b8fa1	inactive	inactive	-
[email protected]		e8b8fa1	launched	launched	43425159.../172.17.8.103
[email protected]		e8b8fa1	launched	launched	2320be18.../172.17.8.106
[email protected]		e8b8fa1	launched	launched	6c66d6fb.../172.17.8.102
[email protected]			094d679	inactive	inactive	-
[email protected]			094d679	launched	launched	43425159.../172.17.8.103
[email protected]			094d679	launched	launched	2320be18.../172.17.8.106
[email protected]			094d679	launched	launched	6c66d6fb.../172.17.8.102



$ fleetctl start nginx.service



$ fleetctl list-units
UNIT				MACHINE				ACTIVE	SUB
nginx.service			010edf2c.../172.17.8.107	active	running
nginx.service			0f1619f3.../172.17.8.101	active	running
nginx.service			2320be18.../172.17.8.106	active	running
nginx.service			43425159.../172.17.8.103	active	running
nginx.service			6c66d6fb.../172.17.8.102	active	running
nginx.service			98b5dead.../172.17.8.105	active	running
nginx.service			e45abe65.../172.17.8.104	active	running
[email protected]	010edf2c.../172.17.8.107	active	running
[email protected]	0f1619f3.../172.17.8.101	active	running
[email protected]		010edf2c.../172.17.8.107	active	running
[email protected]		0f1619f3.../172.17.8.101	active	running
[email protected]		43425159.../172.17.8.103	active	running
[email protected]		2320be18.../172.17.8.106	active	running
[email protected]		6c66d6fb.../172.17.8.102	active	running
[email protected]			43425159.../172.17.8.103	active	running
[email protected]			2320be18.../172.17.8.106	active	running
[email protected]			6c66d6fb.../172.17.8.102	active	running



$ curl 172.17.8.101:80

Ок. Контент от вебсервера через proxy


coreos cluster example