Docker UUID Error

□ Docker UUID Error

 

□ 문제파악

현재 클라우드 VM 상에서 docker의 devicemapper의 위치가 Local이 아니라 NAS에 설정되어있는 상황이었습니다, 또한 VM이 매일 리부팅이 발생하는 상황이었습니다. 서버 안에는 docker 컨테이너 서비스들이 돌아가는 상황이었습니다. 모든 서비들이 안들어가저서 서버에서 docker ps -a 명령어를 통해 container를 확인했는데 아예 docker 명령어를 실행시킬 수 없는 상태였습니다. 

 

□ 오류 메세지

# systemctl start docker

[오류]
Job for docker.service failed because the control process exited with error code. See "systemctl status docker.service" and "journalctl -xe" for details.

□ 오류 상세 확인 [핵심적인 부분만 추렸습니다.]

# journalctl -xe

{[{unix:///run/containerd/containerd.sock 0  <nil>}] <nil>}" module=grpc
May 26 10:36:15 hskim-test dockerd[5110]: time="2020-05-26T10:36:15.006470520+09:00" level=info msg="ClientConn switching balancer to \"pick_first\"" module=grpc
May 26 10:36:15 hskim-cicd kernel: device-mapper: thin: Data device (loop0) discard unsupported: Disabling discard passdown.

May 26 10:36:15 hskim-cicd dockerd[5110]: time="2020-05-26T10:36:15.198587234+09:00" level=error msg="[graphdriver] prior storage driver devicemapper failed: devmapper: Base Device UUID and Filesystem verification failed: devmapper: Current Base Device UUID:e298c53b-84bd-4e00-af3c-6be157514ac1 does not match with stored UUID:89954a3d-f23a-482f
May 26 10:36:15 hskim-cicd dockerd[5110]: failed to start daemon: error initializing graphdriver: devmapper: Base Device UUID and Filesystem verification failed: devmapper: Current Base Device UUID:e298c53b-84bd-4e00-af3c-6be157514ac1 does not match with stored UUID:89954a3d-f23a-482f-94e1-be82f4fe252f. Possibly using a different thin pool tha
May 26 10:36:15 hskim-cicd systemd[1]: docker.service: main process exited, code=exited, status=1/FAILURE
May 26 10:36:15 hskim-cicd systemd[1]: Failed to start Docker Application Container Engine.
-- Subject: Unit docker.service has failed
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
--
-- Unit docker.service has failed.
--
-- The result is failed.
May 26 10:36:15 hskim-cicd systemd[1]: Unit docker.service entered failed state.
May 26 10:36:15 hskim-cicd systemd[1]: docker.service failed.

May 26 10:36:15 hskim-cicd kernel: device-mapper: thin: Data device (loop0) discard unsupported: Disabling discard passdown.
May 26 10:36:15 hskim-cicd dockerd[5110]: time="2020-05-26T10:36:15.150262629+09:00" level=warning msg="Usage of loopback devices is strongly discouraged for production use. Please use `--storage-opt dm.thinpooldev` or use `man dockerd` to refer to dm.thinpooldev section." storage-driver=devicemapper
May 26 10:36:15 hskim-cicd dockerd[5110]: time="2020-05-26T10:36:15.198587234+09:00" level=error msg="[graphdriver] prior storage driver devicemapper failed: devmapper: Base Device UUID and Filesystem verification failed: devmapper: Current Base Device UUID:e298c53b-84bd-4e00-af3c-6be157514ac1 does not match with stored UUID:89954a3d-f23a-482f
May 26 10:36:15 hskim-cicd dockerd[5110]: failed to start daemon: error initializing graphdriver: devmapper: Base Device UUID and Filesystem verification failed: devmapper: Current Base Device UUID:e298c53b-84bd-4e00-af3c-6be157514ac1 does not match with stored UUID:89954a3d-f23a-482f-94e1-be82f4fe252f. Possibly using a different thin pool tha
May 26 10:36:15 hskim-cicd systemd[1]: docker.service: main process exited, code=exited, status=1/FAILURE
May 26 10:36:15 hskim-cicd systemd[1]: Failed to start Docker Application Container Engine.
-- Subject: Unit docker.service has failed
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
--
-- Unit docker.service has failed.
--
-- The result is failed.
May 26 10:36:15 hskim-cicd systemd[1]: Unit docker.service entered failed state.
May 26 10:36:15 hskim-cicd systemd[1]: docker.service failed.

□ 원인 분석

 

▶ UUID가 89954a3d-f23a-482f-94e1-be82f4fe252f 값으로 저장되어 있으나 도커는 e298c53b-84bd-4e00-af3c-6be157514ac1 값으로 인지하여 오류가 발생

Base Device UUID and Filesystem verification failed: devmapper: Current Base Device UUID:e298c53b-84bd-4e00-af3c-6be157514ac1 does not match with stored UUID:89954a3d-f23a-482f-94e1-be82f4fe252f.

▶ 해당 UUID를 도커가 인지하는 uuid 변경

 

UUID 메타정보 저장위치: ../docker/devicemapper/metadata/deviceset-metadata

 

As-is : {"next_device_id":1,"BaseDeviceUUID":"89954a3d-f23a-482f-94e1-be82f4fe252f","BaseDeviceFilesystem":"xfs"}

To-be : {"next_device_id":1,"BaseDeviceUUID":"e298c53b-84bd-4e00-af3c-6be157514ac1","BaseDeviceFilesystem":"xfs"}