[Linux] ShellScript를 이용하여 Application Helth Check 및 프로세스 체크 해서 재기동 하기

작업 사유

 

회사에서 작동 시키던 배치 Application 이 프로세스는 살아 있는데 작동을 안하는 바람에 문제가 좀 생겼다.

그래서 배치 서비스 확인 해서 상황에 따라 자동으로 활성화 하도록 처리하였다.

 

 

생성 스크립트

 

- 배치서비스에 healthCheck 가능한 api 추가

    > 각 배치들의 마지막 작동 시간을 DB Select  후 '년월일분'까지 숫자형태로 리턴하도록 되어 있음

    > healthCheck url 호출 connection timeout 은 10초로 설정

- healthCheck 조건

    > healthCheck 시간이 공백이거나 문자면 오류로 판단 하여 Application 재시작 펑션 호출

    > healthCheck 시간이 10분 전 시간보다 크면 이슈가 있다고 판단하고 Application 재시작 펑션 호출

- Application 재시작 펑션 조건

    > Application 프로세스 체크하여 있다면 강제 kill 후 시작 시작 스크립트 호출

    > Application 프로세스 체크하여 없다면 시작 스크립트 호출

- ShellScript 작성

    # vi /var/local/application_sh/application_health_check.sh

echo ""
echo "####### Application Health Check Start [`date '+%Y-%m-%d %H:%M:%S'`] ########"
echo "# "

CURRENT_DATE=`date '+%Y%m%d%H%M'`
echo "# Current Date => $CURRENT_DATE"

TARGET_DATE=`date '+%Y%m%d%H%M' -d '10 minute ago'`
echo "# 10 minute ago => $TARGET_DATE"

HEALTHCHECK_DATE=`curl --connect-timeout 10 http://127.0.0.1:9000/healthCheck`
echo "# HealthCheckDate=> $HEALTHCHECK_DATE"
echo "# "

function batch_restart() {
        echo "# Application_batch restart !!  "
        echo "# "
        echo "# - APPLICATION COUNT"
        APPLICATION_BATCH_COUNT=`ps -ef | grep application_batch.jar | grep -v grep | wc -l`
        echo "#   > APPLICATION_BATCH_COUNT => $APPLICATION_BATCH_COUNT"
        echo "# "

        if [ "$APPLICATION_BATCH_COUNT" == 0  ]; then
                echo "#   > NOT FOUND APPLICATION_BATCH"
                /var/local/application_batch/bin/startup.sh
                echo "#   > STARTUP !!"
        else
                echo "#   > FOUND APPLICATION_BATCH"
                echo "# - APPLICATION_PID"
                APPLICATION_BATCH_PID=`ps -ef | grep application_batch.jar | grep -v grep | awk '{print $2}'`
                echo "#   > APPLICATION_BATCH_PID => $APPLICATION_BATCH_PID"
                echo "# "

                kill -9 $APPLICATION_BATCH_PID
                echo "#   > kill -9 $APPLICATION_BATCH_PID"
                /var/local/application_batch/bin/startup.sh
                echo "#   > STARTUP !!"
        fi
}

if [[ $HEALTHCHECK_DATE != "" ]]; then
        echo "# Health Check Date is Not Empty => $HEALTHCHECK_DATE"

        if [[ -z ${HEALTHCHECK_DATE//[0-9]/} ]]; then
                echo "# Health Check Date is number => $HEALTHCHECK_DATE"

                if [ "$TARGET_DATE" -le "$HEALTHCHECK_DATE"  ]; then
                        echo "# Health Check Not Problem !!"

                else
                        echo "# Health Check Problem !!"
                        batch_restart
                fi

        else
                echo "# Health Check Date is not number"
                batch_restart
        fi

else
        echo "# Health Check Date is Empty => $HEALTHCHECK_DATE"
        batch_restart

fi

echo "# "
echo "####### Application Health Check End [`date '+%Y-%m-%d %H:%M:%S'`] ########"
echo " "

 

 

로그 저장 스크립트

- ShellScript 작성

    # vi /var/local/application_sh/application_health_check.sh

/var/local/application_sh/application_health_check.sh >> /var/local/application_sh/logs/application_health_check.log.`date +%Y%m%d` 2>&1

 

Crontab 등록

- Crontab 편집기로 내용 입력

    # crontab -e

# 매 시간 15분, 45분 application batch 헬스 체크
15,45 * * * * /var/local/application_sh/sociallink_health_check_log_save.sh

 

로그내용

 

- 이슈 발생한 경우

####### Application Health Check Start [2021-05-18 11:45:01] ########
#
# Current Date => 202105181145
# 10 minute ago => 202105181135
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
^M  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0^M  0    12    0    12    0     0   5725      0 --:--:-- --:--:-- --:--:-- 12000
# HealthCheck => 202105181115
#
# Health Check Date is Not Empty => 202105181115
# Health Check Date is number => 202105181115
# Health Check Problem !!
# Application_batch restart !!
#
# - APPLICATION COUNT
#   > APPLICATION_BATCH_COUNT => 1
#
#   > FOUND APPLICATION_BATCH
# - APPLICATION_PID
#   > APPLICATION_BATCH_PID => 14183
#
#   > kill -9 14183
#   > STARTUP !!
#
####### Application Health Check End [2021-05-18 11:45:01] ########

 

- 이슈 없는 경우

####### Application Health Check Start [2021-05-18 12:15:01] ########
#
# Current Date => 202105181215
# 10 minute ago => 202105181205
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
^M  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0^M  0    12    0    12    0     0     39      0 --:--:-- --:--:-- --:--:--    39^M  0    12    0    12    0     0     39      0 --:--:-- --:--:-- --:--:--    39
# HealthCheck => 202105181215
#
# Health Check Date is Not Empty => 202105181215
# Health Check Date is number => 202105181215
# Health Check Not Problem !!
#
####### Application Health Check End [2021-05-18 12:15:01] ########

 

 

오늘도 이슈가 발생했었는데 다행히 잘 돌았다. ㅋㅋ