LINUX SHELL - 로그 특정 문자 발생시 telegram 알림처리

서버를 운영하다보면 단순하게 서버 어플리케이션에 에러로그가 찍히고 문제 상황이 발생하는 경우도 있지만,

그렇지 않은 경우도 있다.

문제 발생 / 조치 계획

로그상의 특정 패턴의 문자열이 중복으로 발생시 에러 라고 판단해야 함.
모니터링이 필요하다.
sh 파일로 작성 후 crontab 에 등록.
상황 발생시 텔레그램으로 알림 처리.

로그명 : /APP/logs/2022-07-04/application.2022-07-04.log

uniqNo의 값이 중복으로 찍히면 에러 상황 판단하고, 이를 모니터링 하고 싶다.

[07-04 09:20:52.099] INFO [TCP Connection(4)-xxx.xx.12.xx] [application.routing] 81 - uniqNo:NO991842
[07-04 09:21:16.218] INFO [TCP Connection(6)-xxx.xx.3.xxx] [application.routing] 81 - uniqNo:NO991843
[07-04 09:21:49.644] INFO [TCP Connection(4)-xxx.xx.12.xx] [application.routing] 81 - uniqNo:NO991844
[07-04 09:22:24.768] INFO [TCP Connection(6)-xxx.xx.3.xxx] [application.routing] 81 - uniqNo:NO991845
[07-04 09:23:16.802] INFO [TCP Connection(6)-xxx.xx.3.xxx] [application.routing] 81 - uniqNo:NO991846
[07-04 09:23:26.547] INFO [TCP Connection(4)-xxx.xx.12.xx] [application.routing] 81 - uniqNo:NO991848
[07-04 09:23:27.686] INFO [TCP Connection(4)-xxx.xx.12.xx] [application.routing] 81 - uniqNo:NO991849
[07-04 09:23:35.030] INFO [TCP Connection(6)-xxx.xx.3.xxx] [application.routing] 81 - uniqNo:NO991850
[07-04 09:24:15.826] INFO [TCP Connection(4)-xxx.xx.12.xx] [application.routing] 81 - uniqNo:NO991851
[07-04 09:24:42.095] INFO [TCP Connection(4)-xxx.xx.12.xx] [application.routing] 81 - uniqNo:NO991852
[07-04 09:24:55.868] INFO [TCP Connection(4)-xxx.xx.12.xx] [application.routing] 81 - uniqNo:NO991853
[07-04 09:25:05.358] INFO [TCP Connection(6)-xxx.xx.3.xxx] [application.routing] 81 - uniqNo:NO991854
[07-04 09:26:29.145] INFO [TCP Connection(4)-xxx.xx.12.xx] [application.routing] 81 - uniqNo:NO991855
[07-04 09:26:30.044] INFO [TCP Connection(4)-xxx.xx.12.xx] [application.routing] 81 - uniqNo:NO991856
[07-04 09:26:55.030] INFO [TCP Connection(4)-xxx.xx.12.xx] [application.routing] 81 - uniqNo:NO991857
[07-04 09:27:43.128] INFO [TCP Connection(6)-xxx.xx.3.xxx] [application.routing] 81 - uniqNo:NO991858
[07-04 09:28:10.808] INFO [TCP Connection(6)-xxx.xx.3.xxx] [application.routing] 81 - uniqNo:NO991859
[07-04 09:28:10.808] INFO [TCP Connection(6)-xxx.xx.3.xxx] [application.routing] 81 - uniqNo:NO991859
[07-04 09:28:26.989] INFO [TCP Connection(4)-xxx.xx.12.xx] [application.routing] 81 - uniqNo:NO991860
[07-04 09:29:15.464] INFO [TCP Connection(6)-xxx.xx.3.xxx] [application.routing] 81 - uniqNo:NO991861
[07-04 09:29:27.169] INFO [TCP Connection(4)-xxx.xx.12.xx] [application.routing] 81 - uniqNo:NO991862
[07-04 09:29:31.120] INFO [TCP Connection(6)-xxx.xx.3.xxx] [application.routing] 81 - uniqNo:NO991863
[07-04 09:29:49.582] INFO [TCP Connection(4)-xxx.xx.12.xx] [application.routing] 81 - uniqNo:NO991864
[07-04 09:30:47.767] INFO [TCP Connection(6)-xxx.xx.3.xxx] [application.routing] 81 - uniqNo:NO991865
[07-04 09:32:34.547] INFO [TCP Connection(6)-xxx.xx.3.xxx] [application.routing] 81 - uniqNo:NO991867
[07-04 09:33:28.675] INFO [TCP Connection(6)-xxx.xx.3.xxx] [application.routing] 81 - uniqNo:NO991868
[07-04 09:33:32.764] INFO [TCP Connection(4)-xxx.xx.12.xx] [application.routing] 81 - uniqNo:NO991869
[07-04 09:34:15.654] INFO [TCP Connection(4)-xxx.xx.12.xx] [application.routing] 81 - uniqNo:NO991870
[07-04 09:34:39.728] INFO [TCP Connection(6)-xxx.xx.3.xxx] [application.routing] 81 - uniqNo:NO991871
[07-04 09:35:02.884] INFO [TCP Connection(4)-xxx.xx.12.xx] [application.routing] 81 - uniqNo:NO991872
[07-04 09:35:48.667] INFO [TCP Connection(4)-xxx.xx.12.xx] [application.routing] 81 - uniqNo:NO991873
[07-04 09:35:51.966] INFO [TCP Connection(6)-xxx.xx.3.xxx] [application.routing] 81 - uniqNo:NO991874
[07-04 09:35:55.528] INFO [TCP Connection(6)-xxx.xx.3.xxx] [application.routing] 81 - uniqNo:NO991875
[07-04 09:37:26.379] INFO [TCP Connection(6)-xxx.xx.3.xxx] [application.routing] 81 - uniqNo:NO991876
[07-04 09:37:40.887] INFO [TCP Connection(4)-xxx.xx.12.xx] [application.routing] 81 - uniqNo:NO991877
[07-04 09:38:36.148] INFO [TCP Connection(6)-xxx.xx.3.xxx] [application.routing] 81 - uniqNo:NO991878
[07-04 09:39:21.100] INFO [TCP Connection(6)-xxx.xx.3.xxx] [application.routing] 81 - uniqNo:NO991879
[07-04 09:39:27.040] INFO [TCP Connection(6)-xxx.xx.3.xxx] [application.routing] 81 - uniqNo:NO991880
[07-04 09:40:21.986] INFO [TCP Connection(6)-xxx.xx.3.xxx] [application.routing] 81 - uniqNo:NO991882
[07-04 09:41:06.732] INFO [TCP Connection(6)-xxx.xx.3.xxx] [application.routing] 81 - uniqNo:NO991883
[07-04 09:42:34.607] INFO [TCP Connection(4)-xxx.xx.12.xx] [application.routing] 81 - uniqNo:NO991884
[07-04 09:42:49.816] INFO [TCP Connection(4)-xxx.xx.12.xx] [application.routing] 81 - uniqNo:NO991885
[07-04 09:43:08.745] INFO [TCP Connection(6)-xxx.xx.3.xxx] [application.routing] 81 - uniqNo:NO991886
[07-04 09:44:28.117] INFO [TCP Connection(4)-xxx.xx.12.xx] [application.routing] 81 - uniqNo:NO991887
[07-04 09:44:28.356] INFO [TCP Connection(6)-xxx.xx.3.xxx] [application.routing] 81 - uniqNo:NO991888
[07-04 09:44:46.477] INFO [TCP Connection(4)-xxx.xx.12.xx] [application.routing] 81 - uniqNo:NO991889
[07-04 09:45:10.091] INFO [TCP Connection(4)-xxx.xx.12.xx] [application.routing] 81 - uniqNo:NO991890
[07-04 09:45:26.995] INFO [TCP Connection(4)-xxx.xx.12.xx] [application.routing] 81 - uniqNo:NO991891
[07-04 09:45:43.718] INFO [TCP Connection(6)-xxx.xx.3.xxx] [application.routing] 81 - uniqNo:NO991892
[07-04 09:46:37.418] INFO [TCP Connection(4)-xxx.xx.12.xx] [application.routing] 81 - uniqNo:NO991866
[07-04 09:47:40.944] INFO [TCP Connection(6)-xxx.xx.3.xxx] [application.routing] 81 - uniqNo:NO991900
[07-04 09:48:28.150] INFO [TCP Connection(6)-xxx.xx.3.xxx] [application.routing] 81 - uniqNo:NO991893
[07-04 09:48:45.009] INFO [TCP Connection(4)-xxx.xx.12.xx] [application.routing] 81 - uniqNo:NO991894
[07-04 09:48:56.017] INFO [TCP Connection(6)-xxx.xx.3.xxx] [application.routing] 81 - uniqNo:NO991895
[07-04 09:49:14.282] INFO [TCP Connection(4)-xxx.xx.12.xx] [application.routing] 81 - uniqNo:NO991896
[07-04 09:49:14.550] INFO [TCP Connection(6)-xxx.xx.3.xxx] [application.routing] 81 - uniqNo:NO991897
[07-04 09:49:14.906] INFO [TCP Connection(4)-xxx.xx.12.xx] [application.routing] 81 - uniqNo:NO991898
[07-04 09:49:14.911] INFO [TCP Connection(4)-xxx.xx.12.xx] [application.routing] 81 - uniqNo:NO991903
[07-04 09:49:14.920] INFO [TCP Connection(4)-xxx.xx.12.xx] [application.routing] 81 - uniqNo:NO991899
[07-04 09:49:14.923] INFO [TCP Connection(6)-xxx.xx.3.xxx] [application.routing] 81 - uniqNo:NO991902
[07-04 09:49:14.959] INFO [TCP Connection(6)-xxx.xx.3.xxx] [application.routing] 81 - uniqNo:NO991905
[07-04 09:49:14.962] INFO [TCP Connection(6)-xxx.xx.3.xxx] [application.routing] 81 - uniqNo:NO991901
[07-04 09:49:14.970] INFO [TCP Connection(6)-xxx.xx.3.xxx] [application.routing] 81 - uniqNo:NO991904
[07-04 09:49:14.978] INFO [TCP Connection(4)-xxx.xx.12.xx] [application.routing] 81 - uniqNo:NO991906
[07-04 09:49:15.248] INFO [TCP Connection(6)-xxx.xx.3.xxx] [application.routing] 81 - uniqNo:NO991907
[07-04 09:49:26.695] INFO [TCP Connection(4)-xxx.xx.12.xx] [application.routing] 81 - uniqNo:NO991908
[07-04 09:50:06.954] INFO [TCP Connection(6)-xxx.xx.3.xxx] [application.routing] 81 - uniqNo:NO991909
[07-04 09:50:12.690] INFO [TCP Connection(6)-xxx.xx.3.xxx] [application.routing] 81 - uniqNo:NO991910
[07-04 09:51:35.284] INFO [TCP Connection(4)-xxx.xx.12.xx] [application.routing] 81 - uniqNo:NO991911
[07-04 09:52:22.936] INFO [TCP Connection(6)-xxx.xx.3.xxx] [application.routing] 81 - uniqNo:NO991912
[07-04 09:52:42.404] INFO [TCP Connection(4)-xxx.xx.12.xx] [application.routing] 81 - uniqNo:NO991913

SHELL 파일 : app_monitoring.sh 작성

#!/bin/sh
LOG_HOME=/APP/logs
NOW=$(date +%Y-%m-%d)
#NOW=2022-07-01
TIME_STAMP=$(date "+%Y-%m-%d %T")
ROUTE_LOG=$(cat ${LOG_HOME}/${NOW}/application.${NOW}.log | grep 'application.routing' | awk '{print $9}' | uniq -d | uniq)
# cat : 으로 application.log 를 읽는다.
# grep : 명령어를 통해 application.routing 라인을 찾는다.
# awk : 를 통해 9번째 uniqNo:NOxxxxxx를 모두 찾는다.
# uniq -d :  이중에 중복 값만 필터링 한다.
# uniq : 중복값 하나만 출력 


if [ -n "$ROUTE_LOG" ]; then 
# 값이 있는지 체크한다.
	echo "${TIME_STAMP} $ROUTE_LOG"
    # curl 명령어를 통해 텔레그램 API 호출하여, 알림 메세지 전송
    curl -k https://api.telegram.org/bot{BOT_KEY}/sendMessage -d "chat_id=-1234567890" --data-urlencode "text=ROUTE CHECK : $ROUTE_LOG"
else
	echo "$TIME_STAMP ROUTER LOG CHECK OK!"
fi

어떤식으로 알림을 받을까??

고민중에 텔레그램으로 curl 명령어를 통한 전송이 제일 간단해보여서 이렇게 알림 처리를 했다.

linux crontab 등록

*/5 9-18 * * * /APP/bin/app_monitoring.sh  >> /APP/logs/app_monitoring.log 2>&1

1. crontab -e 실행

2. */5 9-18 * * * : 매일 9시부터 18시까지 5분 단위로 실행

3. /APP/logs/app_monitoring.log : 해당 위치에 로그를 남김

원천적인 문제를 해결하기 위해서는 application을 수정하여 버그를 없애는것이 원칙이겠다.

하지만, 당장 조치가 어렵고 운영관점에서는 이를 못본척 할수 없으므로 위와 같은 선제적 알림 처리가 필요했다.

운영서버에 적용하는데 반나절도 안걸렸다는게 핵심이긴 하다.

curl 명령어를 통해 텔레그램에 메세지를 전송하는 방법은 별도로 정리를 하였다.

초간단 curl 명령어를 통해 telegram bot에 메세지 전송

초간단 curl 명령어를 통해 telegram bot에 메세지 전송 가장 이해를 쉽게 하기 위해 캡쳐위주로 정리 한다. 1. 텔레그램 봇 생성 telegram 에 bot father 검색하여 해당 채팅방에 들어간다. 1) /new..

www.appletong.com

저작자표시

애플통 | APPLE TONG

LINUX SHELL - 로그 특정 문자 발생시 telegram 알림처리