Monitor Linux services State

Azure offers us several possibilities to collect data about the state of the services. Firstly, we can collect data from logs indicating which routes you should consult. Secondly, Azure allows us to obtain the logs collected in Windows events and in the Linux Syslog. In this publication, we will focus on the first option.

To begin with, we must create a script that monitors the state of the services and stores the result in a log file. For Azure to be able to collect the information the log file must be in a suitable format. This time the logs we generate will have the following format: one entry per line, each line will start with the date YYYY-MM-DD HH:MM:SS, each field of a line will be separated by a comma and the encoding will be UTF-8. More information about the allowed formats can be found in the official documentation. In both Windows and Linux scripts, the services to be monitored must be modified. Once the script has been modified, it must be added to the task scheduler or cron to make it run every 4 minutes daily. To check that it is running correctly we can go to the route of the logs and see the results. 

Windows :  
 
function writeLog ($logPath, $logFile, $service, $status, $message){ 
$datetime = Get-Date -Format "yyyy-MM-dd HH:mm:ss" 
 
Add-Content -path "$logPath\$logFile" -value "$datetime, Service:$service, Status:$status, Message:$message" -Encoding "utf8" 
} 
 
## CREATE LOG PATH WHICH WILL CONTAIN THE LOG FILES ## 
$logPath = "C:\logs" 
if (!(Test-Path "$logPath")) { 
   New-Item -path $logPath -type "directory" -Force 
}  
 
## CREATE LOG FILE FOR SERVICES if not exist ## 
$date = Get-Date -Format "yyyyMMdd" 
$logPath = "C:\logs" 
$servicelogFile = "services-$date.log" 
if (!(Test-Path "$logPath\$servicelogFile")) 
{ 
   New-Item -path $logPath -name $servicelogFile -type "file" -Force 
}  
 
## SERVICES TO MONITOR ## 
$servicesArray = @('W3SVC','SQLTELEMETRY$SQLEXPRESS','MSSQL$SQLEXPRESS', 'SQLWriter') 
 
## SERVICE STATUS ## 
if ( $servicesArray -ne $null ){ 
foreach ($service in $servicesArray) { 
$serviceStatus = get-service $service -ErrorAction SilentlyContinue 
 
$message = "Running" 
$status = "Success" 
if ($serviceStatus.Status -ne "Running"){ 
$message = $serviceStatus.Status 
$status = "Error" 
} 
writeLog $logPath $servicelogFile $service $status $message 
} 
} 
 
## DELETE LOG FILES OF YESTERDAY ## 
$yesterday = (Get-Date).AddDays(-1).ToString('yyyyMMdd') 
$servicelogFile = "services-$yesterday.log" 
if (Test-Path "$logPath\$servicelogFile") { 
  Remove-Item "$logPath\$servicelogFile" 
} 
 
 
Linux :  
 
 
## CREATE VARS FOR LOG FILES ## 
dateStr=$(date +"%Y%m%d") 
logPath=/var/log/azuremon 
serviceLogFile="services-$dateStr.log" 
errorLogFile="error.log" 
datetime=$(date +"%Y-%m-%d %H:%M:%S") 
 
## CREATE LOG PATH WHICH WILL CONTAIN THE LOG FILES ## 
if [ ! -d $logPath ]; then 
mkdir -p $logPath 
fi 
 
## SERVICES TO MONITOR ## 
servicesArray=(nginx mysqld) 
 
## SERVICE STATUS ## 
if [ ${servicesArray+x} ];then 
for service in "${servicesArray[@]}"; do 
serviceStatus=$(systemctl status $service | grep Active | cut -d ":" -f 2) 
 
message="Running" 
sstatus="Success" 
if [[ $serviceStatus != *"running"* ]];then 
message=$serviceStatus 
if [ -z "$serviceStatus" ]; then 
message="$service service cannot be found" 
fi 
sstatus="Error" 
fi 
datetime=$(date +"%Y-%m-%d %H:%M:%S") 
echo "$datetime, Service:$service, Status:$sstatus, Message:$message" >> $logPath/$serviceLogFile 
done 
fi 
 

## DELETE LOG FILES OF YESTERDAY ## 
yesterday=$(date -d "yesterday" '+%Y%m%d') 
serviceLogFile="services-$yesterday.log" 
if [ -e $logPath/$serviceLogFile ];then 
rm -f $logPath/$serviceLogFile 
fi

Once the scripts are saving information about the state of the services in the log, you have to configure the workspace in Azure so that it goes to the log paths to collect the information. When adding a new route, we will have to follow 4 simple steps. Upload an example log file. As we have previously left the script running, we will already have several log files available for this upload. 

Choose the delimiter of each log. We will select Timestamp with format YYYY-MM-DD HH:MM:SS, this way the logs in Azure will have the time of execution of the script and not the time of collection of Azure.
Indicate the path from where to take the logs. In our case: “C:\logs\services-*.log” in Windows and “/var/log/azuremon/services-*.log” in Linux.

Give a name to the table that will store the data. For example: Services_CL.