Log Analyzer and Report Generator using Shell Script

1. Scenario

You are a system administrator responsible for managing a network of servers. Every day, a log file is generated on each server containing important system events and error messages. As part of your daily tasks, you need to analyze these log files, identify specific events, and generate a summary report.

2. Task

Write a Bash script that automates the process of analyzing log files and generating a daily summary report. The script should perform the following steps:

1. Input: The script should take the path to the log file as a command-line argument.

2. Error Count: Analyze the log file and count the number of error messages. An error message can be identified by a specific keyword (e.g., "ERROR" or "Failed"). Print the total error count.

3. Critical Events: Search for lines containing the keyword "CRITICAL" and print those lines along with the line number.

4. Top Error Messages: Identify the top 5 most common error messages and display them along with their occurrence count.

5. Summary Report: Generate a summary report in a separate text file. The report should include:

   • Date of analysis

   • Log file name

   • Total lines processed

   • Total error count

   • Top 5 error messages with their occurrence count

   • List of critical events with line numbers

6. Optional Enhancement: Add a feature to automatically archive or move processed log files to a designated directory after analysis.

3. Tips

• Use grep, awk, and other command-line tools to process the log file.

• Utilize arrays or associative arrays to keep track of error messages and their counts.

• Use appropriate error handling to handle cases where the log file doesn't exist or other issues arise.

4. Sample Log File

A sample log file named sample_log.log has been provided in the same directory as this challenge file. You can use this file to test your script or sample_log.log

5. Script : log_analyzer.sh

#!/bin/bash

# Here we are showing how to use log file
if [ -z "$1" ]; then
  echo "Usage: $0 <path_to_log_file>"
  exit 1
fi

# Declare variable for first argument
log_file=$1
report_file="summary_report_$(date +%Y%m%d).txt" # Create file using custom timestamp

# Check if log file is exist or not
if [ ! -f "$log_file" ]; then
  echo "Log file not found!"
  exit 1
fi

# Counting lines in log file
total_lines=$(wc -l < "$log_file")
# Counting number of lines containing words ERROR & Failed
error_count=$(grep -cE "ERROR|Failed" "$log_file")
# Finds line include CRITICAL and line number using -n
critical_events=$(grep -n "CRITICAL" "$log_file")

# Declare an associative array to store error_messages
declare -A error_messages

# Read log file line by line
while IFS= read -r line; do
  if [[ "$line" =~ ERROR|Failed ]]; then
    error_message=$(echo "$line" | awk -F']' '{print $NF}')
    ((error_messages["$error_message"]++))
  fi
done < "$log_file"

# Sorting of errors
top_errors=$(for message in "${!error_messages[@]}"; do
  echo "${error_messages[$message]} $message"
done | sort -rn | head -n 5)

# Printing final outout data
echo "Date of analysis: $(date)" > "$report_file"
echo "Log file name: $log_file" >> "$report_file"
echo "Total lines processed: $total_lines" >> "$report_file"
echo "Total error count: $error_count" >> "$report_file"
echo "Top 5 error messages with their occurrence count:" >> "$report_file"
echo "$top_errors" >> "$report_file"
echo "List of critical events with line numbers:" >> "$report_file"
echo "$critical_events" >> "$report_file"

echo "Analysis complete. Report generated: $report_file"

6. Sample Log File : sample_log.log

Find out complete log file sample_log.log

7. Result : Summary Report

$ ./log_analyzer.sh /home/shuhari/linux/sample_log.log

Date of analysis: Fri 12 Jul 2024 07:53:03 PM IST
Log file name: /home/shuhari/linux/sample_log.log
Total lines processed: 2000
Total error count: 13
Top 5 error messages with their occurrence count:
12  - Unexpected exception causing shutdown while sock still open
1  - Unexpected Exception:
List of critical events with line numbers:

Connect me :

LinkedIn | GitHub

Follow me for more amazing content :)