Logging and Monitoring

It’s important to be able observe your web application at runtime, so you can detect issues as they occur and diagnose bugs. To do this effectively, you need to pay attention to how you implement logging and monitoring.

Logging refers to having an application write a record of each event that occurs to a file on disk. These “log files” can be read by administrators to analyse what the application was doing at a given point in time.

Web servers typically log a record of each HTTP request they handle, along with a timestamp, the URL and HTTP method, and the HTTP response code.

You should add logging statements to your code to record important events that occur. Each logging statement added to the log file should have a timestamp, and be traceable to a given code file and line number.

Logging packages allow you tag each log statement with a log level, indicating how significant the event is. This will allow administrators to filter out irrelevant rows form the log file by setting the appropriate configuration.

Log files from different services should be viewable at runtime, which means shipping them to a central log server. Log servers allow administrators to view consolidated log files in realtime, either via the command line or a dedicated web-console.

Log files should also be retained for as long as reasonably possible, since old log files can be analyzed to detect attacks or troubleshoot issues. In many industries, keeping old logs files is required by law, so be sure to back them up!

Be careful not to write confidential information like user passwords or personally identifiable information in log files in case an attacker gets hold of them. Store your log files in a secure location (preferably encrypted) for the same reason.

Log files are very verbose by their nature, so you should use monitoring to detect trends in your logged output. You can’t possibly read all of that data in its raw form!

Monitoring frequently measures key metrics for your web-server, like response times, throughput, server loads and memory usage. This can be rolled up into a monitoring dashboard to make it easy to diagnose the health of the website.

Monitoring can also be used to pick up unexpected or suspicious errors, which can often be an indicator of a cyber-attack. At the very least, you should be collecting records of errors that occur so you can fix any underlying bugs.

It’s not reasonable to expect administrators to observe your website 24/7, so unusual conditions picked up by your monitoring should trigger alerts via email, instant message or text. Make sure the alert “thresholds” are set correctly though, or else you will be getting team-members out of bed constantly!

Observe how the logs for this web-server can be used to detect an attempt to brute-force the login screen. Configure an alert that will cause an the administrator to be notified.

Finally, make sure you have a response plan to follow when alerts are raised! This should be a living document that includes trouble-shooting steps such as restarting servers or adjusting firewalls in response to the problem.

That’s a lot to keep on top of! Let’s break everything down in a more concrete fashion.

tail -f ./application.log
tail -f /var/log/apache2/access.log
tail -f ./application.log
user@localhost:$ ls *

logs-2019-01-01.zip   logs-2019-01-09.zip   logs-2019-01-17.zip  logs-2019-01-25.zip
logs-2019-01-02.zip   logs-2019-01-10.zip   logs-2019-01-18.zip  logs-2019-01-26.zip
logs-2019-01-03.zip   logs-2019-01-11.zip   logs-2019-01-19.zip  logs-2019-01-27.zip
logs-2019-01-04.zip   logs-2019-01-12.zip   logs-2019-01-20.zip  logs-2019-01-29.zip
logs-2019-01-05.zip   logs-2019-01-13.zip   logs-2019-01-21.zip  logs-2019-01-30.zip
logs-2019-01-06.zip   logs-2019-01-14.zip   logs-2019-01-22.zip  logs-2019-02-01.zip
logs-2019-01-07.zip   logs-2019-01-15.zip   logs-2019-01-23.zip  logs-2019-02-02.zip
logs-2019-01-08.zip   logs-2019-01-16.zip   logs-2019-01-24.zip  logs-2019-02-03.zip
tail ./secrets.log
12:59:41.2 INFO sessions.py:124 albert@gmail.com entered password "cowboy77"
12:59:41.8 INFO payments.py:429 User supplied credit card number 4012888888881881
12:59:43.2 INFO profile.py:403  User confirmed social security number is 212-33-1829
12:59:43.8 INFO sessions.py:124 bill@gmail.com entered password "mermaid9"
12:59:48.8 INFO payments.py:810 User supplied CVC code 819
import logging

def establish_session(user_id)
  logging.info("Establishing session for user: {}".format(user_id))

  user = find_user(user_id)

  if not user:
    logging.error("Unable to find user {}!".format(user_id))
    raise UserNotFoundException(user_id)


  logging.info("Established session for user: {}".format(user_id))
  logging.debug("Last logged in: {}".format(user.last_login_time))

  return user
tail -f ./application.log
12:59:41.2 INFO  sessions.py: Establishing session for user: 3892
12:59:41.8 ERROR sessions.py: Unable to find user 3892!
12:59:43.2 INFO  sessions.py: Establishing session for user: 94012
12:59:43.8 INFO  sessions.py:19 Established session for user: 94012
12:59:43.8 DEBUG sessions.py:21 Last logged in: 3 days ago
tail -f ./debug.log
tail -f ./info.log
tail -f ./warnings.log
tail -f ./application.log
Configure alert
Send an alert when log messages containing the text

occur more than time(s) per minute.