Endian banner

The Logs and Reports Menu

In the logs and reports section of the Endian UTM Appliance there are different possibilities to look at and to analyse the log files.

The sub-menu on the left-hand side of the screen contains the following items:

  • Dashboard - the reporting module, a graphical depiction of log files and events.
  • Traffic monitoring - the ntopng graphic interface gives a real time overview of the network traffic using charts.
  • Live Logs - get quick, live view of the latest log entries as they are being generated.
  • Summary - get daily summaries of all logs.
  • System - system logs (/var/log/messages) filtered by source and date.
  • Service - logs from the intrusion detection system (IDS), OpenVPN, and antivirus.
  • Firewall - logs from iptables rules.
  • Proxy - logs from the HTTP, SMTP, and content filter proxies.
  • Settings - customise all the log options.
  • Trusted Timestamping - securely time stamp the log files to verify they have not been altered.

In a nutshell, there are three modalities to access the log from the GUI: Graphically, live and by-service. The first is represented by the reporting module, while in the live mode the log files are visualised as soon as they are created, and in the by-service mode only the logs produced by one daemon or service at a time are displayed.

Dashboard

The reporting GUI module has the purpose to graphically show the occurrence of various types of event on the system.

In a nutshell, the reporting module shows events happened on the Endian UTM Appliance using different widgets and graphs. All events occurring on the system and the information concerning them recorded by the syslog daemon are parsed and used to populate a sqlite3 database. From here, data are gathered according to the options and to the filters applied in the GUI and displayed to the user.

This page is divided into six tabs: Summary, System, Web, Mail, Intrusion attempts, Viruses, and Connections. Except for the first tab, which shows an overview of all events, each of them is dedicated to one service running on the Endian UTM Appliance.

Common elements

All the tabs in this module share the same design: Below the tabs, on the left-hand side there are a date selector on the the left-hand side and a Print button on the right-hand side. Then, a line chart at with an horizontal slider right below, atop one informative boxes (Summary Grid) and a pie-chart. At the bottom, there are one or more tables, depending on the tab and the data shown. The table that is always present is the one displaying the syslog messages related to the events shown.

More in detail, here is a description of all the widget present in the reporting module.

Date selector
At the top left-hand side of the GUI there is an hyperlink that shows the interval within which occurred those events that have been considered for the charts. By clicking on it, a small panel gives access to other choices of intervals. There are two types of choices, the first one concerns events that took place in the last ... days, namely events from the last day, week month, quarter, or year; the second one selects all the events occurred in one of the last 12 months. Upon selecting a new time span, the other widgets are also updated. There is also the possibility to not change the interval shown, by clicking on Cancel.
Print
A click on this button shows a print preview of the current page.

Line Chart and Time Slider

The line chart shows the event happened on the Endian UTM Appliance during the selected time span in a two dimensional graph, in which the x-axis shows the time interval and the y-axis shows the number of occurrences. A coloured line connects events of the same type.

Hint

Different types of event are denoted with different colours.

The time slider is located underneath the chart and allows, within the selected time span, a more fine-grained view of the events, depicted here as histograms. Indeed, the two grey handles on the left and right limits of the slider can be clicked and dragged to reduce the time span shown in the line chart. When reduced, the slider can also be moved by clicking in its middle and dragging it to the left or the right.

Summary Grid

The summary grid has a twofold purpose: On the one side to show the number of occurrences of the various types of events that took place on the Endian UTM Appliance in the selected period, on the other side to filter the type of events shown in the line chart. Its content depends on the tabs it is located in and is not present in the Mail, Intrusion attempts, and Viruses tabs, in which is replaced by tables which group different details about the events.

Pie Chart

The pie chart diagram shows graphically the number of event that took place in the selected time span. When in the Summary tab, each slice can be clicked, to open the tab corresponding to the type of event and show a more detailed representation.

Syslog table

A table that shows the syslog messages extracted from the log files and related to the events shown in the charts. When the table carries lot of messages, these are divided into many pages and can be browsed using the buttons and number at its left bottom. At the right bottom there is an icon that allows to refresh the table’s content.

Summary

The Summary tab gives an overview of all categories of events recorded on the Endian UTM Appliance. The summary grid allows to filter the following types of events:

  • System. The number of Log ins and other events connected with system administration tasks (e.g., uplinks change of status, start and stop of logging, and so on) .
  • Web. The number of pages blocked by the content filter.
  • Mail. The number of spam e-mail received.
  • Intrusion attempts. The events recorded by the IPS.
  • Viruses. The number of viruses found.

Each category can be shown separately, with more information and a higher level of details in the other tabs of the page, see further on.

System

The System tab displays all events that are related to the system efficiency and to system administration. These are all the events shown:

  • Login. The number of logins, both successful and not.
  • Status. The changes in the state of the Endian UTM Appliance.
  • Disk. The events involving disk I/O.
  • Support. Number of accesses and operations donw by the Support Team.
  • Upgrade. Events involving upgrade of system or of packages.
  • Uplink. The times the uplink(s) went online or offline.

A click on the small icon on the left-hand side of each event category shows details about the events that are part of that category and updates the pie chart.

Web

The Web tab displays the number of pages that have been accessed or blocked by the URL filter engine. The summary grid is composed by two tabs: Access report and Filter report.

Access report

This tab shows the domains that have been accessed, grouped into three tables showing respectively the Source IP Address recorded by the HTTP Proxy, the Domain accessed, and the Users who requested the web page, with the total count for every item.

Note

The Access report tab is not present in all appliances.

Filter report

This tab shows to which domains the access has been blocked. In the first table, the following categories are shown, that are those found in the Web Filter (See Menubar ‣ Proxy ‣ HTTP ‣ Web Filter).

  • General Use.
  • Parental Control.
  • Productivity.
  • Security.
  • Uncategorized Sites.

When the checkboxes on the left-hand side of the Blocked category are not ticked, the items in that category are not used to draw the diagram and the pie chart. A click on the icon on the left of the checkbox will expand the category and show all its sub-categories, that can be selected or not, to draw more detailed diagrams.

The other tables at the bottom show the counts of each the blocked objects: The Source IP Addresses and the Domain.

Mail

The Mail tab displays all e-mails blocked as spam.

There is no summary grid in this tab, but three tables displaying counts for:

  • From. The sender(s) of spam e-mails.
  • To. The recipient(s) of spam e-mails.
  • Source IP Address. The IP address from where spam e-mail have been sent.

Intrusion attempts

The Intrusion attempts tab displays all tentative intrusions detected by the IPS (See Menubar ‣ Services ‣ Intrusion Prevention).

The tables at the bottom show counts of the following information:

  • Intrusion attempts: The categories under which falls each attempt.
  • Source IP Address. The IP address from where the attack originated.
  • Destination IP Address. The IP Address to which the attach was launched.

Viruses

The Viruses tab displays all viruses intercepted by the anti-virus engine (See Menubar ‣ Services ‣ Antivirus Engine).

The tables at the bottom show counts of the following information:

  • Virus Name. The name of the virus found.
  • Source IP Address. The IP address where the virus originally was located.
  • Destination IP Address. The IP Address to which the virus was propagated.

Connections

The Connections tab displays the average number of connections started by the users of the Endian UTM Appliance, grouped into:

  • Local connections. Accesses via SSH or console.
  • IPsec users. Clients connected via IPsec.
  • Hotspot users. Users accessing the Hotspot.
  • OpenVPN users. Clients connected using OpenVPN.

Traffic Monitoring

The ntopng software is the successor of the ntop network traffic analyser, which adds a more intuitive interface and more graphical representations of the traffic that flows through the Endian UTM Appliance.

The management interface of ntopng provides now more usability and can be accessed easily accessed from any browser, and therefore has been integrated more tightly with the Endian UTM Appliance interface than in previous versions.

In few words, the abilities of ntopng can be summarised as follows:

  • Real time monitoring of every network interface of the Endian UTM Appliance.
  • Web-accessible management interface.
  • Less resource needed compared to ntop.
  • Integration of nDPI (Application firewall).
  • Traffic analysis according to different parameters (protocol, source/destination).
  • Export of reports in JSON format
  • Storage of traffic statistics on disk.

The ntopng GUI is organised into four tabs: Dashboard, Flows, Hosts, and Interfaces. Moreover, there is also a search box to quickly display information about a given host.

In the footer of each tab, a couple of information are shown: Besides a copyright notice and a link to the ntop home page, there is a chart showing the network traffic over the last 20 seconds, updated in real time, and some numerical data about the current bandwidth used, the number of hosts and flows and the Endian UTM Appliance‘s uptime.

Dashboard

The dashboard shows all connections that interest the Endian UTM Appliance, that is, all established Flows in which the Endian UTM Appliance is involved.

The page is divided into several diagrams, with the first one -a so-called Sankey diagram showing all flows moving on the Endian UTM Appliance, updated in real time. The horizontal flows show the traffic between two hosts, while the vertical width of each flows is proportional to the bandwidth used by that flows, i.e., to the amount of data flowing. The connections -and therefore the direction of the data sent- are shown left to right: Hosts on the left hand-side of the diagram send data to hosts on the right-hand side and are identified by either their IP address or hostname. A click on one host leads to the Overview page in the Hosts tab, which shows several information about that host.

Below the Sankey diagram, four informative-only pie charts show in percentage the items that that generate the most traffic, divided into: Total by host (top left); application protocols (top right), ASNs (bottom left), and live flow senders (bottom right).

Flows

The active flows tab contains a big table with a number of information about the active flows:

  • Info. A click on the icon opens a new page in which more detailed information about that flow is shown.

  • Application. The application causing the flow. nDPI is used to recognise the application, therefore it might be necessary to wait for a couple of packets to see the correct application displayed: In this case, the (Too Early) message appears instead of the application name.

  • L4 Proto The network protocol used by the flow, which is usually TCP or UDP.

  • Client. The hostname and port used by the flow on the client side. Clicking on either the hostname or port, more information will be shown in a new page about the network traffic flowing that host or port.

  • Server. The hostname and port used by the flow on the server side. Like for the Client above, more information is shown when clicking on the hostname or port.

    Hint

    By clicking on the hostname or port, the table shows detailed information about it, opening a sub-tab in the Hosts tab.

  • Duration. The length of the connection.

  • Breakdown. The percentage of traffic generated by the client and by the server.

  • Throughput. The amount of data currently exchanged between the client (on the left, in black) and server (on the right, in green).

  • Total Bytes. The total data exchanged since the connection was first established.

At the bottom of the table, on the left-hand side it is shown the total number of rows shown , while on the right-hand side it is possible to browse the various pages in which the table is split, when the number of rows is higher that the pagination.

A click on the Info icon will give detailed information about that particular flow. Besides those already described above, these additional data are displayed.

  • First Seen. The timestamp when the connection was established, along with the time passed since.
  • Last Seen. The timestamp in which the connection was last active and the time passed since that moment.
  • Client to Server Traffic. The number of packets and bytes sent from the client to the server.
  • Server to Client Traffic. The number of packets and bytes sent from the server to the client.
  • TCP Flags. The TCP states of the current flow.

It is possible to go back to the list of flows by clicking on the Flows hyperlink on the left, right above the table.

Hosts

The Hosts tab allows to view several details about the involved parties of a flow: Host, port, application, flows and their duration, data exchanged, and so on.

Two representation are available: Host List and Top Hosts (Local)

The Hosts List representation shows information about all the hosts involved in some flow with the Endian UTM Appliance and the following data about them:

  • IP Address. The IP address or MAC Address of the host. The latter is shown if the DHCP lease for that host has expired.
  • Location. Whether the host is in the local or in a remote network.
  • Symbolic Name. If available, it is the hostname of the host.
  • Seen Since. The timestamp of the first established connection.
  • ASN.
  • Breakdown. The trade-off between sent and received traffic.
  • Traffic The amount of data exchanged by the host.

A click on the IP address opens an overview of the host, showing several information about it, besides those listed above:

  • Last Seen. The timestamp in which the connection was last active and the time passed since that moment.
  • Sent vs Received Traffic Breakdown. The traffic generated or received by the host.
  • Traffic Sent. The number of packets and bytes sent from the client to the server.
  • Traffic Received. The number of packets and bytes sent from the server to the client.
  • JSON. Download information about the host in JSON format.
  • Activity map. How many flows have seen the host involved at a given timestamp. Each square shows a minute and the darker the colour, the more flows have taken place in that minute.

From here it is also possible to open additional informative tabs about that host. Each tabs contains one or more pie charts (except for the Contacts and Historical tabs) above a textual summary of the data displayed.

  • Traffic. The network protocol used by the host. (TCP, UDP and ICMP being the most common).
  • Packets. The length in packets of each flow. (note: just my guess)
  • Protocols. The application protocol used by the host.
  • Flows. The table with all the network flows from the hosts.
  • Talkers. The Sankey diagram of the connections, very similar to the one shown in the Dashboard, which however shows only the most active flows.
  • Contacts. This tab is slight different from the others. It shows on top an interaction maps and on the bottom a list of connection that have the host as client or receiver.
  • Historical. An interactive graph that shows the history of the
    traffic flown form and to the host in a given timespan (up to one year), that can be selected above the graph.

The Top Hosts (Local) representation shows a real-time graphic of the hosts that have active connections to the host. It displays the last 30 minutes.

Interfaces

The Interfaces tab allow to select the network interface, among the active ones, whose traffic should be displayed.

Note

It is currently not possible to select flows and/or hosts from different interfaces

Live

When entering in the Live Logs section, a box is shown, containing the list of all the log files available for real time viewing. Any number of logs to see can be chosen by ticking the corresponding checkboxes and displayed in a new window upon clicking on the Show selected logs button.

IT is also possible to watch all the log files at once, by ticking the Select all bottom checkbox and click on the Show selected logs button.

Single log files can be viewed by simply clicking on the Show this log only link on the right-hand side of the list.

The window that opens contains two boxes, Settings at the top and Live logs at the bottom.

Warning

The list of log entries can become nearly unreadable if many logs are showed, due to the possible high number of log entries produced (especially by the firewall or proxy log, which can generate several log entries per second in case of heavy traffic). In this cases, the logs to be displayed can be configured in the Settings box.

Settings

This box allows to modify the settings of the log viewer, including which of the log files to show, their colour and options to highlight or find specific keywords.

On the right-hand side of the box appears the list of the logs that are currently displayed, and the colour with which they are highlighted, while on the left-hand side some additional control elements are shown, that help limit the output:

Filter
Only the log entries that contain the expression in this field are shown.
Additional filter
Like the filter above, but applied to the output of the first filter. In other words, only log entries containing both expressions are shown in the log.
Pause output
Clicking on this button will prevent new log entries from appearing on the live log. However, after clicking the button once more, all new entries will appear at once, quickly scrolling the old ones.
Highlight
All the log entries that contain this expression will be highlighted in the chosen colour. The difference with the filtering option is that all the content is still displayed and the log entries containing the expression will be highlighted with a coloured background.
Highlight color
Clicking on the coloured square gives the choice to select the colour that will be used for highlighting.
Autoscroll
This option is only available if the Sort in reverse chronological order option in the Menubar ‣ Logs ‣ Settings section is turned off. This causes all the new entries to be shown at the bottom of the page: If this option is enabled, the list is scrolled upwards to show the latest entries at the bottom of the page, otherwise only the older entries are show and the scrollbar on the right should be used to see the new ones.

To add or remove some log from the display, click on the Show more link right below the list of the log files on the top right. The controls will be replaced by a table from which the desired log files can be selected by ticking or unticking their respective checkboxes. To change the colour of a log file, click on the colour palette of that log type and then choose a new colour. To show the controls again, click on one of the Close links below the table or below the list of the displayed log files.

Live logs

The logs chosen for viewing are shown in this box, which consists of a table divided in three columns.

Left column
This column contains the log name, that is, the daemon or service producing the log entry.
Middle column
The time stamp (date and time) of the event that has been recorded.
Right Column

The actual message generated by the service or daemon and recorded in the log files.

Note

Some log messages -especially Firewall entries- span more than one line. To show the whole message, click on it or on the expand button at the right of the message.

Finally, there is also the chance to increase or decrease the window size by clicking on the Increase height or Decrease height buttons, respectively, which are situated on the heading of the box.

Common actions

The sub-menu entries System, Service, and Firewall show log files for different services and daemons, grouped by similar characteristics. Several controls are available to search within the log in these three items, with the System menu item that has one additional option. These sub-menu entries have also a common structure of their pages, organised in two boxes: Settings at the top, that contains the following options, and Log at the bottom that contains the actual messages of the log file..

Filter
Only the lines that contain the entered expression are shown.
Jump to Date
Directly show log entries from this date.
Jump to Page
Directly show log entries from this page in the result set. The number of entries shown per page can be modified on the Menubar ‣ Logs ‣ Settings page.
Update
After changing any of the settings above, a click on this button refreshes the page content.
Export
When clicking on this button the log entries are exported to a text file.
Sign log
When clicking on this link, the current log is signed. This button is only available if Trusted Timestamping is enabled.
Older, Newer
These two buttons are present in the Log box and show up whenever the number of entries grows too much and are divided into two or more parts. They allow to browse older or newer entries of the search results by clicking on them.

Note

A message at the top of the page informs if on a given date there are no logs available: This can happen either if the daemon or service were not running, or if they did not produce any message.

In the remainder of this section, all the services and their peculiar settings are presented.

Summary

This page presents summaries for the logs produced by the Endian UTM Appliance, separated by days and generated by the logwatch log monitoring software. Unlike the other parts of the log section, it has its own settings to control the level of details shown. The following control elements are available in the first box at the top of the page.

Month
Select from this drop-down menu the month in which the log messages were generated.
Day
The second drop-down menu allows to pick the day in which the log messages were generated.
<<, >>
Browse the history, moving to the previous or next day. The content of the page will be automatically refreshed.
Update
Immediately refresh the content of the page when the month/day combination has been changed.
Export
When clicking on this button, a text version of the summary is shown and can be saved on a local filesystem.

Below the Settings box, a variable number of boxes appears, depending on the running services that have log entries. The Disk Space box should at least be visible, showing the available disk space on the chosen date, while other boxes that can show up include Firewall, DHCP Server, and SSHD.

Note that the summaries are not available for the current day, as they are generated nightly from the log files generated the day before.

System

In this section appears the log viewer for the various system log files. The Settings box, besides the common actions, one more option is available:

Section
Choose from the drop-down menu which logs should be displayed, either All or only those related to a given service or daemon. Among others, they include kernel messages, SSH access, NTP, and DHCP

Following the choice of the section, click on the Update button to refresh the logs displayed in the Log box at the bottom of the page, in which the Older and Newer buttons allow to browse the pages.

Service

In this section appear the log entries for three important services provided by the Endian UTM Appliance, each in its own tab: IDS, OpenVPN, and the anti-virus, either ClamAV, Panda, or both. Only the common actions are available.

Firewall

The firewall log viewer contains the messages that record the firewall’s activities. Only the common actions are available.

Each line in the table is a connection recorded by the firewall, together with a number of information:

Time
The timestamp at which the message was generated.
Chain
The chain through which the packet has passed, including the policy applied to the packet.
Iface
The interface through which the packet has passed.
Proto
The protocol of the packet.
Source, Src port
The IP address and port from which the packet has arrived.
MAC address
The MAC address of the source interface.
Destination, Dst port
The IP address and port to which the packet has arrived.

Proxy

The proxy log viewer shows the logs for the four daemons that use the proxy. Each of them has its own tab: squid (HTTP), icap (Content filter), sarg (HTTP report), and smtpd (SMTP, email proxy).

HTTP

In addition to the common actions, the log viewer for the HTTP proxy allow these values to be specified:

Source IP

Show only the log entries containing the selected source IP address, chosen from a drop-down menu.

Note

The IP addresses available in the drop-down menu depend on those recorded in the log files.

Ignore filter
A regular expression that filters out all the log entries that contain it.
Enable ignore filter
Tick this checkbox to enable the ignore filter.
Restore defaults
A click on this button will restore the default search parameters.

The log entries shown are those produced by the squid software.

Content filter

This tab contains the same settings as the previous HTTP tab and shows log entries of the content filter engine.

HTTP Report

The HTTP report tab has only one option:

Enable
Tick the checkbox to enable the proxy analysis report generator and clicking on the Save button.

Once the report generator is activated, periodic reports are generated and accessible by clicking on the Daily report, Weekly report, and Monthly report links.

SMTP

Only the common actions are available in the tab of the postfix daemon. The log entries shown are those from the log files generated by the postfix daemon.

Settings

This page contains global configuration options for the Endian UTM Appliance‘s logging facilities, organised into four boxes: Log viewing options, Log summaries, Remote logging, and Firewall logging.

Log viewing options

Number of lines to display
The pagination value, i.e., how many lines are displayed per log-page.
Sort in reverse chronological order
If this checkbox is ticked, then the newest log entries will be displayed first.

Log summaries

Keep summaries for __ days
How long should the log summaries be stored on disk before deletion.
Detail level
The detail level for the log summary: the higher the level, the more log entries are saved and showed. The drop-down menu allows three levels of detail: Low, Medium, and High.

Remote logging

Enabled
Ticking this box allows to enable remote logging. The next option allows to enter the hostname of the syslog server.
Syslog server
The hostname of the remote server, to which the logs will be sent. The server must support the latest IETF syslog protocol standards.
Protocol
Select from the drop-down menu if the communication to the remote syslog server should use UDP or TCP.

Firewall logging

Log packets with BAD constellation of TCP flags
If this option is enabled the firewall will log packets with a bad constellation TCP flag (e.g., all flags are set).
Log NEW connections without SYN flag
With this option enabled, all new TCP connections without SYN flag will be logged.
Log accepted outgoing connections
To log all the accepted outgoing connections this checkbox must be ticked.
Log refused packets
All the refused packets will be logged by the firewall, if this option is enabled.

Growing Logging Files and Disk Space Management

The standard policy for storing log files on Endian UTM Appliance has been the following. Every night, log files are rotated and saved as daemonname.nnn.gz, while newer messages are written in a new log file. nnn is a progressive number, starting from 1. On some appliances, especially on the New Mini ARM, disk space may be quickly filled up, especially if many daemons are actively logging.

This policy has been changed after the 2.5 release. Until the 2.4 version, indeed, the log’s storage policy of the Endian UTM Appliance was to keep log files for 365 days for each service and delete them afterwards. The new policy, after the release of the 2.5 version is to delete older log files, to make room for newer ones, when the partition storing the logs is about to run out of space. To be more precise, the packages in which first the policy changed are: efw-syslog-2.6.5-1.endian9.noarch.rpm (2.4-ARM), efw-syslog-2.9.8-1.endian9.noarch.rpm (2.5).

The new policy can be modified or even reverted, to suite different needs.

See also

More information about the policies about logging can be found in this article.

See also

Some guidelines to free space on a Endian UTM Appliance can be found here.

Trusted Timestamping

Trusted timestamping is a process that log files (but in general any document) undergo in order to track and certify their origin and compliance to the original. In other words, trusted timestamping allows to certify and verify that a log file has not been modified in any way by anyone, not even the original author. In the case of log files, trusted timestamping proves useful for example, to verify the accesses to the system or the connections from the VPN users, even in cases of independent audits.

Trusted timestamping is not enabled by default, but its activation only requires a click on the grey switch. When it turns green, some configuration options will show up.

Timestamp server URL

The URL of the timestamp server (also called TSA) is mandatory, since it will be this server that signs the log files.

Note

A valid URL of a valid TSA is needed to be able to use trusted timestamping. Several Companies can supply this kind of service.

HTTP authentication
If the timestamp server requires to authenticate, tick the box below the HTTP authentication label.
Username
The username used to authenticate on the timestamp server.
Password.
The password used to authenticate on the timestamp server.
Public key of the timestamping server
To ease and to make the communication with the server more secure, the server’s public key can be imported. the certificate file can be searched on the local computer by clicking on the Browse... button, and then uploaded to the Endian UTM Appliance by clicking on the Upload button. After the certificate has been stored, next to the Public key of the timestamping server label, a Download link will appear, that can be clicked to retrieve the certificate, for example if it should be installed on another Endian UTM Appliance.

After clicking on the Save button, the settings are stored and, on the next day, a new button will appear in the Logs section, on the right-hand side of the Settings box:

Verify log signature
When clicked it will show a message in a yellow callout to inform about the status of the log.

See also

The official OpenSSL timestamping documentation and RFC 3161, the original definition of the Time Stamp Protocol.