ConfigServer Outgoing Spam Monitor

ConfigServer Outgoing Spam Monitor (osm) has been designed to use multiple methods to monitor outgoing email and SMTP connections for activity that could indicate a spammer is active on a server.

With the proliferation of web scripts in shared hosting environments that are often poorly maintained or badly written, the chances of a hacker exploiting vulnerabilities in scripts is at an all time high. Additionally, end-user PC's and other devices that send email through a server (relay) that have been compromised and used as a spam source has always been a problem. These issues along with spammers deliberately targeting hosting providers by purchasing accounts simply to send out spam have kept the diligence required to prevent spam from being sent from servers all the more difficult.

osm Features:

- Outgoing email sent via exim is tracked by cPanel account
- Matching Subject headers for outgoing email sent via exim is tracked by cPanel account
- Script path location (cwd) is tracked by cPanel account
- Matching script path location (cwd) is tracked by cPanel account
- Outgoing SMTP connections to remote servers (that bypass exim) are tracked by cPanel account
- Matching script path location for outgoing SMTP connections to remote servers (that bypass exim) are tracked
- Authenticated outgoing email is tracked by email account and connecting IP address
- osm uses real-time Packet Inspection to track SMTP connections, this is primarily useful if you cannot use the csf SMTP_BLOCK or cPanel provided equivalent feature
- Configurable trigger counts for each type of tracking by cPanel account on a per email/connection per second basis
- Apache Status information us used to link outgoing email with actual scripts being used
- Multiple actions can be performed once a report is raised after a trigger count is reached:

- Send an email report of the events
- Store the report of events to view in the WHM UI
- Hold outgoing email from the cPanel/email account in the exim queue
- Discard outgoing email from the cPanel/email account
- Suspend the whole cPanel account
- Prevent the email account from logging in
- Rename the reported path
- Run the custom script configured in the WHM UI
- Rename the file determined from the Apache Status
- Block the IP address (AUTHRELAY, ALWAYSRELAY, POPRELAY, Apache Status) in csf

- Custom action script is configurable and can be sent JSON, YAML, XML and PERL data structures to allow for client specific actions
- Inheritance rules are used to configure all trigger counts for each cPanel account plus the default settings

Please also see the osm FAQ for additional information.

osm Daemon
osmd

You can enable/disable the osm daemon that does all of the work for this product. If osmd is disabled, no monitoring will be performed. What osm monitors is configurable in the Main Configuration and Event Configuration pages.

SMTP Packet Interception (packet)

osm uses a hook into the Linux kernel to intercept outbound (or to localhost) SMTP connections on port 25, 465 and 587. While this technique cannot count the number of actual emails (it cannot look inside the packets), it does provide information about the outgoing connection including the source IP and port and the destination IP and port. Using this information osm determines which process is using this connection and obtains information from the process that provides that details needed to track the emails.

osm keeps track of the number of outgoing connections made by each cPanel account in use as well as the path used by the process sending the email. If the number of connections made by a single cPanel account exceeds the configured limit, a report is triggered.

See possible restrictions on this functionality in the Limitations section.

Additionally, the packet inspection process also tries to match the running process with related URL's being accessed to try and determine the possible script in use (see Apache Status section).

SMTP Packet Interception Path Matching (packet:cwd)

As part of the Packet Interception process, osm determines the path being used by the process sending email. osm counts matching paths used by connections and if the number for the same path exceeds the configured limit, a report is triggered.

Exim Log Line Processing (logline)

osm monitors the exim email log for lines generated when exim sends out email. osm uses the information from these log lines to determine how the email is being sent. If it is via an authorised email account login, that account will be tracked and if it exceeds the configured limit, a report is triggered. The source IP address of the account relaying email through the server is also noted.

If the log lines are being generated from a local source (script) then the cPanel use account that is used is tracked and if it exceeds the configured limit, a report is triggered.

Additionally, the Log Line process also tries to match the running process with related URL's being accessed to try and determine the possible script in use (see Apache Status section).

Exim Log Line Subject Matching (logline:subject)

As part of the Exim log line processing, osm keeps track of matching email Subjects being used and if the number for the same Subject exceeds the configured limit, a report is triggered.

Exim Log Line Path Processing (cwdcheck)

In addition the logline processing, osm also monitors the log lines generated by exim that specify the path (cwd=/path/to/script/) that exim determines is used by the process sending out email. osm counts the cPanel user that owns the path being used and if the number for the same account exceeds the configured limit, a report is triggered.

Additionally, the Exim Log Line Path Processing process also tries to match the running process with related URL's being accessed to try and determine the possible script in use (see Apache Status section).

Exim Log Line Path Matching (cwdcheck:cwd)

As part of the Exim Log Line Path Processing, osm keeps track of matching paths being used and if the number for the same path exceeds the configured limit, a report is triggered.

Exim Log Line Multiple Recipient Processing

Where there are multiple recipients on a single exim log line, osm will reprocess the line for the total number of recipients listed. This ensures that for this type of detection a single email sent to hundreds of recipients will not count as a single event, but the total number of recipients. This option can be disabled if each log line should be treated as a single event.

Apache Status

If enabled in settings, whenever one of the three event types is triggered osm tries to match the event with URL's currently being accessed by the cPanel account involved. If it can determine matching connections, it will use these to list possible scripts that are currently in use and provides this information to the generated report. See possible restrictions on this functionality in the Limitations section.

Note: We do not recommend enabling the Perform actions on Apache Status results setting as this could easily lead to false-positive IP address blocks and script renaming. This feature should only be used for reporting and investigation purposes
If Apache is not used or Apache Status does not function as normal for a cPanel Server, then the setting Perform Apache Status lookups to try and identify scripts and connecting IPs should be disabled.

osm WHM User Interface
Main Configuration for Settings

All configuration for osm is done through the WHM UI. This includes configuration settings as well as Default and per cPanel account report trigger counts and actions (based on an inheritence system).

Event Configuration for Reports/Triggers/Actions

When the number of events being monitored by osm exceeds the configured trigger count within the configured time interval, a report is generated and any configured actions are performed based on the information provided in the report from each of the events that exceeded the trigger count.

Events are discarded when their age exceeds the configured time interval or if they are used in a report (but only for the trigger exceeded).

One or more actions can be performed when a report is generated depend on the information obtained by osm for each event.

Actions types:

Name | Action
Email | Send an email report of the events
Store | Store the report of events to view in the WHM UI
Hold | Hold outgoing email from the cPanel/email account in the exim queue (local deliveries are not affected)
Disable | Discard outgoing email from the cPanel/email account (local deliveries are not affected)
Suspend | Suspend the whole cPanel account
Login | Prevent the email account from logging in
Renamepath | Rename the reported path
Custom | Run the custom script configured in the WHM UI
Renamefile | Rename the file determined from the Apache Status
Firewall | Block the IP address (AUTHRELAY, ALWAYSRELAY, POPRELAY, Apache Status) in csf

Custom Action

This is configured in the osm Settings. It must point to a pre-existing excecutable `chmod +x` script that contains a valid `shebang!` line. When a report is configured to run this custom action, the script is executed and passed the report data in a format chosen in the Main Configuration page.

Valid formats for the passed data include:

Name | Data Type
JSON | https://www.json.org/
YAML | http://yaml.org/
XML | https://www.w3.org/XML/
PERL | https://perldoc.perl.org/Data/Dumper.html

Trigger Levels

This is configured in the Main Configuration.

This option provides the ability to have multiple trigger settings for each event type and therefore different actions based on the levels set. For example, if this setting is set to 3:

Each event type will now three trigger settings, so:

1. if 100 emails are sent every 300 seconds which triggers a logline report, perform actions Email and Store
2. if 500 emails are sent every 300 seconds which triggers a logline report, perform actions Email and Store and Hold
3. if 1000 emails are sent every 300 seconds which triggers a logline report, perform actions Email and Store and Suspend

Ignore Events

Events can be ignored by adding entries in the Main Configuration.

Simply place one entry per line for cPanel user, email address, group, IP address, directory path or script (with path) to ignore.

All entries in the ignore list are checked against all event fields for a match and if found, the event will be ignored.

Note: Wildcards and regular expressions can not be used

Requirements

- cPanel/WHM (supported versions)
- Server with static IPv4 address (for licensing)
- Redhat/CentOS/CloudLinux Linux v6/7
- Apache with mod_status required for the Apache Status feature
- Pcap Kernel access via libpcap required for SMTP Packet Interception
- csf for IP address blocking

Limitations

- Without mod_status configured via Easyapache, the Apache Status feature cannot be used
- mod_rewrite rules in local htaccess files may break Apache Status functionality
- IP address event triggers (by (AUTHRELAY, ALWAYSRELAY, POPRELAY email events) are controlled by the "Default" settings in Event Configuration
- Duplication of reports will occur between logline and cwdcheck report types as they are often referring to the same email event. However, each event type offers different triggers to detect outgoing spam patterns
- The SMTP Packet Interception feature will not function on Virtuozzo/OpenVZ Servers as the kernels do not support Pcap access
- See the osm FAQ for additional information