close
Mailspect Documentation
Distributed Message Stores for Email Archive and Spam Quarantine

Email Store using ESMTP and Internal Queue

Introduction

Mailspect MPP has a distributed model for storing spam quarantines and archives that is designed for larger environments. In our distributed model remote instances of MPP make quarantine/archive decisions based on your configuration and forward the email for storage via ESMTP to a centralized instance of MPP. The centralized instance of MPP maintains connections to the quarantine/archive message store and manages writes to the system.This is advantageous because it removes direct database connections from instances of MPP that are involved in the flow of email, thus increasing throughput for busy systems and isolating failures in all environments. Our distributed architecture is compatible with all message store types.

"Email Store" features refer to a set of functionality that enables two instances of MPP to work as quarantine/archive client/server. In such configuration first instance scans incoming email and decides whether to send it to quarantine/archive. If it decides so it sends email to second instance of MPP via network socket. Second instance puts email to Internal Queue on disk and replies with success to first one. The it retrieves queued email, processes and submits email to database. The key feature of this configuration is that second instance queues email and then processes it. In that way client instance doesn't wait for potentially "heavy" processing and submission procedures so their scanning tasks runs smoothly. Email Store feature should be effective when client (or multiple clients) and server are on different machines.

Use-case View

This section contains brief description of the features from usage point of view. Entities referenced in this section are described in "Structural View" section.

Client Side of Email Store

Client side of Email Store is configured by specifying special URI's for QUARANTINE and ARCHIVE options. Client uses "Email Store Protocol" to communicate with server.

Server Side of Email Store

Server side of Email Store is configured by enabling:

- Email Store Protocol as an input protocol

- Internal Queue

- Quarantine/Archive

Structural View

URI's for QUARANTINE and ARCHIVE options

Client MPP instance could use "smtp" URI's for QUARANTINE and ARCHIVE as follow:

smtp://inet:10027@192.168.10.1 smtp://unix:/var/run/quarantine.socket

First form is used for remote communication.Second is for local communication via UNIX socket. When such URI is specified message data and scanning results will be transmitted to quarantine/archive using Email Store Protocol.

Email Store Protocol

Email Store Protocol is a ESMTP protocol with custom XMPPSCANINFO and XMPPSCANDETAIL extensions.

During extensions negotiations server reports what extensions and what extension elements it supports for each extension. This is standard mechanism but non-standard keywords for extensions.

client: EHLO client.host.com server: 250-server.host.com server: 250-XMPPSCANINFO HOST GROUP STORE STATE server: 250 XMPPSCANDETAIL VIRUSINFO SPAMSCORE SPAMLEVEL HARASSSCORE

Before each MAIL command client passes scan result to server:

client: XMPPSCANINFO HOST=host.that.scanned.com GROUP=scan_profile server: 250 Ok client: XMPPSCANINFO STORE=Quarantine STATE=virus,spam,unauthorized server: 250 Ok client: XMPPSCANDETAIL VIRUSINFO Arbitrary text about infection server: 250 Ok client: XMPPSCANDETAIL SPAMSCORE Arbitrary text about spam server: 250 Ok client: XMPPSCANDETAIL SPAMLEVEL Arbitrary text about level server: 250 Ok client: XMPPSCANDETAIL HARASSSCORE Arbitrary text about harass server: 250 Ok client: XMPPSCANDETAIL REASON Arbitrary text about quarantine reason server: 250 Ok

HOST - is the host that scanned mail.

GROUP - is scanning profile that was used.

STORE - is type (Archive, Quarantine) of storage that is targeted.

STATE - is scan state as comma-separated (without any whitespace) list of flags: virus,spam,harass,unauthorized,error,max_recursion,max_size,acl,encrypted,malformed. Other are self-explained.

Email Store Protocol as an input protocol'

The following options should be specified to forces server to use Email Store Protocol as an input protocol:

EMAIL_SERVER is postfix EMAIL_SERVER_IN_PROTOCOL is smtp EMAIL_SERVER_IN_PROTOCOL_EXTENSIONS contain mailstore EMAIL_SERVER_IN_SOCKET should specify input socket address

Server should never relay mail via its output channel. To be safe it should be configured to some dummy value:

EMAIL_SERVER_OUT_SOCKET is "unix:/non/existing/file/name"

Internal Queue

Please refer to Internal Queue for queue configuration.

Quarantine/Archive

Usual quarantine/archive URI's should be specified. ON_QUARANTINE_SUCCESS and ON_ARCHIVE_SUCCESS should be "discard" to avoid unwanted relaying to server's output channel.

1. MPP quarantine server and MPP client are on different hosts

For this example, we will consider: 192.168.1.2 as our MPP scanner/quarantine client and 192.168.1.3 as MPP quarantine server using file://quarantine. Custom configuration file with Email Store / Internal Queue options enabled is available from: ftp://ftp.mailspect.com/pub/mpp5/beta/mppd-quarantine.conf.xml

1.1. Quarantine server configuration

The following options should be defined in /usr/local/MPP/mppd.conf.xml on 192.168.1.3 host:

Bellow is a list of options required for mppd-quarantine.conf.xml:

<mppd>

<common>
 ......
   <email_server>postfix</email_server>
   <email_server_in_protocol>smtp</email_server_in_protocol>
   <email_server_in_protocol_extensions>mailstore</email_server_in_protocol_extensions>
   <email_server_in_queue_enable>yes</email_server_in_queue_enable>
   <email_server_in_queue_media>/usr/local/MPP/working/queue</email_server_in_queue_media>
   <email_server_in_socket>inet:10027@192.168.1.3</email_server_in_socket>
   <email_server_out_socket>unix:/non/existing/file/name</email_server_out_socket>
   <email_server_in_threads_min>2</email_server_in_threads_min>
   <email_server_in_threads_max>4</email_server_in_threads_max>
   <quarantine type="primary">file:///usr/local/MPP/quarantine/%TYPE%/%RCPT_DOMAIN%/%RCPT_NAME%</quarantine>
   <log_dirlog_template>/var/log/MPP/$YEAR$$MONTH$$DAY$-quarantine.log</log_dirlog_template>
 .......
</common>
<group id="default">
...
 <actions>
  ...
   <on_quarantine_success>discard</on_quarantine_success>
   <on_archive_success>discard</on_archive_success>
  ...
 </actions>
...
</group>
....

</mppd>

Start: /usr/local/MPP/mppd -f /usr/local/MPP/mppd.conf.xml
Stop: /usr/local/MPP/mppd -s
Restart: /usr/local/MPP/mppd -r 


1.2. Quarantine client configuration

In /usr/local/MPP/mppd.conf.xml on 192.168.1.2 host user: <quarantine type="primary">smtp://inet:10027@192.168.1.3</quarantine>

Start: /usr/local/MPP/mppd -f /usr/local/MPP/mppd.conf.xml
Stop: /usr/local/MPP/mppd -s
Restart: /usr/local/MPP/mppd -r 

2. MPP quarantine server and MPP client on the same host

Multiple mppd instances can be started now in the same time, each using a different configuration file and another PID file location. Configuration file and PID file location should be provided as options for mppd. Since 2 MPP instances will run on the same machine: mppd instance which would scan and act as quarantine client, and other mppd instance which would act as quarantine server.

2.1. Quarantine server configuration

Custom configuration file with 'EmailStore'/'InternalQueue' options enabled is available from: ftp://ftp.mailspect.com/pub/mpp5/beta/mppd-quarantine.conf.xml

Bellow is a list of options required for mppd-quarantine.conf.xml:

<mppd>
 <common>
  ......
   <email_server>postfix</email_server>
   <email_server_in_protocol>smtp</email_server_in_protocol>
   <email_server_in_protocol_extensions>mailstore</email_server_in_protocol_extensions>
   <email_server_in_queue_enable>yes</email_server_in_queue_enable>
   <email_server_in_queue_media>/usr/local/MPP/working/queue</email_server_in_queue_media>
   <email_server_in_socket>inet:10027@127.0.0.1</email_server_in_socket>
   <email_server_out_socket>unix:/non/existing/file/name</email_server_out_socket>
   <email_server_in_threads_min>2</email_server_in_threads_min>
   <email_server_in_threads_max>4</email_server_in_threads_max>
   <quarantine type="primary">file:///usr/local/MPP/quarantine/%TYPE%/%RCPT_DOMAIN%/%RCPT_NAME%</quarantine>
   <log_dirlog_template>/var/log/MPP/$YEAR$$MONTH$$DAY$-quarantine.log</log_dirlog_template>
  .......
 </common>
 <group id="default">
 ...
  <actions>
   ...
   <on_quarantine_success>discard</on_quarantine_success>
   <on_archive_success>discard</on_archive_success>
   ...
  </actions>
 ...
 </group>
 ....
</mppd>
Create mppd-quarantine symlink: ln -s /usr/local/MPP/mppd /usr/local/MPP/mppd-quarantine 
Start: /usr/local/MPP/mppd-quarantine -f /usr/local/MPP/mppd-quarantine.conf.xml -p /var/run/mppd-quarantine.pid
Stop:  /usr/local/MPP/mppd-quarantine -s -p /var/run/mppd-quarantine.pid
Restart: /usr/local/MPP/mppd-quarantine -r -p /var/run/mppd-quarantine.pid

2.2. Quarantine client configuration

MPP quarantine client will be regular mppd instance, using action 'quarantine' for virus/spam using "smtp://" protocol.

<quarantine type="primary">smtp://inet:10027@127.0.0.1</quarantine>

Start: /usr/local/MPP/mppd -f /usr/local/MPP/mppd.conf.xml
Stop: /usr/local/MPP/mppd -s
Restart: /usr/local/MPP/mppd -r 
Restart: /usr/local/MPP/mppd -r 

Bellow is a list of options required for mppd-quarantine.conf.xml:

<mppd>
 <common>
  .....
   <email_server>postfix</email_server>
   <email_server_in_protocol>smtp</email_server_in_protocol>
   <email_server_in_protocol_extensions>mailstore</email_server_in_protocol_extensions>
   <email_server_in_queue_enable>yes</email_server_in_queue_enable>
   <email_server_in_queue_media>/usr/local/MPP/working/queue</email_server_in_queue_media>
   <email_server_in_socket>inet:10027@127.0.0.1</email_server_in_socket>
   <email_server_out_socket>unix:/non/existing/file/name</email_server_out_socket>
   <email_server_in_threads_min>2</email_server_in_threads_min>
   <email_server_in_threads_max>4</email_server_in_threads_max>
   <quarantine type="primary">file:///usr/local/MPP/quarantine/%TYPE%/%RCPT_DOMAIN%/%RCPT_NAME%</quarantine>
   <log_dirlog_template>/var/log/MPP/$YEAR$$MONTH$$DAY$-quarantine.log</log_dirlog_template>
  ......
 </common>
 <group id="default">
  ...
  <actions>
   ...
   <on_quarantine_success>discard</on_quarantine_success>
   <on_archive_success>discard</on_archive_success>
   ...
  </actions>
 ...
 </group>
 ....
</mppd>
Create mppd-quarantine symlink: ln -s /usr/local/MPP/mppd /usr/local/MPP/mppd-quarantine 
Start: /usr/local/MPP/mppd-quarantine -f /usr/local/MPP/mppd-quarantine.conf.xml -p /var/run/mppd-quarantine.pid
Stop:  /usr/local/MPP/mppd-quarantine -s -p /var/run/mppd-quarantine.pid
Restart: /usr/local/MPP/mppd-quarantine -r -p /var/run/mppd-quarantine.pid

2.2. Quarantine client configuration

MPP quarantine client will be regular mppd instance, using action 'quarantine' for virus/spam using "smtp://" protocol.

<quarantine type="primary">smtp://inet:10027@127.0.0.1</quarantine>

Start: /usr/local/MPP/mppd -f /usr/local/MPP/mppd.conf.xml
Stop: /usr/local/MPP/mppd -s
Restart: /usr/local/MPP/mppd -r