Postfix mail queues Postfix has four main queues: maildrop, incoming, active and deferred (click the upper left-hand icon for the big picture). Locally-posted mail is deposited into the maildrop, and is copied to the incoming queue after some cleaning up. The incoming queue is for mail that is still arriving or that the queue manager hasn't looked at yet. The active queue is a limited-size queue for mail that the queue manager has opened for delivery. Mail that can't be delivered goes to the deferred queue, so that it does not get in the way of other deliveries. The queue manager keeps information in memory about the active queue only. The active queue size is limited on purpose: the queue manager should never run out of working memory because of a peak message workload. Whenever there is space in the active queue, the queue manager lets in one message from the incoming queue and one from the deferred queue. This guarantees that new mail will get through even when there is a large backlog. In addition to the queues mentioned above Postfix also maintains two parking spaces. The hold queue is for mail that is frozen in the queue; no delivery attempts are made until someone releases these messages with the postsuper command. The corrupt directory is for damaged queue files. Rather than discarding these, Postfix leaves them here for human inspection. No thundering herd Implementing a high-performance mail system is one thing. However, no-one would be pleased when Postfix connects to their site and overwhelms it with lots of simultaneous deliveries. This is an issue especially when a site has been down and mail is backed up elsewhere in the network. Postfix tries to be a good network neighbor. When delivering mail to a site, Postfix will initially make no more than two simultaneous connections. As long as deliveries succeed, the concurrency slowly increases up to some configurable limit (or until the host or network is unable to handle the load); concurrency is decreased in the face of trouble. For those familiar with TCP/IP implementation details, Postfix implements its own analog of the TCP slow start algorithm Fairness Apart from the thundering herd controls, the Postfix delivery strategy is based on round-robin selection. The queue manager sorts message recipients in the active queue by destination, and makes round-robin walks along all destination queues. On the average, Postfix will do simultaneous deliveries to the same domain only when there is not enough work to keep all outbound SMTP channels busy. So, when AOL goes off-line and comes back, it should not stop the system from delivering to other sites. When mail arrives faster than Postfix can deliver it, Postfix will favor new mail over delayed mail. The idea is that new mail should be delivered with as little delay as possible; delayed mail can be delivered while the system would otherwise be idle. Exponential backoff Postfix implements per-message exponential backoff. When a message cannot be delivered upon the first attempt, the queue manager gives the queue file a time stamp that is offset into the future by some configurable amount of time. Queue files with future time stamps are normally ignored by the queue manager. Whenever a repeat delivery attempt fails, the queue file time stamp is moved into the future by an amount of time equal to the age of the message. Thus, the time between delivery attempts doubles each time. This strategy effectively implements exponential backoff. Destination status cache The Postfix queue manager maintains a limited, short-term list of unreachable destinations. This list helps it to avoid unnecessary delivery attempts, especially with destinations that have a large mail backlog. Background info: Postfix queue directories The following sections describe Postfix queues: their purpose, what normal behavior looks like, and how to diagnose abnormal behavior. The "maildrop" queue Messages that have been submitted via the Postfix sendmail(1) command, but not yet brought into the main Postfix queue by the pickup(8) service, await processing in the "maildrop" queue. Messages can be added to the "maildrop" queue even when the Postfix system is not running. They will begin to be processed once Postfix is started. The "maildrop" queue is drained by the single threaded pickup(8) service scanning the queue directory periodically or when notified of new message arrival by the postdrop(1) program. The postdrop(1) program is a setgid helper that allows the unprivileged Postfix sendmail(1) program to inject mail into the "maildrop" queue and to notify the pickup(8) service of its arrival. All mail that enters the main Postfix queue does so via the cleanup(8) service. The cleanup service is responsible for envelope and header rewriting, header and body regular expression checks, automatic bcc recipient processing and guaranteed insertion of the message into the Postfix "incoming" queue. In the absence of excessive CPU consumption in cleanup(8) header or body regular expression checks or other software consuming all available CPU resources, Postfix performance is disk I/O bound. The rate at which the pickup (8) service can inject messages into the queue is largely determined by disk access times, since the cleanup(8) service must commit the message to stable storage before returning success. The same is true of the postdrop(1) program writing the message to the "maildrop" directory. As the pickup service is single threaded, it can only deliver one message at a time at a rate that does not exceed the reciprocal disk I/O latency (+ CPU if not negligible) of the cleanup service. Congestion in this queue is indicative of an excessive local message submission rate or perhaps excessive CPU consumption in the cleanup(8) service due to excessive body_checks. Note, that once the active queue is full, the cleanup service will attempt to slow down message injection by pausing $in_flow_delay for each message. In this case "maildrop" queue congestion may be a consequence of congestion downstream, rather than a problem in its own right. Note also, that one should not attempt to deliver large volumes of mail via the pickup(8) service. High volume sites must avoid using content filters that reinject scanned mail via Postfix sendmail(1) and postdrop(1). A high arrival rate of locally submitted mail may be an indication of an uncaught forwarding loop, or a run-away notification program. Try to keep the volume of local mail injection to a moderate level. The "postsuper -r" command can place selected messages into the "maildrop" queue for reprocessing. This is most useful for resetting any stale content_filter settings. Requeuing a large number of messages using "postsuper -r" can clearly cause a spike in the size of the "maildrop" queue. The "hold" queue The administrator can define "smtpd" access(5) policies, or cleanup(8) header/ body checks that cause messages to be automatically diverted from normal processing and placed indefinitely in the "hold" queue. Messages placed in the "hold" queue stay there until the administrator intervenes. No periodic delivery attempts are made for messages in the "hold" queue. The postsuper(1) command can be used to manually release messages into the "deferred" queue. Messages can potentially stay in the "hold" queue for a time exceeding the normal maximal queue lifetime (after which undelivered messages are bounced back to the sender). If such "old" messages need to be released from the "hold" queue, they should typically be moved into the "maildrop" queue, so that the message gets a new timestamp and is given more than one opportunity to be delivered. Messages that are "young" can be moved directly into the "deferred" queue. The "hold" queue plays little role in Postfix performance, and monitoring of the "hold" queue is typically more closely motivated by tracking spam and malware, than by performance issues. The "incoming" queue All new mail entering the Postfix queue is written by the cleanup(8) service into the "incoming" queue. New queue files are created owned by the "postfix" user with an access bitmask (or mode) of 0600. Once a queue file is ready for further processing the cleanup(8) service changes the queue file mode to 0700 and notifies the queue manager of new mail arrival. The queue manager ignores incomplete queue files whose mode is 0600, as these are still being written by cleanup. The queue manager scans the incoming queue bringing any new mail into the "active" queue if the active queue resource limits have not been exceeded. By default, the active queue accommodates at most 20000 messages. Once the active queue message limit is reached, the queue manager stops scanning the incoming (and deferred, see below) queue. Under normal conditions the incoming queue is nearly empty (has only mode 0600 files), with the queue manager able to import new messages into the active queue as soon as they become available. The incoming queue grows when the message input rate spikes above the rate at which the queue manager can import messages into the active queue. The main factor slowing down the queue manager is transport queries to the trivial- rewrite service. If the queue manager is routinely not keeping up, consider not using "slow" lookup services (MySQL, LDAP, ...) for transport lookups or speeding up the hosts that provide the lookup service. The in_flow_delay parameter is used to clamp the input rate when the queue manager starts to fall behind. The cleanup(8) service will pause for $in_flow_delay seconds before creating a new queue file if it cannot obtain a "token" from the queue manager. Since the number of cleanup(8) processes is limited in most cases by the SMTP server concurrency, the input rate can exceed the output rate by at most "SMTP connection count" / $in_flow_delay messages per second. With a default process limit of 100, and an in_flow_delay of 1s, the coupling is strong enough to limit a single run-away injector to 1 message per second, but is not strong enough to deflect an excessive input rate from many sources at the same time. If a server is being hammered from multiple directions, consider raising the in_flow_delay to 10 seconds, but only if the incoming queue is growing even while the active queue is not full and the trivial-rewrite service is using a fast transport lookup mechanism. The "active" queue The queue manager is a delivery agent scheduler; it works to ensure fast and fair delivery of mail to all destinations within designated resource limits. The active queue is somewhat analogous to an operating system's process run queue. Messages in the active queue are ready to be sent (runnable), but are not necessarily in the process of being sent (running). While most Postfix administrators think of the "active" queue as a directory on disk, the real "active" queue is a set of data structures in the memory of the queue manager process. Messages in the "maildrop", "hold", "incoming" and "deferred" queues (see below) do not occupy memory; they are safely stored on disk waiting for their turn to be processed. The envelope information for messages in the "active" queue is managed in memory, allowing the queue manager to do global scheduling, allocating available delivery agent processes to an appropriate message in the active queue. Within the active queue, (multi-recipient) messages are broken up into groups of recipients that share the same transport/nexthop combination; the group size is capped by the transport's recipient concurrency limit. Multiple recipient groups (from one or more messages) are queued for delivery via the common transport/nexthop combination. The destination concurrency limit for the transports caps the number of simultaneous delivery attempts for each nexthop. Transports with a recipient concurrency limit of 1 are special: these are grouped by the actual recipient address rather than the nexthop, thereby enabling per-recipient concurrency limits rather than per-domain concurrency limits. Per-recipient limits are appropriate when performing final delivery to mailboxes rather than when relaying to a remote server. Congestion occurs in the active queue when one or more destinations drain slower than the corresponding message input rate. If a destination is down for some time, the queue manager will mark it dead, and immediately defer all mail for the destination without trying to assign it to a delivery agent. In this case the messages will quickly leave the active queue and end up in the deferred queue. If the destination is instead simply slow, or there is a problem causing an excessive arrival rate the active queue will grow and will become dominated by mail to the congested destination. The only way to reduce congestion is to either reduce the input rate or increase the throughput. Increasing the throughput requires either increasing the concurrency or reducing the latency of deliveries. For high volume sites a key tuning parameter is the number of "smtp" delivery agents allocated to the "smtp" and "relay" transports. High volume sites tend to send to many different destinations, many of which may be down or slow, so a good fraction of the available delivery agents will be blocked waiting for slow sites. Also mail destined across the globe will incur large SMTP command- response latencies, so high message throughput can only be achieved with more concurrent delivery agents. The default "smtp" process limit of 100 is good enough for most sites, and may even need to be lowered for sites with low bandwidth connections (no use increasing concurrency once the network pipe is full). When one finds that the queue is growing on an "idle" system (CPU, disk I/O and network not exhausted) the remaining reason for congestion is insufficient concurrency in the face of a high average latency. If the number of outbound SMTP connections (either ESTABLISHED or SYN_SENT) reaches the process limit, mail is draining slowly and the system and network are not loaded, raise the "smtp" and/or "relay" process limits! Especially for the "relay" transport, consider lower SMTP connection timeouts (1-5 seconds) and higher than default destination concurrency limits. Compute the expected latency when 1 out of N of the MX hosts for a high volume site is down and not responding, and make sure that the configured concurrency divided by this latency exceeds the required steady-state message rate. If the destination is managed by you, consider load balancers in front of groups of MX hosts. Load balancers have higher uptime and will be able to hide individual MX host failures. If necessary, dedicate and tune custom transports for high volume destinations. Another common cause of congestion is unwarranted flushing of the entire deferred queue. The deferred queue holds messages that are likely to fail to be delivered and are also likely to be slow to fail delivery (timeouts). This means that the most common reaction to a large deferred queue (flush it!) is more than likely counter- productive, and is likely to make the problem worse. Do not flush the deferred queue unless you expect that most of its content has recently become deliverable (e.g. relayhost back up after an outage)! Note that whenever the queue manager is restarted, there may already be messages in the active queue directory, but the "real" active queue in memory is empty. In order to recover the in-memory state, the queue manager moves all the active queue messages back into the incoming queue, and then uses its normal incoming queue scan to refill the active queue. The process of moving all the messages back and forth, redoing transport table (trivial-rewrite(8) resolve service) lookups, and re-importing the messages back into memory is expensive. At all costs, avoid frequent restarts of the queue manager. The "deferred" queue When all the deliverable recipients for a message are delivered, and for some recipients delivery failed for a transient reason (it might succeed later), the message is placed in the deferred queue. The queue manager scans the deferred queue periodically. The scan interval is controlled by the queue_run_delay parameter. While a deferred queue scan is in progress, if an incoming queue scan is also in progress (ideally these are brief since the incoming queue should be short), the queue manager alternates between bringing a new "incoming" message and a new "deferred" message into the queue. This "round-robin" strategy prevents starvation of either the incoming or the deferred queues. Each deferred queue scan only brings a fraction of the deferred queue back into the active queue for a retry. This is because each message in the deferred queue is assigned a "cool-off" time when it is deferred. This is done by time- warping the modification times of the queue file into the future. The queue file is not eligible for a retry if its modification time is not yet reached. The "cool-off" time is at least $minimal_backoff_time and at most $maximal_backoff_time. The next retry time is set by doubling the message's age in the queue, and adjusting up or down to lie within the limits. This means that young messages are initially retried more often than old messages. If a high volume site routinely has large deferred queues, it may be useful to adjust the queue_run_delay, minimal_backoff_time and maximal_backoff_time to provide short enough delays on first failure, with perhaps longer delays after multiple failures, to reduce the retransmission rate of old messages and thereby reduce the quantity of previously deferred mail in the active queue. One common cause of large deferred queues is failure to validate recipients at the SMTP input stage. Since spammers routinely launch dictionary attacks from unrepliable sender addresses, the bounces for invalid recipient addresses clog the deferred queue (and at high volumes proportionally clog the active queue). Recipient validation is strongly recommended through use of the local_recipient_maps and relay_recipient_maps parameters. When a host with lots of deferred mail is down for some time, it is possible for the entire deferred queue to reach its retry time simultaneously. This can lead to a very full active queue once the host comes back up. The phenomenon can repeat approximately every maximal_backoff_time seconds if the messages are again deferred after a brief burst of congestion. Ideally, in the future Postfix will add a random offset to the retry time (or use a combination of strategies) to reduce the chances of repeated complete deferred queue flushes. ~~~~~~~~~~~~~~ How /var/spool/postfix looks like: l /var/spool/postfix/ total 128K drwxr-xr-x 16 root root 4.0K Nov 30 09:30 . drwxr-xr-x 16 root root 4.0K Nov 30 09:30 .. drwx------ 17 postfix root 4.0K Dec 26 14:04 active drwx------ 6 postfix root 4.0K Dec 16 13:05 bounce drwx------ 2 postfix root 4.0K Nov 23 21:11 corrupt drwx------ 5 postfix root 4.0K Jan 4 20:45 defer drwx------ 5 postfix root 4.0K Jan 4 20:45 deferred drwx------ 2 postfix root 4.0K Nov 23 21:11 flush drwx------ 2 postfix root 4.0K Nov 23 21:11 hold drwx------ 18 postfix root 4.0K Jan 4 20:43 incoming drwx-wx--- 2 postfix postdrop 4.0K Nov 23 21:11 maildrop drwxr-xr-x 2 root root 4.0K Dec 16 12:42 pid drwx------ 2 postfix root 4.0K Jan 4 19:38 private drwx--x--- 2 postfix postdrop 4.0K Jan 4 19:38 public drwx------ 2 postfix root 4.0K Nov 23 21:11 saved drwx------ 2 postfix root 4.0K Nov 23 21:11 trace