A unix style mail setup

These days you should not use anything else. I want to have access to email from different devices, mostly my phone and my laptop and statuses of mails should always be in sync.

• Allows me to write mails with emacs (or whatever text editor I like best): In my setup I use the mutt mail client which naturally allows you to use the editor of your choice, in my case an emacsclient.
• Works offline and online: In this setup there is a clear separation between two things:
  • the handling of your mail locally (Writing, reading, moving, archiving, …). This is all done in your local maildir and completely independent of any internet connection.
  • Transfer of mail to and from the server. This is done by a cron job in the background and independent of the local handling of mail. If you finished composing a new mail it will be queued and automatically sent the next time you are online.
• Seamless integration of multiple identities In this setup I use mutt's folder-hooks to change many aspects like outgoing mailserver, signature, identity.
• PGP support: Just choose a mail client that supports it.
• IMAP on multiple servers: Here I use offlineimap that syncs quickly in parallel from many different sources.
• Fast: Speed is something that everybody wants and here we have it throughout.
  • Using the mutt mail client is fast a priori, e.g. because you never touch the mouse.
  • Fetching mail is much faster with offlineimap. The full sync of all folders on all my IMAP accounts takes around 12 seconds, but the additional flexibility in this setup allows for easy customization: I do the full sync only every 30 minutes and usually only update Inboxes every 3 minutes. This quick sync takes < 2 seconds.
  • Using a local maildir for the mail handling is much faster than doing IMAP queries constantly when you switch folders.
  • Sending mail through a relay script, even if you are always online, is much faster: If you send a 3MB mail it will just be saved on hard disk and you can immediately continue working. Within the next minutes it is automatically sent in the background.
  • Using mu for mail search on a local maildir is a gazillion times faster for indexing and slightly faster for searching
• Has on-the-fly address book: Mutt has support for external address sources so we can use a specialized tool that does this job well.
• Highly customizable: Things you see here can be adapted to your needs easily.

For ages I have been using Mozilla Thunderbird for my mail but over the time it became slower and slower and I was quite frustrated about some things not being customizable, Keyboard shortcuts being the most prominent example. For long time I always wanted to use emacs to write my mail. Some of its editing features are already hardwired in my brain.

The two parts of mail, namely getting your mail and sending mail are delegated to separate programs here. In more integrated environments you don't see any difference between these two, but having them in split programs gives you much more flexibility.

In the examples you will find configuration for two dummy accounts "acc1" and "acc2" which will be synced to sub-folders of a local Maildir "~/Mail". The layout is as follows with "folderX" representing mail folders on the IMAP server. The special mu folder will become clear below.

~/Mail
  -> acc1
    -> acc1 
    -> acc1.folder1
    -> acc1.folder2
    ...
  -> acc2
    -> acc2 
    -> acc2.folder1
    -> acc2.folder2
  -> mu-search

To save login information in gnome-keyring we will use two scripts. This has to be done only once. The first one will store your smtp passwords and comes with the vanilla distro of msmtp but is finally shipped with gentoo msmtp. You can also get it from the tarball that comes with this page. For every smtp server you want to use to send mail do the following:

./msmtp-gnome-tool.py -s --server "mail.acc1.com" --username "acc1user"

This will allow us to use msmtp to send mail later. Now for fetching mail we will use a slightly different script that you can fetch here or find in the tarball. I have learned this trick from here.

./gnome-keyring-add-imap-password.py

Run it for every imap server you want to connect to and enter your login credentials for these servers. What we have done is basically a more modern variant of a netrc file.

offlineimap is a python script for bidirectional sync between a local maildir and imap servers. Here is a sample configuration file. It has two accounts that are synced. You should read the example configuration file now and set up your accounts accordingly. The example file will output information using the TTYUI user interface that is suited for calling it from the command line. Later we will run it from a cron job and choose the silent UI.

[general]
accounts = acc1, acc2
maxsyncaccounts = 5
ui = TTY.TTYUI
pythonfile=~/bin/offlineimap-helpers.py
socktimeout = 90

[Account acc1]
localrepository = acc1local
remoterepository = acc1remote
autorefresh = 2

[Account acc2]
localrepository = acc2local
remoterepository = acc2remote
autorefresh = 4

[Repository acc1local]
type = Maildir
localfolders = ~/Mail/acc1

[Repository acc2local]
type = Maildir
localfolders = ~/Mail/acc2

[Repository acc1remote]
type = IMAP
remotehost = imap.acc1.com
remoteusereval = get_username("imap.acc1.net")
remotepasseval = get_password("imap.acc1.net")
nametrans = oimaptransfolder_acc1
ssl = yes

# Folders to get:
folderfilter = lambda foldername: foldername in [
             'INBOX', 'Drafts', 'Sent', 'archiv']

[Repository acc2remote]
type = IMAP
remotehost = imap.acc2.net
remoteusereval = get_username("imap.acc2.net")
remotepasseval = get_password("imap.acc2.net")
nametrans = oimaptransfolder_acc2
ssl = yes
port = 993

One of the coolest things about offlineimap is that you can inject arbitrary python code. The file specified with

pythonfile=~/bin/offlineimap-helpers.py

contains python functions that I used for two purposes: Fetching passwords from the gnome-keyring and translating folder names on the server to local foldernames. The python file should contain all the functions that are called here. get_username and get_password are part of the interaction with gnome-keyring and not printed here. Find them in the example file that is in the tarball or here. The folderfilter is a lambda term that, well, filters which folders to get. oimaptransfolder_acc2 translates remote folders into local folders with a very simple logic. The INBOX folder will simply have the same name as the account while any other folder will have the account name and a dot as a prefix. offlineimap handles the renaming correctly in both directions:

import re

def oimaptransfolder_acc1(foldername):
    if(foldername == "INBOX"):
        retval = "acc1"
    else:
        retval = "acc1." + foldername
    retval = re.sub("/", ".", retval)
    return retval

def oimaptransfolder_acc2(foldername):
    if(foldername == "INBOX"):
        retval = "acc2"
    else:
        retval = "acc2." + foldername
    retval = re.sub("/", ".", retval)
    return retval

After setting this up you should backup your email (do it now, don't skip it!) and try running offlineimap from the command line. If everything works, you should see informative output and find your local maildir populated with the appropriate subdirectories. By default offlineimap keeps running and resyncs the accounts after a waiting time that you can configure in .offlineimaprc. You can kill it with C-c. In this guide we will restart offlineimap for every single run and manage the timing via cron.

At this point you should set up your mail client to read a local maildir. I use mutt, but there are other good ones like gnus, or even Mozilla Thunderbird. Yes, right! You could do all this voodoo here and then use Thunderbird to work on the local maildir and see how fast it actually can be if it is not tight to keeping several IMAP connections alive.

Here is some rudimentary mutt configuration:

# Set up mutt to use our local maildir:
set mbox_type=Maildir
set folder=$HOME/Mail
set spoolfile="+/acc1/acc1"  # This is your 'primary inbox'

mailboxes ="acc1/acc1" ="acc2/acc2" ="acc1/acc1.folder1" # ... Keep going with more. 

# One very useful feature are folder hooks which change
# options depending on the folder you are viewing:

# for instance you could do
folder-hook acc1 'set pgp_autosign postponed="+acc1/acc1.drafts" record="+acc1/acc1.sent"
folder-hook acc1 'set from="Egon Olsen<egon@acc1.com>" signature="~/.signature-acc1"
folder-hook acc1 'set sendmail="msmtp-enqueue.sh -a acc1"; 
macro index,pager a "s=acc1/acc1.archive<enter>"'

You get the point. Read the manual for more information about mutt configuration. Two points deserve mention:

1. We set sendmail to a script msmtp-enqueue.sh which accepts an account option -a and still has to be set up. See below
2. I'm really hooked to keeping all my mail forever so I introduced the 'a' key for archiving mail in the appropriate folder. This macro implements this: When pressing a on the index or in the pager the mail is moved to the respective archive folder.

You should now be able to read the synced email. You can also try to delete some email or move it around and sync your changes back to the server by running offlineimap again (if it is not still running).

msmtp is a software the gets mail from your computer to an smtp server that will send it. In Thunderbird this is just part of the big bulky package. msmtp is configured through the file ~/.msmtprc

# Set default values for all following accounts.
defaults
tls on
tls_trust_file /etc/ssl/certs/ca-certificates.crt
logfile ~/.msmtp.log

# Passwords are stored in gnome-keyring,

# A mail service
account acc1
host mail.acc1.net
from me@acc1.com
tls_starttls off
port 465
auth on
user me@acc1.com

# Another sendmail provider
account acc2
from me@acc2.com
host smtp.acc2.com
# tls_certcheck off
tls_min_dh_prime_bits 512
auth on
user me
port 587

# Set a default account
account default : acc1

See the manual and various web pages for more examples of configuring msmtp. One of my mail servers sends too short primes for recent tls libraries. It took me quite a while to figure out what is going on so I left the respective option in here, just in case you run into this trouble. Basically you can pass on tls options through msmtp's configuration (like min_dh_prime_bits in this case). When everything is set up you should be able to send mail with the well named program 'mail': Try it with

echo "... ok, well... it is. Sorry for the noise" | mail -s "This is not a drill!!!" your@mail.com

Now, as a homework, think about a cool script that sends email!

When you run the above bit you will notice a short delay for sending the email. I find this a bit annoying and also the above will only work when you are online. Remedy comes through mail-relay scripts which will simply cache the calls to mail on you hard drive. Copy msmtp-runqueue.sh, msmtp-enqueue.sh, msmtp-listqueue.sh to ~/bin and modify them in case you want your queue to be saved in a location different from $HOME/.msmtpqueue. Now queing email to be sent from acc1 is as as simple as running "msmtp-enqueue.sh -a acc1" instead of 'mail' or sendmail. Configure your mail program to use 'msmtp-enqueue.sh -a accountname' as the way to send email. You can view the queue with msmtp-listqueue.sh of course and actually send the mail with msmtp-runqueue.sh. We will take care of automatization in the next step:

I wrote the following script to implement all the mail transfer if an internet connection is present. One thing that I always wanted and that is quite easy to implement now is a quick sync for the INBOXes (every 3 minutes, say) and a more heavy sync for everything (archive folders with several GB) only every 30 minutes. This script will check if offlineimap is not running and an internet connections exists. In this case it will check the file ~/.offlineimap_lastlongsynctime for the last (unix-) time of a full sync. If this is shorter than ${longsyncdiff} ago then do a quick sync, otherwise do a long one. Additionally it will send mail that has been saved using msmtp-queing every time it finds an internet connection. Read the script and understand what it is doing before you run it!

#!/bin/bash

## WARNING: No guarantees. This might eat your mail in the case that it wrongly detects that
## offlineimap is not running and runs it again. (although offlineimap will also try to catch that case)

# -o run only once
# -f specify folders
# -u choose interface to be quit except in case of errors
# -l log to ~/.offlineimap.log

longcommand="offlineimap -o -u Noninteractive.Quiet"
shortcommand="offlineimap -o -f INBOX,IML -u Noninteractive.Quiet"
sendmailcommand="/home/tom/bin/msmtp-runqueue.sh"
# Time in seconds between long syncs:
longsyncdiff=1800 # 30 Minutes

# Check if offlineimap is running:
pid=$(pgrep -f "/usr/bin/offlineimap")
if [[ ${pid} -gt 0 ]] ; then
    echo "Offlineimap is running with pid ${pid}"
    exit 1
fi

# Check connection status
if ! ping -c1 www.google.com > /dev/null 2>&1; then 
    # Ping could be firewalled ...
    # '-O -' will redirect the actual html to stdout and thus to /dev/null
    if ! wget -O - www.google.com > /dev/null 2>&1; then
        # Both tests failed. We are probably offline 
        # (or google is offline, i.e. the end has come)
        exit 1;
    fi
fi

# We are online: So let's get mail going first
(${sendmailcommand} &> ~/.msmtp-queue.log) &

# Now we determine what to do based on the last time we did things:

if [ -e ~/.offlineimap_lastlongsynctime ] ; then 
    # Last sync file exists
    lastlongsync=$(<~/.offlineimap_lastlongsynctime) >/dev/null # The unix time of the last long sync
    curtime=$(date +%s)
    timediff=$(( curtime - lastlongsync ))
    if [ ${timediff} -gt ${longsyncdiff} ]; then
        echo ${curtime} > ~/.offlineimap_lastlongsynctime
        exec ${longcommand}
    else
        exec ${shortcommand}
    fi
else
    # Do the long sync
    echo ${curtime} > ~/.offlineimap_lastlongsynctime
    exec ${longcommand}
fi 

This script should be run every 3 minutes and we use cron for this. One minor obstacle is that a job run from cron will not inherit certain variables from the environment resulting in the nasty feature that your gnome-keyring cannot be contacted. You can read about the gory details here. Upshot: You should export the session id of your dbus session to somewhere in your home directory on login. Create a script export_x_info.sh

#!/bin/bash
# Export the dbus session address on startup so it can be used by cron
touch $HOME/.Xdbus
chmod 600 $HOME/.Xdbus
env | grep DBUS_SESSION_BUS_ADDRESS > $HOME/.Xdbus
echo 'export DBUS_SESSION_BUS_ADDRESS' >> $HOME/.Xdbus

And execute it upon (graphical) login. Now you can finally use 'crontab -e' to edit your users crontab and add the offlineimapdaemon script that transfers all mails. I've set it to run every three minutes:

*/3 * * * * source ~/.Xdbus; /home/tom/bin/offlineimapdaemon.sh

With this the basic setup is finished. Up next: mail search and address book.

mu (maildir utils) is a mail search engine based on xapian. It has many features and a very helpful upstream. After installing it we need to build the index:

mu index --maildir "~/Mail"

This will take while when run first, subsequent runs are faster. It will leave behind a xapian database in the ~/.mu directory (but this is of course configurable if you like).

To perform a basic search, run 'mu find ${searchstring}':

$ mu find penguin
Mon 04 Jun 2007 09:11:26 PM CEST rc5-request@lists.distributed.net rc5 Digest, Vol 39, Issue 3
Wed 06 Jun 2007 02:34:56 PM CEST rc5-request@lists.distributed.net rc5 Digest, Vol 39, Issue 4
Tue 28 Aug 2007 09:46:40 PM CEST rc5-request@lists.distributed.net rc5 Digest, Vol 41, Issue 5
Sun 02 Sep 2007 02:32:47 AM CEST rc5-request@lists.distributed.net rc5 Digest, Vol 42, Issue 1
Wed 11 Jun 2008 11:13:15 AM CEST rc5-request@lists.distributed.net rc5 Digest, Vol 48, Issue 2

The output is a list of mails containing the word penguin. Note that this will not find 'penguins', as the search is word-based. (Substring search is expansive!)

To use this from mutt we will use mu's option to create a Maildir with symlinks to the results. This is the folder 'mu-search' that showed up in the Maildir above. Here is a snippet of the mutt configuration that shows details.

mailboxes [... all the other mailboxes ...] "=mu-search"

# mu search:
macro index "\Cs" "<shell-escape>mu find --clearlinks --linksdir ~/Mail/mu-search --format links "
macro index "<Esc>s" "<change-folder-readonly>~/Mail/mu-search\n" "display mu-find results"

The last two lines define macros. As C-s is hard-wired to 'search' in my brain, I also want to use it in mutt. It will present me a command line already containing the call tu mu with the relevant options "–clearlinks" for clearing left over links from the last search and "–linksdir" and "–format" doing the obvious things. The second macro binds Esc-s to changed to the results folder. Note that you cannot operate on the mail but only browse it. Therefore the folder will be opened readonly.

The final step is to edit your crontab to rebuild the index regularly.

I use lbdb, the little brother's database. When search is cheap it is best to have a database that is filled with a lot of addresses. My heuristics is to include every address that I ever sent mail to. lbdb maintains a list of email addresses and provides them to mutt as a completion function. First we fill the database:

To fill lbdb from cron I use the following script refreshaddress.sh:

#!/bin/sh

acc1=~/Mail/acc1/acc1.sent/cur
acc2=~/Mail/acc2/acc2.Sent/cur

parsemail () {
    cat $1 | lbdb-fetchaddr
}

parsemaildir () {
    for mailfile in $( find $1 -type f -mtime -5 ) ; do
        parsemail ${mailfile}
    done
}

# The IFS variable saves the file name separator 
# which we will temporarily set to \n so that the
# spaces in Gmail folders will work

for i in "${acc1}" "${acc2}" ; do 
    o=${IFS}
    IFS=$(echo -en "\n\b")
    parsemaildir "${i}"
    IFS=o
done

lbdb comes with a program lbdb-fetchaddr which fetches email-addresses from a mail header which it reads from stdin. What this script will do is cycle through all the outboxes and let lbdb-fetchaddr process all mails that are not older than 5 days. When you run this the first time, choose a longer period or no period to fill the database. One thing to not worry about: duplicates will be auto-removed when searching using lbdb. Now setup your mail client to use lbdb for address look up. Here is how to do it in mutt:

# Address look up with lbdb
set query_command="lbdbq '%s'"

Now you can just complete addresses and names by pressing C-t wherever mutt asks you for an email address.

My user's crontab finally looks as follows.

*/3 * * * * source ~/.Xdbus; ~/bin/offlineimapdaemon.sh
13 11 * * * mu index --maildir ~/Mail
15 11 * * * ~/bin/refreshaddress.sh

This means that every three minutes the mail update script is run and daily at 11:13 mu updates its database while at 11:15 the address database is refreshed.