A unix style mail setup

Bitcoin donations accepted: 17BdguYxvaL2gsmogFgUkzjqnGCFGTBYWi

Introduction and Rationale

Why IMAP

These days you should not use anything else. I want to have access to email from different devices, mostly my phone and my laptop and statuses of mails should always be in sync.

What's New

Jan 2013: Improved mu search command, thanks Frederik Fischbach

July 2012: Overhaul of the entire guide in progress

Feb 2011: The mail search engine is now mu instead of mairix.

Some requirements for my setup:

  • Allows me to write mails with emacs (or whatever text editor I like best): In my setup I use the mutt mail client which naturally allows you to use the editor of your choice, in my case an emacsclient.
  • Works offline and online: In this setup there is a clear separation between two things:
    • the handling of your mail locally (Writing, reading, moving, archiving, …). This is all done in your local maildir and completely independent of any internet connection.
    • Transfer of mail to and from the server. This is done by a cron job in the background and independent of the local handling of mail. If you finished composing a new mail it will be queued and automatically sent the next time you are online.
  • Seamless integration of multiple identities In this setup I use mutt's folder-hooks to change many aspects like outgoing mailserver, signature, identity.
  • PGP support: Just choose a mail client that supports it.
  • IMAP on multiple servers: I use offlineimap that syncs quickly in parallel from many different sources.
  • Fast: Everybody wants speed. I don't want to wait for an internet connection when I hit the "Send" button (which, in fact, is not a button…)
    • Using the mutt mail client is fast a priori, e.g. because you never touch the mouse.
    • Fetching mail is much faster with offlineimap. The full sync of all folders on all my IMAP accounts takes around 10 seconds, but the additional flexibility in this setup allows for easy customization: I do the full sync only every 30 minutes and usually only update Inboxes every 3 minutes. This quick sync takes less than 2 seconds.
    • Using a local maildir for the mail handling is much faster than doing IMAP queries constantly when you switch folders (especially on an SSD).
    • Sending mail through a relay script, even if you are always online, is much faster: If you send a 3MB mail it will just be saved on hard disk and you can immediately continue working. Within the next minutes it is automatically sent in the background.
    • Using mu for mail search on a local maildir is a gazillion times faster both for indexing and for searching.
  • Has on-the-fly address book: Mutt has support for external address sources so we can use a specialized tool that does this job well, in this case it will also be realized using mu.
  • Highly customizable: Things you see here can be adapted to your needs easily.

For ages I have been using Mozilla Thunderbird for my mail but over the time it became slower and slower and I was quite frustrated about some things. For instance I would want to customize keyboard shortcuts. I also want to write my email with emacs, but maybe not go the entire way of using a lisp based mail client. These days mu in fact comes with a small mail client inspired by notmuch.

The two parts of mail, getting your mail and sending mail are delegated to separate programs here. In more integrated environments you don't see any difference between these two, but having them in split programs gives you much more flexibility.

For the examples we will assume that there are two IMAP accounts "acc1" and "acc2" and each of them is synced into a separate sub-folder of a local Maildir "~/Mail". The local directory layout is as follows with "folderX" representing mail folders on the IMAP server. The special mu folder will become clear below.

~/Mail
  -> acc1
    -> acc1 
    -> acc1.folder1
    -> acc1.folder2
    ...
  -> acc2
    -> acc2 
    -> acc2.folder1
    -> acc2.folder2
  -> mu-search

Disclaimer

Please double check everything you find here before applying it to your mail. Keep backups! Read the scripts before running them! Especially the bi-directional sync can lead to surprises. If you delete a lot of mail locally, for instance because you synced it to your local machine twice, running offlineimap on this local repository will delete the mail remotely. I take no responsibility for any harm to your pets or email. All scripts without other notice are copyrighted by myself and distributed under the GPLv2 or later at your choice. This guide is published under a Creative Commons Attribution-ShareAlike 3.0 Unported License.

The tools

These are the tools that will be used in no particular order. They have been selected for implenting the unix philosophy of having one tool for one job. All of them are available in Gentoo.

  • offlineimap Clone IMAP accounts to your local machine and back.
  • msmtp Send mail easily via a mail-relay.
  • vixie-cron Run jobs at a specified time of the day
  • mutt The mail client that sucks less
  • emacs The versatile real time display editor
  • python The glue.
  • gnome-keyring In lack for a proper 'generic' tool.
  • mu Finding email quickly

Implementation

The first task to solve is the safe storage of passwords. They should not be saved in plaintext on the harddisk. The tools that we use here all support plain text passwords in their configuration files, but this is not an option that you should consider at all. We use gnome-keyring to save passwords somewhat more securely. For KDE users, at some point kwallet may be supported, but I don't know how to do that at the moment. Maybe at some point there will be a unified password safe?

Gnome Keyring

To save login information in gnome-keyring we will use two scripts. This has to be done only once. The first one will store your smtp passwords and comes with the vanilla distro of msmtp and also with Gentoo's msmtp package (it is at /usr/share/msmtp/msmtp-gnome-tool). Usage is simple: for every smtp server you want to use to send mail do the following:

./msmtp-gnome-tool.py -s --server "mail.acc1.com" --username "acc1user"

If everything works out your password will be asked for and stored in the keyring. This will allow us to use msmtp passwordless later. For fetching mail we will use a different script that you can fetch here or from github. I have learned this trick from here.

./gnome-keyring-add-imap-password.py

Run it for every imap server you want to connect to and enter your login credentials for these servers. (One could argue that we implemented are modern variant of a netrc file.)

Collecting Email with offlineimap

offlineimap is a python program for bidirectional sync between a local maildir and imap servers. Its documentation is not always up to date, but it has a very friendly mailing list to which you can subscribe here. In any case you should read 'man offlineimap' (which, in fact, contains bits from this guide).

Let's look at a sample configuration file with two accounts that are synced. The example file will output information using the 'ttyui' user interface that is suited for calling it from the command line. Later we will run it from a cron job and choose the 'silent' UI.

[general]
accounts = acc1, acc2
maxsyncaccounts = 5
ui = ttyui
pythonfile=~/bin/offlineimap-helpers.py
# Die after 120 seconds of nothing. Probably the connection died ...
socktimeout = 120

[Account acc1]
localrepository = acc1local
remoterepository = acc1remote
autorefresh = 2
quick=0

[Account acc2]
localrepository = acc2local
remoterepository = acc2remote
autorefresh = 4
quick=2

[Repository acc1local]
type = Maildir
localfolders = ~/Mail/acc1
nametrans = localtransfolder_acc1

[Repository acc2local]
type = Maildir
localfolders = ~/Mail/acc2
nametrans = localtransfolder_acc2

[Repository acc1remote]
type = IMAP
remotehost = imap.acc1.com
remoteusereval = get_username("imap.acc1.net")
remotepasseval = get_password("imap.acc1.net")
nametrans = oimaptransfolder_acc1
ssl = yes
# insert your server certificate's fingerprint here if needed
# cert_fingerprint=1c671ac670b1fbbfddeddd3541dbcc06148061e2
# But also see this thread: 
# http://comments.gmane.org/gmane.mail.imap.offlineimap.general/2312

# Folders to get:
folderfilter = lambda foldername: foldername in [
	     'INBOX', 'Drafts', 'Sent', 'archiv']

[Repository acc2remote]
type = IMAP
remotehost = imap.acc2.net
remoteusereval = get_username("imap.acc2.net")
remotepasseval = get_password("imap.acc2.net")
nametrans = oimaptransfolder_acc2
ssl = yes
port = 993

One of the coolest things about offlineimap is that you can use arbitrary python code to fill the variables. The file containing the python code is specified with the line

pythonfile=~/bin/offlineimap-helpers.py

In this simple example we have code for fetching passwords from the gnome-keyring and translating folder names on the server and local back and forth. For instance, get_username and get_password are part of the interaction with gnome-keyring and not printed here. You find them in the example file that is on github or here.

One can also specify python code directly in the configuration file. For instance, folderfilter is a lambda term that, well, filters which folders to get. The functions to translate folders use a very simple logic. The INBOX folder should have the same name as the account while any other folder will have the account name and a dot as a prefix. Then offlineimap handles the renaming correctly in both directions:

import re

def oimaptransfolder_acc1(foldername):
    if(foldername == "INBOX"):
	retval = "acc1"
    else:
	retval = "acc1." + foldername
    retval = re.sub("/", ".", retval)
    return retval

def localtransfolder_acc1(foldername):
    if(foldername == "acc1"):
	retval = "INBOX"
    else:
	# remove leading '.acc1'
	retval = re.sub("acc1\.", "", foldername)
    retval = re.sub("\.", "/", retval)
    return retval

One must make sure that these two functions are exact inverses of each other. See these threads for more information.

After preparing the configuration you should backup your email (do it now, don't skip it!) and try running offlineimap from the command line. If everything works, you should see informative output and find your local maildir populated with the appropriate subdirectories. By default offlineimap keeps running and resyncs the accounts after a waiting time that you can configure in .offlineimaprc. You can kill it with C-c. In this guide we will restart offlineimap for every single run and manage the timing via cron. This will allow us to run it with different parameters on different schedules (quick sync!) and also avoid issues with hung instances after network issues.

Setting up your mail client

At this point you should set up your mail client to read a local maildir. I use mutt, but there are other good ones like gnus, or even Mozilla Thunderbird. Yes, right! You could do all this voodoo here and then still use Thunderbird to work on the local maildir and see how fast it actually can be if it is not keeping several IMAP connections alive.

Here is some rudimentary mutt configuration:

# Set up mutt to use our local maildir:
set mbox_type=Maildir
set folder=$HOME/Mail
set spoolfile="+/acc1/acc1"  # This is your 'primary inbox'

# I manually manage the mailbox command so that my mailboxes come up
# in the order I want.
mailboxes ="acc1/acc1" ="acc2/acc2" ="acc1/acc1.folder1" # ... Keep going with more. 

# One very useful feature are folder hooks which change
# options depending on the folder you are viewing:

# for instance you could do
folder-hook acc1 'set pgp_autosign postponed="+acc1/acc1.drafts" record="+acc1/acc1.sent"
folder-hook acc1 'set from="Egon Olsen<egon@acc1.com>" signature="~/.signature-acc1"
folder-hook acc1 'set sendmail="msmtp-enqueue.sh -a acc1"; 
macro index,pager a "s=acc1/acc1.archive<enter>"'

You get the point. Read the manual for more information about mutt configuration. Two points deserve mention:

  1. We set sendmail to a script msmtp-enqueue.sh which accepts an account option -a and still has to be set up. See below
  2. I'm really hooked to keeping all my mail forever so I introduced the 'a' key for archiving mail in the appropriate folder. This macro implements this: When pressing a on the index or in the pager the mail is moved to the respective archive folder.

You should now be able to read the synced email. You can also try to delete some email or move it around and sync your changes back to the server by running offlineimap again (if it is not still running).

Sending email with msmtp

msmtp is a software the gets mail from your computer to an smtp server that will send it. msmtp is configured through the file ~/.msmtprc

# Set default values for all following accounts.
defaults
tls on
tls_trust_file /etc/ssl/certs/ca-certificates.crt
logfile ~/.msmtp.log

# Passwords are stored in gnome-keyring,

# A mail service
account acc1
host mail.acc1.net
from me@acc1.com
tls_starttls off
port 465
auth on
user me@acc1.com

# Another sendmail provider
account acc2
from me@acc2.com
host smtp.acc2.com
# tls_certcheck off
tls_min_dh_prime_bits 512
auth on
user me
port 587

# Set a default account
account default : acc1

See the manual and various web pages for more examples of configuring msmtp. One of my mail servers sends too short primes for recent tls libraries. It took me quite a while to figure out what is going on so I left the respective option in here, just in case you run into this trouble. Basically you can pass on tls options through msmtp's configuration (like min_dh_prime_bits in this case). When everything is set up you should be able to send mail with the well named program 'mail': Try it with

echo "... ok, well... it is. Sorry for the noise" | mail -s "This is not a drill!!!" your@mail.com

Now, as a homework, think about a cool script that sends email!

When you run the above bit you will notice a short but annoying delay while the mail is sent over the network. Remedy comes through a mail-relay scripts which will cache the calls to mail on you hard drive. To install it, copy msmtp-runqueue.sh, msmtp-enqueue.sh, msmtp-listqueue.sh to ~/bin and modify them in case you want your queue to be saved in a location different from ~/.msmtpqueue. The scripts should come with your distribtuion. On Gentoo they are located in usr/share/msmtp/msmtpqueue. Now queing email to be sent from acc1 is as as simple as running "msmtp-enqueue.sh -a acc1" instead of 'mail'. Configure your mail program to use 'msmtp-enqueue.sh -a accountname' as the way to send email. You can view your current queue with msmtp-listqueue.sh and actually send the mail with msmtp-runqueue.sh. We will take care of automatization in the next step.

The mail-update script

I wrote the following script to implement all the mail transfer if an internet connection is present. The script runs a 'quick sync' of the INBOXes (every 3 minutes, say) and a more heavy sync for everything (archive folders with several GB) only every 30 minutes. It will also check if another instance of offlineimap is running and if an internet connections exists. In this case it will check the file ~/.offlineimap_lastlongsynctime for the last (unix-) time of a full sync. If this is shorter than ${longsyncdiff} ago then do a quick sync, otherwise do a long one. Additionally it will send mail that has been saved using msmtp-queing every time it finds an internet connection. Read the script and understand what it is doing before you run it!

#!/bin/bash

## WARNING: No guarantees. This might eat your mail in the case that
## it wrongly detects that offlineimap is not running and runs it
## again. (although offlineimap will also try to catch that case)

# -o run only once
# -f specify folders
# -u choose interface to be quit except in case of errors
# -l log to ~/.offlineimap.log

longcommand="offlineimap -o -u Noninteractive.Quiet"
shortcommand="offlineimap -o -f INBOX,IML -u Noninteractive.Quiet"
sendmailcommand="/home/tom/bin/msmtp-runqueue.sh"
# Time in seconds between long syncs:
longsyncdiff=1800 # 30 Minutes

# Check connection status
if ! ping -c1 www.google.com > /dev/null 2>&1; then 
    # Ping could be firewalled ...
    # '-O -' will redirect the actual html to stdout and thus to /dev/null
    if ! wget -O - www.google.com > /dev/null 2>&1; then
	# Both tests failed. We are probably offline 
	# (or google is offline, i.e. the end has come)
	exit 1;
    fi
fi

# We are online: So let's get mail going first
(${sendmailcommand} &> ~/.msmtp-queue.log) &

# Check if offlineimap is running:
pid=$(pgrep -f "/usr/bin/offlineimap")
if [[ ${pid} -gt 0 ]] ; then
    echo "Offlineimap is running with pid ${pid}"
    exit 1
fi

# Now we determine what to do based on the last time we did things:

curtime=$(date +%s)
if [ -e ~/.offlineimap_lastlongsynctime ] ; then 
    # Last sync file exists
    lastlongsync=$(<~/.offlineimap_lastlongsynctime) >/dev/null # The unix time of the last long sync
    timediff=$(( curtime - lastlongsync ))
    if [ ${timediff} -gt ${longsyncdiff} ]; then
	echo ${curtime} > ~/.offlineimap_lastlongsynctime
	exec ${longcommand}
    else
	exec ${shortcommand}
    fi
else
    # Do the long sync
    echo ${curtime} > ~/.offlineimap_lastlongsynctime
    exec ${longcommand}
fi

To run this script every 3 minutes we use cron. One minor obstacle is that a job run from cron will not inherit certain variables from the environment that your shell lives in. Therefore from the program run by cron, your gnome-keyring may be unreachable. You can read about the gory details here. Upshot: You should export the session id of your dbus session to somewhere in your home directory on login. To do so we create a script export_x_info.sh

#!/bin/bash
# Export the dbus session address on startup so it can be used by cron
touch $HOME/.Xdbus
chmod 600 $HOME/.Xdbus
env | grep DBUS_SESSION_BUS_ADDRESS > $HOME/.Xdbus
echo 'export DBUS_SESSION_BUS_ADDRESS' >> $HOME/.Xdbus

This script should be executed once after (graphical) login.

Finally we use 'crontab -e' to edit the cron table and add the offlineimapdaemon script that transfers all mails. I've set it to run every three minutes:

*/3 * * * * source ~/.Xdbus; /home/tom/bin/offlineimapdaemon.sh

With this the basic setup is finished. We can now send mail and our local maildir is synced on a regular basis. Up next: mail search and address book.

Searching mail with mu

mu (maildir utils) is a mail search engine based on xapian. It has many features and a very helpful upstream. After installing it we need to build the index:

mu index --maildir "~/Mail"

This will take while when run first but subsequent runs are much faster. The result is a xapian database, saved in the ~/.mu directory (but this is of course configurable if you like).

To perform a basic search, run 'mu find ${searchstring}':

$ mu find penguin
Mon 04 Jun 2007 09:11:26 PM CEST rc5-request@lists.distributed.net rc5 Digest, Vol 39, Issue 3
Wed 06 Jun 2007 02:34:56 PM CEST rc5-request@lists.distributed.net rc5 Digest, Vol 39, Issue 4
Tue 28 Aug 2007 09:46:40 PM CEST rc5-request@lists.distributed.net rc5 Digest, Vol 41, Issue 5
Sun 02 Sep 2007 02:32:47 AM CEST rc5-request@lists.distributed.net rc5 Digest, Vol 42, Issue 1
Wed 11 Jun 2008 11:13:15 AM CEST rc5-request@lists.distributed.net rc5 Digest, Vol 48, Issue 2

The output is a list of mails containing the searched string. Note that this will not find 'penguins', as the search is word-based. (Substring search is expansive!) The query language is the xapian query language.

A useful feature of mu is that it can create a Maildir containing symlinks to the search results. We will use this feature to connect mu to mutt. (Remember the folder 'mu-search' that showed up in the Maildir above?). Here is a snippet of the mutt configuration that shows details.

mailboxes [... all the other mailboxes ...] "=mu-search"

# mu search:
macro index "\Cs" "<enter-command>unset wait_key<enter><shell-escape>read -p'mu query: ' x; \
      mu find --clearlinks --linksdir ~/Mail/mu-search --format links \$x<enter>\
      <change-folder-readonly>~/Mail/mu-search\n" "mu find"
macro index "<Esc>s" "<change-folder-readonly>~/Mail/mu-search\n" "display mu-find results"

The last two lines define macros. As C-s is hard-wired to 'search' in my brain, I also want to use it in mutt. It will present a command line asking for the search query and switch to the results folder automatically. The options to mu find are "–clearlinks" for clearing left over links from the last search and the obvious "–linksdir" and "–format". The second macro binds Esc-s to jump to the results folder. Note that you cannot operate on the mail but only browse it. Therefore the folder will be opened readonly.

The final step is to edit your crontab to rebuild the index regularly, e.g. on every 13th minute of the hour.

*/3 * * * * source ~/.Xdbus; ~/bin/offlineimapdaemon.sh
13 */1 * * * mu index --maildir ~/Mail

Address book.

To build an addressbook we will also use mu which also finds contacts using cfind:

$ mu cfind Linus
Linus Torvalds torvalds@linux-foundation.org

When search is cheap it is best to have a database that is filled with lots of addresses. My heuristics is to include every address that I ever sent mail to. So the trick will be to have a second database that is an index of only the sent mail folders. It will be saved in "~/.mu-sent-index".

#!/bin/sh

# Set up an array with outboxes
sentboxes=( "~/Mail/acc1/acc1.sent/ ~/Mail/acc2/acc2.Sent" )

# Run mu on each of them
for dir in "${sentboxes[@]}"
do
    mu index --nocleanup --maildir="${dir}" --muhome=~/.mu-sent-index
done

Now we setup the mail client to use mu for address lookup. Here is how to do it in mutt:

# Adress lookup with lbdb
set query_command="mu cfind --muhome=~/.mu-sent-index --format=mutt-ab '%s'"

Now you can just complete addresses and names by pressing C-t wherever mutt asks you for an email address.

Cron

My user's crontab finally looks as follows.

*/3 * * * * source ~/.Xdbus; ~/bin/offlineimapdaemon.sh
13 */1 * * * mu index --maildir ~/Mail
15 11 * * * ~/bin/refreshaddress.sh

Consequently, every three minutes the mail update script is run, at every 13th minute of the hour the mail index will be rebuilt, and daily at 11:15 mu updates its address database.

Questions ?

I offer support for this guide via email: tomka@g.o and will post questions and answers on this page.

Creative Commons License
This work is licensed under a Creative Commons Attribution-ShareAlike 3.0 Unported License.

Date: 2013-01-22

Validate