If you wake up looking forward to booting your shiny new Windows 10 with your flashy outlook, well this is way too advanced for you, pass on. One advice though: get a grown-up OS ;-)
Since I moved out of gmail (for obvious privacy reasons), I was using simultaneously mutt and roundcube to access my emails.
mutt is a really powerful tool to help you quickly get things done (reading, writing, searching emails). Roundcube on the other hand allowed me to access my emails from anywhere without having to setup all the needed tools.
Those tools were nice but I was missing some features. I thus decided to move to something faster, more complete with specific features:
- TUI: I hate using the mouse and want something fast with great keybindings preferably vim-like
- fast search: Ability to search in all my emails quickly (independently if the searched term is in the body or the header)
- tags: Ability to tag my emails
- webmail-compatible: Retro-compatibility with webmail (roundcube or similar) such that I could still access my emails from another computer without having to setup all needed tools
- Auto-tag: Ability to auto-tag my emails (specific mailing-list to a specific tag/folder, ...)
So in short I wanted to be able to read my emails from a TUI that allows me to tag (auto-tag) my emails but also allows in some way to synchronize the changes (tags, flags, ...) with the remote IMAP server. Such that I would be able, if needed, to read my emails remotely and still be able to find my way across the structure (folder hierarchy for roundcube).
I first came across sup which is a really nice mail user agent. Sup didn't match my criteria for two reasons:
- inability to synchronize tags on IMAP server (especially regarding folder structure).
- ruby (nothing against ruby but I'm more into Python)
I then discovered notmuch which is a rewrite of sup with performance in mind. Notmuch is surrounded with very neat tools in the UNIX-philosophy "Do one thing and do it right". notmuch is not as complete as sup, it does not allow to read or write emails but only cares for indexing, searching and tagging emails. The rest is done by other specific tools as described below.
Before I dig into the different blocks that are now building my email stack I would like to mention that there are three ways (with pros and cons) to synchronize your email's tags with a remote IMAP servers:
-
using X-Keywords This method will add (or complete) the header's key named X-Keywords with a list of tags (separated by a comma).
-
using X-labels As for previous solution, but by using the header's key named X-labels with tags separated by a space.
-
using folders Tags are considered as virtual folders and thus translated into folders in maildir (filesystem) and on the remote mail server (IMAP server) and vice-versa.
I chose to use latest method for the following reasons:
- immutability of email: I didn't feel comfortable changing email's content
- webmail (retro-)compatibility: Roundcube is using folders to structure emails and has no understanding of X-labels or X-keywords. It thus would have been quite a mess when accessing my emails from a browser (flat structure, ...)
- sync only subset: To quickly sync your emails, you might want to only synchronize a specific subset of them, for example INBOX. If your emails are structured in folders on the remote IMAP server (maildir or similar), this is much easier to do
Now let's get down to the email stack itself. It is made of multiple tools, each of them doing a specific thing (and doing it right). It might seems a bit awkward at first but once setup, it is a killing email setup !
- offlineimap: sync emails with a remote IMAP server
- notmuch: index all emails and provide tagging and searching features
- notmuch auto-tagging: auto-tag new emails
- maildir-notmuch-sync: sync tags to folders and vice-versa
- msmtp: send emails
- alot: view, search and tag emails in a nice ncurses interface
- abook: store contacts and provide auto-completion when writing new emails
Now let's view all these tools in details ...
offlineimap - the email fetcher
offlineimap downloads emails from an IMAP server and store them locally. It also sync your changes back to the remote server.
Here's my config file:
# author: deadc0de
[general]
accounts = personal
pythonfile = TODO # path to below script
fsync = false
[Account personal]
localrepository = personal-local
remoterepository = remote
presynchook = maildir-notmuch-sync pre TODO # path to your maildir
postsynchook = maildir-notmuch-sync post TODO # path to your maildir
# use sqlite for quicker sync
status_backend = sqlite
# number of concurrent connection to imap server
maxconnections = 3
# auto-refresh every X minutes
autorefresh = 5
# quick refresh N times between the autorefresh
# won't update the flags though
quick = 10
[Repository personal-local]
type = Maildir
localfolders = TODO # maildir path
nametrans: local_to_remote # ext call
[Repository remote]
type = IMAP
remotehost = TODO # remote IMAP server
remoteuser = TODO # login username
remotepasseval = mailpasswd() # ext call
ssl = yes
realdelete = no
remoteport = TODO # remote port
sslcacertfile = /etc/ssl/certs/ca-certificates.crt
nametrans: remote_to_local # ext call
idlefolders = ['INBOX']
The script maildir-notmuch-sync used as a pre- and
post-hook is described in its own section below.
And the python script providing the functions for translating folder's names
(local_to_remote and remote_to_local)
and retrieving the password using gpg2 (mailpasswd):
# author: deadc0de
import re
import subprocess
# upload name translator
def local_to_remote(folder):
# TODO - do your translations with re here
return folder
# download name translator
def remote_to_local(folder):
# TODO - do your translations with re here
return folder
# password helper
def mailpasswd():
path = "TODO" # gpg file path
return subprocess.check_output(["/usr/bin/gpg2", "--batch", "-d", path]).strip()
For more information on how to configure/use offlineimap, see the official doc.
In order to avoid putting my password in clear text in the config, I used gpg-agent to query the password from a gpg encrypted file created with
# gpg key
echo "YOURPASSWORD" | gpg2 --encrypt --recipient "YOURNAME" -o <password-file>.gpg
or
# symmetric
echo "YOURPASSWORD" | gpg2 -c > <password-file>.gpg
Offlineimap can easily be used as a service to continuously refresh and sync your emails with following systemd unit:
[Unit]
Description=Offlineimap daemon
Requires=network-online.target
After=network.target
[Service]
User=TODO
ExecStart=/usr/local/bin/offlineimap
KillSignal=SIGUSR2
Restart=always
[Install]
WantedBy=multi-user.target
notmuch - the indexer
notmuch is a rewrite of sup with performance in mind. It allows to tag emails and search among them. It is very fast and able to handle very large quantity of emails.
Its configuration is very easy. You simply provide it with basic information on
your emails and specify the tags with which each new emails is to be tagged
with. This will be useful for maildir-notmuch-sync used below as well as for
auto-tagging.
notmuch is to be setup using notmuch setup. Then each time you have
synchronized your emails (with offlineimap), simply run notmuch
new to index new mails.
notmuch auto-tagging script
notmuch itself can be used to auto-tag new emails. The easiest way
is to add a specific tag to new emails (tofilter for example) and then
process all new emails right after calling notmuch new.
First make sure you add a specific tag to new emails through
notmuch's config tags=.... For the following example, the entry in notmuch
config would be:
...
tags=new,tofilter
...
This means that after running notmuch new, any new email will be
automatically tagged with two tags: new and tofilter. tofilter will be
used here to auto-tag emails while new will be used by maildir-notmuch-sync
described in following section.
Now let's say you want all emails coming from
debian-bugs-rc@lists.debian.org to be tagged with debian.
notmuch tag -tofilter +debian -- tag:tofilter and from:debian-bugs-rc@lists.debian.org
This will remove the tag tofilter and add the tag debian to all emails which are tagged tofilter and are from the debian bugs mailing list. For more information, see the doc about initial tagging.
Since a new tag is added to all new emails (tofilter in this example), make sure you remove it from all emails that you don't process. Otherwise you will have mails hanging around with the tag tofilter:
notmuch tag -tofilter -- tag:tofilter
Theses steps can be put in a bash script and run from maildir-notmuch-sync through the TAG_SCRIPT config.
maildir-notmuch-sync - the tags/folders sync script
This script was originally written by Ethan Schoonover. It is available on github under this link. The original script's purpose is to synchronize notmuch tags with remote IMAP servers and more specifically with gmail.
I modified it to be able to translate notmuch's tags into a maildir folder and thus get my emails structured on the IMAP server for access with webmail. My modified version is available here.
The modified version will create a new folder (locally in maildir and remotely through offlineimap) when a new tag is used for the first time. Also emails will be copied around depending on their tags.
Let's say for example that you tag a new email with two tags: family and toremember. This script will copy the email to two different folders in the maildir, the family folder and the toremember folder (and create them if they don't exist). When offlineimap will be run, those will be copied to the remote IMAP server to their respective folders.
It is very convenient but also has its drawbacks. A single email might co-exist in several different folders on your maildir and on your IMAP server.
alot - the MUA
alot is the interface to your emails. It is terminal based and fully compatible with notmuch.
Its main features (to me at least):
- modular and implemented in Python
- based on ncurses (through the urwid toolkit)
- vim/pentadactyl keybindings (although fully customizable)
- slick and powerful interface (buffer-based, colors, ...)
- PGP/MIME support
Some useful configs:
-
sync INBOX folder and refresh when hitting comma
, = shellescape --refocus=true 'offlineimap -o -f INBOX'; search tag:inbox OR tag:unread OR tag:flagged -
edit new email with vim
editor_cmd = 'vim -c "set textwidth=72" -c "set wrap" -c "set spell" -c "set nocp"' -
use msmtp to send emails
[accounts] [[personal]] realname = TODO # your real name address = TODO # your email address sendmail_command = msmtp --account=TODO -t sent_box = maildir://TODO # the sent maildir folder draft_box = maildir://TODO # the drafts maildir folder -
use abook to search for contacts
[accounts] [[[abook]]] type = abook abook_contacts_file = "TODO" # path to abook file
You can find more themes (with screenshots) here.
This (old) screencast shows the use of alot. A good resource is the alot documentation which lists all needed configuration options.
msmtp - the email sender
msmtp is a SMTP client that allows to easily transfer emails to an SMTP server. It is easily integrated into alot by providing an account in its config files as mentioned here.
Here's the complete manual on msmtp to help correctly configure msmtp.
You can also use gpg2/gpg-agent to provide a password to msmtp without writing it in clear text in the config file.
passwordeval "gpg2 --quiet --for-your-eyes-only --no-tty --decrypt <password-file>.gpg"
abook - the contact manager
One last thing missing to get a complete email stack is a contact manager. I found abook to fill all needed requirements. Moreover it is fully compatible with alot which uses it to auto-complete TO when redacting.
BTW this script converts vcard to abook.
Conclusion
I've been using that setup for some weeks now and am very happy with it. It is fast and the search feature is really doing it ! It takes some time to setup all the right configs and keybindings but once settle, it's a bomb ! Last but not least, TUI rocks !
Some more references: