Read and Compose Email in Emacs with Notmuch

It has been 18 months that I read & write my emails in Emacs. No need to say I have enjoyed the mouse-free experience brought by Emacs. Recently, I had to keep track a new email account. So I came across my old note written back then which I enhanced in this post. I made lots of updates subsequently including password management through pass, multi-accounts support etc. to make it as complete as possible.

Introduction

A full back-and-forth cycle of email consists to

1. Receive email through a program which synchronize emails locally from an email server.
2. Read email through a program (MUA) whose the UI offers an organized & handy presentation of emails.
3. Compose email in whatever editor.
4. Send email with a mail transfer agent or an interface of it.

I made the following choices which I will detail the configuration throughout this post:

1. Receive email: offlineimap.
2. Read email: notmuch.el.
3. Compose email: Emacs message mode.
4. Send email: smtpmail-multi to send email through multiple SMTP servers.

As you may have seen, except the reception of email, the remaining can be done within Emacs.

Receiving email

As stated above, we use offlineimap to fetch emails from potentially multiple mailbox. But most importantly, we store all emails locally for two purposes: 1. to be able to read emails offline, 2. to not mess up tags synchronization which may cause data loss. You may think that your huge mailbox would take a tremendous place in your disk. Well, I can say that if you pay attention to keep only one copy of your emails¹ After setting up offlineimap correctly, you can check this information by running fdupes -mr . under ~/.email/. , it should not take much. For instance, I have 2.6k emails taking 460MB of the disk.

Configure offlineimap

Install offlineimap with your favorite package manager² Note that, at the time of writing, offlineimap is in the process to port from python2 (2020-01-01 :coffin:) to python3, see OfflineIMAP/offlineimap3. . Then copy the minimal configuration (the path depends on your distribution).

1
cp /usr/share/doc/offlineimap/examples/offlineimap.conf.minimal ~/.offlineimaprc

Here is the relevant part of my configuration (~/.offlineimaprc) for reference. See the documentation in /usr/share/doc/offlineimap/examples/offlineimap.conf or Archwiki for more information. Note the postsynchook option at the account level: it’s an email tagging script which is run as soon as new email arrives. We will come soon to its content in this section. Remember what I said regarding the space taken by locally stored email? Well, they remain tiny provided that they are not duplicated elsewhere. That’s not always the case, for instance Gmail may store an email in the folder [Gmail].Important beside [Gmail].All Mail. You may consider to filter out the extra folders you don’t want as below with a python’s lambda expression or a function (see the documentation).

1
# ~/.offlineimaprc
2
[general]
3
accounts = Acc1_Gmail, Acc2, Acc3 # comma-separated list of accounts
4

5
[Account Acc1_Gmail]
6
localrepository = LocalAcc1
7
remoterepository = RemoteAcc1
8
postsynchook = ~/.email/postsync.sh # notmuch tagging script
9
utf8foldernames = yes
10

11
[Repository LocalAcc1]
12
type = Maildir
13
localfolders = ~/.email/my-acc1@gmail.com
14

15
[Repository RemoteAcc1]
16
type = Gmail
17
remoteuser = my-acc1@gmail.com
18
remotepass = password
19
sslcacertfile = /etc/ssl/certs/ca-certificates.crt
20
readonly = true # readonly if you don't want mess up with the 'unread' tag...
21
folderfilter = lambda foldername: foldername in ['[Gmail].All Mail']
22

23
[Account Acc2]
24
...  ...

Launch offlineimap automatically at boot

You would certainly want to launch automatically offlineimap at boot. This can be done with systemd. In my case, I have three accounts, it’s advised ³ OfflineIMAP community’s website : No, I’m not using maxconnections to create three separated systemd services and set maxsyncaccounts = 1 in ~/.offlineimaprc as we have done above.

Instead of write three different system service files, I write the following template unit file where the variable %i will match later with an account name in .offlineimaprc. (Note that you should avoid ”@” in the account name since systemd gives it a precise meaning).

1
# ~/.config/systemd/user/offlineimap@.service
2
[Unit]
3
Description=Sync mail with offlineIMAP for Account %i in .offlineimaprc
4
Documentation=man:offlineimap(1)
5

6
[Service]
7
ExecStart=/usr/bin/offlineimap -a %i -u basic
8
Restart=always
9
RestartSec=60
10

11
[Install]
12
WantedBy=default.target

Then run systemctl daemon-reload to load the new service file. The following commands enable the auto-start on boot and launch the service right now.

1
systemctl enable --user --now offlineimap@account-1.service
2
systemctl enable --user --now offlineimap@account-2.service
3
systemctl enable --user --now offlineimap@account-3.service

Note that account-* is the account name appeared in each [ Account XXX ] section. If everything goes well, offlineimap will sync emails on the next boot automatically.

Auto-tagging with notmuch

The next thing to do is email auto-tagging, without this feature your mailbox will be a nightmare. Again, install notmuch with your favorite package manager. We will write the script aforementioned so that email be filtered as soon as they are synced locally.

Configurate notmuch

Before starting to use notmuch, you must configure it. In particular, you have to set the database path, your email accounts which appeared in ~/.offlineimaprc, tagging rule for incoming email and tag to exclude by default when searching.

1
# ~/.notmuch-config
2
[database]
3
path=/home/firmart/.email
4

5
[user]
6
name=Firmin Martin
7
primary_email=my-acc1@gmail.com
8
other_email=my-acc2@gmail.com; my-acc3@gmail.com
9

10
[new]
11
tags=inbox;unread;
12
ignore=
13

14
[search]
15
exclude_tags=deleted;
16

17
[maildir]
18
synchronize_flags=true

Expose highly active addresses

The following command lists email-senders address sorted by decreasing amounts of emails sent⁴ Technically you can aggregate all duplicate email addresses with jq, but you would have to handle the case, comma-separated addresses, and the "First Last <first.last@gmail.com>" notation. It’s merely an example after all. .

1
notmuch show --format=json --body=false --entire-thread=false "*"
2
| jq '.[] | .[] | .[0].headers.From'
3
| sort | uniq -c | sort -n

Replace From by To to expose highly active mailing list.

The snippet above help us to find out the best contributors of our inbox to tag them properly.

Auto-tagging script

Here is my little shell script ~/.email/postsync.sh which is run once offlineimap finished to sync my emails. You might want to take a look at notmuch help search-terms to understand the syntax of tagging commands.

I identify several visibility categories of emails:

1. I don’t want to see them at all and they’re harmful => spam
2. I don’t want to see them at all => blacklisted
3. I want to see them but it doesn’t matter when => move out inbox
4. It’s important! => keep them in the inbox and tag them more

1
#!/usr/bin/env bash
2
# ~/.email/postsync.sh
3

4
# tag_new <tags> <search-term>
5
function tag_new { notmuch tag $1 -- tag:inbox and $2; }
6

7
# blacklist <search-term>
8
function blacklist { tag_new "-inbox -unread +deleted" $1; }
9

10
# spam <search-term>
11
function spam { tag_new "-inbox -unread +spam +deleted" $1; }
12

13
# security <search-term>
14
function security { tag_new "-inbox +Security" $1; }
15

16
# update : let notmuch process new mails
17
notmuch new
18

19
# blacklisting
20
notmuch tag -inbox -- tag:deleted and tag:inbox
21
blacklist "from:/.*@.*[.]pinterest[.]com/"
22
blacklist "from:/.*@linkedin[.]com/"
23
blacklist "from:/.*@quora[.]com/"
24
blacklist "from:noreply@medium.com"
25
blacklist "from:noreply@youtube.com"
26
## this list continue with 100+ addresses ...
27

28
# ... and spams
29
# `+spam' can't be found at all in notmuch if `exclude_tags=deleted;spam;'
30
# is set in the [search] section of `.notmuch-config'.
31
spam "from:esf@cnnsimail.com"
32
# ...
33

34
# Family first
35
tag_new "+family" "from:dad@gmail.com or from:mom@gmail.com"
36

37
# Friends
38
# ...
39

40
# Co-workers
41
# ...
42

43
# Mailing list
44
tag_new "-inbox +CoqClub" "to:coq-club@inria.fr or [Coq-Club]"
45

46
# Newsletter
47
tag_new "-inbox +SE.newsletter" "from:do-not-reply@stackoverflow.email"
48

49
# Universities
50
# ...
51

52
# Security (accounts/verification code/email confirmation/... etc.)
53
security "from:no-reply@accounts.google.com or accounts-noreply@google.com"
54
security "from:account-security-noreply@account.microsoft.com"
55

56
# and more ...
57

58
# From me
59
tag_new "-inbox -unread +FromMe" "from:my-main-gmail@gmail.com or from:univ-account@my-univ.fr or from:my-second@gmail.com"

Reading mail

notmuch.el

Follow the instructions given on the official website.

Key-bindings

The key-bindings I use are from evil-collection. They are quite different from the default ones. You can define new keybindings for different notmuch views (tree, show, hello, search, message) as below, but usually I rarely tag manually an email (except flagging important one). Instead, I add a new tagging rule as depicted above.

1
(define-key notmuch-show-mode-map "S"
2
  (lambda ()
3
    "delete message and move on"
4
    (notmuch-show-tag '("+deleted" "-unread"))
5
    (notmuch-show-next-open-message-or-pop)))

Compose email

Simply press C-x m (compose-mail) in Emacs to compose an email to send. Normally, the From:=/=To: fields can be autocompleted.

Send email

Unless your local system is configured for sending email using sendmail, you may want to access a remote SMTP server.

SMTP configuration

Below is a fragment of my SMTP setup. You should acquire this information from the host (Gmail⁵ If your Google account has 2-step verification activated, you will likely have to create and use an app password instead of your regular password. , your institution, your company etc.). Using smtpmail is not enough to sending email with different accounts. Fortunately, the package smtpmail-multi made the task easier.

1
(use-package smtpmail-multi
2
  :ensure t
3
  :config
4
    (setq smtpmail-multi-accounts
5
    '((host . ("firmin.martin@host.fr" "smtp.host.fr" 587 "firmin.martin@host.fr" nil nil nil nil))
6
    (gmail-main . ("firmin.martin@gmail.com" "smtp.gmail.com" 587 "firmin.martin@gmail.com" nil nil nil nil))))
7

8
    (setq smtpmail-multi-associations
9
    '(("firmin.martin@host.fr" host)
10
    ("firmin.martin@gmail.com" gmail-main)))
11

12
    (setq smtpmail-multi-default-account 'gmail-main)
13
    (setq message-send-mail-function 'smtpmail-multi-send-it)
14

15
    (setq smtpmail-debug-info t)
16
    (setq smtpmail-debug-verbose t))

Then you have to put your credentials somewhere. Such places are designated by the variable auth-sources which defaults to ("~/.authinfo" "~/.authinfo.gpg" "~/.netrc").

For instance, put the following in ~/.authinfo.

1
machine smtp.host.fr login firmin.martin port 587 password abc123
2
machine smtp.gmail.com login firmin.martin port 587 password abc123

Patch: Fully-Qualified Domain Name (FQDN)

You may encounter issue regarding the FQDN when sending email. I have the following patch in my configuration coming from here.

1
(when (>= emacs-major-version 25)
2
  (setq smtpmail-local-domain (car (split-string (shell-command-to-string "hostname -f")))))

Bonus: passwords encryption with pass

You may have seen a security hole which would hopefully make you uncomfortable: we have written credentials in plain text. Let’s fix it. I assume in the following that the reader has already setup gpg (2.1+) and pass.

Remember, we have stored passwords in ~/.offlineimaprc to pull emails locally with offlineimap and in ~/.authinfo so that Emacs is able to send email.

~/.offlineimaprc

Quoting ArchWiki:

1. Create a password for your email account.

   1
   pass insert email/myaccount

2. Create a python function that retrieves the password (in ~/.offlineimap/pass.py for instance).

   1
   #! /usr/bin/env python3
   2
   from subprocess import check_output
   3
   
   4
   def get_pass(account):
   5
       return check_output("pass email/" + account, shell=True).splitlines()[0]

3. In .offlineimaprc, under the general section, indicate the python file

   1
   [general]
   2
   # ...
   3
   pythonfile = ~/.offlineimap/pass.py

   and replace each remotepass = password by the next one.

   1
   remotepasseval = get_pass("myaccount")

auth-source-pass

To make Emacs read credentials through pass, we use the package auth-source-pass which exactly do the job for us. The configuration is simple.

1
(use-package auth-source-pass
2
  :ensure t
3
  :config
4
  (auth-source-pass-enable))

You should create a <smtp host>.gpg with pass --edit email/<smtp host> for each smtp server. For instance, the entry of .authinfo

1
machine smtp.host.fr login firmin.martin port 587 password abc123

corresponds to ~/.password-store/email/smtp.host.fr.gpg

1
abc123
2
user: firmin.martin
3
host: smtp.host.fr
4
port: 587

Cache gpg passphrase

By now, offlineimap and Emacs will retrieve your passwords through pass. Great! But, if you setup gpg without extra configuration, you will be prompted the passphrase every two hours. Why? The answer lies in the gpg agent options default-cache-ttl and --max-cache-ttl. The documentation says

1
--default-cache-ttl n
2
    Set the time a cache entry is valid to n seconds. The default is 600
3
    seconds. Each time a cache entry is accessed, the entry’s timer is reset. To set
4
    an entry’s maximum lifetime, use max-cache-ttl. Note that a cached passphrase
5
    may not be evicted immediately from memory if no client requests a cache
6
    operation. This is due to an internal housekeeping function which is only run
7
    every few seconds.
8

9
--max-cache-ttl n
10
    Set the maximum time a cache entry is valid to n seconds. After this time a
11
    cache entry will be expired even if it has been accessed recently or has
12
    been set using gpg-preset-passphrase. The default is 2 hours (7200 seconds).

That is, by default, the passphrase is cached 10 minutes and can be extended each time it is accessed up to 2 hours. As we set RestartSec=60 in ~/.config/systemd/user/offlineimap@.service, it ensures that we reach the maximum cache time. To increase the cache time permanently to one day, add the line below in ~/.gnupg/gpg-agent.conf.⁶ If you retrieve your emails at a frequency lower than every 10 minutes, then you should also assign the same value of max-cache-ttl to default-cache-ttl.

1
max-cache-ttl 86400

You should restart gpg-agent to see the effect (gpg -K should be enough). At this point, not only are your passwords secure to some extent, but no one can see and write emails on your behalf after one day without enter the passphrase.

Addendum: general workflow

I summarize below how one maintains this email workflow.

• Tagging emails. Update ~/.email/postsync.sh when necessary (usually to blacklist some addresses).

• Changing password. Modify adequately ~/.offlineimaprc and .authinfo, or if you use pass as above, update the passwords with pass edit email/<account>. Beware, if you only change your password remotely, you won’t be able to receive and (possibly) write any email.

• Adding new email. Each time you want to add a new email account in you workflow, you should

  • update .offlineimaprc: update accounts, add one more account section plus

  associated local/remote sections;

  • update .notmuch-config: update primary_email or other_email;
  • update smtpmail-multi-accounts and smtpmail-multi-associations if that account may be used to write email;
  • update credentials with pass;
  • run systemctl enable --user --now offlineimap@ACCOUNT-NAME.service;
  • (optional) update postsync.sh to tag the email written by yourself.