{"id":283,"date":"2008-11-17T00:40:31","date_gmt":"2008-11-16T22:40:31","guid":{"rendered":"http:\/\/www.thomaskeller.biz\/blog\/?p=283"},"modified":"2010-09-24T13:38:44","modified_gmt":"2010-09-24T12:38:44","slug":"server-side-email-filtering","status":"publish","type":"post","link":"https:\/\/www.thomaskeller.biz\/blog\/2008\/11\/17\/server-side-email-filtering\/","title":{"rendered":"Server-side email filtering (update)"},"content":{"rendered":"<p>If you&#8217;re having more than one computer where you look regularily for your emails (f.e. at home, at work and while you&#8217;re on the way) and you get a reasonable amount of (non-spam) emails every day, you probably know the problem: Client-side email filters just don&#8217;t do it.<\/p>\n<p>Being a novice with all the mail software stuff my initial simple idea was &#8220;hey, lets look for a web-based procmailrc frontend&#8221; &#8211; but all I found didn&#8217;t really catch it. So I looked a bit further and stumbled across the sieve mail filtering language (<a href=\"http:\/\/www.ietf.org\/rfc\/rfc3028.txt\">RFC<\/a>). Here is an example sieve file (taken from <a href=\"http:\/\/libsieve-php.sourceforge.net\/\">libsieve-php<\/a>, a PHP Sieve library):<\/p>\n<p>    require [&#8220;fileinto&#8221;];<\/p>\n<p>    if header :is &#8220;Sender&#8221; &#8220;owner-ietf-mta-filters@imc.org&#8221;<br \/>\n            {<br \/>\n            fileinto &#8220;filter&#8221;; # move to &#8220;filter&#8221; mailbox<br \/>\n            }<br \/>\n    elsif address \\:DOMAIN :is [&#8220;From&#8221;, &#8220;To&#8221;] &#8220;example.com&#8221;<br \/>\n            {<br \/>\n            keep;                # keep in &#8220;In&#8221; mailbox<br \/>\n            }<br \/>\n    elsif anyof (NOT address :all :contains<br \/>\n                    [&#8220;To&#8221;, &#8220;Cc&#8221;, &#8220;Bcc&#8221;] &#8220;me@example.com&#8221;,<br \/>\n                  header :matches &#8220;subject&#8221;<br \/>\n                    [&#8220;*make*money*fast*&#8221;, &#8220;*university*dipl*mas*&#8221;])<br \/>\n            {<br \/>\n            fileinto &#8220;spam&#8221;;    # move to &#8220;spam&#8221; mailbox<br \/>\n            }<br \/>\n    else<br \/>\n            {<br \/>\n            fileinto &#8220;personal&#8221;;<br \/>\n            }<\/p>\n<p>To let this work you need to setup an LDA (Local Delivery Agent) for your MTA (Mail Transfer Agent, such as Exim) which puts incoming emails into the local user&#8217;s mailboxes. This LDA reads in the Sieve script (which f.e. resides in the user&#8217;s home directory) and evaluates the expressions to figure out where it should go.<\/p>\n<p>Now while the script itself is already easier to read and understand than a cryptic procmail script, it&#8217;s far from being perfect:<\/p>\n<p>* Non-technical people will still have a hard time to write these rules<br \/>\n* Users need physical (i.e. ftp \/ shell) access to the mail server to edit the script, this is especially problematic if your email users are virtual<\/li>\n<h4>The solution: ManageSieve<\/h4>\n<p>ManageSieve is a <a href=\"http:\/\/ietfreport.isoc.org\/idref\/draft-martin-managesieve\/\">protocol specification<\/a> which is relatively new and still pretty much in flux, but gains support pretty quickly. It especially targets problem number two, i.e. it allows the management of sieve scripts without giving a user shell access to the machine. ManageSieve clients authenticate via the IMAP login credentials and run their commands against a dedicated server port (usually 2000).<\/p>\n<p>KMail already supports the ManageSieve protocol since KDE 3.5.9 and there is a <a href=\"http:\/\/sieve.mozdev.org\/\">Thunderbird plugin<\/a> in the work. KMail was not an option, being on Mac and while Thunderbird is my main email client, its not the client of my girl (which uses Apple&#8217;s Mail). Even if I&#8217;d have been able to persuade her using Thunderbird again, it would have been a no-go area for her anyways: The Thunderbird plugin has no nice end user interface as of now, but merely comes across as a managed script editor (though a &#8220;real&#8221; UI is planned).<\/p>\n<p>So I was very happy to find out that somebody at least wrote a <a href=\"http:\/\/alec.pl\/roundcube\/managesieve\/\">ManageSieve plugin<\/a> for my webmail client of choice, <a href=\"http:\/\/roundcube.net\/\">roundcube<\/a>.<\/p>\n<h4>The setup: Dovecot&#8217;s ManageSieve server + Dovecot&#8217;s Sieve plugin for deliver + Roundcube&#8217;s managesieve patch<\/h4>\n<p>Since the ManageSieve standard is not yet completed, implementations tend to differ. The roundcube managesieve implementation was built around and only tested with Dovecot, a popular POP3\/IMAP server, so my initial setup (the Exim\/Courier IMAP tandem) didn&#8217;t fit. I quickly read dovecot&#8217;s docs with respect to Exim integration and decided to give the Courier replacement a try since it seemed well supported. This was supposed to be the easiest part, `sudo apt-get remove courier-imapd &#038;&#038; sudo apt-get install dovecot-imap`, until I noticed that the installed dovecot version in Hardy (1.0.10) did not include the needed sieve patches, so I had to compile and patch everything myself (again, since the ManageSieve specification is not yet finished, its not part of the main dovecot distribution, either). Luckily, the exact workflow &#8211; downloading and patching Dovecot, downloading managesieve, installing and configuring everything &#8211; is documented <a href=\"http:\/\/wiki.dovecot.org\/ManageSieve\">here<\/a>.<\/p>\n<p>The final missing piece now was roundcube. Downloading and applying the <a href=\"http:\/\/alec.pl\/roundcube\/managesieve\/managesieve-rc0.2beta.tar.gz\">patch<\/a> was a no-brainer and worked out as expected &#8211; after patching a new &#8220;Filters&#8221; menu popped up in roundcube&#8217;s settings view:<\/p>\n<div><a href=\"http:\/\/www.thomaskeller.biz\/blog\/wp-content\/uploads\/2008\/11\/managesieve.png\"><img loading=\"lazy\" decoding=\"async\" class=\"aligncenter size-medium wp-image-288\" title=\"roundcube ManageSieve plugin\" src=\"http:\/\/www.thomaskeller.biz\/blog\/wp-content\/uploads\/2008\/11\/managesieve-300x157.png\" alt=\"\" width=\"300\" height=\"157\" srcset=\"https:\/\/www.thomaskeller.biz\/blog\/wp-content\/uploads\/2008\/11\/managesieve-300x157.png 300w, https:\/\/www.thomaskeller.biz\/blog\/wp-content\/uploads\/2008\/11\/managesieve.png 994w\" sizes=\"auto, (max-width: 300px) 100vw, 300px\" \/><\/a><\/div>\n<p>Fine, until my first test showed that the rules weren&#8217;t applied. So I checked back into my server &#8211; everything seemed to be in place:<\/p>\n<p>    $ ls -lh .dovecot.sieve* sieve<br \/>\n    lrwxrwxrwx 1 me me   21 2008-11-16 12:09 .dovecot.sieve -&gt; sieve\/roundcube.sieve<br \/>\n    -rw&#8212;&#8212;- 1 me me  560 2008-11-16 12:16 .dovecot.sievec<\/p>\n<p>    sieve:<br \/>\n    total 16K<br \/>\n    -rwx&#8212;&#8212; 1 me me  459 2008-11-16 12:09 roundcube.sieve<br \/>\n    drwx&#8212;&#8212; 2 me me 4,0K 2008-11-16 12:09 tmp<\/p>\n<p>(The ManageSieve specification allows to activate and deactivate multiple existing sieve scripts, dovecot&#8217;s implementation does this by symlinking to the correct one from .dovecot.sieve into sieve\/&lt;scriptname&gt;. The .dovecot.sievec is the compiled, i.e. syntax checked version of the script, another implementation detail.)<\/p>\n<p>And yes, my rules editing from within roundcube found their way into the file:<\/p>\n<p>    $ cat sieve\/roundcube.sieve<br \/>\n    require [&#8220;fileinto&#8221;];<br \/>\n    # rule:[spam]<br \/>\n    if anyof (header :contains &#8220;Subject&#8221; &#8220;*****SPAM*****&#8221;)<br \/>\n    {<br \/>\n        fileinto &#8220;Trash&#8221;;<br \/>\n    }<br \/>\n    [&#8230;]<\/p>\n<p>Looking into the logfile of `deliver` (dovecots LDA) shed light into the darkness:<\/p>\n<p>    deliver(me): 2008-11-16 02:20:28 Info: msgid=: save failed to Trash: Unknown namespace<br \/>\n    deliver(me): 2008-11-16 02:20:28 Info: sieve runtime error: Fileinto: Generic Error<br \/>\n    deliver(me): 2008-11-16 02:20:28 Error: sieve_execute_bytecode(\/home\/me\/.dovecot.sievec) failed<br \/>\nSince dovecot 1.1 `deliver` respects the IMAP `prefix` setting in dovecot.conf, which I had to set during my courier-imap -&gt; dovecot transition. This basically &#8220;virtually&#8221; adds a string prefix like &#8220;INBOX.&#8221; or something else to all mailbox names. (The actual use case is to have distinct &#8220;public&#8221; and &#8220;private&#8221; IMAP folders with namespaces, but I don&#8217;t use that.)<\/p>\n<p>A simple example: If your Maildir folder structure looks like this on harddisk<\/p>\n<p>    cur<br \/>\n    new<br \/>\n    tmp<br \/>\n    .Foo<br \/>\n    .Foo.Bar<\/p>\n<p>than this means that your mail client actually gets reported this structure on IMAP&#8217;s LIST command:<\/p>\n<p>    INBOX<br \/>\n    INBOX.Foo<br \/>\n    INBOX.Bar<\/p>\n<p>This was actually what was reported to roundcube as well, but roundcube&#8217;s IMAP code removes the INBOX.-prefix for some reason, thus reporting the ManageSieve plugin the wrong mailbox path, &#8220;Trash&#8221; instead of &#8220;INBOX.Trash&#8221;.<\/p>\n<p>After diving a bit through roundcube&#8217;s PHP code I could fix the issue with the rather ugly usage of a meant-to-be private function of roundcube&#8217;s IMAP API (<a href=\"http:\/\/pastebin.ca\/1258305\">patch is available here for the interested<\/a>), but wohoo, now finally everything works as expected!<\/p>\n<p>And again, a weekend is gone. The outcome? I can filter emails server-side and &#8211; I wrote this blog. I feel its hell about time to do some more substantial things again&#8230;<\/p>\n<p><strong>Update:<\/strong> If you managed to set up server-side filtering and wonder why your favourite Mail reader Thunderbird does not show you new emails in various IMAP target folders even though you&#8217;ve subscribed to them, ensure you&#8217;ve set the preference &#8220;<em>mail.check_all_imap_folders_for_new<\/em>&#8221; to &#8220;<em>true<\/em>&#8220;<strong> <\/strong>(<a href=\"http:\/\/kb.mozillazine.org\/Checking_for_new_messages_in_other_folders_-_Thunderbird\" target=\"_blank\">source<\/a>).<strong><br \/>\n<\/strong><\/p>\n","protected":false},"excerpt":{"rendered":"<p>If you&#8217;re having more than one computer where you look regularily for your emails (f.e. at home, at work and while you&#8217;re on the way) and you get a reasonable amount of (non-spam) emails every day, you probably know the problem: Client-side email filters just don&#8217;t do it. Being a novice with all the mail &hellip; <a href=\"https:\/\/www.thomaskeller.biz\/blog\/2008\/11\/17\/server-side-email-filtering\/\" class=\"more-link\">Continue reading <span class=\"screen-reader-text\">Server-side email filtering (update)<\/span><\/a><\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[24],"tags":[],"class_list":["post-283","post","type-post","status-publish","format-standard","hentry","category-debian-ubuntu-administration"],"_links":{"self":[{"href":"https:\/\/www.thomaskeller.biz\/blog\/wp-json\/wp\/v2\/posts\/283","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.thomaskeller.biz\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.thomaskeller.biz\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.thomaskeller.biz\/blog\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/www.thomaskeller.biz\/blog\/wp-json\/wp\/v2\/comments?post=283"}],"version-history":[{"count":14,"href":"https:\/\/www.thomaskeller.biz\/blog\/wp-json\/wp\/v2\/posts\/283\/revisions"}],"predecessor-version":[{"id":295,"href":"https:\/\/www.thomaskeller.biz\/blog\/wp-json\/wp\/v2\/posts\/283\/revisions\/295"}],"wp:attachment":[{"href":"https:\/\/www.thomaskeller.biz\/blog\/wp-json\/wp\/v2\/media?parent=283"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.thomaskeller.biz\/blog\/wp-json\/wp\/v2\/categories?post=283"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.thomaskeller.biz\/blog\/wp-json\/wp\/v2\/tags?post=283"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}