Where are the kick ass e-mail clients?
Published June 5th, 2006 in Mac OS X, General, Software, KiwiE-mail clients are interesting in that it’s a somewhat religious decision, comparable to choosing a text editor or operating system; everyone has an opinion and their own feature that’s always a deal breaker. I’ve used quite a few clients, and I’ve never been happy with the IMAP implementations, atleast with regards to Mac e-mail clients, there might be some shining stars on other platforms that I don’t know of, (if you know of one let me know). I’ve been told that Mulberry had the best IMAP support, but screenshots of Mulberry were enough to keep me away. Sadly, the company went out of business and nothing more came of the client. There isn’t much in the way of great IMAP clients. Mail.app is slowly getting better, Thunderbird is probably the best out there right now, but it’s not completely OS X native, and most other clients don’t even support IMAP. This brings up the interesting question:
Where are the kick ass clients?
Part of the reason few excellent IMAP clients exist is due to the complexity of the IMAP spec. POP3 has been a lower fruit for developers to pick, and consequently POP3 support has been much more widespread. Not only is it difficult to get IMAP completely right, but it is a totally different paradigm then POP3, so in order to support IMAP and POP3 increased complexity is required. In fact, many e-mail clients start out POP3 only and add IMAP as patch later, which is causes some of the wonky IMAP behavior. However, it’s understandable that developers don’t conquer IMAP once you compare the specifications of IMAP vs. POP3:
IMAP Spec (RFC 3501)
6051 Lines
POP3 Spec (RFC 1939)
1291 Lines
The IMAP spec is far larger than the corresponding POP3 spec, so it’s not surprising that it’s a bitch to get right. Even then, that is just one RFC; there are other RFC’s for IMAP which add extensions like IDLE support, UIDPLUS, the UNSELECT command, and more (check out IMAP Connection if you’re interested).
Those 6000 lines describe just the basic protocol, people also expect stability, speed, and, disconnected support in a modern client. IMAP has proven itself to be stable, and fast, but disconnected support is a curious issue. IMAP wasn’t designed to support disconnected access, but rather an always on connection, but wait…
What’s online, offline and disconnected?
Let me regress for a moment and explain. An offline client uses POP3, you connect to the server, and download your mail to the local machine. Whenever you read your e-mail it’s being fetched from your local computer. An online client is what IMAP was first designed as, when you read your e-mail you always have an active internet connection, your e-mail stays on the server and messages are downloaded on demand. A disconnected client is a fusion of the two mail models. In a disconnected client your local copy of e-mail is synchronized with the server. You can still manipulate your mail and the client is smart enough to copy your changes to the server when you have an internet connection again. Because of the flexibility it’s not surprising that many users have come to expect disconnected mode.
Where does this leave us?
Disconnected mode is very convenient but it’s difficult to get right. The IMAP protocol is very difficult to use in disconnected mode because it wasn’t originally designed for that purpose. In IMAP, every message has a UID (Unique Identifier) and a every folder has a UIDValidity value, the combination of these two is unique across the account. This does make it easier to program a disconnected e-mail client, but the problem is that the UID’s aren’t always static, and some servers are better with this than others. What can happen is that the server might decide that it’s time to change the message UID’s for whatever reason, and then you need to purge your local cache and download everything in that mailbox again. If the developer fails to verify the UID before executing a command on the server, some very nasty things can happen. For example, you might have a UID stored for a message that the user would like to delete, so you send a delete command to the server, but the UID validity is never checked. If the UID has changed, in the best case the delete operation will fail, but in the worst case it could remove a message that is now using the other message’s old UID. This is of course, not cool, and it would be nice if UID’s didn’t change. Some servers are crappy in that they change the UID’s far more than is necessary while others attempt to keep the UID constant for as long as possible. And just as it is hard to build an IMAP client, it’s even more of a bitch to build the server.
Another problem with IMAP is that it’s very time consuming to update all off the message flags. What do I mean by message flags? I’m taking about the flags that mark if a message has been read, replied to, forwarded, marked for deletion etc. There are some tricks out there using UID’s to only download new messages, but to make sure you have the latest flags you have to iterate over every single message, and this can be very time consuming for a large mailbox with thousands of messages.
What next?
What I propose instead of time consuming iterations is a time stamp. You send the server a time stamp, like for example the last time you connected to the server, and it plays back every change it has made to the mailbox since that time. So syncing becomes much easier, connect to the server and send the time stamp, the server sends you everything that has changed since that time. No more time consuming flags updates, or UID tricks. It would be even nicer if we could do away with IMAP and replace it with something simpler, and more generic. Why not write a generic synchronization server like .Mac that you can use to synchronize objects with? Then add support for time stamps and we can start syncing everything on to a central server. Then we can sync all kinds of data across computers just by using the generic protocol. Oh, and let me know if I missed an IMAP detail, or there is some magic trick I don’t know about it.
I thought you were writing it?
I think “most” people haven’t attacked the email client issue because of Microsoft outlook. That obviously doesn’t apply to Mac users so well, but on the PC side of things they’ve had the email client thing pretty sewn up with a “it’s mostly functional” piece of software that in my mind typifies the crap that most PC users are willing to put up with.
Buy one? Why on earth would you? This other thing that comes with my machine does pretty much what I want and it’s “free”.
Mail.app isn’t great, but it mostly works. There’s plenty of space to improve, but instead of people wanting more functionality, the vast majority have been happy with less - the whole web email client thing.
So yeah - it’s hard, and there’s mostly-functional stuff out there. I thing that’s why there aren’t any kick ass email clients out there.
Well, until you’re done, I suppose.
-joe
Now, we have libraries that will simply development of imap clients and IMAP already exists. We shouldn’t try to invent something people will never use since IMAP already fits the needs.
Though, you’re right. It’s not as simple as POP.
What you are proposing is to keep on the server a client state information, which may not scale. A synchronization server is not that easy to write. An IMAP server is not that much complicated to write. Your idea of synchronization server for email will introduce some other problems.
How about a checkbox in the preferences (”Never leave mail on the server”) which causes all mail on an IMAP account to always be downloaded to the client and then immediately deleted and purged from the server as soon as it’s written and comitted to the local disk? Treating IMAP as a glorified POP3 eliminates the remote flag updates problem, at least.
Reasoning: Many people use IMAP because they don’t have a choice (thanks, Microsoft Exchange!) and would prefer POP3. (Other people want the promised ‘convenience’ and ‘benefits’ of IMAP’s online mode, but, being in the former camp — and having tried out every email client available with IMAP on the Mac for the past ten years — I keep wondering what’s wrong with those particular people. I mean, let’s all just chew on glass for a while.)
Get a look at Polymer a very fast cross-plateform IMAP client, currently in developement.
Amongst other things, it only fetches what is needed, is very fast and supports tags.
Matt, I look forward to seeing and using Kiwi. I have never liked Mail. I am not not a Microsoft fan but I have to admit that Entourage is the best most flexible email client availaible for the Mac right now. I deployed it for a company in which I was a Macintosh System Administrator and it worked quite well. One feature that I could not live without that Entourage supports is a leave on server setting when you setup an IMAP account. This allows me to locally view, respond to, and save all emails from the mail server but everything is still left on the mail server until I delete or change it locally. For example a locally deleted email is deleted on the mail server the next time I connect to the mail server. This means that I do not have to backup my emails because they are all store in two places; my computer and the mail server at all times. This setup also allows me to connect to my email server from any computer on any platform and view all of my email though a webmail interface.
Good luck with Kiwi, I would love to be able to use something other then a Microsoft product!
Glenn Perez
What sort of “other problems” would be introduced by a synchronisation server? It seems like an elegant solution to me (but I have never coded a mail app).
I would love to see a way to tie a particular “From” address to a particular email account and to a particular user/set of users. I have 3 accounts setup in Mail.app with 5 different “From” addresses. Certain recipients only get email from me from particular addresses and it would be nice to automate that.
synchronization server needs either an infinite disk space on server and/or much computation.
Maybe you might want to start to write a spec for a mail synchronization server to encounter the problems.
Chris,
Decisions like which account is used to respond to an e-mail address I want to put into a series of Lua scripts. That way, users can (or atleast the more technically minded) users can customize that behavior.
See RFC4551 an IMAP Extension for Conditional STORE Operation or Quick Flag Changes Resynchronization. Also, there are a number lemonade drafts which will help clients quickly reconnect and get notifications.
I would love to see a way to tie a particular “From” address to a particular email account and to a particular user/set of users. I have 3 accounts setup in Mail.app with 5 different addresses. Certain recipients only get email from me from particular addresses and it would be nice to automate that.
hydrocodone online hydrocodone online
I never understood the allure of “connected” IMAP email in a world of disconnecting laptop users. These days it’s a lot easier to get wifi, but when laptop purchases exploded 5-7 years ago, most people connected by modem or lan. It was pretty easy to sit in the park with your POP mail, review everything you downloaded, write a bunch of messages, then connect and send once you get home. With IMAP, there was no access to the recently downloaded emails. Now we have wifi in some parks and many people have wifi in their living room, so they can do IMAP while physically disconnected.
Clearly I am all for disconnected clients. I am going to check out Glen Perez’s suggestion about using Entourage to save email locally with an IMAP setup.
As to Joe and his comments that users are content with the stufff that is available; well it may be true that people put up with it, but they are complaining every day to and at their computer and email programs. Email is the most used application in the world, it should be getting a lot more attention in so many areas: UI, usability, functionality, features, in addition to building in full word processing capability, and building new protocols and standards for true disconneted clients.