Subtle Errors and CoreData
Published December 18th, 2005 in Cocoa, KiwiIdeally in Kiwi, everything would be pulled from the server during launch, but this limits Spotlight integration and it makes offline reading of e-mail impossible. I originally toyed around with storing e-mail in the MH mail format because it’s an open format and every message has it’s own file. Persistence is never easy, especially when people demand stability, and implementing the MH format is no exception. Knee deep into writing file I/O code, I took a step back and rewrote with CoreData.
Previously, the interface fed off of instance variables in objects via bindings. It was clear that this was making persistance and multithreading more complex than necessary. So I packaged all the code together that performs IMAP calls and sat it on top of CoreData. Now, instead of writing retrieved IMAP data to local instance variables, it writes directly to the CoreData store. The interface sits on top of a copy of the CoreData store and feeds off of it with Cocoa bindings.
If I had used the MH format, I would have had to manage all of the relationships between Folders and Messages and more importantly, I also would have to insure consistency in the event of crash. CoreData has been a huge help in this regard, it maintains the referential integrity for me and it keeps changes in memory until they are committed. Now users of Kiwi can be confident that their data is stored in a SQLlite database that is consistent across sessions.
Moving from a graph of NSObjects in memory to CoreData has not been without trial and error, though. For about the past day, I’ve been trying to figure out why one table hasn’t been displaying data that exists in the database. I double checked all of my connections, I tripled checked my methods and still, the table was displaying a lone and empty cell. Finally, today I figured it out: I had forgotten to switch the controller in Interface Builder from Class Mode to Entity mode. I made the simple switch, and my tableview (thankfully) came to life.
I have to say that these new technologies, like Bindings and CoreData, solve problems but they also introduce yet another learning curve for developers on the Mac platform. In the end however, I think CoreData is going cause alot of developers out there to take a second look at their persistence code and wether CoreData and Bindings can help them spend more time on other features.
Sounds very cool - but what does that mean for Spotlight? Will it be possible to find single messages?
Yes, it will be possible to find single messages. I’m not sure exactly how we are going to implement it yet, I haven’t looked deeply into it. However, I know people have integrated spotlight with monolithic databases in the past, even Apple has done this with iCal.
Don’t forget that when storing the message in any format and if you want some performance when reading headers information, for instance, when you display the list of messages, you’ll have to build a cache about parsed headers. That’s what you can find in high-level API of libetpan but since you rely on low-level API, you’ll have to implement that.
It’s interesting to test with mailboxes with at least 10000 messages. I wrote this about mail format performances :
http://libetpan.sourceforge.net/mailstorage.html
About Spotlight, you’ll have to remember that each result will be identified by a file. That means that for a mailbox, you’ll have to generate as much files as messages if you consider that a result will be a single message.
iCal does this by generating a file for each event, look at ~/Library/Caches/Metadata/iCal.
Hi,
Why not use the Maildir format? Its a standard, every message has its own file, every mail folder has its own folder. It has been used for years (decades?). It is supported by many mail client and servers. I use it with offline imap (sync imap to maildir) and mutt-ng (reads maildir).
http://en.wikipedia.org/wiki/Maildir
Maildir is a standard and makes mail very portable between clients and operating systems.
With regards to maildir… if performance is an issue caching is an easy solution as Hoa mentioned.
Hm, iCal is really a not so good example. Everything has to be here twice: in the monolithic database-file and as cached event in the filesystem.
But on the other hand - if CoreData speeds up development and provides fast access to single messages, it would be ok
Greetings,
I recently discovered Kiwi while looking at IMAP clients for OS X to replace Mulberry after ISAMET/Cyrusoft filed for bankruptcy a couple months ago. Although Kiwi is still too immature to be my primary mail client I’ll be watching its evolution and suspect other OS X [ex-]Mulberry users might be interested, too. If you’re not familiar with Mulberry (often avoided for its ugly, obscure UI), it has some uniquely powerful features that I haven’t found in other mail apps (at least for OS X). More about that another time.
Re: Spotlight integration with Core Data
I’m not a developer (although I’ve been accused of being one in denial) but this looked like it might be helpful:
Spotlight: That Nifty Open Trick
PS: Any chance of adding comment previewing to this weblog?
Trying to post that Spotlight URL…
Spotlight: That Nifty Open Trick