imap client duplicate/synchronization problem

imap client duplicate/synchronization problem

Post by erikre » Mon, 14 Jun 2004 01:23:36

I have a problem related to imap with mozilla-1.5/netscape 7.1 or any
recent mozilla version. It relates to the state if the client/server
connection, duplicate INBOX messages, the semantics of the INBOX.msf
file and such. Please read on:

1. I have an external email account with an IMAP interface
and clients on multiple machines.

2. Because of space limitations, I periodicaly must delete messages on
the server (not using the client, but directly via the email provider
web interface).

3. Hence my local INBOX contains more messages than the server, which
is all fine and dandy (most of the time).

Here is the problem:

Every once in a while the client/server get into a funny mutual state
and the client (or maybe it is the server) decides to download ALL my
messages again. Now I have an INBOX which is 1.5X the size it should be
(remember, not 2x, since I have deleted some messages on the server). So
I fire up "mutt -f INBOX" and delete all the duplicate messages with
some mutt wizardry. Now of course INBOX.msf is wrong, so I have to
restore it somehow. So I remove INBOX.msf, restart the client, and hope
that netzilla will rebuild it for me. But no, it uses the server-side
(incomplete) INBOX as source, and now I have a complete INBOX with an
incomplete INBOX.msf, and I cannot see all the messages in the INBOX
from the client (I still can from mutt, but I want netzilla!).

My global question is simple: After doctoring the INBOX, how can I get
netzilla and the server both into a good state where the INBOX.msf is in
good shape and everyone is happy?

Some related questions are:

Q1. How can I get netzilla to create the INBOX.msf from the local INBOX
file instead of the server INBOX file?

Q2. ...and at the same time avoid that the server thinks I'm out of sync
and dump more duplicate mail at me?

I know one way of solving Q1 (make a copy of INBOX to local folder/file
dummyINBOX, delete the duplicates in mutt, delete dummyINBOX.msf,
restart netzilla, watch it recreate dummyINBOX.msf, copy to
INBOX.msf). The problem is when I replace INBOX.msf, netzilla gets in
the funny state again, and starts wanting to dump email one me.

Q3. How can I doctor the INBOX.msf file so that the server thinks I'm
up to date??

and secondarily

Q4. Is there a standalone/commandline inbox2msf program anywhere?

Q5. Where is the INBOX.msf format documented?

Q6. How does he client/server decide from INBOX.msf that all the
messages need to be transferred again.

I did hunt around on the web and usenet quite a bit to try and answer
but no luck. Are there and netscape/mozilla/imap wizards out there that
can lend a hand?


Bonus question: Why doesn't the mozilla/netscape email client allow
quick selection of messages that are TO some user, only FROM some user?
The msf file seems to contain all the email addresses, so why not?

imap client duplicate/synchronization problem

Post by Neil » Tue, 15 Jun 2004 22:32:08

I'm guessing here that you've decided to download messages for offline
use. So what happens if you start Mozilla in offline mode and try to
open the doctored inbox?

It will do, but only in the drafts, templates and sent messages folders.

Warning: May contain traces of nuts.


imap client duplicate/synchronization problem

Post by erikre » Wed, 16 Jun 2004 15:15:13

Hi Neil,

Yes, I'm using the download for offline option.

About starting in offline mode: Good idea, but nothing happens. The
INBOX simply comes up as empty, and the msf does not get generated.

I wonder why this is restricted to certain folders...


imap client duplicate/synchronization problem

Post by Neil » Thu, 17 Jun 2004 00:07:20

OK, I've done some digging. You cannot recreate an IMAP msf from an
offline store. In fact, messages that you deleted from the server should
be marked as deleted in the offline store so that even if you could
rebuild the msf it would skip those messages. The best you can do is to
copy the offline store to a file in your local folders and doctor it to
remove the deleted status from all messages.

Because those tend to be the ones containing messages from you, whereas
other folder tend to contain messages to you...

Warning: May contain traces of nuts.

imap client duplicate/synchronization problem

Post by erikre » Thu, 17 Jun 2004 17:02:25

Hi Neil,

Thanks for looking into this. I see you are involved in,
with numerous enhancements and bugfixes to your credit.

Would it make sense for me to submit a bugzilla for an enhancement in
the direction of what I'm looking for?

I understand that there is some logic in "deleting" messages both on
server and client side when server messages are deleted. But on the
other hand, it is becoming common for people to use free imap services
with limited amounts of storage (mine has a 10MB limit), and it would be
bad if several 1MB emails would dispappear locally just because I need
to make room on the server.

I know there are plenty of workarounds in the form of periodically
copying/moving email to a local folder, but I can't help thinking that I
ought to be able to collect a year or two worth of email before I
archive into a local folder (maybe one per year?), even with a 10MB imap
account limit.

But I'd settle for learning how to hack the msf file, if need be :-)
(any pointers??)

PS: Regarding the "view: subject or sender", I'd still speak up for
being able to do "view: subject or sender or recipient", because I tend
to keep also *sent* messages in my inbox (bcc: ), for easier linear
perusal of complete email exchanges.

imap client duplicate/synchronization problem

Post by Neil » Thu, 17 Jun 2004 21:00:27

Given the number of workarounds I doubt it would be worthwhile
submitting a bug except to track your own implementation of the enhancement.

Sorry, I don't know the code in that area.

I would not be surprised if you find that this one has already been filed.

Warning: May contain traces of nuts.