UW IMAP - Message Status Changes not Propogated

UW IMAP - Message Status Changes not Propogated

Post by danso » Sat, 07 May 2005 00:45:28


[This is actually based on a reply to an older thread but I then
thought it deserves one of it's own]

Hello All:

I am finding that IMAP (UW server) is not updating the status of
messages until the connection is closed. This causes problems on
clients with peristent connections (such as mail clients Thunderbird,
Outlook, Applemail etc).

So the following happens:

1. You Log in.

2. View mail list - see 2 new mails.

3. Read the first mail - whilst you do this the client sends a command
to teh server to mark it as seen (ie to update the Status or X-Status
field with "RO" tags). This update is not committed until the
connection closes.

4. You go to the next message - whilst reading this the client rechecks
the mailfolder (say you're on a 5 minute check cycle). It sees the
first mail as still marked as "new" (the equivalent of Status: N or a
missing Status header) and refreshes the folder list to show it as such
(so it's bold etc).

The original "mark it as read" request seems to get lost at this point
so even when you do log out the command is not executed.

The above also applies to Deleted messages (Status: D) so they appear
to "rise from the dead" after a few minutes. Even sometimes after
Purging the folder.

Anyone ever experienced this? Or have you somehow managed to get UW
IMAP to commit status changes immediately? What is the expected
behaviour?

Losing hair on this one. Thanks for any help!
Daniel
 
 
 

UW IMAP - Message Status Changes not Propogated

Post by Mark Crisp » Sat, 07 May 2005 02:32:33

You are using traditional UNIX mailbox format, which only allows one
process to access it at a time. All locks in traditional UNIX mailbox
format are exclusive. You are correct that status updates in that format
are postponed until the next checkpoint (check, expunge, or close) but
that doesn't matter because only one process is accessing it.

From your description, it sounds like either you, or your client, are
spawning a second process to access the mailbox. This second process
smashes the exclusive lock held by the first process, thus cancelling
everything that the first process did, in order to seize the lock for
itself.

If you are wondering why UW imapd does this, it is so that if you forget
and leave your client running in the office and go home, you can get to
your mail. Without the lock-smash procedure, you wouldn't have access to
your mail until you get back to the office.

However, it seems that some badly-written IMAP clients assume that if a
lock is exclusive, it will stay held and prevent a second process from
breaking the lock. Therefore, these poorly-written clients rationalize,
it is alright to spawn a second IMAP session to do something like check
for new mail, since the lock will just cause an error. Then, when their
client does this bad thing, they blame the server.

There is no reason, other than a lazy programmer, for a client to spawn
multiple sessions to the same mailbox. The IMAP protocol makes it
completely unnecessary.

UW imapd supports other mailbox formats which allow multiple access, and
then you won't see the effect of the poorly-written client. Or you can
use a well-written client, such as Pine or Mulberry.

-- Mark --

http://www.yqcomputer.com/
Science does not emerge from voting, party politics, or public debate.
Si vis pacem, para bellum.

 
 
 

UW IMAP - Message Status Changes not Propogated

Post by danso » Sat, 07 May 2005 04:10:08

gt; You are using traditional UNIX mailbox format, which only allows one
mailbox
format
but

Ok so at I now know that the postponement of updates is expected
behaviour.






Indeed - I am using Thunderbird. This does have a "Number of
Connections to Cache" settings.

A quote from thread about thunderbird:
"This is not uncommon with IMAP clients; the intent is to be able to do
multiple things in parallel. The IMAP standard permits this."

I'm guessing you refute that? (I haven't got round to reading the
100page rfc yet).

There is something you said that contradicts my findings though:
I can access my same mailbox via Thunderbird and via Squirrelmail.
Indeed if I open up a folder in Squirrelmail with Thunderbird looking
at it the connection is indeed "killed lost mailbox lock"ed.
However THE UPDATES ARE THEN DONE.

just to be 100% unambigious:
1. cd /var/home/mail/test [1 message inside the test mbox]
2. watch 'cat test' [this just shows the contents on the screen and
refreshes every 2 seconds]
Status: RO
3. Connect to test folder in thunderbird. See message there marked as
read.
4. Click on message and select "Delete".
Status: RO [no change]
5. Open up Squirrelmail and login
6. Click on the test folder. Instantly:
Status: D
(also maillog reports killed mailbox lock at this instant).
7. See message list with one "greyed out" message.

Thus the update is happening at the instant the connection is killed.
This also goes for marking messages as read/unread etc. The update is
done when the connection is killed.

I can also reproduce this by clicking FILE>OFFLINE>WORK OFFLINE in
thunderbird. Instantly the Status is updated.

Ahh from your reply i tried this as well: change the status of a
message and then hit purge. The status updates! Even if it's not a
delete action. As you said, the checkpoint "Purge" allows you to push
through the updates.

forget
to
access to

Yep I understand this.

a
from
rationalize,
check
their

I know Outlook (Express) was badly written but I'd have thought
Thunderbird would have been written to a good standard. No idea about
Applemail. In

spawn

Come to think of it - I actually think the multiple threads option is
so you can open multiple sessions to different folders. This would be
reasonable.

******************************************
In any case the problem here seems to be the fact that IMAP buffers the
updates for too long. Surely it would make sense to have in place some
timeout - after an update, if no further updates happen within x
seconds then commit the changes?
******************************************

I finally have a good picture of the problem. With one thread, when you
change away from that folder the single thread must do a close and then
open a new folder. The close action commits the changes.
With multiple threads, a new connection is made to the second folder.
The first connection maintains the first folder open. No check point
has been hit - no changes committed.
Now 5 mins later the client does a check of subscribed folders. What
happens (i believe) is that a NEW connection is made to the first
folder. This new connection reads the folder, sees the status of the
message pre-commit and then displays them in the old state.

The only bit i don't understand is why, when the new connection is made
to the first folder, are the changes not made? Given the webmail test
above, the act of destroy
 
 
 

UW IMAP - Message Status Changes not Propogated

Post by Sam » Sat, 07 May 2005 04:42:01

This is a MIME GnuPG-signed message. If you see this text, it means that
your E-mail or Usenet software does not support MIME signed messages.

danson writes:


There's nothing wrong with using another connection to browse another
folder. Well, nothing terribly wrong, but it still indicates a lack of full
grasp on reality. For some strange reason, my IMAP client can do everything
that Thunderbird can, from an IMAP perspective, and I never see any
particular need to open a second connection for

But, opening a second connection, and then using it to access the same
folder you have already opened, is S T U P I D. It indicates the lack of
even the most basic programming and design skills. There's absolutely
nothing you can accomplish with that second connection that you can't
already do with your existing one.


-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.4 (GNU/Linux)

iD8DBQBCencIx9p3GYHlUOIRAnwEAJ94HcSbvTUM5ov+kkvr3h3IWwlmTwCfXsk0
RZ/9Ore5cIwhiKF6fwmO3wc=
=i1m6
-----END PGP SIGNATURE-----
 
 
 

UW IMAP - Message Status Changes not Propogated

Post by Mark Crisp » Sat, 07 May 2005 06:57:01

n Thu, 5 May 2005, danson wrote:

1) It is perfectly reasonable to operate on multiple mailboxes in parallel
through multiple sessions.

2) It is senseless to operate on the same mailbox in parallel through
multiple sessions from a single client.

3) The only sensible reason for operating on the same mailbox in parallel
through multiple sessions is to serve *different* clients. For example,
consider a helpdesk mailbox that a team of 10 people each has open. The
traditional UNIX mailbox format does not support that type of access, but
most other mailbox formats do.

The authors of certain poorly-written clients confuse points (2) and (3)
above. If the mailbox format allows the helpdesk-type access of (3), then
the poorly-written client gets away with the senseless behavior in (2);
the only consequence is that it wastes resources (both on client and
server).


Whether or not the updates happen depends entirely upon good management
and good luck. The software tries very hard to do the updates when it
gets the Kiss Of Death from another process that is smashing the lock to
seize it for itself. Whether it succeeds depends upon a great many
factors.


The old Netscape Messager (predecessor to Thunderbird) was perhaps the
worst IMAP client ever written. I know that it was cleaned up
considerably in the decade since, but I don't know how thorough of a job
they did. I haven't tracked it much.

As for *how* bad it was, let's just say that it was *so* bad that I
uninstalled Netscape in favor of IE. That's pretty damn bad.... :-)

The main problem with Outlook Express is that it's a dead product; it has
known bugs but they aren't ever fixed. Outlook's problem is that it is
first and foremost an Exchange client; its IMAP support is an afterthought
and it shows.

A better comparison is with Entourage (Microsoft's IMAP/POP client for Mac
OS X); the first version was pretty bad. Entourage 2004 looks quite a bit
better, but I haven't studied it in enough detail yet.

My own preferred IMAP client is Pine.


This is under the control of the client. IMAP has a CHECK command for
this purpose. The CHECK command in IMAP is not to "check for new mail";
*all* IMAP commands check for new mail automatically. Rather, CHECK is a
checkpoint operator. EXPUNGE, CLOSE, and LOGOUT are also checkpoint
operators, but they do other things.

Since a checkpoint can potentially be time-consuming, you typically find
IMAP clients doing something like "pinging" the mailbox after a couple of
minutes of inactivity with a NOOP command (which does nothing, but since
it's a command a check for new mail happens), and maybe every 15 to 20
minutes does a CHECK.

On servers which do not need a checkpoint, the CHECK command acts just as
a NOOP.


The first connection should have received a Kiss Of Death from the second
connection. The second connection shouldn't be able to acquire the lock
otherwise.


Probably correct. The first connection must have died without being able
to save the changes. There are known ways in which this can happen,
especially if a network filesystem (e.g. NFS) is involved.

The usual scenario is that the first connection never got a Kiss Of Death,
and the second connection smashed the lock held by the first connection
anyway then seized it. The first connection eventually gets something to
wake it up, and at that point it discovers t
 
 
 

UW IMAP - Message Status Changes not Propogated

Post by Paul Rubi » Sat, 07 May 2005 07:04:41

Mark Crispin < XXXX@XXXXX.COM > writes:

I think sometimes multiple sessions from one client makes sense. Just
now I got a big file attachment that I had to open, which triggered a
slow download. While the download was in progress it would have been
nice to be able to keep on reading and responding to other messages
from the same client. The most obvious way to do that is with
multiple connections. Am I missing something?


If you really want to disparage multiple connections from the same
client, it would be good to say something to that effect in the RFC.
Otherwise, implementers will follow their own ideas on the issue.
 
 
 

UW IMAP - Message Status Changes not Propogated

Post by Mark Crisp » Sat, 07 May 2005 07:15:35


Yes, you are. :-)

The message reading is on the same pipe as the download. If you want to
download "in the background", your client can use IMAP partial fetches for
the download and give the user the opportunity to do something else in
between the partial fetches.


The RFC *does* say something to that effect.

The problem is that people don't read RFCs; or rather, they read them
selectively. One of the biggest problems, impacting all protocols, is
that people do not use the syntax, and instead infer syntax from examples.

-- Mark --

http://www.yqcomputer.com/
Science does not emerge from voting, party politics, or public debate.
Si vis pacem, para bellum.