Mail Lists vs Empty Inbox?

[Oogiem]

While I can handle the keep the inbox of general regular email empty pretty well by putting messages into folders that are filed and done or into waiting for or do later mailboxes where I seem to get totally bogged down is my large number of e-mail lists and the mail they create.

I subscribe to many e-mail lists, some are work related (grazing management, sheep management, fiber work, lists for board of directors I am on etc.) some are hobbies (scrapbooking and knitting) and some are general info or chat lists. I have mail rules set so all new messages go into the appropriate lists' mailbox My "regular" e-mail runs to about 150 messages a day and I am managing to keep the inbox to zero at the end of the day, the do later mailbox is hovering at about 30 messages while my waiting for mailbox is about 5 or 10 at most.

My list mail runs to somewhere between 300-500 messages per day depending on the season. I only read/process messages for a list when I have time to handle the replies I know I will need to do. So a particular mailbox may have open, unread messages in it for a while if it's a list I know I will be needing to respond to and I don't have time right now. I also defer reading lists when I am swamped with work and can't get to them, my hobby and chat lists fall into this category.

Where I run into problems is the archiving of messages from those lists. Some threads are useful forever. On some lists I send similar messages over and over so it's helpful to pull up a past message, tailor to the new question and send it off. On others the threads are useful for a while but eventually they lose their usefulness. I've been leaving the various messages in the mailbox for the list and over time it builds up quite a set of messages, some are clearly still useful, some may have value for me as I do my personal scrapbooks (what was I thinking/doing on a particular date) and some are junk. Some were valuable while the debate and discussion was going on but are now not so important.

Historically I've had many mailboxes with the various categories of messages for my regular e-mail (slowly combining into a few larger mailboxes of related items) and one each for my lists. I currently use Eudora on the Mac. I hate all web mail apps I've seen and refuse to use them so solutions that include gmail or yahoo are not appropriate. I am not at all comfortable depending on an ISP to contain my message archive. I'm looking at moving to a new mail client, either Apple Mail or Odysseus, since Eudora has been orphaned and my mailboxes are larger than those systems are used to dealing with. A typical list mailbox for me has between 15,000 and 25,000 messages.

I see 2 problems:

1) How do I prune the list archive when a particular thread is no longer active and I don't need to keep it for reference? The time line of when this happens varies, for some lists it may be days others may be months so date based systems don't work. I tend to forget to go back and delete no longer needed messages once a thread has died.

2)How do I archive huge reference message sets in current mail apps? Have I missed one I should look at? (I have already looked at and rejected Thunderbird) One reason I've stuck with Eudora is the robust mail handling, lots of separate folders and clean and fast searching. Neither currently identified possible replacement is as robust in that area and it's frustrating to try to adapt to a new mail app just because my favorite is being orphaned.

Thanks for any help and suggestions

 

[kewms]

Many large mailing lists maintain their own archives online. Can you store pointers to the list archives instead of maintaining your own database?

In most long email threads, most of the posts are junk. You could probably cut the volume by 70% or more if you only kept the best bits of any given thread. Similarly, are you sure that a mail program is the best place for this archive? Search and retrieval might be easier if you pulled the key information out to an open format (such as text files) or a dedicated research tool. (On the Mac, I can particularly recommend DevonThink.)

For that volume of mail, you're likely to choke most individual-focused software. Have you looked at business-focused programs? Lawyers and accountants in particular need to archive large quantities of mail for long periods of time.

Katherine

 

[Oogiem]

Many large mailing lists maintain their own archives online. Can you store pointers to the list archives instead of maintaining your own database?

I don't trust any mail archive I do not control, so no that is not an option for me. I require that all my archives be stored on my own personal machine. That some also exist as backups on a web system is fine but inadequate for my primary access. We don't always have net access so I must have the ability to reference everything while offline.

In most long email threads, most of the posts are junk. You could probably cut the volume by 70% or more if you only kept the best bits of any given thread.

But what are the best bits to keep is not clear until time has passed. During the throes of a major discussion on say the details of the knitting stitches and toe formation of Gunnister man's stockings and other related archeological finds there may be many side branches, off shoots and detours. Eventually there is a consensus or it's clear that some posts are garbage or can be deleted because they duplicate others who are more eloquent. But you can't say what's actually important until the thread has finished.

Similarly, are you sure that a mail program is the best place for this archive?

No I'm not but it comes in e-mail and I think of it as e-mail so I expect to be able to handle it as e-mail.

Have you looked at business-focused programs?

No but only because I have no idea what such solutions might be.

 

[kewms]

But what are the best bits to keep is not clear until time has passed. During the throes of a major discussion on say the details of the knitting stitches and toe formation of Gunnister man's stockings and other related archeological finds there may be many side branches, off shoots and detours. Eventually there is a consensus or it's clear that some posts are garbage or can be deleted because they duplicate others who are more eloquent. But you can't say what's actually important until the thread has finished.

Of course. But by the time you reach 15,000 accumulated messages it's probably safe to toss a large chunk of them. The more you toss, the easier it will be to find things in what's left. And the smaller the archive file is, the more email tools are likely to be available for you.

Katherine

 

[Todd V]

Get Out of Mail Programs

I've learned that entrusting your important reference data to an application that may or may not have long-term support is a bad idea. I remember the Quarterdeck Mail days when I had tons of well-organized folders for years of emails. Then suddenly it's no longer supported and I had all of that reference material in a place it became increasingly more difficult to access.

What I do now is the following:

(1) I use Apple Mail.

(2) All of my email is setup for POP - I like having complete control of my emails and hate having duplicates on a server for longer than a week.

(3) I configure rules to cut the channels flowing to my email I don't want to read - and wrote an applescript to take care of any the Mail rules can't seem to filter properly.

(4) I process my inbox to zero every day - My goal is to trash every email I've responded to if I can. Anything that needs to be kept I save into the file system and store in an appropriate place - then delete the email.

--> (5) For every single action found in an email I need to do - I make a clipping of that text by selecting the text and then option-dragging it to a folder on my desktop called "Inbox" - and I process them all later. This is an excellent habit to get into since simply filing emails doesn't separate out actionable from non-actionable content - you always have to re-read the entire email to figure out what you have to do again.

I know there is value to wanting to see the threads, etc. with a mail program, but I've become more and more convinced that any mail program should be treated more as an inbox and less like a reference system. And the more I've been treating it that way the more peace of mind I have knowing that I'm not going to have 15,000 important things I can no longer access any more.

That's what's working for me.

 

[jimmo]

But by the time you reach 15,000 accumulated messages it's probably safe to toss a large chunk of them. The more you toss, the easier it will be to find things in what's left.

I have to second that. In fact, it is not only safe, but a good idea. You are never going to go through that many messages. In my 20 years of mailing lists, the likelihood of looking at messages that are over a week old is close to nil. If you have that many messages in a single mailing-list, they need to create an archive, not you.

My email client is configured to automatically toss things out after 7-10 days depending on the importance to me of that list. Definitely get out of the habit of saving mailing-list messages "just in case".

 

[dschaffner]

My solution is to keep everything in outlook, which autoarchives to another file on a regular basis.

The autoarchives files are split by year to keep file size manageable.

I use google desktop if I ever need to find anything. It hasn't let me down yet.

- Don

 

[Oogiem]

I have to second that. In fact, it is not only safe, but a good idea. You are never going to go through that many messages. In my 20 years of mailing lists, the likelihood of looking at messages that are over a week old is close to nil.

We must function differently then. I often read and refer to messages that are years or decades old. I just yesterday had to go find a series of messages about some pedigree data I researched first and reported on back in 1997. I had a new question about that bloodline. I was able to find the original question, find my research up to that point and within half an hour had caught it up to the current generation (I'm dealing with sheep pedigrees so generations turn quickly) reposted all the old and new data and went on. If I'd not saved the old thread I'd have had to start at the beginning again and that would mean researching over 35 years of pedigrees. The original research took almost a week. I can't afford to redo work that's been done before hence my need to save the archive of messages that document that type of stuff.

In a hobby area, I often want to go back and review what certain authors have said or discussed about particular archeological find. When some new data is published it is useful to review the previous research and conclusions and see how the new data fit into the large picture. Those types of messages can be very old. I just went back to review ones originally posted in 1985 about a particular subject that has new data. Just for grins I went to try to find another archive of that data and the oldest stuff I found is only 3 years old so the only copies of the original messages that exist are ones I and other similar folks saved. There is no overriding archive for some of this stuff.

So it's not accurate to say that the stuff is never useful or not referenced or has a backup somewhere else. At least not the list traffic I care about.

For me list e-mail *is* a huge reference database. And I really don't want to print the relevant messages and file them in my paper reference file that would take up way too much filing space and waste too much paper.

 

[kewms]

For pedigree data specifically, I'd use a genealogy-oriented database. That would seem a better fit than email to the natural structure of the data. I don't know what's available for animal breeders, but there's tons of software for family genealogy that you could probably adapt.

For general research topics (archaeology in your case, integrated circuit manufacturing in mine), I use DevonThink (Mac). It's got very powerful tools for finding connections among related topics.

You said in an earlier post that this material comes to you as email, and so you'd like to manage it in email. The problem is that most email programs simply aren't designed for large scale data management. You're trying to muck out the Augean stables with a pitchfork, when what you really need is a river.

Katherine

 

[Oogiem]

For pedigree data specifically, I'd use a genealogy-oriented database.

Oh the detailed data are already in a major pedigree system. The summaries of what the data means are what I send via e-mail. Having the old e-mail means I don't have to start at the beginning of a line to answer new questions I can pick up where the last message ended even if it's months or years ago.

I'll took a quick look at DevonThink but at first glance it doesn't seem like it's really applicable. I guess I don't understand how you use it or how I'd adapt it to my needs.

 

[dhlesq]

I subscribe to a number of email lists for my profession.

This is essentially read/review material.

I created a rule in my email inbox to send all of those messages to a read/review folder. That way they don't even appear in my inbox when I am processing my email to empty.

Periodically, I go through the read/review folder and delete the emails that pertain to subjects that don't concern me. For the threads that I want to keep, I delete the useless "me-too" messages in the thread. The useful messages get moved to a reference folder for that email list.

Then when I need to review old messages on a subject I do a Google desktop search to get all of the relevant info.

I currently have 12,000 messages in the read/review folder. This is not uncommon. When I review it I can process about 3,000/hr as the vast majority are immediately deleted as having no value (i.e., watercooler type threads that get into the hundreds) or are on subjects that are not valuable to me professionally.

David

 

[Oogiem]

What are you using as a mail client?

Do you dump all your messages from all lists into a single folder?

How often do you prune the folder to get rid of the junk?

Do you ever have the case that a set of messages are not junk for a period of time, like several months or a few year, but then are junk eventually? How do you prune the reference files?

At least you're dealing with similar volumes of stuff

 

[dhlesq]

What are you using as a mail client?

Do you dump all your messages from all lists into a single folder?

How often do you prune the folder to get rid of the junk?

Do you ever have the case that a set of messages are not junk for a period of time, like several months or a few year, but then are junk eventually? How do you prune the reference files?

At least you're dealing with similar volumes of stuff

I use Microsoft Outlook for my entire GTD implementation. I don't use any add-ons for it. I use it because it can handle all digital buckets, and sync them to my smartphone.

I have two sub-folders for the initial intake of the list emails. I'm an attorney so one folder is for lists that are law-related. The other folder is for non-law-related stuff. Since this is read/review material it does not have any kind of commitment associated with it, as it does not pertain to me or my clients personally. This means that if I am doing some read/review, I can read from whichever folder I feel like at the moment.

I don't prune the inbox portion that often, as it is a relatively unimportant task. When I do prune it, I usually don't do it all in one sitting as that would just be too much time spent on something that is relatively unimportant. If it gets too big, I sometimes just wholesale delete all of it without even reading it.

I rarely prune the reference file. I really only do it when my Outlook starts running slowly. With Google desktop its always easy to find messages on a given topic or search term. The only reason to prune it would be if Outlook is starting to choke on its own data.

I regularly have the case where a set of messages becomes junk either because the law changed or because a more succinct thread came up months later on the same topic. When I need to refer to the information, if I find a thread that is out-of-date, not particularly useful, or duplicative of later threads I just delete it at that point. For instance, every month or two a thread may come up on the best software to help prepare estate planning documents. During read/review, most of these go to the reference folder. But if I am actually using the reference folder to search for that topic, many of those threads will be out-of-date or duplicative. It is when I am actually looking at the reference material on the subject that I trash anything that is a waste of space.

For me, the key to this part of my implementation is accepting that none of these emails actually matter to my life. They are there for only one reason. To make my life quicker and easier when I need information.

Think of the packrat who saves all old plastic containers. It is true that plastic containers can come in handy at times, but if you save every single one of them forever, your house fills up with them to the point that there is no room to move. Instead you just save a few that seem really useful and recycle/trash the rest. It's true that you may one day say, "that one I threw away would be really handy right now," but you just can't get attached to everything that might have some unspecified use in some unspecified future.

Email listservs are there for your information (FYI), which is not to be mistaken with actual importance, in my opinion.

David