All of the major third-party PIM add-ons (AgendaOne, Agenda Fusion, Pocket Informant) give you the option to set up your task list sorted by category. I use AgendaOne, with the default setting to have all of the categories collapsed. While I can't change categories by repeatedly pressing the button I've assigned to open the task list, I find it just as fast to scroll to and select the category I want to expand.
Art