It's hard to give a clear answer because NAs are so personal. A few thoughts, though...
Not writing them down doesn't make them go away. Write *everything* down.
Yes, 164 is probably too many, unless they are all very tiny. How long would it take you to actually do all of them, assuming no interruptions and no new tasks? If the answer is more than a week or two, you might need to rethink your commitments: you're probably not going to be able to meet all of them. FWIW, I currently have 78 NAs, not including recurring tasks, and I'm feeling completely swamped.