Second concussion in 3 weeks

I'm just wondering when should I get worried about long term issues or permanent damage. It's been 3 weeks with very minimal recovery. I'm still feeling out of it (short term memory issues, concentration, attention, awareness/alertness, feeling slow) are all worse than prior to second concussion. Still have intermittent headaches as well. Doctor's tell me there's not much that can be done right now.