There are occasional 'insanity' effects in D&D, but as one of my players has said, trying to model actual mental illnesses in a roleplaying game mechanically can come off as all manner of insulting, ignorant, insensitive, and so on (as one example of many, the AD&D tables use such outdated terminology that the name of the condition and what is actually described do not even match).
My approach to the matter is to have insanities be gonzo mental mutations/curses that are primarily focused on making the game more interesting rather than modeling anything from real life. The most common means of obtaining an insanity is from learning forbidden knowledge where the insanity is the price you pay for the knowledge/abilities gleaned, and if the insanity is removed, so too is the knowledge lost. This allows for roleplaying the insanity to the specifics- one example would be from my own campaign, where players had access to a scroll that showed those who read it flashes of futures, one of which was the fall of the human world to brain wasps in a distant future if their current sorceress-princess was driven out of the society. Upon rolling 'Homicidal Mania,' Ser Arsem became convinced he needed to murder certain people to prevent this terrible future from occurring, and it was a pretty useful plot point that led to a lot of interesting misadventures. It was up for debate if he was acting on prophecy or insanity. I think leaving the precise details to the players is the best thing to do, so it becomes an aspect of the character rather than random 'ok you are NPC now and don't get to play' thing.
- Chaotic- You cannot take the same action twice in a row, be it during combat or downtime. At the GM's option, repeating plan may be impossible as well if any pattern of behavior is being established. Any attempt to predict your actions (such as reading your mind to see what you are thinking of doing next, or time-travel) will be bamboozled.
- Elvishness- Due to ruinous nostalgia of past lives, excessive interest in the aesthetics of a scenario, a sudden desire to recite philosophical poetry or create art, or talking to spell-wisps, ghosts, bugs, etc, you have a 1/20 chance of losing touch with the urgency of any crisis situation and instead focusing on something whimsical instead. If you create something artistic to share with the group from this whimsy, you are eligible for a small bonus of XP, along the lines of an X from Die Trying at GM discretion.
- Hypersanity- What if it's not you, but everyone else? Once per session, upon encountering some common mundanity, like 'sitting in chairs to eat' or 'shaking hands,' declare that an act of insanity and refuse to partake in it henceforth, regardless of social consequences. If your list becomes unmanageably long do not fret about what you forget now and then, just try to make up for it upon remembering (like throwing away all the chairs to prevent a similar slip-up).
- Parallaxis- Your mind, attuned to realities congruent and less so, cannot help but provide you with extra information about these parallel worlds. You do not believe yourself to be lying, but everything you say is embellished with things that you forget are not locally true, which makes it difficult for people to believe even the true parts when you mention a town that was never founded being vital for this month's crop supply. These false ideas are not entirely useless- to use the prior example, the unfounded crop-town may be a valley of rich soil unclaimed by anyone else.
- True Love- Pick a character, or perhaps even an idea (It need not be romantic love). You would do anything for them, and if they perish, it might as well be the end of the world for you. You are immune to any effect that would turn you against them.
- Jekyllism- You have an alternate self, the shadow, the id, maybe a cannibalized twin, whatever you call it, and you can call it forth, and it you, at will, though altered mental states sometimes cause an involuntary swap (such as a Fear effect causing one personality to retreat, calling forth the other). It is your opposite, or near to it, and your stats are reversed and even your class is different (Fighter-MU and Cleric-Thief being the default 'swaps').
The other personality will seek to avoid this 'insanity' being removed, or even seek to remove you and become the only self, but is not unreasonable and acts in their own best interests and will cooperate on important matters.
Depending on context, the other self may not be a facet of yourself, but a dream, or a fictional character, or a ghost or something, in which case they are not so much 'opposite' as 'just another person' and may have knowledge completely unknown to you. - Homicidal Mania- Upon meeting new people (having them talk to you in enough depth to learn each others names is the rough trigger here) you have a 1-in-20 chance to decide they need to die. You may scheme for subtle assassination plans rather than defaulting to instant stabbing, but if delayed for too long you will lose the ability to gain XP, and eventually the ability to sleep.
- Nightmare Host- Whenever you sleep, you become a Nightmare Incursion, a dungeon of sorts that can be entered and altered from within, and is host to monsters symbolic of yourself. This effect occurs and is permanent upon death, and those slain within/killed by the monsters of your nightmares become nightmare incursions themselves. The wandering encounter rates are such that if you are asleep, no one near you will be for long.
As it may soon be relevant, sleepless nights reduce all stats by 1 and prevent any healing, recovery of spells and abilities, and so on. - Determined- Pick a goal you have- you are obsessed with its completion and cannot take downtime activities in the service of anything else, and should justify most actions within this context, or grumble mightily when doing unrelated things. If you are slain before it is completed, you immediately become undead and continue trying to accomplish this goal devoid of the weaknesses of the living (as a PC or NPC depending on preferences). You gain bonus XP for following your goal, however, and pick a new goal if you complete one.
- MegaMasochism- You register pain as pleasure, and gain 1d6 temporary HP whenever struck for damage. This temporary HP is lost at the end of a combat or otherwise defined scene, as it is a brief high and your wounds are still wounds. You also lose any dodge-based AC bonuses, and show up with 1d6 less HP after any instance of downtime due to self (or otherwise) flagellation.
Previously pleasurable activities lose their luster and so things like food, drugs, soft beds, and even healing effects (unless the healing process is painful/does not numb pain) provide only half their previous benefits, rounding down.
Naturally, any pain-based attempts at coercion are entirely ineffective. - Awakened Sleeper- While asleep, you function as though mostly awake, though your speech is unintelligible and you must simultaneously contend with the dream realm as well as the waking world, (assuming the GM is too busy to run both realms at once, apply a 50% chance to act as Confused each action taken). While useful for night-time ambushes, sleep spells, and so on, this is largely an annoyance and it may be easiest to bind yourself upon rest so you do not wander.
- Darktongue- You constantly mutter and babble, though less so if you occupy your mouth with a pipe, or food, or a hand over it. Fragments of out-of-character chatter make their way into diegetic conversation, impairing auditory stealth and conversation, and your muttering may disturb others- if something especially odd is said, it may reroll reaction or morale of friends and enemies alike. Spellcasting is most direly affected- when you cast a spell, it is subject to wild magic, temporary spell mutations, etc etc
- P-Zombie- You lose your soul and your mind, have no internal experience of being, and are immune to any such effects reliant on you being more than a chemo-electrical meat-puppet. However, the empty thing that I will refer to as 'you' for sake of conversation behaves as though 'you' still had consciousness, and acts as though 'you' were affected by such effects. Mind-reading or 'detect' spells targeting you will yield no results, however, as there is 'nothing there.' Philosophers must save vs nervous sweat in your presence.
- Fight/Flight- When put into crisis situations/threatened with physical harm, you must make a Will save (or whatever save is most appropriate) or you must either enter a berserker rage in which you deal and take maximum melee damage and must attack(or at least take violent offensive action like throwing Fireballs or chasing your foes bellowing) until all threats are eliminated, or you may flee as fast as possible until you have evaded, hidden from, and otherwise lost all pursuing threats and given yourself a good 10 minutes/1 dungeon turn to calm down.
- Amnesia- You forget the specifics of your past- you know what a human is, but you don't know WHO a human is. You know what spaghetti is, but not if you like it. All that remains is a dim recollection of 'ah yes, I was a human*, I did, human things.' The memories are retained within the soul or the brain (depending on lore) but cannot be accessed until the seal upon them is removed. Some abilities and skills may be lost, others may remain via muscle memory and unconscious influence.
*Obviously replace human with dog if you are a dog or something - Hyperbolic Optimism/Pessimism- Upon waking, you have a 50% chance to be either rabidly optimistic or nihilistically pessimistic for the rest of the day. While initially amusing it soon begins to wear on those exposed, and each subsequent day of the same ludicrous cheer or crushing gloom reduces the morale of retainers by 1 until the swap is made back for a change of pace.
For downtime, you may take two downtime actions, but each has double chance of mishaps occurring due to hasty decisions and self-sabotage. - Blabbering Honesty- You cannot lie, and if someone is lying and you think you know the truth, save or blurt it out.
If you are an Elf or similar fey critter, you instead lose all your magic powers if you lie overtly (though lies by omission are fine). - Nightmare Curse- You are not quite possessed, but infested by a nightmare creature that manifests as aberrant behavior. Roll your random curse on page 236
- Contrarianism- When told to do something, save or attempt to do the opposite, or in the case of an exact opposite being untenable, something counterproductive to the intended order. Your own intended actions can trigger this (ie, jumping into a pit instead of over it), but only if you have gone a session without causing some zany misadventure and are now jonesing for contrarian wackiness.
- Darkmind- Exposure to forbidden elements has altered your mind-concept in alien ways incomprehensible to mere diegetic beings. Your character is now the character of at least one other player in addition to yourself, providing them with a wealth of new ideas and behaviors.