In 1986, California state assemblyman John Vasconcellos came up with what he believed could be “a vaccine for major social ills” like teen pregnancy and drug abuse: a special task-force to promote self-esteem among Californians. The effort folded three years later, and was widely considered not to have accomplished much.

To Kristin Neff, a psychology professor at the University of Texas, that’s not surprising. Though self-esteem continues to reverberate as a pop-psych cure-all, the quest for inflated egos, in her view, is misguided and largely pointless.

There’s nothing wrong with being confident, to answer Demi Lovato’s question. The trouble is how we try to achieve high self-regard. Often, it’s by undermining others or comparing our achievements to those around us. That’s not just unsustainable, Neff argues, it can also lead to narcissism or depressive bouts during hard times.

Neff proposes a better path: Self-compassion. In other words, treating yourself just like you would your best friends, even when they (you) screw up.

I recently interviewed Neff about how self-esteem fails us and how we can boost our compassion for ourselves instead. An edited version of the conversation follows.

Olga Khazan: What are some contexts in which we usually hear about boosting self-esteem?

Kristin Neff: Well, it seems like it's just deeply permeated, especially American culture, where we have very high levels of self-esteem and narcissism. I think because of the big self-esteem movement, people just got it in their heads that the key to psychological health was self-esteem. Jean Twenge and Keith Campbell showed that because of this emphasis on self-esteem, we actually got a generation of narcissists. I think it’s generally out there in the culture, but maybe especially among parents and educators.

Jenny Crocker—she’s one of the best people who talks about this. She says, you have to stop the costly pursuit of high self-esteem. It's not having high self-esteem is the problem, it's pursuing it, which is usually based on feeling special and above-average or better than others. The best way to think about the problem of self-esteem is not whether or not you have it, but what you do to get it. That's where the issues really come in.

Khazan: So what's wrong with telling people to have better self-esteem?

Neff: When you take it too seriously, you become a narcissist. And we know narcissists tend to have problems with relationships, they push people away, so there are definitely maladaptive consequences to narcissism.

The other thing is, it's pretty common, at least in American society, that in order to have high self-esteem, you have to feel special and above-average. If someone said, "Oh, your performance was average," you would feel hurt by that, almost insulted.

When we fail, self-esteem deserts us, which is precisely when we need it most.

And so the problem is we're constantly comparing ourselves to others. We try to puff ourselves up. We have what's called self-enhancement bias, where we see ourselves as better in almost any culturally valued trait. There's a large body of research showing that bullying is largely caused by the quest for high self-esteem—the process of feeling special and better-than.

So if I can pick on the weird, nerdy kid, I actually get a self-esteem boost. Then, if you look at things like prejudice, at least some element playing a role in prejudice is if I feel that my religious group or my ethnic group is better than yours, that's one way to make a social comparison, and I am actually boosting my self-esteem. So that's a problem. And also the fact that on some level, someone is always going to be doing it better.

When I teach workshops I say, it's logically impossible for everyone to be above average at all times, so we're basically predicating ourselves with a logical impossibility. Eventually that's going to hit reality. Maybe somebody does do that better than me. Do I accept that or am I destabilized by that?

Usually, self-esteem is highly contingent on success. And the three domains it’s contingent on are, first, peer approval. That's what other kids at school and other people of work think of me, which is a really lousy source of information, because a) they don't know you very well and b) you don't know what they think of you very well.

And then, perceived appearance, which for women is especially damning, and it's also the most important domain for self-esteem for women. One of the reasons boys don't suffer as much from low self-esteem is that boys, growing up, they think they're pretty attractive. They rate their own attractiveness pretty high. The standards of beauty are much higher for girls than for boys. For girls, from the third grade, you start seeing a nose-dive in how attractive they think they are. Starting in third grade think, girls think, "I'm fat," and "I'm not pretty enough," and start comparing themselves to high standards and their self-esteem takes a hit. Boys stay pretty stable.

The final one is success. The real problem with that is self-esteem is only available when we succeed. But when we fail, self-esteem deserts us, which is precisely when we need it most. And some people argue that the instability of self-esteem going up and down is more damaging than the level of self-esteem itself.

Khazan: So what is self-compassion? How is it better?

Neff: It means treating yourself with the same kind of kindness, care, compassion, as you would treat those you care about—your good friends, your loved ones.

One component is self-kindness, which is in a way the most obvious. But it also entails a recognition of common humanity—in other words, the understanding that all people are imperfect, and all people have imperfect lives. Sometimes, when we fail, we react as if something has gone wrong—that this shouldn't be happening. “I shouldn't have failed, I shouldn't have had this issue come up in my life.” And this sense that “this shouldn't be happening,” as if everyone else in the world were living perfectly happy, unproblematic lives. That type of thinking really causes a lot of additional suffering, because people feel isolated and separated from the rest of humanity.

So, when we have self-compassion, when we fail, it's not “poor me,” it's “well, everyone fails.” Everyone struggles. This is what it means to be human. And that really radically alters how we relate to failure and difficulty. When we say, "Oh, this is normal, this is part of what it means to human," that opens the door to the grow from the experience. If we feel like it's abnormal, this shouldn't be happening, then we start blaming ourselves.

Self-compassion also entails a mindfulness. In order to have self-compassion, we have to be willing to turn toward and acknowledge our suffering. Typically, we don't want to do that. We want to avoid it, we don't want to think about it, and want to go straight into problem-solving.

And in fact, I would argue that self-compassion also provides a sense of self-worth, but it's not linked to narcissism the way self-esteem is. It's not linked to social comparison the way self-esteem is, and it's not contingent, because you have self-compassion both when you fail and when you succeed. The sense of self-worth that comes from being kind to yourself is much more stable over time than the sense of self-worth that comes from judging yourself positively.

Khazan: What would someone who experienced a major setback say to themselves if they were being self-compassionate?

Neff: I write a lot about this in my book because my son was diagnosed with autism. He's 14 now. In my book, I talk about, thank God I had many years of self-compassion practice under my belt, because immediately when it happened, I knew what I needed to do. Instead of just going into “what type of therapy, what type of treatment?” I knew I had to acknowledge that this was difficult for me, that it was emotionally painful. I had to really think about being kind and caring and understanding to myself, letting myself feel whatever feelings were coming up, whether or not I thought I should be having them.

I remember one time I was at the playground and there were all these mothers with their kids and they were all laughing and interacting, and my son was off in a corner not interacting. I started going down the path of self-pity. You know, “why me, why me ...” But when I remembered common humanity, I had this very powerful experience where I remembered, wait a second, maybe these mothers are not dealing autism, but every single one of these mothers will have challenges with their child in some form or another. Maybe a mental health issue or a physical issue or maybe they'll have a very conflict-filled relationship. Once I had made the switch from “poor me” to “this is what motherhood's all about—we have challenges with our children and we love them anyway,” it really radically reframed how I related to my own emotional difficulty. It made it much more easy to cope.

Khazan: I noticed that you found this works for romantic relationships, for body image ... what are some of the various contexts that you found that this works in?

Neff: One is coping and resilience. A lot of people think self-compassion is weak. Well, it's not. For instance, there's some work with combat vets, on their level of self-compassion— are they an inner enemy or an inner ally? The vets who were an inner ally instead of an inner enemy cope much better and are much less likely to develop PTSD symptoms. It helps people cope with divorce, pain, age.

A big one, which a lot of people just can't quite believe, is that it enhances motivation. People who are more self-compassionate, when they fail, they're less afraid of failure. There was a study where helping people be more self-compassionate about failure [on a test], later on when they had a chance to study for a second test, they actually studied longer than people who were not told to be self-compassionate. Because, basically, it creates an environment where it's safe to fail, so self-compassionate people are often more likely to try again. They also have more self-confidence, because they aren't cutting themselves down all the time.

There's some work on physical health, showing that self-compassion is linked to better immune function. Studies show that it stabilizes glucose levels in diabetes patients, another one looking at telomere lengths—it's associated with longer telomeres. [Self-compassionate] people are healthier, they take better care of themselves, they are more likely to exercise and eat well, more likely to go to the doctor. Self-compassion is caring about yourself and not wanting yourself to suffer.

Khazan: One of your findings is the men have more self-compassion than women. Why is that?

Neff: It's a very small difference, but it's consistent: Women tend to be less self-compassionate than men. Now, we're doing research looking at gender role orientation, and androgynous women—women who draw equally on their masculine and feminine sides— have exactly the same level of self-compassion [as men]. It seems to be the feminine women ... when you think about it, when you really identify with norms of self-sacrifice, “I should always be meeting the needs of others,” a lot of those problems that come from identifying with the traditional female stereotype. They're the ones who seem to suffer more. This is kind of new data, I haven't even published this data yet, it's kind of interesting but it makes sense to me. Women are told they should not take care of themselves; that they should always be outwardly focused.

Khazan: Is there a risk, though, that you can sort of forgive yourself for too much?

Neff: That's another surprising finding. People who are more self-compassionate are more likely to take personal responsibility for harming others and are more likely to apologize. When it's safe to make a mistake and you have the resources to say, “I can't believe I said that...” Self-compassion gives you the resources to acknowledge that and see yourself clearly, because you're not saying you're a horrible person, you're just saying, “Wow, I was out of line there.” And that actually increases your ability to take responsibility and apologize.

You might think that I'm batting all of these concerns away, but the research is pretty clear now. All the fears we have of self-compassion are pretty much based on misconceptions. And the research shows the opposite. Self-compassion helps you be motivated, it helps you take responsibility. It's not self-indulgent, it's not selfish, it leads to better relationships. I find it's quite remarkable how much research there is supporting these ideas.

Khazan: That’s great. So, how do you cultivate it if you’re not a naturally self-compassionate person?

Neff: One of the easiest ways is, "What would I say to a close friend I cared about in this situation?" So most of us have a lot more experience being compassionate to others than to ourselves.

Another one we talk about, actually, believe it or not, is physical touch. The physiological compassion system is triggered by the main three triggers, which is physical warmth, gentle touch, and soothing vocalizations (an ahhh sound), so it's amazing what you can do with a gesture. You know, putting your hands on your heart, or something for you that's supportive. What happens is your physiology calms down and the caregiving system gets activated and helps facilitate the talking to yourself in a kinder way. I always tell people, yes it is touchy-feely, but don't underestimate it because we are mammals at the end of the day.

Khazan: One reason this is hard for me to conceptualize is that I just don't think that's what friends say when their friends mess up? I think friends are more likely to say, “oh, it wasn’t really that bad,” or “that person deserved it.” To sort of minimize, rather than acknowledge the friend’s wrongdoing.

Neff: I think that's a good point. I think, in some ways, a parent might be a better example, because a parent is really invested in the well-being of their child. And a parent really wants to make sure that child grows up not doing things that are going to be harmful to them.

People have access to the language more easily when they think about how they treat a friend, because they are just much more experienced at it. But again, that mindfulness has to be there. If the compassion is used to say, “Oh, it's no big deal,” if it doesn't acknowledge the big deal ... you have to be aware when you use self-compassion that you aren't using it in a tricky way or in a superficial way, to make the pain go away and pretend there's not a problem when there really is one. You have to fully acknowledge that there's a problem here, a mistake was made. And once you do that, framing it in a larger compassionate perspective helps you to simultaneously acknowledge it, hopefully do some healing, and move on.