When Brett, a 69-year-old man from St. Paul, Minnesota, went to visit his dying mother in a nursing home a few years ago, he began to worry in earnest about his later years.

“I’ve been in institutions most of my life,” says Brett (his last name has been withheld to protect his privacy), who spent his teenage years in a series of foster homes and juvenile detention centers, estranged from his family. “It never scared me. But I won’t get to be in a nursing home like my mom was.” His own experience, he fears, will be much different.

Brett, who is transgender, has a full beard, a low speaking voice, and has had his breasts removed. But he never had sex-reassignment surgery, meaning that his transgender status would quickly become obvious to a nursing-home aide charged with bathing or dressing him. Like many transgender seniors, he worries what this will mean for him once he enters a nursing home or assisted-living facility.

Currently, there are more than 1.5 million LGBT people over 65 in the U.S., a number expected to double over the next 15 years as the population ages. But precise statistics on older transgender adults—or, for that matter, transgender people of any age—are hard to come by. One 2011 study using health-survey data estimated that the country’s transgender population was around 700,000; this past May, the Census Bureau published a study that analyzed the number of “likely transgender individuals” based on the people who had changed their name (around 90,000) or sex (around 22,000) with the Social Security Administration.

These estimates vary so wildly in part because there’s no reliable means of tracking when people change their gender: The Census Bureau still offers only male and female, and many trans people haven’t completely transitioned into living full-time as their expressed gender. Others have so successfully suppressed their history that there’s little evidence they ever lived as anyone else.

One thing, though, is clear: For transgender people, aging into the later years of life can present a unique set of challenges.

Most transgender people have not surgically transitioned—for reasons that include prohibitive cost and decreased sexual function—so when they disrobe in a medical setting, they’re automatically outed, explained Loree Cook-Daniels, the founder of the Milwaukee-based Transgender Aging Network. “That inability to closet even if they want to means we have a much bigger problem in getting trans people to health care,” she said.

Past research has found that many transgender people avoid seeing a doctor for fear of being ostracized. Of those who do seek health care, a 2011 survey of 6,456 transgender people across the U.S. found that 28 percent of respondents had suffered harassment in medical settings, including ridicule or rough treatment. Nineteen percent said that they had been denied care altogether by doctors and other providers, and 50 percent reported having to educate their medical providers on transgender care.

In a more recent study, Tarynn Witten, the director of research and development at Virginia Commonwealth University’s Center for Biological Complexity, examined the concerns of 1,963 transgender survey respondents about their later-life and end-of-life plans. Many participants, she wrote, “stated that they would prefer to die at home rather than be in a palliative care or nursing home facility,” often out of “[fear] of the type of care that they would receive; would they receive the right painkillers, would the care be respectful, would they be abused or violated, would their gender identities be respected, would they be allowed to live their last moments with grace and dignity [?]”

“As I am only part-time in each gender, I am worried that I will be in some situation that will force me to be considered totally masculine,” one of Witten’s study participants wrote, like “being assigned to ‘the boy’s room,’ meaning exile from femininity.”

In nursing homes, gender-segregated spaces like bedrooms and housing wings can be of particular concern to transgender seniors. As Alia Wong recently reported for The Atlantic, the U.S. Department of Education has affirmed that the protections of Title IX extend to transgender students; in July, citing Title IX, the Justice Department filed a statement in support of a transgender student suing a Virginia school district after being required to use an “alternative” restroom. But, Cook-Daniels said, there are currently no similar anti-discrimination regulations in place for nursing homes and other assisted-living facilities.

Cook-Daniels and other TAN employees regularly host workshops and trainings on transgender aging issues for health-care professionals, elder-care providers, and national aging organizations. The most common questions, she said, are ones about segregated living spaces: “People want to know, ‘What gender roommate do I give [a transgender senior]?,’ ‘What bathroom do they use?’,” she said. “There’s still a basic lack of understanding of how gender identity trumps biological sex, and that people should have access to facilities that match their gender identity.”

The roommate question, in particular, can be fraught with complications. Brett recently experienced what life might be like for him in a long-term care facility when he checked in for a temporary stay in a St. Paul nursing home to recover from a back injury. After a week in a single room, he said, he was assigned a male roommate—but he worried that a roommate of either gender would soon discover that he was transgender, a fact he didn’t want to be publicized.

“A female would have a problem with me, and I’d have a problem with a male,” he said. “I wouldn’t want him to know about me … And with the gossip in nursing homes, that secret would last about a week.”

Rather than take a roommate, Brett left the nursing home and went to stay with his daughter, who took over caring for him as he recovered. But many in Brett’s situation wouldn’t have that same option: Past research has found that LGBT seniors are only half as likely as their heterosexual peers to have a support network of close relatives, and in a recent survey of 6,450 transgender people in the U.S., more than half reported that they were estranged from their families—a significant disadvantage in a country where around 80 percent of older people with disabilities rely partially or exclusively on family caregivers.

Mitigating the effects of social isolation experienced by older LGBT adults is a goal of the advocacy group Services and Advocacy for Gay, Lesbian, Bisexual, and Transgender Elders (SAGE). One of the organization’s current priorities is working to secure recognition for the LGBT community in the Older Americans Act (OAA), the source of federal funding for senior-care programs across the country, as a population of “greatest social need,” deserving of dedicated funds for training, outreach, and services. (The current interpretation of the term, as explained by the U.S. Administration on Aging, is vague: “In some communities, such isolation may be caused by minority religious affiliation. In others, isolation due to sexual orientation or gender identity may restrict a person’s ability to perform normal daily tasks or live independently. Each planning and service area must assess their particular environment to determine those populations best targeted based on ‘greatest social need.’”)

On July 13, the SAGE executive director Michael Adams was one of four delegates representing LGBT concerns at the White House conference on aging, a conversation that happens once every 10 years and influences the contents of the OAA. The National Resource Center on LGBT Aging, a training center run by SAGE, currently receives direct federal funding, Adams said, but he hopes that the delegates’ presence at the conference will make groups like his more of a priority: “Designating older LGBT adults as a population of greatest social need in the OAA could open the door for [other training organizations] to receive federal funds,” he said.

In the meantime, many similar groups rely on private donations, small state grants, and fees paid by the service providers who hire them. Rajean Moone runs one such organization; as the executive director of the St. Paul-based Training to Serve, he regularly works with facilities that want to improve their inclusivity. “It’s not about repainting your building with a rainbow flag,” he said. TTS recommends simple steps like changing intake forms to leave questions about gender open-ended, and putting mechanisms in place for residents to report bullying or discrimination.

“We’re not there to change anyone’s beliefs,” says Moone. “We’re there to help them give high-quality care to people who deserve dignity.”