Department of Housing and Urban Development Secretary Julián Castro at National Journal's event, "Sustainable Homeownership: The Future of Housing Finance" at the Newseum in Washington, D.C. on June 3, 2015.National Journal

The housing market could look very different later this year if certain players get their way.

Among the possible changes coming to the housing sector: a reduced role for Fannie Mae and Freddie Mac in families' mortgages, with the hope that more private lenders feel confident enough to give out loans again.

"The president has previously stated he supports a comprehensive housing-finance reform, centered on the need "¦ to require more private capital in the system," Rep. Ed Royce said at National Journal's "Sustainable Homeownership: The Future of Housing Finance" conference, underwritten by 1st Alliance Lending, on Wednesday afternoon at the Newseum. "Exactly. And I think Congress should take him up on this and put a bill on his desk that does just that."

Royce, a California Republican who sits on the Financial Services Committee, lauded a recent bill from Senate Banking, Housing, and Urban Affairs Chairman Richard Shelby that would phase out Fannie Mae and Freddie Mac, calling them "the unfinished business of the financial crisis."

Since the housing bubble burst in 2007, credit—and by extension, homeownership—has declined, something that Housing and Urban Development Secretary Julián Castro said must be fixed in order to buoy the rest of the economy.

"Responsible homeownership remains a fundamental pillar of the American Dream," Castro said. "It allows folks to invest in their own communities, it sparks a wave of economic activity, "¦ and for many hardworking Americans, buying a home is the best avenue to build wealth."

Castro in part endorsed Royce's outlook, saying that President Obama wanted a proposal from Congress to encourage the private sector to take on some of the risky loans now held by the quasi-nationalized Fannie Mae and Freddie Mac. However, he said, "the proof is going to be in the pudding."

"I agree with him that ... the administration would like to see housing-finance reform happen," Castro said, "and I think that we agree on some of the common elements of that, including taking the taxpayers off the hook, getting more private capital into the system."

Such a proposal has some backing in the business community, which Royce said was ready to engage and lend again given the right opportunity. The banking industry could support a slimmed-down Fannie Mae and Freddie Mac.

"Love? Maybe. Like? Sure," said Joseph Pigg, senior vice president and senior counsel of the American Bankers Association. "It's definitely a better vision of the future than where we are today. "¦ Right now we view them as pretty much the only game in town."

Panelists—which included Julia Gordon, senior director of housing and consumer finance at the Center for American Progress, and Andrew Jakabovics, senior director of policy development and research at Enterprise Community Partners—also discussed the proper role of a regulator to make sure government agencies providing loans didn't take on too much risk, repeating the same conditions that led to the financial collapse.

"If you're going to create that moral hazard "¦ you have to have a regulatory system that controls the risks," said Mark Calabria, director of financial-regulation studies at the Cato Institute.

Royce admitted that his optimism about reform becoming law would surprise the audience. But as he rushed back to the Capitol for a vote, he had a few parting words for Castro: "Take a look at the Senate bill."

We want to hear what you think about this article. Submit a letter to the editor or write to letters@theatlantic.com.