But he acknowledged that in the wake of the leaks by Edward Snowden, there is now a widespread view that the agency is invading people's privacy.
"We've got to find a way to get the confidence of the American people back so they will respect NSA as much as they respect the military," he said.
He argued that his plan would bolster privacy protections while maintaining the NSA's ability to uncover terrorist plots.
Ruppersberger proposal is in line with President Obama's goal of giving up NSA control of the phone database while maintaining the program's capability. The administration is currently reviewing several options for overhauling the program, including having phone companies hold the data and giving the data to a third-party group.
The White House is expected to announce its plan for the program before March 28.
But Ruppersberger warned that no matter what plan the White House comes up with, the program could expire next year when the Foreign Intelligence Surveillance Act is up for renewal.
He said he is working with House Intelligence Committee Chairman Mike Rogers on legislation that would revamp and extend the law.
"I'm not sure whether we could get the votes to pass another FISA bill," Ruppersberger said. "Mike and I realize we have to make a change."
But he acknowledged that the House Intelligence chairman is not on board yet with his proposal to limit the NSA phone sweeps.
In an emailed statement, Rogers said he continues to work with Ruppersberger and other lawmakers "to craft a proposal that will address the concerns around bulk data storage, protect civil liberties, increase transparency and confidence in the government's intelligence-collection activities, and maintain a targeted capability for counterterrorism operations."
Caitlin Hayden, a White House spokesperson, said the president hasn't decided yet on his plan for the program.
Ruppersberger's proposal would not strengthen the standard NSA analysts need to meet before reviewing phone records. Currently, the NSA collects millions of records, but only accesses the database if there is a "reasonable, articulable suspicion" that a phone number is connected to terrorism.
Under the USA Freedom Act, a tougher bill from GOP Rep. Jim Sensenbrenner and Democratic Sen. Patrick Leahy, the NSA would need to show that a record is relevant to a terrorism investigation and pertains to an agent of a "foreign power."
But Ruppersberger argued that the USA Freedom Act's standard is too restrictive.
"In my opinion that would put our country at risk," he said. The Maryland Democrat argued that intelligence agents are trying to thwart terrorist attacks and they shouldn't be held to the same standard as police or prosecutors trying to obtain evidence for a trial after the crime has already been committed.
In a statement, Sensenbrenner applauded Ruppersberger for agreeing that bulk data collection should end and urged him to sign on to the USA Freedom Act.
"It strikes the proper balance between security and privacy, and I am confident it has the votes to pass," Sensenbrenner said.