“To quote William Wallace in Braveheart,” Paul Ryan said on Wednesday morning, “we have to unite the clans.”

This was the money quote of the House speaker’s address to a conservative conference in Washington. The one his office dished out to reporters in advance, a canned sound bite designed to pull back, ever so briefly, the ears of a Beltway political crowd transfixed by the intensifying presidential campaign.

Just about 12 hours earlier, Ryan met for beers with the most troublesome of those “clans”—the House Freedom Caucus, that group of a few dozen conservatives who shoved Ryan’s predecessor, John Boehner, out of office and who gave him only the most lukewarm of blessings to take his place. According to The Huffington Post, the meeting “did not go well.” The topic was the budget, and the insistence of the Freedom Caucus members that House Republicans abandon the two-year agreement that Boehner struck with the White House by cutting billions of dollars in spending for fiscal 2017.

Ryan endorsed that deal and implemented it by shepherding to passage an omnibus appropriations bill that funds the federal government through next September. A top priority for him and Senate Majority Leader Mitch McConnell is to use the next several months of budget peace to return Congress to what insiders call “regular order”—the process of passing individual appropriations bills through the House and Senate to fund government agencies rather than lumping them all together into a massive, year-end bill that conservatives typically despise. (In December, 95 Republicans voted against the omnibus, and that was considered a victory for leadership.)

Conservatives prize regular order, too, but they don’t like the higher spending levels that were included in the budget agreement. And they were spooked when the projected deficit shot up by more than $100 billion, partially a direct result of the spending increases and tax increases Congress passed at the end of 2015. “Paul Ryan has two choices,” Representative Mo Brooks of Alabama told The Huffington Post. “He can either support a financially responsible path that rises to the challenges that America faces, or not.”

For Ryan and the Freedom Caucus, this confrontation was only a matter of time. House conservatives like Raul Labrador, Brooks, and others largely rolled their eyes at the adulation poured on the new speaker by Republicans who practically begged him to take the job last year. They warned Ryan would have a brief honeymoon, and now it’s over. Ryan didn’t need their votes when he struck bipartisan agreements with Democrats at the end of last year, but the annual budget resolution is another matter. It’s traditionally a party-line vote and a particular point of pride for Ryan, a former chairman of the Budget Committee. GOP leaders can’t afford to lose more than a couple dozen Republican votes, so conservatives are trying to exert their leverage to get more spending cuts. Yet a budget that doesn’t adhere to last fall’s agreement would push Democrats away and threaten to unravel Ryan and McConnell’s grand plan for a normal appropriations process.

The relatively arcane budget process, of course, is really just a small fissure compared with the much bigger tactical divisions in the Republican Party, and that’s what Ryan tried to confront on Wednesday. He was speaking to Heritage Action, the rabble-rousing, right-wing advocacy group that emboldened conservatives inside Congress and made so much trouble for Boehner and his lieutenants in the leadership. Boehner would never have appeared at the Heritage summit, having spit hot fire at the group for much of his last two years in office.

Ryan showed up, if only to ask them to stand down. He argued that Democrats had been trying to sow division in the GOP for years, and in this critical election year, conservatives had to stay united. “What I want to say to you today is this: Don’t take the bait,” he said.

Don’t fight over tactics. And don’t impugn people’s motives. It’s fine if you disagree. And there’s a lot that’s rotten in Washington. There’s no doubt about that. But we can’t let how you vote on an amendment to an appropriations bill define what it means to be a conservative. Because, it’s setting our sights too low. Frankly, that’s letting the president define us. That’s what he wants us to do. That’s defining ourselves as an opposition party, instead of a proposition party.

Will conservatives heed that call? Unlikely. Later in the afternoon, at a panel on the “State of the Conservative Movement,” Representative Dave Brat offered something of a response. Brat is the Virginian who took out Eric Cantor, defeating the Republican majority leader in perhaps the biggest primary upset in House history. He’s now a member of the Freedom Caucus, and he urged Heritage and its funders to keep the pressure on GOP leaders, no matter who they were. “It’s absolutely crucial that you guys pile on politically in order to outweigh the special-interest power,” Brat said.

Ryan wants to use 2016 as a test run for the GOP in Congress, hoping he can begin to unify the party so that it can govern effectively in 2017 if a Republican president is elected. And as a popular young conservative himself, he undoubtedly has a better chance to succeed where Boehner fell short. Yet if the emerging budget battle is any indication, he’s off to a bumpy start.