I don’t know about you, but every morning for the past month I’ve woken up, crawled from bed to check the news, then consoled myself with the thought: “Only X more days until this hot mess of an election is done.” Losers want to riot? Bring it. At least the public can go back to ignoring pollsters, political consultants, and Newt Gingrich, the way God intended.

For one long-suffering soul, however, Election Day will not bring relief. Almost everyone agrees that, whatever happens November 8, Paul Ryan is looking at a lousy 2017. For starters, he’ll be contending with a vengeful Donald Trump, who has proved himself ungracious in both defeat and victory. But what really stinks for the speaker: Trump will likely be less of a headache than members of Ryan’s own conference.

For months, the ultraconservative Freedom Caucus has made no secret of its displeasure with Ryan. Some members are unhappy about his treatment of Trump. Others are disappointed at how few of their issues he prioritized this session. Still others blame him for Tea Partyer Tim Huelskamp’s primary loss. (Don’t ask.) Bottom line: Plenty of Freedom Caucusers are itching to teach the high and mighty Ryan a lesson.

The most direct rebuke would be for a caucus member to run in the upcoming speakership election. Alas, the rabble-rousers’ mission is complicated by the fact they lack a challenger well-positioned to garner the necessary support. (This is, ironically, a by-product of the caucus’s tendency to stick a thumb in the eye of fellow Republicans). Indeed, the dearth of hard-right challengers in the House has led to the bizarre development of Ryan’s failed primary opponent, businessman Paul Nehlen, announcing a bid for speaker. (Yes. A non-House member technically can hold the gavel.)

And so conservative diehards have been pondering ways to increase their leverage over Ryan. Some would like to delay leadership elections until after the lame-duck session. This would let them keep the heat on Ryan during the looming appropriations battle, discourage him from holding a vote on the controversial TPP, and maybe even make him more amenable to rule changes aimed at increasing conservative members’ influence.

In short, the Freedom Caucus is determined—eager even—to make Ryan’s life as painful as possible in the coming Congress.

For their part, less revolutionary-minded lawmakers are looking to give Ryan a bit of breathing room. Representative Devin Nunes (a former Boehner ally who has called Freedom Caucusers “right-wing Marxists”) thinks he has hit on one possibility: Strip individual lawmakers of the ability to hold the speaker hostage using a “privileged motion” to “vacate the chair.”

A privileged motion is a parliamentary procedure that allows any member to push his issue to the head of the line, superseding regular House business and forcing a floor vote within two days. A motion to vacate the office of the speaker was how Freedom Caucuser Mark Meadows shoved John Boehner out the door in 2015. In that case, Meadows specifically designated his motion “non-privileged,” as sort of a warning shot. But a motion to vacate is typically privileged and, as such, a powerful threat to hold over a speaker.

Currently, such a motion can be filed by any lawmaker, regardless of how much support his proposal has.

Nunes wants to do away with this sort of blackmail. At a September 14 hearing of the Rules subcommittee, he argued for changing House rules so that, before a privileged motion to vacate the chair could be filed, it would first need the support of a majority of lawmakers from either party’s conference.

“Obviously, we know the turmoil that can be created with a huge leadership vacuum in the middle of a Congress,” Nunes told the subcommittee. “We saw that this Congress where any member, if they want to go out and offer something, if it becomes popularized over the internet, becomes a fundraising tool, and the next thing you know you have people attacking leaders of our respective parties.”

Nunes’s office says he’ll introduce the change after the election “as part of a larger group of reforms.” It’s likely to receive a fair amount of support. His presentation was warmly received by the Rules subcommittee. Democrat Jim McGovern said of the existing system, “I don’t think it’s good for the institution. It creates uncertainty. And it creates a climate here where it makes it much more difficult to get things done.”

Before he took the gavel in October 2015, Ryan himself expressed a desire to reform the rule. As his spokesman Brendan Buck told the GOP conference at the time, “No matter who is speaker, they cannot be successful with this weapon pointed at them all the time.”

As you might imagine, the Freedom Caucus is less than thrilled. Representative Dave Brat has slammed Nunes’s proposal as “a move to protect the existing power structure” that “goes in the opposite direction of what we need.” And caucus spokeswoman Alyssa Farah assured me, “Any changes to or moves to eliminate the Motion to Vacate are a non-starter for the Freedom Caucus.”

Conservatives beyond Capitol Hill are joining the battle, too. The Tea Party group FreedomWorks is loudly opposing Nunes’s proposal. In an October 21 op-ed in Washington Examiner, the group’s president and CEO, Adam Brandon, argued: “The motion to vacate the chair is the sword of Damocles that hangs over the head of whoever happens to hold the office. If a speaker begins to lose his or her way and act in a manner that harms the House, this procedural tool serves as a vital accountability mechanism.”

Brandon called on the Speaker to come out against Nunes’s proposal now, noting ominously that “with a diminished House Republican majority expected in the 115th Congress, any rules package that guts the motion to vacate the chair would create unnecessary tension in the House Republican Conference and lead to the defeat of the rules on the floor. All Ryan has to do to avoid this is come out against Nunes' proposal.”

This is a not so gentle reminder to Ryan that, if Republicans lose House seats on Election Day, he will have less wiggle room in the new Congress and will be even more dependent on his ultraconservative members.

For now, at least, the beleaguered speaker refuses to play.  His office informed me that he is focused solely on “protecting our majority” and has no intention of discussing rules changes until after the election.

The Freedom Caucus intends to be ready when he is.

So much for Ryan getting a break in the New Year.