Weathering conservative attacks on his immigration reform work "has been a real trial," Sen. Marco Rubio (R-Fla.) said Wednesday, in a speech aimed at the tea party activists who helped elect him and are now upset by his work on the issue.

Speaking on the Senate floor, Rubio sought to reassure conservatives who are concerned that the legislation won't include serious border security measures.

"Getting to this point has been very difficult. To hear the worry, anxiety and growing anger in the voices of so many people who helped me get elected to the Senate, who I agree with on virtually every other issue, has been a real trial for me," he said. "I know they love America, and they are deeply worried about the direction this administration and the political left are trying to take our country."

The immigration system is broken and needed to be fixed, he argued, "and I wasn't going to leave it to the Democrats alone to figure out how."

Rubio also pushed back against those who accuse him of getting involved in immigration reform for political reasons. On the contrary, he argued that trying to pass the legislation has earned him the ire of "so-called 'pro-immigrant' groups," "the Beltway media," and "many conservative commentators."

"Truthfully, it would have been far easier to just sit back, vote against any proposal and give speeches about how I would have done things differently," he said. "This most certainly isn’t about gaining support for future office."

The senator has been struggling for months to woo skeptical conservatives. If the bill he has worked so hard to pass dies, it will be a blow to his standing as a presidential contender in 2016. If the effort succeeds, it would be a huge boon in the general election. But he would still have to worry about potential conservative anger in the primary.

The Senate passed an amendment earlier in the day designed to bolster security along the U.S.-Mexico border. Rubio supported that amendment; 29 Republicans opposed it, and the path to final passage of an immigration bill is still uncertain.