Consider for a moment how the Democrats throw fits over the hubris of the Green Party to practice democracy. In Montana the Dems sued in court to keep the Greens off the ballot. In California they changed the election laws to squeeze them out.
Now compare that to the Republicans, who rarely complain about the far larger Libertarian Party.
In fact, the Libertarian Party surged by 92% after 2008, and yet Republicans rarely point fingers of blame like the Democrats do.
While the Dems try to shame Green Party voters, Republicans court Libertarian voters.

Libertarians smashed their previous records for votes and membership in 2016.

However, the glory days of the Libertarian Party might already be coming to an end.

At the end of January 2017 the party hit its highest membership number of this decade, 18,908. That number dropped 24 percent to 14,321 as of the end of August.
...Only 300 new paying members joined the party in August while over 2,000 failed to renew their membership.

Now much of the drop in 2017 was simply a natural decline in an off-year.
However, I found this article which might explain the big drop in August.

In a press release on August 15, 2017, in the wake of the violent protests in Charlottesville, Virginia, the LP’s executive director Wes Benedict reiterated the party’s stance on bigotry:

“We condemn bigotry as irrational and repugnant. Government should neither deny or abridge an individual human right based on sex, wealth, ethnicity, creed, age, national origin, personal habits, political preference, or sexual orientation.”

The press release went further, and asked any members who identified as white nationalists to leave the party:

“There is no room for racists and bigots in the Libertarian Party. If there are white nationalists who — inappropriately — are members of the Libertarian Party, I ask them to submit their resignations today. We don’t want them to associate with the Libertarian Party, and we don’t want their money. I’m not expecting many resignations, because our membership already knows this well.”

Maybe the membership didn't know this well.

Meanwhile, on the opposite side of the political spectrum, the Democratic Socialists of America have surged in the post-2016 era.

Following the 2016 political campaign season, the party’s membership had gone from a relatively steady membership of roughly 6,000 to a membership of over 32,000. As such, it is believed to be the current largest socialist political party in the country.

I'm gonna go out on a limb and say that the rise of the DSA and the decline of the Libertarians are both partly due to Trump.