Between Rexburg and Yellowstone are a few more towns where people live year-round, including St. Anthony and Ashton. (Beyond that is largely seasonal, as the elevation is high and snow last a long, LONG time.) Just outside of St. Anthony is a very large series of sand dunes. The sand is fine and soft, and it wafts over basalt (lava) at the base of some hills. If you have ever seen the movie Napoleon Dynamite, this is where the grandma gets hurt fourwheeling and where the uncle films himself playing football.
Not far out of Ashton you can see Mesa Falls. This is the Snake River coming over the cliff. It's quite scenic and beautiful, although it's not a hiking destination. You drive there, you pay your $5 for parking, and you walk over to the falls and look at them. Which is too bad, as I was hoping for an actual hike. I think if you come in the right season, you might be able to go inside the building, though, which might offer a little more for your money.
After Ashton you come into Island Park, a very long, narrow "town" that goes on for like, 20 miles. It's mostly resorts. Very pretty in summer, but I doubt many people live there in winter. This is what it looked like on April 30 of this year (click to enlarge and see the snow levels):