The integration of variable and intermittent renewable energy generation into the power system is a grand challenge to our efforts to achieve a sustainable future. Flexible demand is one solution to this challenge, where the demand can be controlled to follow energy supply, rather than the conventional way of controlling energy supply to follow demand. Recent research has shown that electric building climate control systems like heat pumps can provide this demand flexibility by effectively storing energy as heat in the thermal mass of the building. While some forms of heat pump demand flexibility have been implemented in the form of peak pricing and utility demand response programs, controlling heat pumps to provide ancillary services like frequency regulation, load following, and reserve have yet to be widely implemented. In this paper, we review the recent advances and remaining challenges in controlling heat pumps to provide these grid services. This analysis includes heat pump and building modeling, control methods both for isolated heat pumps and heat pumps in aggregate, and the potential implications that this concept has on the power system.