Byproducts of energy use are among the leading drivers of emissions that can lead to climate change. Heating and cooling represent the largest source of energy use in urban infrastructure; thus, significant policy efforts have been implemented to reduce energy demand. Estimates of the efficacy of these policies are often difficult as they rely on self-reporting or detailed utility data, which is not available broadly and challenging to collect in areas where utility infrastructures are unavailable. Buildings' heating, ventilation, and air conditioning (HVAC) can be used to measure energy consumption from the demand side to evaluate end use and potential energy policy-induced reductions. Therefore, we are proposing a novel, non-intrusive method of energy use monitoring that uses proximal infrared remote sensing of building envelopes to find patterns of heating and cooling use. Our results show that the technique can discern on-and-off patterns of externally facing cooling units (ACs) in a dense urban scene at distances up to ~0.5 miles. We have tested multiple cases of residential buildings in New York City. We collected ~337,000 infrared images at 10-second intervals of the buildings’ facades taken from June 11 to July 18, 2018, between 500 m and 1 km. We have applied computer vision techniques to that sequence of images to identify exterior-venting HVAC units and generate their infrared time series, which correlates directly with their temperature. We determine each AC unit's on/off transitions using a one-dimensional edge detection algorithm and identify aggregated and disaggregated patterns of end-user behavior that are direct proxies for total heating and cooling use. Finally, we demonstrate that this new methodology is sufficiently sensitive to assess policy interventions to reduce daily peak demand effectively.