Oracle field tested a Target Rank module in their Home Energy Reports for residential customers with the aim of providing greater savings motivation than their Neighbor Comparison module.
Target Rank was an alternative user experience that was deployed for 18,000 income eligible customers already receiving electronic Home Energy Reports (eHER). During the pilot, the Neighbor Comparison module was replaced with a Target Rank module for six months; the new module provided the customer with a short-term achievable energy-saving target (called a “challenge”) in the format of a score on a 100-point scale. The pilot impacted 38,000 total customers receiving eHER; 18,000 received the Target Rank module as a treatment group, and the rest served as a control.
Results and Outcomes
Due to challenges in report configuration and roll out, the test took place from June 2019 to December 2019, with analysis and survey work conducted in Q1 2020. While several promising indicators were observed, the differences between the treatment and control groups were not large enough (or the sample size was not large enough) to achieve statistical significance and prove/disprove the null hypotheses. One contributing factor was fewer than 200 participants completed both the pre and post satisfaction survey, limiting the reliability of those survey results. Given these results, the Residential Programs and Oracle teams will consider future inclusion of the Target Rank module but may not proceed with immediate integration.
Despite this, the treatment (Target Rank) groups reported more positive responses to questions like, “[the report] helps me better understand my energy usage” and “[the report] provides value to me”. Both Target Rank groups had triple the click-through rate compared to the eHER control group (1.7% vs 0.6%), which is an indicator of higher customer engagement. There seemed to be overall movement among the treatment group from a left-skewed distribution of Target Rank scores at the beginning of the pilot to a more equal distribution by the end; however, the control group showed a similar trend. Finally, participants receiving the Target Rank experience rated it as more useful compared to those receiving a customized Low Income marketing module, but the small number of responses made strong conclusions difficult to make.