Will Virtual Audits be the new normal? Why data science is opening new doors in the energy efficiency industry
WRITTEN BY team energyx ∙ TORONTO, ONTARIO
Originally appeared in AESP Strategies.
For most buildings, the first step towards energy efficiency is an on-site energy audit by a certified professional. The result of this audit is a detailed analysis of the building's energy performance completed using sophisticated modeling software. These on-site assessments are accurate and comprehensive, but they can also come with a high cost – particularly for residential buildings where audit costs represent a greater proportion of total retrofit project costs.
Because of the reliance on experts to collect and process data, on-site audits also present challenges in terms of scalability and consistency. Scaling audits to more buildings essentially requires deploying more energy auditing experts, which does little to reduce the cost to audit an individual building. Meanwhile, variations in building conditions, data collection, and auditor experience present challenges in providing a consistent audit experience across all buildings.
Finding ever more accurate, rapid, cost effective, and scalable auditing techniques is vital for the advancement of energy efficiency, and the need for viable alternatives to on-site audits has recently been amplified by the COVID-19 pandemic. Digital audits have existed for decades, and they run the gamut from customer engagement and education tools through to data-based energy assessments and building modeling designed to rival expert advice.
Data science is fueling the next generation of digital audit technology and the term ‘Virtual Audit’ is now being used to characterize audits completed with limited or no in-person interactions. This article explores the state of Virtual Audit technology today and presents potential use-cases for data-based Virtual Audits into the future.
What are ‘Virtual Audits’ and why are they useful?
When it comes to how we define energy efficiency tools, the waters are muddier than ever: Audit descriptors such as online, virtual, digital, disaggregated, AI-powered, remote and more all seem to ambiguously overlap. A distilled definition of what we think of as a ‘Virtual Audit’ would read something like this:
“A systematic assessment of a building’s energy performance and improvement potential conducted via software using available data collected without visiting the building site.”
This definition maintains the key elements of any holistic audit and clarifies that the audit should attempt to analyze the entire building and not only specific elements of it. Additionally, it provides qualification for the term ‘virtual’ that helps to separate a truly Virtual Audit from close cousins such as “remote audits” which utilize software that helps keep professional auditors off-site. Remote audits ultimately still require someone on-site to collect the data, whether it’s the building owner providing information over the phone or capturing video of their surroundings. In contrast, online ‘self-guided’ audits and consumption disaggregation go so far as to eliminate the auditor completely, but they fall short in their ability to accurately and holistically assess the building’s performance and improvement potential. All of these solutions exist as complements to or replacements of the traditional on-site audit, and each of them have strengths and weaknesses.
For Virtual Audits to be truly useful and valuable, they need to outperform existing options. Table 1 illustrates a qualitative comparison of Virtual Audits against on-site, remote, and online audits. Virtual audits need to provide high accuracy and be cost effective in terms of dollars paid (i.e. the real cost) and operation hassle incurred, such as occupants taking the day off work (i.e. the transactional cost). To take full advantage of this economy of “high accuracy for less cost” they should also be scalable and deliver consistent results.
More audits = more efficiency
With over 10 million single-family homes in North America1,2, scalability and consistency are of critical importance. While the typical residential building has the potential to reduce its energy consumption by more than 25%, most homeowners struggle to understand how to unlock these savings through building improvements; hence the value of energy assessments in providing specific recommendations. By identifying efficient building retrofits, energy audits drive the adoption of energy efficiency. Put simply: more audits = more efficiency.
Today, on-site audits only reach about 1% of the population each year 3,4. Industry conversion rates suggest that online audits and other self-guided tools can reach another 5% to 15% of the population in areas where they are available. This leaves a staggering 84% or more of the population unaudited. To grow the energy efficiency sector and to achieve our collective efficiency policy goals, we have to reach the remaining 84%. Virtual audits can help.
Despite the fact that a vast amount of residential building data has been accumulated in recent years, the data sets are large and complex and can’t be effectively utilized by relational database management systems. Data science technology such as Machine Learning has been introduced as the solution to harness the potential of this data, allowing us to capture, analyze, update, query, and visualize information. In the context of energy efficiency, modern Machine Learning algorithms such as “Deep Learning” can leverage data that utilities have been collecting for years, including building characteristics and historical energy consumption, combined with numerous publicly available data sources, to open new doors that will revolutionize energy efficiency delivery.
Are Virtual Audits really accurate?
Despite the potential Virtual Audits present for the energy efficiency sector, they have been challenged on their ability to deliver the same level of granularity and savings potential as on-site audits. In contrast to the more ‘white-box’ nature of the physics-based calculations utilized by traditional energy modeling software, data-based Virtual Audits are more ‘black box’ and are developed by correlating inputs (e.g. building information) with desired outputs (e.g. annual building energy consumption). These correlations are developed and established through a process known as ‘model training’.
Model training can be used to ensure that Virtual Audits behave in a way that inherently accounts for and reduces the risk of making inaccurate predictions about building performance. Figure 1 shows that a ‘sweet spot’ exists where data-based Virtual Audits can essentially balance risk and accuracy. Through model training and management of model complexity, data scientists are able to achieve the ‘sweet spot’: accurate ‘Virtual Audit models’ that work for large portions of a population to produce consistently accurate predictions within acceptable error limits.
To examine the accuracy of Virtual Audits, we collected and processed robust sets of building data to determine correlations between building features and energy consumption. Virtual audits were able to predict the energy consumption of a home with the same statistical accuracy as an on-site audit, even without looking at the results of the blower door test (Figure 2).
Machine-learning based Virtual Audits also offer the flexibility to replicate the software output generated from an on-site assessment under ‘standard operating conditions’, or to calibrate the predictions to more closely predict actual consumption without requiring any additional inputs. From a practical perspective, this allows the Virtual Audit to achieve a closer estimate to actual energy consumption than what is generated by the typical building analysis software used in an on-site audit. These comparisons are illustrated in Figure 3. The red and green lines compare on-site audits (red line) and Virtual Audits (green line) from the perspective of standard operating conditions. The congruence of the lines indicates that Virtual Audits are as accurate as on-site audits in this regard.
The actual consumption of the building (blue line) represents the actual occupancy condition. When the Virtual Audit is asked to target actual ‘occupied’ consumption (yellow line) it effectively deviates from the standard operating conditions and predicts consumption closer to actual (Figure 3).
New Doors - Use-cases for Virtual Audits
The utility sector has already benefited from advanced analytics in grid management, demand response, and predictive maintenance. Similarly, Virtual Audits offer utilities new opportunities to rapidly advance in key aspects of their business.
Door 1: Market segmentation through ‘bottom-up’ potential studies
Virtual audits offer accuracy and consistency at scale. Once a model is developed for a particular region, any number of buildings can be audited so long as the basic inputs to the model are available. This means that the number of audits that can feasibly be performed in a service territory is limited only by available data.
In recent projects, we have found that readily available data can be used to virtually audit approximately 20% of the residential building stock annually without any customer engagement. This presents new opportunities for market segmentation where Virtual Audit results essentially validate market potential before an efficiency program has even been launched. Utility program designers and government planners can utilize Virtual Audits to segment markets based on building nuances normally perceptible only through a bottom-up, building-by-building analysis.
Door 2: Customer Engagement
Instead of hoping that the ideal customers participate in energy efficiency programs, Virtual Audits offer the opportunity for new customer engagement tactics. ‘Push marketing’ strategies can be used to target only the buildings that are the most likely to benefit from improvement opportunities identified by Virtual Audits.
With Virtual Audits, mass-marketing is no longer about informing all customers of efficiency programs and incentives that are available, but instead providing automatic enrollment and even pre-approval for specific customers that are identified to be top candidates. In this way, push marketing strategies can be used to both increase program participation as well as overall program effectiveness. EnergyX research has shown that optimal customers (i.e. top quartile in terms of energy saving potential) can deliver up to four times the energy savings of “average customers” - a significant boost to any program’s cost effectiveness test.
Door 3: Claiming and Adjudicating Savings
Program effectiveness and attribution remains a persistent industry challenge. In no small part, this is because auditing and collecting the data to evaluate all program savings is an impossible task given the conventional means available. Today, governments and program evaluators rely on surveys and limited audits conducted for handfuls of actual program participants, when Virtual Audits could do the same job for all program participants at a fraction of the cost and time.
Because Virtual Audits can be calibrated to target actual consumption and to account for any confidence interval or error limit, they seem poised to redefine how program effectiveness is claimed and adjudicated. Their mathematical and algorithmic nature brings the assumptions and biases of both operators and evaluators to the forefront with statistical transparency. Together these factors will increase reporting accuracy, which ultimately leads to a better understanding of how to optimize efficiency initiatives in the future.
Conclusion
From population segmentation to individual energy consumption prediction, Virtual Audits provide a new lens for evaluating building performance at scale. Although they use a different approach, the outcome is comparable to an on-site audit performed by professional auditors. Through sophisticated analysis of building characteristics and other data, most energy saving opportunities can be identified without needing to physically visit a building. This presents an obvious benefit: Virtual Audits can be applied to the majority of a population, only deploying professional energy auditors when and where they are needed most.
Virtual Audits aren’t designed to eliminate the role of professional auditors and building scientists in energy efficiency, but rather grow our collective potential in this sector. By finding the complex patterns in large data sets, we strive to audit more buildings, reach more people, and maximize our impact.
References:
1Statistics Canada. 2016. Census in Brief, Dwellings in Canada (https://www12.statcan.gc.ca/census-recensement/2016/as-sa/98-200-x/2016005/98-200-x2016005-eng.cfm)
2The United States Census Bureau. 2020. (https://www.census.gov/housing/hvs/files/currenthvspress.pdf)
3Residential Energy Services Network (RESNET®) (https://www.hersindex.com/benefits/)
4Statistics Canada. 2011. Environment Accounts and Statistics Division, Households and the Environment Survey (survey number 3881). https://www150.statcan.gc.ca/n1/pub/11-526-x/2013001/t053-eng.htm
