Review and analysis to verify model quality (QA/QC)

Quality Assurance (QA) and Quality Control (QC) are are critical in BEM analysis, ensuring the reliability and validity of both input and output data. Poor data quality, whether due to inaccuracies, inconsistencies, or incompleteness, can lead to fundamentally flawed analyses and erroneous designs. QA processes establish the framework and procedures to prevent errors from the outset, focusing on data collection methods, validation rules, and proper documentation. QC, on the other hand, involves the hands-on inspection, testing, and monitoring of data throughout the analysis lifecycle to identify and correct any deviations or defects. Without rigorous QA/QC, design decisions could be based on unreliable information, jeopardizing project success.

This section provides guidance on how to effectively review a model's inputs and outputs.

Review input data

Some simulation programs provide reports that summarize the model inputs. Other programs require users to review the model input files. It is important to ensure that inputs used in the model match your intended inputs (the building you are trying to analyze) as closely as possible. Below are several examples of model input checks to perform.

Project location

Verify that the project location used for the analysis is consistent. Check to make sure that the input file location, and design sizing are using the same location. Also verify that the weather file used for the annual simulation is consistent with the input file location. An example is provided below. Note that there are likely to be variations in naming conventions between the different location/weather inputs, but the city name is generally featured in the names of them all. The city names should match, or they should be geographically near each other with the same climate conditions.

Site Location: CHICAGO_IL_USA TMY2-94846

Sizing Data: CHICAGO_IL_USA Annual Heating 99% Design Conditions DB

Sizing Data: CHICAGO_IL_USA Annual Cooling 1% Design Conditions DB/MCWB

Weather File: Chicago Ohare Intl Ap IL USA TMY3 WMO#=725300

Model geometry

Perform a visual inspection of the BEM model geometry and confirm that it looks correct when compared to architectural drawings and renderings. Many BEM tools have the ability to view the BEM model geometry directly within the tool. Other tools allow for the geometry to be exported to an external tool like CAD, BIM, or other 3D drawing tools.

The visual inspection should include reviewing the model in isometric views, and plan views of each building story. The isometric views can be compared to architectural elevation drawings or architectural renderings. The plan views can be compared to architectural floor plans.

Building envelope area

Check floor area, wall area, roof area and window area. In addition to checking total areas, it can be useful to look at wall and window areas based on cardinal orientation to see how they compare. Below is an example of this type of analysis - wall areas were summed up for each orientation and the areas of opposite orientations are compared.

The example below is for a rectangular building, so the opposing wall areas are the same. In order to determine the areas for each orientation, the model inputs were analyzed as follows:

• North walls are oriented at 0 degrees azimuth angle, +/- 45 degress
• East walls are oriented at 90 degrees azimuth angle, +/- 45 degrees
• South walls are oriented at 180 degrees azimuth angle, +/- 45 degrees
• West walls are oriented at 270 degrees azimuth angle, +/- 45 degrees

Orientation	Wall Area (ft2)
East	490.83
North	984.89
South	984.89
West	490.83

QA/QC Notes:

OK: North and South wall areas differ by 0.0%

OK: East and West wall areas differ by 0.0%

The analysis could also be expanded to include window area and window-to-wall ratio.

Orientation	Wall Area (ft2)	Window Area (ft2)	WWR
East	490.83	80.00	16.3%
North	984.89	300.00	30.5%
South	984.89	300.00	30.5%
West	490.83	80.00	16.3%

While the analysis above is for the 4 cardinal directions, the ordinal directions (NE, NW, SE, SW) could also be included by specifying the 8 orientations +/- 22.5 degrees.

Horizontal areas (Floors, Roofs, Ceilings)

Check that horizontal areas of surfaces at different elevations align. If not, this may indicate a problem with model geometry not drawn correctly, or missing zones. The example below was generated by summing the building model's floor, ceiling, and roof areas at each elevation. The example building is a simple 2 story rectangular building. Here we can see that the areas match at each elevation, and that the ceiling and floor areas between the first and second match (as expected).

Elevation (ft)	Surface Type	Area (ft2)
0.00	Floor	4990.15
7.87	Ceiling	4990.15
7.87	Floor	4990.15
9.84	Roof	4990.15

Floor-to-floor heights

Check that the floor-to-floor heights for each building story are accurate.

In the example below, floor-to-floor heights are determined by calculating the bottom and top elevations of each zone, and listing all unique values. Note that, if the plenum (cavity space above the occupied area) is modeled as a separate thermal zone, then this would appear as a short "story."

Floor Elevation (ft)	Ceiling Elevation (ft)	Zone Height (ft)	Number of Zones	QA/QC Note
0.00	7.87	7.87	5	OK
7.87	9.84	1.96	1	Warning: Low ceiling. Is this a plenum?

Envelope thermal properties

Verify envelope component performance is as expected by checking the modeled U-factor for walls, and the U-factor, SHGC, and VLT for windows. These values are typically reported on the software output reports.

Zone inputs

Internal gains

Review inputs for interior lighting, equipment, and occupancy for each zone in the model. Verify that the areas for each zone are accurate. The internal gains can be checked agains "Standard" inputs based on energy codes or standards. Refer to the BEMcyclopedia lookup tables for internal gains as one resource for this information.

The example below shows a table report summarizing the internal gains for a simple building model in a manner that is easy to review and verify.

Zone Name	Area (ft2)	People (ft2/Person)	Lighting (W/ft2)	Elec Equipment (W/ft2)
PLENUM-1	4990.15	-	-	-
SPACE1-1	1067.35	97.03	1.48	0.98
SPACE2-1	460.00	92.00	1.48	0.99
SPACE3-1	1038.50	94.40	1.52	1.01
SPACE4-1	460.00	92.00	1.48	0.99
SPACE5-1	1964.30	98.20	1.50	1.00

Infiltration rates

There are several approaches for modeling infiltration in BEM. A suggested resource for learning more about recommended best practices is a report by PNNL.

Ventilation rates

Ventilation rates at the zone level should be compared against the building code, or standards such as ASHRAE 62.1. Read more about specifying appropriate ventilation rates: Ventilation rates - calculations and inputs for mechanical ventilation.

Operating Schedules

Plots of operational schedules for visual QA/QC check.

Review plots of the schedules used in the model by performing a visual inspection. Aim to answer these questions with your review:

• Do loads operate as expected according to building occupancy?
• Are different types of schedules operating in a way where they are roughly aligned with each other (i.e. systems turn on/off or are turned down at the same time)?

HVAC Inputs

• Air handlers - Verify system capacities, fan power, and cooling and heating efficiency match the design specifications. (Verification of sizing discussed below)
• Chilled water and hot water systems - Verify that system capacities, equipment efficiency, and loop pressure loss match the design specifications.

Review output data

QA/QC of the model outputs are performed to assist modelers in verifying that the modeled performance is aligned with how the building is expected to perform. This analysis utilizes the simulation output results, and some checks may compare output data to input data to verify alignment. Analysis is performed by parsing output results files—annual, monthly, and hourly to review the results at various levels of granularity. Graphical and tabular presentation of the output data can both be useful.

This type of analysis should be able to be applied across multiple modeling software programs that generate hourly outputs, although the names of output variables are likely to vary between software tools.

Annual outputs

Annual outputs should be reviewed to provide a "big-picture" sense of the modeled performance and identify any potential red flags before moving on to more granular review of the output data.

Annual heating and cooling energy consumption profiles

Hourly plot of annual heating and cooling energy consumption.

Review a plot of the annual heating and cooling energy consumption hourly data to answer the questions:

• Do the shapes of the energy consumption profiles make sense?
• Do you see heating energy when outdoor temperatures are low, and cooling energy when outdoor temperatures are high?

The example below shows a line plot of hourly energy consumption for heating and cooling. This example project is located in Chicago, IL where there are expected to be high heating loads in the winter, and high cooling loads in the summer. The plot shows some load for both heating and cooling during the shoulder seasons (spring, fall) and there are minimal heating loads in the summer due to VAV reheat operation. This may indicate that VAV reheat control strategies could be evaluated to minimize the need for reheat in the warmer seasons.

Boiler and chiller operation hours by load ranges

Review how many hours of the year the systems operate in different load ranges. If the system never operates near full capacity, this may indicate the systems are oversized, or perhaps the weather data is not representative of the project's climate.

The example below shows a histogram of this data for both boiler and chiller operation. The chiller loads never exceed 70% which may indicate oversizing. The boiler load shows a large number of hours in the 0-10% range which can be explained by low loads associated with VAV reheat throughout the year. This may indicate that VAV reheat control strategies could be evaluated to minimize the need for reheat in the warmer seasons.

Energy use intensity (EUI)

Review the modeled EUI and compare to benchmark data for the project's building type.

The example below shows a project's EUI, broken out by each end use, and total for the modeled building. This EUI is compared to a benchmark (in this case, using results from the PNNL prototype models, but other benchmarks can be used instead). Because the difference in modeled results differs from the benchmark by a large percentage, it is flagged as a potential issue for further review.

	EUI (kBTU/ ft² - yr)
Heating (Nat Gas)	10.45
Cooling (Elec)	3.08
Interior Lights (Elec)	15.43
Equipment (Elec)	9.06
Fans (Elec)	1.66
Pumps (Elec)	0.48
Total	40.16

QA/QC Notes:

Benchmark EUI for building type "OffS" in CZ 5A is 26.2 (source: PNNL 90.1-2019 Prototype Models)

Warning: Percent difference between project and benchmark EUI is 41.9%.

Unmet load hours

Review the number of hours heating and cooling setpoints are not met to ensure that system capacities have been sized appropriately. A small number of hours is likely not a problem, but higher numbers indicate that the HVAC capacity may not be properly sized or that there is some other issue such as a problem with the inputs for HVAC control. Most software tools produce a report of unmet load hours. Alternatively, this can be determined by inspecting the hourly results and comparing zone thermostat temperatures to the setpoints.

In the example below, this check is performed at both the zone and the whole-building level. For the whole-building calculation, if more than one zone experiences an unmet setpoint during the same hour, it will only count this as one hour in the table. Energy codes and standards such as ASHRAE 90.1 limit the number of allowable unmet load hours to 300 hours per year, so in this example a warning is raised that the limit has been exceeded.

	Heating UMLH	Cooling UMLH
PLENUM-1	0	0
SPACE1-1	285	22
SPACE2-1	186	9
SPACE3-1	260	0
SPACE4-1	218	10
SPACE5-1	270	0
Building	303	41

QA/QC Notes:

Warning: The building exceeds 300 total unmet load hours (total is 344.0 hours). This indicates that the system capacities may be undersized, or there is a problem with the system controls.

Model with Design Cooling Airflow as a relative metric (cfm-ft2)

Zone Heating & Cooling Loads with Spreadsheet Conditional Formatting

The fundamental concept underlying the UMLH test is that the ability to maintain space temperatures within the throttling range is an indication that the simulated heating and cooling equipment and zone airflow are adequately sized and appropriately controlled. If they are not, this will result in UMLHs. A useful check to perform is to access if each zone has sufficient airflow and heating/cooling capacity to address the conditioning requirement and maintain comfort conditions. Often a model-view can be used for such a task. It is recommended to use a relative metric, e.g. CFM/ft2 or l/sec-m2.

Peak demand for each end-use

Review the peak demand values to make sure their magnitudes are appropriate, and that they occur during the expected time of the year. Note that for some enduses that do not typically experience seasonal variation, e.g., lighting and plug equipment, the peaks are likely to occur on the first day of the simulation period (usually January 1).

	Peak Demand (kWh)	Peak Date/Time	QA/QC Note
Heating (Nat Gas)	27.75	2017-02-06 08:00:00	OK: Peak heating occurs in winter
Cooling (Elec)	7.37	2017-07-19 15:00:00	OK: Peak cooling occurs in summer
Interior Lights (Elec)	7.50	2017-01-01 10:00:00	OK: Peak is not seasonally dependent
Equipment (Elec)	4.50	2017-01-01 09:00:00	OK: Peak is not seasonally dependent
Fans (Elec)	0.64	2017-07-17 14:00:00	OK: Peak Fans (Elec) occurs in summer
Pumps (Elec)	0.59	2017-07-16 19:00:00	OK: Peak Pumps (Elec) occurs in summer

QA/QC Notes:

OK: Peak lighting demand is 100.0% of installed lighting power)

OK: Peak equipment demand is 90.0% of installed equipment power

Additional checks could be performed to compare the peak demand of heating and cooling to the capacities of the heating and cooling systems.

Monthly outputs

Monthly energy by end-use totals

Bar charts comparing the magnitude of monthly energy use for each end use

Monthly outputs should be reviewed to verify that seasonal variations in modeled performance are appropriate for the building design. This type of analysis can show the relative magnitude of each end-use.

The example image of bar charts show how heating and cooling energy vary depending on the season, while lighting and receptacle loads do not (minor variations due to the number of days in each month). Fan and pump energy also show seasonal variation however the magnitude is much less than for heating and cooling.

The graphical depiction is useful for "seeing" any anomalies. A tabular version of this data (see below) is also useful to understand the actual energy consumption metrics.

Month	Heating (Nat Gas) (kWh)	Cooling (Elec) (kWh)	Interior Lights (Elec) (kWh)	Equipment (Elec) (kWh)	Fans (Elec) (kWh)	Pumps (Elec) (kWh)
Jan	3626.82	1.99	1978.14	1139.61	276.29	14.97
Feb	2893.00	6.82	1729.51	1016.01	249.55	13.55
Mar	1581.97	44.22	1904.27	1122.41	279.28	12.36
Apr	693.38	249.76	1821.39	1081.21	117.08	44.61
May	337.81	491.49	1978.14	1139.61	134.14	83.65
Jun	155.29	797.35	1821.39	1081.21	131.15	110.57
Jul	123.53	1150.81	1904.27	1122.41	143.93	143.79
Aug	185.10	905.41	1978.14	1139.61	142.49	129.74
Sep	269.49	545.66	1747.51	1064.01	119.38	88.61
Oct	658.74	213.59	1978.14	1139.61	290.26	26.07
Nov	1745.23	94.10	1895.27	1098.41	269.84	19.24
Dec	3018.34	0.61	1830.39	1105.21	276.21	12.13

Compare outputs to typical benchmark values

If model results differ unexpectedly from benchmarks then it’s a good idea to spend some time investigating whether or not the difference is reasonable or is due to some inappropriate inputs. The most common benchmarks are annual whole building energy consumption per square foot, while some sources also provide energy consumption broken down by end use.

Review energy consumption end use breakdown

The fraction of total energy estimated for each end use, such as cooling, lighting or plug loads, will vary between buildings for many reasons. Therefore, there are no firm rules for judging whether or not end use fractions are reasonable. But even a quick review of end-use results can identify problems such as zero heating or cooling energy, which typically indicates a problem with the model.

Review seasonal patterns of end-use energy consumption

Plots of monthly energy consumption by end use are helpful for checking for unusual results, such as cooling in winter or heating in summer. While there may be valid reasons for these results, sometimes it indicates a problem with HVAC control inputs or some other issue.

Review HVAC system and component autosizing

Most of the HVAC system components in a simple box model will be automatically sized by the simulation program. Since sizing affects energy consumption, it is important to check the sizing results to make sure they are reasonable. Two useful approaches are to 1) compare the sizing results to rules of thumb and 2) compare the autosized capacity to the peak simulated load. Common rules of thumb can be applied to air flow (cfm/ft2), cooling capacity (ft2/ton) and heating capacity (Btu/hr-ft2). To compare sizing to simulated load, some simulation programs have standard output reports that show how many hours per year that, for example, a chiller operates in different part-load ranges. If the equipment never operates close to full capacity, that is an indication that it is oversized.

Review hourly or sub-hourly results

Most simulation programs can output hourly or sub-hourly values of energy consumption and many other results. Even a quick review of the hourly data can be useful to identify potential problems. A time series plot of hourly electricity consumption will show whether the systems are operating on the intended schedule and whether there is expected daily, weekly and seasonal variation. Plots of hourly chilled water and hot water load can help identify issues with HVAC system operation. An hourly report of outdoor ventilation air flow for an air handler can be used to verify that economizer cooling is being used and that desired ventilation rates are being used.

Additional Resources

Some of the materials to support developing, documenting, and checking model input values are listed below and have been developed by practitioners. They have not been formally vetted by the modeling community.

• Example Building Model eQUEST/DOE-2 Checklist by RMI

Description:  A checklist to support quality assurance checking of a whole-building simulation model. Some of the items are eQUEST specific. https://rmi.org/our-work/buildings/deep-retrofittools-resources

• eQUEST Quality Control Checklists by Jeff Hirsch and Associates

Description:  Outlines general quality assurance principles and suggests values to check in DOE-2 output reports.

• Office Building Benchmark Values by IBPSA-USA BEM Library

Description:  Lists typical values of key performance metrics typical for a large office building. Three categories of efficiency are considered: inefficient existing, efficient, and low-energy.

https://web.archive.org/web/20170918004209/http://bemlibrary.com/

• Building Performance Metrics by James Waltz

Description:  From Waltz’s book, Computerized Building Energy Simulation Handbook

https://rmi.org/wp-content/uploads/2017/04/Pathways-to-Zero_WaltzGuide_2010.pdf

• Recommended Metrics for QC Benchmarking by IBPSA-USA BEM Library

Description:  A list of parameters helpful for completing quality control checking of model input and output values. Practitioners can consider including them as part of their QC tools.  Developers can consider having software calculate and report these values.

• A User Guide for Understanding and Minimizing Unmet Load Hours by IES Ltd.

Description:  An in-depth user-guide explaining sources and approaches to address unmet load hours. Unmet-Load-Hours-IES-User-Guidance-Part-1-Understanding-Unmet-Load-Hours-.pdf

Links to external websites are provided as a convenience for further research, but do not imply any endorsement of the content or the operator of the external site, as detailed in BEMcyclopedia's general disclaimers.