Posts Tagged ‘simulation’

Live Fire Simulations: Key Elements of Fidelity examined some of the important elements in physical fidelity, the extent to which the simulation looks and feels real. This post will begin the process of identifying key aspects of functional fidelity, the extent to which the simulation works and reacts realistically.

Maintaining the Balance

One important factor to consider in live fire training simulation is that despite the fact that it is a training exercise, the fire is real. While I too use the terms training fire and real fire, the combustion processes and fire dynamics in live fire training are the same as encountered in a structure fire. What is different is the type, amount, and geometry of the fuel used and the ventilation profile.

Providing complete functional fidelity (as well as physical fidelity) simply requires that structural environment, fuel loading, and ventilation be the same as would be encountered in an actual incident. However, this substantially increases variability of outcome and the risk to participants.

Use of a purpose built structure allows control of variability and provides the ability for repetitive and ongoing training. Purpose-built structures are often designed to use either solid Class A fuel or gaseous Class B fuel. Facility design and selection of fuel type should be based on a wide range of factors including provision of adequate fidelity for the type of training to be conducted, environmental issues, health and safety of participants, anticipated duty cycle (i.e., frequency and duration of training activity) and life-cycle cost (e.g., initial purchase price, ongoing maintenance costs).

Figure 1. Live Fire Simulation with Gas (left) and Class A Fuel (right)

As illustrated in Figure 1, there can be obvious and substantial differences in physical fidelity when gas fired props are used to simulate a typical compartment fire. Depending on the purpose of the simulation these physical differences can be important. However, differences in functional fidelity may be even more important.

Functional Fidelity

As stated earlier, functional fidelity is the extent to which the simulation works and reacts realistically. I have tentatively identified five subsystems related to functional fidelity of live fire simulation as illustrated in Figure 2.

Figure 2. Functional Fidelity Concept Map

This concept map is a simple and preliminary look at the elements of functional fidelity. Each of the concepts illustrated can be further refined and elaborated on to provide a clearer picture of physical fidelity in live fire training.

Physiology & Personal Protective Ensemble

Live fire simulation places the participant in a hostile environment requiring the use of personal protective clothing and self-contained breathing apparatus. Functional fidelity in these subsystems and their interaction with the thermal environment may or may not be a critical element of context, depending on the intended learning outcomes of the simulation. However, insulation of firefighters from the thermal environment encountered in structural firefighting delays, modifies, and my limit perception of critical thermal cues (e.g., high temperature, changes in temperature). Overcoming this challenge requires training in a realistic thermal context.

Fire Suppression System

One of the most critical elements in functional fidelity of the fire suppression system involves interaction with fire dynamics: Does the fire and fire environment (e.g., hot gas layer) react appropriately to extinguishing agent application? In some respects there may be a conflict between the desire for physical fidelity of the fire suppression system and functional fidelity of the interaction between extinguishing agent application and fire dynamics. For example, it may be desirable for participants to use the same flow rate (providing realistic nozzle reaction) in simulations as will be used in the structural firefighting environment. However, the limited fuel load typically used to provide a safe training environment do not result in the same required flow rate for fire control as fire in a typical residential or commercial compartment. Does use of a high flow handline in live fire simulation with limited fuel load create an unrealistic expectation of the performance under actual incident conditions? Which is more effective, realistic flow rate with a limited fuel load or matching of flow rate and fuel load to provide a realistic interaction between the fire and fire attack? This question remains to be answered.

The type of nozzle used presents a simpler issue related to functional fidelity in live fire simulation. Most combination nozzles have similar operational controls for controlling the flow of water (i.e. on and off) and pattern adjustment. However, there are subtle differences such as the extent of movement needed to adjust from straight stream to wide angle fog. Flow control mechanisms vary more widely from fixed flow rate, to variable flow and automatic nozzles. Firefighters must be able to select pattern and flow as necessary based on conditions encountered in the fire environment and intended method of water application. Use of a substantively different nozzle in training than during incident operations is likely to result in less than optimal performance. However, as with the question of flow rate, the extent to which this is a concern is unknown.

Fire Dynamics

Likely the greatest concerns with regards to functional fidelity are in the area of fire dynamics. If firefighters are to learn how fires develop and the impact of changes to ventilation profile and application of extinguishing agents, the fire must behave as it would under actual incident conditions (or as close to this ideal as can be safely and practically accomplished).

Most fuels encountered in structure fires are solids (e.g., furniture, interior finish, structural materials). Class A fuel used in live fire training, often has a lower heat of combustion and heat release rate than typical fuel in the built environment, but is similar in that it must undergo pyrolysis in order to burn; providing similar (but not identical) combustion performance. Combustion of Class A fuel also results in generation of significant smoke, another similarity to typical fuels in the built environment. Class B (gas) fuel used in structure fire simulations has a high heat of combustion with heat release rate controlled through engineering and design of the burner system. However, this fuel generally burns cleanly, necessitating introduction of artificial smoke to provide higher fidelity. Flaming combustion and smoke production must be mechanically controlled by a computerized system, a human operator, or both. The nature of the fuel and design of the combustion system also impact on the fires reaction to changes in ventilation and application of extinguishing agents such as water.

Changes to ventilation will not substantively influence fire behavior if the fire is fuel controlled. However, if the fire is ventilation controlled, changes in ventilation can have a significant impact on fire behavior (which may or may not be desirable, depending on the intended learning outcomes).

With Class A fuel, water applied to cool the hot gas layer or to fuel packages has a similar impact as it would in actual structural firefighting operations. The degree of similarity is dependent on the design of the compartment in which the training is being conducted as well as fuel factors and ventilation profile. With Class B fuel, temperature sensors, computerized controls, and a human operator all must interact to ensure that water application results in appropriate changes to combustion.

System Latency

In general, live fire training simulations are conducted in real time. However, time lag due to system limitations in Class B (gas) fired props or delay in instructor or operator perception of changing conditions in either Class A or B fueled props can influence system latency, resulting in faster or slower than normal interaction between  fire dynamics and fire control subsystems.

Closing Thoughts

While I admit I am (currently) biased in favor of live fire training simulation using Class A fuel, there is no reason why other systems such as those using Class B (gas) fuel could not be designed in such a way to provide higher fidelity. However, this would likely increase both complexity and cost. I have been also discussing the potential of computer based (non-live-fire) fire training systems to develop some (but likely not all) of the skills necessary to safely operate in the structural firefighting environment. This is something else to think about!

I will come back to this topic again from time to time as I gain additional insight into the elements of physical and functional fidelity in live fire training.

Remember the Past

Last Tuesday, was the second anniversary of the deaths of Captain Mathew Burton and Engineer Scott Desmond of the Contra Costa County Fire Protection District in California.

Figure 1. 149 Michele Drive-Alpha/Delta Corner

Note: Contra Costa Fire Protection District (Firefighter Q76) Photo, Investigation Report: Michele Drive Line of Duty Deaths. This photo illustrates conditions shortly after 0159 (Q76 time of arrival).

July 21, 2007
Captain Matthew Charles Burton
Fire Engineer Scott Peter Desmond
Contra Costa County Fire Protection District, California

Captain Burton, Engineer Desmond, and another engineer were the crew of Engine 70. At 0143 hours, Engine 70 was dispatched to a residential fire alarm. As additional information was received, the incident was upgraded to a structural fire response with the addition of two engines, a quint, and a command officer.

Engine 70 arrived on the scene at 0150 hours and reported heavy fire and smoke from a small single-family residence. Firefighters reported that they had confirmed reports that two occupants of the home were still inside.

Captain Burton and Engineer Desmond advanced an attack line into the structure and flowed water on the fire. They reported that the fire had been knocked down and requested ventilation at 0155 hours. Captain Burton and Engineer Desmond exited the structure temporarily to retrieve a TIC, then re-entered the structure and went to the left toward the bedrooms with an attack line, while another crew went to the right without an attack line.

The engineer for Engine 70 placed a PPV fan at the front door. One of the civilian fire victims was located by the crew that had gone to the right; her removal was difficult and firefighters had to exit the building to ask for help. During this time, the fire inside the house advanced rapidly.

Firefighters had difficulty venting the roof due to multiple roofs, built-up roofing materials, and the type of construction.

A command officer arrived on the scene at approximately 0202 hours and began to look for Captain Burton to assume Command. The command officer tried to contact the Engine 70 crew by radio but was unsuccessful. A second alarm was requested, and a report of a missing firefighter was transmitted at approximately 0205 hours.

The fire had advanced within the structure and had to be controlled before firefighters could search for the missing crew. Captain Burton and Engineer Desmond were located and removed from the structure between 0212 and 0226 hours. The firefighters were found in a bedroom.

For additional information, see my earlier posts on this incident, the Contra Costa County Fire District Investigative Report, and National Institute for Occupational Safety and Health (NIOSH) Death in the Line of Duty Report.

Contra Costa County LODD: Part 1

Contra Costa County LODD: Part 2

Contra Costa County LODD: What Happened

Contra Costa County Fire Protection District Investigation Report: Michele Drive Line of Duty Deaths

National Institute for Occupational Safety and Health (NIOSH) Death in the Line of Duty Report F2007-28

Ed Hartin, MS, EFO, MIFireE, CFO

Several earlier posts (Training Fires Versus Real Fires, Live Fire Training: Important Questions) introduced the concepts of live fire training as simulation, physical fidelity, and functional fidelity. This post will dig a bit deeper into what aspects of fidelity may be important in live fire training.

Interesting Puzzle

Physical fidelity is the extent to which the simulation looks and feels real. Functional fidelity is the extent to which the simulation works and reacts realistically. In Live Fire Training: Important Questions, I presented a puzzle provided by Roy Reyes of the Swedish Civil Contingencies Agency. He forwarded me the following photo (Figure 1) from a fire behavior instructor course that he had conducted in Valencia, Spain and posed two questions, one quite general and the other very specific:

1. What do you see in the photo?
2. Why are the flames in the hot gas layer in the center, and not across the entire width of the compartment?

Figure 1. Participants Conducting Fire Behavior Demonstration 2

Physical and functional fidelity are potentially quite important in developing firefighters understanding of fire behavior and skill in application of fire control techniques. The two questions that Roy asks are important (and I will get back to them). However, two more fundamental questions could be asked: 1) To what extent is the fire behavior in this container based prop reflective of conditions that would be encountered in a “real” fire? 2) Does it matter (given the learning outcomes intended for this training session)?

Physical Fidelity

Physical fidelity is important in providing visual, audible, and tactile cues that are essential to developing and maintaining situational awareness. In addition, physical fidelity is a key component in firefighters’ perception of the realism of the simulation.

The concept of physical fidelity is simple. However, when you start to think about the live fire training environment, it quickly becomes more complex. The elements of fidelity related to flight simulation discussed in Live Fire Training: Important Questions as a starting point. Physical fidelity may include the firefighters’ personal protective ensemble, tools, and equipment as well as visual, audio, and thermal aspects of the environment. As illustrated in Figure 2, a number of these elements of physical fidelity are interrelated.

Figure 2. Physical Fidelity Concept Map

This concept map is a simple and preliminary look at the elements of physical fidelity. Each of the concepts illustrated can be further refined and elaborated on to provide a clearer picture of physical fidelity in live fire training.

Functional Fidelity

While physical fidelity is important, functional fidelity; realistic functioning of the simulation, is likely even more important. Development of critical skills and the ability to read the impact of tactical action is dependent on adequate functional fidelity.

As with physical fidelity, the concept is straight forward, the simulation should function in a realistic manner. However, this is likely to be even more complex than simply looking realistic.

Roy’s puzzle provides an interesting starting point to think about the nature, function, and importance of functional fidelity.

The first question asked, what do you see in the photo? Firefighters are engaged in a training session in a container based CFBT cell with a fire located in the front on the right side. A well defined hot gas layer has developed with flames extending through the hot gas layer at the center of the compartment.

The second question is more significant. Why are the flames extending in the hot gas layer in the center of the compartment and not across the full width of the compartment? There could be a number of possible explanations, but it is likely that the metal walls of the container are acting as thermal ballast. Energy used to increase the temperature of the metal compartment walls (which have excellent thermal conductivity) is not being used in the combustion process (preventing flaming combustion next to the walls). The same phenomenon can be demonstrated by placing a coil of copper wire into a candle flame. This causes a reduction in flaming combustion, and in many cases the wire absorbs sufficient energy to extinguish the flame.

So, the thermal conductivity of the container walls can at times influence the behavior of flaming combustion in CFBT cells. Does this present a problem or is it simply an opportunity to present the puzzle to the learners and engage in a discussion about thermal ballast?

A subsequent post will examine this concept in greater depth and present a preliminary concept map illustrating key dimensions of functional fidelity.

Ed Hartin, MS, EFO, MIFireE, CFO

In several recent posts (Training Fires and “Real” Fires and Live Fire Training in Purpose Built Structures, I emphasized that all live fire training is a simulation. Fidelity is the extent to which the simulation replicates reality.

Figure 1. Training in an Acquired Structure

Note: Ed Hartin Photo

The Questions

Some firefighters and fire officers subscribe to the belief that use of acquired structures with realistic fuel loading is the only way to develop the necessary competence and skills to operate safely and effectively on the fireground. However, current standards such as National Fire Protection Association (NFPA) 1403 Standard on Live Fire Training (2007) places specific constraints on fuel types and loading. Some departments are faced with environmental constraints that preclude burning Class A fuel for structural live fire training and consequently use gas fired structures (or don’t conduct live fire training at all). Most departments who have access to purpose built structures and props for structural live fire training are limited to a single type of facility (due to economic constraints). This gives rise to an interesting set of questions:

• What degree of simulation fidelity is necessary to develop the knowledge and skills necessary for safe and effective operation on the fireground?
• What are the key elements of fidelity for various learning outcomes such as 1) developing understanding of fire development in a compartment, 2) dynamic risk assessment, inclusive of recognizing critical fire behavior indicators, 3) selecting appropriate fire control techniques, 4) developing competence and confidence when operating in a hazardous environment, 5) developing skill in nozzle operation and technique, 6) evaluating the effect of tactical operations.
• Is live fire training the only or most effective simulation method for achieving these learning outcomes? If so, what type of simulation will safely provide the required degree of fidelity? If not, what other simulation method may be used in place of, or in addition to live fire training to provide the required degree of fidelity?

I believe that effective performance under stressful conditions requires substantial training in a realistic context. However, the answers to the preceding questions have not yet been determined. What we have is a great deal of strongly held opinion without supporting discipline or task specific evidence.

Dimensions of Fidelity

As discussed in Training Fires and “Real” Fires, fidelity can be examined in a number of different ways, but one simple approach is to consider physical and functional characteristics of the simulation. Physical fidelity is the extent to which the simulation looks and feels real. Functional fidelity is the extent to which the simulation works and reacts realistically.

Figure 2. Two-Dimensional Fidelity Matrix

However, this simple model provides limited guidance when examining questions related to live fire training. Here it is necessary to consider: What are the key elements of physical and functional fidelity necessary to support the specific learning outcomes intended from a given training evolution?

In A Handbook of Flight Simulation Fidelity Requirements for Human Factors Research, Rehman (1995) describes three purposes of aircraft flight simulation: 1) provide practice on specific skills, 2) reinforce acquisition and use of job-relevant knowledge, or 3) to evaluate a system or new concept. The fidelity requirements for each of these three purposes may be quite different. In addition, fidelity applies to the simulator itself, the participants, and related or events external to the simulator. In a flight simulator, each subsystem of the simulator (e.g., cockpit layout, audio, motion) has specific fidelity characteristics that must be considered as illustrated in Figure 3.

Figure 3. Flight Simulator Subsystem Fidelity Characteristics

Note: Adapted from A Handbook of Flight Simulation Fidelity Requirements for Human Factors Research.

How might these concepts be applied to evaluating fidelity requirements for live fire training? Determining the answers to the questions posed in this post will require a significant research effort (and related funding). However, the first step in this process is to clarify, refine, and tightly focus the questions that this research needs to answer.

My next post will examine this interesting topic a bit further.

An Interesting Puzzle

Closely related to the topic of simulation fidelity, I was provided with an interesting puzzle by my friend Roy Reyes of the Swedish Civil Contingencies Agency. He forwarded me the following photo (Figure 4) from a fire behavior instructor course that he had conducted in Valencia, Spain. His first question was what do you see in the photo?

Figure 4. Participants Conducting Fire Behavior Demonstration 2

Note: Roy Reyes Photo

The second question is a bit more specific, why are the flames in the hot gas layer in the center, and not across the entire width of the compartment?

The answer to this question provides an important learning opportunity related to how simulator and simulation design impact on fidelity and the importance of the instructor in establishing context.

I will come back to these questions in my next post!

Ed Hartin, MS, EFO, MIFIreE, CFO

References

National Fire Protection Association (NFPA). (2007) NFPA 1403 Standard on Live Fire Training Evolutions. Quincy, MA: Author.

Rehman, A. (1995). A handbook of flight simulation fidelity requirements for human factors research, Report Number DOT/FAA/CT-TN95/46. Retrieved July 6, 2009 from http://ntl.bts.gov/lib/000/800/858/tn95_46.pdf

The theme for the 2009 meeting Institution of Fire Engineers (IFE) Compartment Firefighting Special Interest Group (SIG) in Sydney, Australia was Finding the Common Ground. The 15 participants represented 12 fire service organizations from Australia, New Zealand, Sweden, the UK, Spain, Croatia, China, Canada, and the United States.

Figure 1. 2009 IFE Compartment Firefighting SIG Participants

Understanding & Application

The dominant common theme identified by the participants is the need for firefighters and fire officers to have a solid understanding of fire dynamics and the ability to apply that knowledge in an operational context. Achieving this goal cannot be accomplished simply by delivering a course or training program, it requires a fundamental shift in perspective and ongoing effort to support individual and organizational learning.

Simply achieving knowledge of fire dynamics and skill in task and tactical activity is necessary but not sufficient. Achieving increased safety and effectiveness requires that firefighters and fire officers effectively apply this knowledge on the fireground. Facilitating this transfer from training to operational context is a challenge is a significant challenge.

Dr. Stefan Svensson of the Swedish Civil Contingencies Agency posed the question: How do we get learners to understand the differences between training fires and “real fires”. This is an interesting question in that training conducted in a container, burn building, or acquired structure is in fact a “real fire”, but has considerably different characteristics than a fire occurring in a house, apartment, or commercial building. Improperly designed training may provide the learner with an inaccurate perspective on the fire environment which can lead to disastrous consequences. The challenge is managing risk while developing a realistic understanding of fire behavior.

What is the Difference?

Compartment fires in the training environment differ from those encountered during emergency operations differ on the basis of compartment characteristics, fuel, ventilation profile, heat release rate, and time scale. In addition to differences related to fire dynamics, firefighters and fire officers also encounter psychological stress resulting from a sense of urgency, organizational and community expectations (particularly in situations where persons are reported to be trapped in the building).

Other than acquired buildings, structures used for fire training are generally designed and built for repetitive use and not for regular human habitation. Structural characteristics that make a durable live fire training facility are considerably different than most if not all other structures in the built environment. Density, thermal conductivity, and specific heat of training structures can be considerably different than a dwelling or commercial structure, which has a significant impact on fire behavior.

The ventilation profile of a purpose built prop or burn building is also likely to have significantly different compartmentation and ventilation profile than a typical residential or commercial structure. Live fire training facilities often (but not always) are designed with burn compartments. This speeds fire development and minimizes both initial and ongoing cost. However, fire behavior and the impact of fire control tactics can be considerably different in a large area and/or high ceiling compartment. Many modern structures are designed with open floor plans that are challenging to duplicate in the training environment. Energy efficient structures limit ventilation (air exchange), while training structures are often quite leaky, particularly after extensive use. This can have a significant influence on development of a ventilation controlled burning regime and influence of ventilation on the concentration of gas phase fuel in smoke. Failure of glass windows in ordinary structures should be anticipated, as this changes the ventilation profile and resulting fire behavior. Training structures on the other hand provide a more consistent ventilation profile as durable (e.g., metal) windows do not present the same potential for failure.

While structural characteristics, compartmentation, and ventilation differ between typical structures in the built environment and those used for live fire training, one of the most significant differences lies in the types, quantity, and configuration of fuel.

National Fire Protection Association (NFPA) 1403 Standard on Live Fire Training is fairly explicit regarding fuel characteristics and loading for live fire training evolutions. Most of these provisions can be tied directly to incidents in which participants in live fire training exercises lost their lives. Unfortunately, there are not the same provisions in fire and building codes. Fuel load is considerably higher in most residential and commercial occupancies than is typically used in live fire training, even in advanced tactical evolutions.

Together these differences provide considerably different fire dynamics between the training and operational environments. How much and in what ways does this impact on the effectiveness of compartment fire behavior training (CFBT)?

Fidelity

As discussed, CFBT, even when conducted in an acquired structure does not completely replicate fire conditions encountered in an operational context. All CFBT involves simulation. The extent to which a simulation reflects reality is referred to as fidelity:

The degree to which a model or simulation reproduces the state and behavior of a real world object or the perception of a real world object, feature, condition, or chosen standard in a measurable or perceivable manner; a measure of the realism of a model or simulation; faithfulness… 2. The methods, metrics, and descriptions of models or simulations used to compare those models or simulations to their real world referents or to other simulations in such terms as accuracy, scope, resolution, level of detail, level of abstraction and repeatability. (Northam, n.d.)

CFBT can involve a wide range of simulations, from the use of photos and video, non-fire exercises, small scale props such as doll’s houses, single and multi-compartment props, and burn buildings, and acquired structures. Each provides differing degrees of fidelity.

Fidelity can be described in a number of different ways. One fairly simple approach is to examine physical and functional fidelity (see Figure 2). Physical fidelity is the extent to which the simulation looks and feels real. Functional fidelity is based on the extent to which the simulation works and reacts realistically.

Figure 2. Two-Dimensional Fidelity Matrix

Note: Adapted from Fidelity Versus Cost and its Effect on Modeling & Simulation (Duncan, 2007)

While describing fidelity of a simulation as low, moderate, or high, this is likely to be inadequate. A more useful description of fidelity includes both qualitative and quantitative measures on multiple dimensions. But what measures and what dimension? In a compartment firefighting simulation, key elements of physical fidelity will likely include fire behavior indicators such as Building, Smoke, Air Track, Heat, and Flame (B-SAHF). Important aspects of physical fidelity would include the characteristics of doors and windows (e.g., opening mechanism), hose and nozzles, and influence of tactics such as gas and surface cooling on fire behavior.

On the surface it makes sense that increased fidelity would result in increased effectiveness and transfer of knowledge and skill. However, it is important to remember that “All models are wrong, but some models are useful” (Box & Draper, 1987, p. 424). The importance the various aspects of fidelity depend on the intended learning outcome of the simulation. In fact, a simulation that focuses on critical contextual elements may be more effective than one that more fully replicates reality.

Figure 3. Door Entry Drill

For example, teaching the mechanics and sequence of door entry procedures (see Figure 3) might be more effectively accomplished using a standard door without smoke and flame than under more realistic live fire conditions. On the other hand, reading fire behavior indicators at the door and effectively predicting interior conditions is likely to require substantively different elements of context. However, at this point, we simply have unsupported opinion and in some cases anecdotal evidence of the effectiveness or lack of effectiveness of current training practices. The key to this puzzle is to clearly define the intended learning outcomes and identify the critical elements of context that are required.

Questions Remain

The IFE Compartment Firefighting SIG identified the need for a greater emphasis on fire behavior training at all levels (e.g., entry level firefighters, incumbent firefighters, and fire officer) as well as ongoing professional development and skills maintenance. However, a number of interesting questions remain, including:

• What are the most effective methods of developing firefighters understanding of compartment fire behavior?
• What is necessary to effectively facilitate transfer of this knowledge from training to the operational context?
• What level of fidelity is necessary in live fire training do develop and maintain critical skills?
• How can technological simulation (computer or video based) simulation be used to augment live fire training to maintain proficiency?
• To what extent might non-live fire simulation (e.g., CFBT for the Wii) be used to develop compartment firefighting competencies?

Professor David Morgan of Portland State University observes that “A successful research project requires two things: Meaningful research questions and appropriate means to answer those questions” (Morgan, 2005, p. 1-2). One of the greatest potential benefits resulting from collaboration between members of the IFE Compartment Firefighting SIG is the integration of the skills of academics and practitioners, scientists and firefighters. During the 2009 workshop, SIG member Steve Kerber from Underwriters Laboratory (formerly with the National Institute for Standards and Technology) emphasized the importance of scientists and engineers doing research with, not simply for the fire service. This has the potential to not only identify meaningful questions, but also to provide the knowledge and skills necessary to answer them.

Ed Hartin, MS, EFO, MIFireE, CFO

References

Northam, G. (n.d.). Simulation fidelity – Getting in touch with reality. Retrieved May 2, 2009 from http://www.siaa.asn.au/get/2395365095.pdf

Box, G. & Draper, N. (1987). Empirical model-building and response surfaces. San Francisco: Wiley.

Duncan, J. (2007). Fidelity versus cost and its effect on modeling & simulation. Paper presented at Twelfth International Command and Control Research and Technology Symposium (12^th ICCRTS), 19-21 June 2007, Newport, RI.

Morgan, D. (2005). Introduction [to integrated methods] (Unpublished Manuscript). Portland, OR: Portland State University.