This paper reports on research that asked chief officers from Australia’s fire and emergency services what they identify as a measure of success. When identifying appropriate measures of success, community members need to consider and acknowledge multiple, sometimes competing issues. Accordingly this research cannot give a definitive answer to ‘what is the measure of success?’ but it is argued that emergency services, political leaders and at-risk communities need to engage in a more meaningful discussion about what can realistically be expected from each other. The outcomes of those discussions have to move past the rhetoric that ‘this should never happen again’ and need to be reflected in the policy and legislative goals that instruct emergency managers and in the ongoing communication about risk and responsibility for managing risk.
In his review of the 2011 Perth Hills bushfires former Australian Federal Police Commissioner Mick Keelty (2011, p. 3) said:
‘There remains one question the answer to which eluded the Special Inquiry but it is an answer that requires further examination and that is: What is the measure of success of the outcome of a bushfire? Is the loss of no lives the only performance measure? If so, how many houses is an acceptable number to lose?’
In a review of the response to the 2013 Tasmania Fires, former South Australian Police Commissioner Malcolm Hyde (2013, p 198) said:
‘How do you judge success in emergency response operations? This was a question considered in the Special Inquiry into the Perth Hills Bushfire 2011, and it was not able to be answered… This conundrum is not lost on chief fire officers. At an Executive Forum this year, chief fire officers and commissioners considered this very issue, noting there were different ways to measure success; they too were unable to answer the question.’
Keelty said that finding the answer to this question requires further examination. To contribute to that examination a sample of senior officers from Australia’s fire and emergency services were canvassed on what they used, or could use, as their measures of success. The research does not attempt to, and cannot, give a definitive answer to ‘what is (or should be) the measure of success?’, rather it reports the views of the sample, and presents a series of arguments to inform further research and discussion.
Current Australian policy calls for responsibility for natural hazards is to be ‘shared’ (COAG 2011). In order to share responsibility governments and its agencies need an articulated view of its role and how to define ‘success’. Determining the policy and management objectives for fire and emergency management is complex, messy, and political but if emergency services, and those who lead them, wish to avoid being judged by unknown, vague or conflicting criteria, they need to engage in discussions between themselves, their staff, their community, and their political leaders to explain what they see as success and failure. In order to start that discussion, researchers from the Australian National University (funded by the Bushfire Cooperative Research Centre) asked a sample of chief officers from Australian fire and emergency services organisations what they considered to be appropriate measures of success.
Semi-structured interviews of 30-120 minutes were conducted with chief officers who attended the 2011 Australasian Fire and Emergency Services Authorities Council (AFAC) Command Forum, and who agreed to take part. These officers were either the Commissioner of the agency, who was both chief executive and principal operations officer, or, in those agencies where operational responsibility was separate from the administrative role, officers charged with managing response operations. These included operations officers, chief fire officers, and fire control officers. Regardless of the formal title, all participants are referred to here by the term ‘chief officer’.
There were 36 chief officers at the 2011 AFAC Command Forum representing 27 separate fire and emergency services agencies from each Australian jurisdiction as well as New Zealand. Interviews were conducted with 18 officers (50 per cent), representing 16 agencies (60 per cent), and seven of the nine (including New Zealand) jurisdictions (seven per cent).
The interviews were recorded and transcribed except in two cases where technical failures led to reliance on manual transcriptions made during the interview. The responses were analysed to identify what the chief officers saw as the measure of success when responding to an emergency and, in particular, a catastrophic event such as the Black Saturday bushfires in 2009. The research was approved by the Australian National University Human Research Ethics Committee (ANU 2011). Although all participants were asked common stimulus questions, only representative quotes are set out in this paper. The interviews were given with a commitment to anonymity and research participants are not disclosed. A discussion paper reporting the larger research project and including quotations and inferences drawn from the interviews was circulated to a wider officials group, including the interviewees, to communicate results and verify the analysis as valid. The conclusions reported here were presented to the 2013 Sydney AFAC Command Forum to inform their discussion on identifying measures of success.
Before reporting the chief officers’ views, some background will help identify why this issue is important to emergency managers. Following major natural hazard events such as the Victorian Black Saturday bushfires in 2009, the Queensland and Victorian floods in 2011, the Perth Hills bushfires in 2011, and the Tasmanian fires in 2013, Australia has used formal, complex, post-event inquiries to identify how the tragedy occurred and what can be done to prevent a similar occurrence in the future. (The authors will report, elsewhere, on research into the current carriage of such inquiries and possible future alternatives (Eburn & Dovers, forthcoming)). A problem facing post-event inquiries is identifying the standard by which emergency services are to be judged: that is, what does success actually look like? (Keelty 2011, Hyde 2013). Neither the Commonwealth, nor the states and territories, have a clear statement on what emergency management policy is meant to achieve. They fail to state either the policy objective or how achievement of that objective will be monitored and evaluated. For example the objectives of emergency management are described in legislation as:
The Emergency Management Act 2013 (Vic) has been passed by the Victorian Parliament but, at the time of writing, is not yet in force. Once commenced, the Act will convey the express objective of fostering ‘a sustainable and efficient emergency management system that minimises the likelihood, effect and consequences of emergencies’. The statutory objective, in those terms, recognises that there are limitations in emergency management (hence the goal to ‘minimise’ rather than avoid, the impact of emergencies) and trade offs (hence the need to ensure arrangements are both sustainable and efficient).
Statements that agencies ‘can take appropriate and timely action to prevent or mitigate, respond to and recover from emergencies’ (Emergency Services Commissioner 2009, p. 1.5) or that an agency is required to take ‘… all necessary steps for the prevention and suppression of fires and for the protection of life and property in case of fire’ (Country Fire Authority Act 1958 (Vic) s 20) imply that such control, action or necessary steps can, in fact, be taken (Interviewee #14); but this is not always the case.
Policy objectives that refer to ‘effective’ or ‘adequate’ measures are unhelpful as they are devoid of meaning. A goal to have ‘effective’ or ‘adequate’ measures begs the question of ‘effective or adequate for what purpose?’ The fire and emergency services are ‘adequate’ for most events; events that are ‘routine’ and even rare but ‘normal’ ‘non-routine’ events, but not for ‘complex unbounded’ events (Handmer & Dovers 2013). In the event of overwhelming events, the resources and response of emergency services will always be ‘inadequate’ even if they save many, but not all, lives.
Objectives ‘to protect and preserve life’ or to ‘control’ or ‘prevent’ the impact of an event are also unhelpful, as they imply that all lives can be protected or control can be exercised. If a life is lost or the fire or hazard is not controlled, prevented or supressed, then there has been failure regardless of what is saved and preserved. It follows that the current range of policy objectives is not necessarily helpful in either guiding action or informing post-event evaluation and does not provide an answer to the critical question – ‘What is the measure of success of the outcome of a bushfire (or other natural hazard)?’
In light of Keelty’s question, the chief officers interviewed were asked to explain the measures they applied, or thought should be applied, to determine whether the response to an emergency, and in particular a bushfire, had been a success. The chief officers nominated a number of possible measures, which are reported under three themes each having its own limitations.
The majority of chief officers believed that response is measured by what is lost, rather than what is saved.
Theme 1: Measure what is saved rather than what is lost |
|---|
… if you save 500 houses and you lose 10, should you be satisfied? Yes, on one hand you would say, well, you know there was 510 houses that could have been lost. We only lost 10. (Interviewee #3) |
… if you look at the extent of the impact and how many people could have potentially died, and how many people were in that area and didn’t die, and you got it right down to 173 out of say potentially 10,000 people. That to me is probably quite successful. You know, people see 173, they don’t see how many people were in the area and affected and impacted. (Interviewee #8) |
… the media’s sensationalised the fire fighting and they never really … look at all the houses that didn’t burn down … No, it’s all about doom and gloom. The guy going through all his possessions that are burnt … create a bit of controversy over that… that’s their measure. (Interviewee #11) |
… what’s always reported is the losses rather than the saves … ultimately we’re judged usually by a couple of hours on a Saturday afternoon … Where all your prevention and preparation works essentially counts for naught. (Interviewee #12) |
Generally most events, we measure success in a negative context … it’s about measure of loss. How many houses were lost, or how many people died... there’s got to be some measure there that relates efforts to things that have been saved, so people and houses. (Interviewee #16) |
The difficulty with this measure is identifying what is saved. If a house does not burn, or better yet, if effective hazard reduction activities coupled with pro-active policing means no fire happens, even on a day of catastrophic fire weather conditions, then it’s hard to claim credit for the absence of fire. In 2012 the South Australian power provider, ETSA, disconnected the power on a day of extreme fire danger. There was criticism of their actions (ABC 2012); in particular Broome and Smith (2012) argued that the risk to human health and safety from disconnecting the power exceeded the reduced risk of death and injury from bushfire. If a post-event review had found there had been a spike in the number of deaths due to heat related effects that may have been avoided had the electricity supplied been maintained and air conditioners operated, then it would be possible to say ETSA’s decision ‘caused’ those extra deaths. But it is impossible to say how many people did not die from the bushfire that did not, and may never have, happened.
Further, to put the issue in harsh terms 173 people died in 2009 during the Black Saturday fires but that may well have been a successful outcome if the objective had been to ‘minimise’ the number of deaths. The agency response may have saved many more people than died but it is unacceptable to stand before a community, particularly one that was affected by fire, and claim that the loss of 173 lives was evidence of successful firefighting and community engagement, or even a reasonable outcome when measured against realistic expectations or possible outcomes. Again, the counter-factual of deaths avoided is difficult, if not impossible, to establish.
Another suggested measure of success, is to ask whether the agency and staff did all, or the best, that they could? Did they stick to the plan and meet their objectives?
Theme 2: We stuck to the plan and did our best |
|---|
… our Commissioner would be satisfied if we could demonstrate that we have followed our procedures and we had done everything that - you know, we ticked the boxes... (Interviewee #3) |
The expectation is … everyone’s done the best job they can with the resources they have and the knowledge and skill sets that they have. (Interviewee #5) |
So, to me, the measure of success from an agency perspective is… can we tick all the boxes and say we did everything possible…. (Interviewee #6) |
If you set the objectives for the operation based on the context of what’s unfolding and you meet those objectives that’s probably a reasonable measure of success… If [the objective was to] … minimise the loss of life and you did, so if you lost four out of 5000, that’s minimised it. |
… what were your objectives? Were your objectives met? Now, I think that is the only measure you can come up with and if you say my objective was to save the life of the people living here, and if that objective was achieved then I would say yes, it was a success because that was what you were intending to do and if you achieved it then you have succeeded. (Interviewee #9) |
Objectives are unlikely to be met in every event as circumstances may overwhelm resources and because the fire or other event may not behave as expected. Decisions are made in a dynamic environment that is information poor. A decision may be the best decision given the information available but it does not follow that the outcome will be as expected or that the objective will be achieved or that, in the circumstances, the response was a failure.
Further, adhering to the plan can also lead to an unsuccessful outcome. A Scottish Sherriff (the equivalent of the Australian Coroner) criticised incident controllers at a rescue scene because ‘they rigidly stood by their operational guidelines’ (Leslie 2011) and delayed a rescue with the effect that a trapped person died from the complications caused by the delay. The Sheriff said that the Incident Controller considered that the operation was a success because he struck rigidly to the fire service policy, which required him to withdraw his officers and wait for the arrival of the Police Mountain Rescue Squad. According to the Sheriff however, ‘this was not a successful operation: a woman died who had not only sustained survivable though life threatening injuries, but who had also ultimately suffered and died from acute hypothermia...’ (Leslie 2011). Relying on pre-defined guidelines and procedures, sticking to the plan and focussing on pre-set objectives, leads to a situation where managers are ‘damned if they do; and damned if they don’t’, with the assessment depending on the outcome rather than the processes followed.
An argument that success is achieved when ‘everyone’s done the best job they can’ is also doomed to fail, at least in the face of media and political scrutiny and often in inquiries. After catastrophic events there are post-event inquiries and each one will find areas of breakdown in communications, resources, or decision-making. Each inquiry is able to point to examples where, with hindsight, someone did not do ‘the best’ that they could have done (Schapel u.d, Doogan 2006, Hope u.d).
It should be noted that courts of law, unlike the media and, arguably post-event inquiries, can be much more sympathetic to the circumstances of an emergency. In deciding questions of legal liability the courts have recognised that an emergency warrants prompt action that, in hindsight, may not have been the best decision. It has been said that a rescuer, ‘acting under the pressure of emergency, is to be judged leniently as to the reasonableness of his conduct’ (Wallis v Town of Albany (1989) Aust Torts Reports 80-283 , ¶69,011) and that:
[A] man is not to be charged with negligence if he … finds himself faced with a situation which requires immediate action of some sort and if, in the so called “agony of the moment”, he makes an error of judgment and takes a step which wiser counsels and more careful thought would have suggested was unwise.
(Leishman v Thomas (1958) 75 WN(NSW) 173 , p. 175)
Finally:
The law appreciates that a rescuer may act – and may feel impelled to act – under the pressures of the moment, where delay may be considered vital to the safety of those he is considering protecting from risk. It is not appropriate to subject a rescuer’s actions, or his subjective view of the risks involved to himself and/or to others, to fine scrutiny in the court room.
(Tolley v Carr [2010] EWHC 2191, ¶22-¶23).
As the Canberra bushfires in 2003 show, a post-event inquiry into a catastrophic event may be very critical but the court findings may not be the same. After that event the ACT Coroner made adverse comments about the then Minister and the performance of three senior officers in the Emergency Services Authority (Doogan 2006), but the litigation over those fires settled with a verdict in favour of the Territory Government (Andrews & Doherty 2012).
A consistent view of interviewees was that no responder deaths was a measure of success. Even so, some officers recognised that the community may expect that emergency responders will put themselves at risk to help others.
Theme 3: No responder deaths |
|---|
I suspect that a community would think [if no civilians died, but a fire fighter died] well, they’re fire fighters; they’re like front line troops. They are putting themselves in danger to tragedy; that fire fighters died; but that’s one of the risks of doing the job that they do. (Interviewee #1) |
Our success rates in fire fighter safety are very high. So that has to be an indicator…. The aspirational goal is no loss of life, but not at the cost of more lives. … [A fire fighter death] will always scar that operation. It’s no longer successful because there’s been a fire fighter death. … the fire fighter death brings it right down. It brings it right down because … fire fighters are meant to be trained to avoid all of that. To be calculated and risk savvy … So when they die something has clearly gone wrong; clearly gone wrong… (Interviewee #8) |
Look I think the community are accepting [of fire fighter deaths] ….the community will probably say well, you choose to do that and you do accept the risk that you might die doing it. I think they’d perhaps probably say that to fireys. But I don’t think they’d be keen for us to say it back the other way. You choose to live on the side of a hill with trees all around you - you’ve got to accept the risk that you’re going to die. I don’t think there’s too many people would really agree with that. (Interviewee #10) |
This measure is supported by modern health and safety legislation. This was highlighted in the UK by the (ultimately unsuccessful) criminal prosecution of incident controllers who responded to a warehouse fire where four firefighters died (Ellicott 2011, Hayes 2011). In Australia, uniform workplace, health and safety laws require a ‘person conducting a business or undertaking’ to ‘so far as is reasonably practicable’ ensure the health and safety of workers (for example, Work Health and Safety Act 2011 (NSW) s 19(1)). While chief officers, incident controllers, and first responders must consider the interests of individuals who are at risk from fire or other hazard, the law is clear that the primary duty is to firefighter safety.
Although the chief officers saw firefighter safety as a measure of success, they were pragmatic that the community may not share that view, expecting firefighters and emergency workers to put themselves at risk to protect others. A recent UK Coroner said, when delivering a critical review of the actions of two paramedics who refused to enter a water filled ditch to try to rescue a trapped driver, ‘I was brought up in a country where men risked their own lives to save the lives of others. That was a period in our history which has almost ceased’ (Robinson 2013).
There is no assumption that there should, or could, be a single measure of success when reviewing the response to a natural hazard. What constitutes a success is contested, and each potential measure is not without its difficulties. Different stakeholders may all want and expect different outcomes depending on their role and responsibilities, and each event is dynamic with considerable uncertainty. The problems of measuring success were identified, even in this initial exploration, and what represents a success will vary with each stakeholder’s position. This may not be news to those engaged in the broader study of politics and political science but may be news to the chief officers and their staff who find themselves subject to regular criticism after each event, as the following sample of recent Australian newspaper headlines shows:
This study has gathered the views of the leaders of a substantial sample of Australia and New Zealand fire and emergency services. They represent one perspective. Communities, media commentators, social and environmental researchers, governments, courts and others may have different views. It is argued that the measures of success proposed by the chief officers who took part in these interviews may also be flawed and therefore ineffective or unacceptable as measures of success. What this shows is that just as the post-event inquiry does not have clear measures of success by which to judge the preparation for, and response to, a major event (Keelty 2011, Hyde 2013), neither do the officers charged with leading the response to the emergency.
This discussion has not answered Keelty’s question and does not identify what are reasonable measures of success, rather it reveals the absence of clear measures of success. Research per se cannot identify ‘the’ measures of success, they are not waiting to be discovered; rather they need to be negotiated between stakeholders. Current Australian policy calls for responsibility for natural hazards is to be ‘shared’ (COAG 2011). In order to share responsibility it is vital that governments and its agencies have an articulated view of what it and the fire and emergency services organisations would consider a successful outcome. There are no clear measures of success and the suggested measures identified by chief officers are themselves problematic and identify critical policy gaps. It falls on agencies and their political leaders to engage with stakeholders to identify and explain what they see as success and failure. If communities and individuals better understand what they can expect from emergency services organisations, they can make a more informed judgement on what they need to do to protect themselves.
This research was stimulated by Commissioner Keelty’s question ‘What is the measure of success of the outcome of a bushfire?’ As Commissioner Keelty noted, this question ‘requires further examination’ and this research forms part of that further examination. Identifying the views of chief officers is important as they lead their agencies and their views and intentions affect the operational decisions at the front line. The major finding from this research is that there are no clear measures of success and that the suggested measures identified by chief officers are themselves problematic and are unlikely to stand up to detailed scrutiny in the next post-event inquiry. It has been argued that identifying some measures of success is essential in order to inform those at risk as to what they may expect from emergency service agencies and what they must do for themselves; and to give those agencies at least some starting point for evaluating performance after the next significant event. The chief officer views can form the starting point of a discussion within and between agencies and the community. In the longer term, realistic statements identifying negotiated measures of success and acknowledging the necessary trade-offs and tensions, could be expressed in policy statements and legislative materials, and communicated more widely.
The authors acknowledge the contribution of the Bushfire Cooperative Research Centre that funded this research and the officers who took part in the interviews.
ABC 2012, ‘ETSA defends decision to cut power’. At: www.abc.net.au/news/2012-01-03/etsa-defends-power-cut-decision/3756354 [3 January 2012].
Andrews L & Doherty M 2012, ‘Fire litigation ends for ACT’ The Canberra Times (online). At: www.canberratimes.com.au/act-news/fire-litigation-ends-for-act-20120920-268d2.html [20 September 2012].
ANU 2011, Australian National University Human Research Ethics Committee, Human Ethics Protocol 2011/480, 26 September 2011.
Broome R & Smith W 2012, ‘The definite health risks from cutting power outweigh possible bushfire prevention benefits’ 197(8) Medical Journal of Australia pp. 440-441.
Council of Australian Governments (COAG) 2011, National Strategy for Disaster Resilience, Commonwealth of Australia, Canberra.
Doogan M 2006, ‘The Canberra Firestorm: Inquests and Inquiry into Four Deaths and Four Fires between 8 and 18 January 2003’, ACT Coroners Court, 2006.
Eburn M & Dovers S (Forthcoming) ‘Learning lessons from disasters: alternatives to Royal Commissions and other quasi-judicial inquiries’ Australian Journal of Public Administration (accepted for publication).
Ellicott C 2011, ‘Three brigade bosses to be charged with manslaughter over deaths of four firefighters in warehouse blaze’. At: www.dailymail.co.uk/news/article-1361454/Fire-brigade-bosses-charged-manslaughter-warehouse-blaze.html.
Handmer J & Dovers J 2013, ‘Handbook of Disaster Policies and Institutions: Improving Emergency Management and Climate Adaptation’ (2nd edition). Earthscan-Routledge, London.
Hayes S 2011, ‘Atherstone fire charges could be dropped’ Lemington Observer (Online). At: http://leamingtonobserver.co.uk/2014/02/12/news-Atherstone-fire-deaths-caused-by-major-organisational-failings-96070.html.
Hope AN (u.d) ‘Inquest into the deaths of Trevor George Murley, Lewis Kenneth Bedford and Robert Wayne Taylor’ (Western Australia Coroners Court, undated).
Hyde M 2013, ‘2013 Tasmanian Bushfires Inquiry: Volume One’ (State of Tasmania: Hobart, Australia).
Keelty M 2011, ‘A Shared Responsibility: The Report of the Perth Hills Bushfire February 2011 Review’ (Government of Western Australia, Perth).
Leslie DJ 2011, ‘Fatal Accident Inquiry under the Fatal Accidents and Sudden Deaths Inquiry (Scotland) Act 1976 into the Death of Allison Hume’ [2011] FAI 51. At: www.scotcourts.gov.uk/opinions/2011FAI51.html [16 November 2011].
Office of the Emergency Services Commissioner 2009, ‘Emergency Management Manual Victoria; Part I Emergency Management in Victoria’ (Government of Victoria, Melbourne).
Robinson M ‘Coroner condemns paramedics who refused to save drowning man in ditch ‘for health and safety reasons’ MailOnline. At: www.dailymail.co.uk/news/article-2368415/Coroner-condemns-paramedics-refused-save-drowning-man-ditch-health-safety-reasons.html#ixzz2b4ERinRp) [18 July 2013].
Schapel AE (u.d.) ‘Inquest Into The Deaths Of Star Ellen Borlase, Jack Morley Borlase, Helen Kald Castle, Judith Maud Griffith, Jody Maria Kay, Graham Joseph Russell, Zoe Russell-Kay, Trent Alan Murnane and Neil George Richardson’ (Coroners Court of South Australia undated).
Teague B 2009, ‘Victorian Bushfires Royal Commission, Chairman’s Opening Remarks’. At: www.royalcommission.vic.gov.au/getdoc/ee8eec7c-da6c-4c97-9ec8-3553d705f102/Opening-Remarks---Chairman.
The Age 1939, ‘Commission on Bush Fires’.
Dr Michael Eburn is an Associate Professor in the ANU College of Law and visiting fellow at the Fenner School of Environment and Society. He is the Chief Investigator on a Bushfire and Natural Hazards CRC funded research project on ‘Policies, institutions and governance of natural hazards’.
Professor Steve Dovers FASSA is Director of the Fenner School of Environment and Society, Australian National University, and a Lead Researcher with the Bushfire and Natural Hazards CRC.
The latest editions from the Australian Emergency Management Handbook and Manual Series are now available online at
https://ema.infoservices.com.au/collections/handbook.
Electronic copies can be downloaded at no cost, and hard copies can be purchased.
cover-1_fmt.png)
cover-1_fmt.png)