Skip to main content
Skip to secondary navigation
Menu
Australian Government - Office of the Australian Information Commissioner - Home

From annual reports to open data

Professor John McMillan, Australian Information Commissioner, Address to the Institute of Public Administration Australia (ACT), Annual Reports Award Dinner, Canberra, 23 May 2013

During my academic career I was an avid collector of glossy annual reports that adorned bookshelves and supported impressive footnote references; for a few years I served on one of the IPAA judging panels for the annual report awards; and over the last ten years in multiple statutory roles that require the completion of annual reports to Parliament I've been closely involved in preparing about thirty annual reports. Each year I count down to a time when I will never have to write another Foreword!

Many of you will have had a disquieting experience in preparing an annual report. The most recent for me is that the first annual report of the Office of the Australian Information Commissioner was to be built around one our three corporate colours — orange — but came back from the printer in luminescent watermelon. You won't find that version on your bookshelves!

The keenest lesson I have learnt is that however much time you devote to crafting and perfecting your corporate reflections on the previous year's work and challenges ahead, some colleagues and friends will be more interested in counting the number of times you are photographed in the report. A scorecard comparing you to the most photographed agency heads will soon arrive. (Though none of us will ever outclass the Irish Ombudsman whose annual report one year contained a glamorous shot of her, her husband and their five children.)

Another analytical pastime, ably assisted by web technology, is to check whether a colleague's annual report photograph reveals the same stress lines and age profile displayed in the colleague's latest Google pic.

I'll leave it to the IPAA to decide whether to widen the criteria for annual report awards based on those observations.

The intrinsic (as opposed to the entertainment) value of an annual report lies in the information it conveys about the activity, objectives and performance of an agency. Annual reports bolster government transparency and accountability. They provide a valuable contemporary as well as historical record of the government of the nation. Importantly this record becomes publicly available to the community, nationally and globally.

Annual reports to parliament were one of the earlier and recognised mechanisms for providing public access to the record of government. In 1976 the Report of the Royal Commission on Australian Government Administration recommended that departmental reports become the practice of government, 'to ensure that departments would make available for public information and discussion, material which would not otherwise be released'. With a quaint eye to the future, the Commission foresaw that annual reports might contain 'discussion of issues of real interest and concern', reveal 'some of the innovative thinking' in departments, and 'help to breakdown the prevalent image of stuffiness and remoteness'.[1]

Barely six years later, the Royal Commission's views on public access were overtaken by a more radical development, the commencement of the Freedom of Information Act 1982 (a development supported only in a minority report of the Commission). The FOI Act was a radical reform in two ways: it gave members of the public a legal right to specify which government documents they wanted to read; and it specified criteria and procedures for externally and independently resolving disputed public access decisions. Thereupon ended a tradition of secrecy, a presumption of confidentiality, and the absolute discretion of government to decide what information to release.

The fundamental importance of the FOI Act is unchanged. Tens of thousands of information access requests are made to agencies each year; FOI releases figure prominently and frequently in media reports and on agency disclosure logs; and FOI exemption criteria strongly influence all debate on drawing the line between government transparency and secrecy.

The relevance of the FOI Act was confirmed in 2010 when it was substantially reformed. The reforms included a new objects clause declaring a modern philosophy for government ('information held by Government is to be managed for public purposes, and is a national resource'[2]); a new Information Publication Scheme that supplements annual reports by requiring web publication of accurate and up-to-date information on agency structures, functions, personnel, reports, consultation arrangements and the like; and the creation of my own office, the Office of the Australian Information Commissioner (OAIC), to promote open government, safeguard the privacy of personal information, and advise government on information policy.

In other respects, however, the public access scheme modelled in the FOI Act is being overtaken by technology. Technology and digitisation have changed the way government records information, stores information, shares information, provides access upon request, publishes information, places conditions on public access, and deals with threats to the security and confidentiality of information.

Technology also underpins new concepts that capture this cultural change. We now speak of public sector information, big data, cloud computing, open data, e-government, connected government, digital government and online engagement. As the Gov 2.0 Taskforce observed in 2009, 'Open public sector information is…an invitation to the public to engage, innovate and create new public value'.[3]

This sentiment has been echoed by world leaders. The UK Cabinet Office Minister, Francis Maude, has characterised open data as 'the raw material of the new industrial revolution'.[4] In the same vein US Secretary of State Hillary Clinton observed that 'In the 21st century...one of the most significant divisions among nations will not be north/south, east/west, religious, or any other category so much as whether they are open or closed societies...[C]ountries with open governments, open economies, and open societies will increasingly flourish.'[5]

This explains why governments around the world are fundamentally changing the way that information is valued, managed, used and shared with others. Through the web we are enjoying an unparalleled release on open access terms, of information and raw data relating to economic performance, educational achievements, health outcomes, spatial patterns, weather forecasts, government contracting, government spending, consumer choices, crime patterns, traffic movements, political donations, agricultural activity, flood mapping, and licensing approvals. We no longer need to turn to hard-copy government reports for that information.

Every revolution penetrates to the core of society, and the digital age, no less, throws up a host of technical, policy and cultural challenges. One way of teasing those out is to contrast the assumptions and practices that characterise annual reports, with those that must guide an open government culture of open information, open data and open dialogue.

First, a stand-out feature of annual reports is that they are perfected publications. The record of government contained in a report is carefully drafted, reviewed and refined before it is publicly released. Understandably, government agencies take care to present a message which they feel is accurate and balanced. The same care is taken with assembling data for graphs, tables and performance measures. Understandably, agencies wish to guard against the obvious risk of data being misconstrued, misunderstood and perhaps misrepresented. Government may be blamed for any mistakes or misapprehensions.

This cultural attitude is regarded by some as a significant impediment to an open data culture. Preparing faultless data may mean it is never released. Earlier this month the Shakespeare Review of public sector information practices in Britain called for 'a clear, visible, auditable plan for publishing data as quickly as possible, defined by bottom-up market demand and top-down strategic thinking'.[6] The Review recommended a twin-track release process — Track 1 would be an 'early even if imperfect track'; while Track 2 would be a National Core Reference Data highly-quality track. Information would move from Track 1 to Track 2 through government, industry, academic and community consultation.

Secondly, annual reports are based on a large amount of information and data that may be assembled for that singular purpose, of preparing an annual report. Once that task is completed some of the information may be consigned to the physical or electronic filing cabinet.

In an open data culture, raw data and information must be discoverable, accessible and useable by the community. Principle 1 of the Principles on open public sector information published by my office states that open access to public sector information must be the default position in government. Working from the premise that 'Information held by Australian Government agencies is a valuable national resource…[a]gencies should use information technology to disseminate public sector information, applying a presumption of openness and adopting a proactive publication stance'.

Inherent in that proposition is that people should have a choice as to how they use government information. Importantly, they should be able to use it for a different purpose to that of the agency that first collected or assembled the information. Indeed, the true value of information can be realised only if others can access it, build upon it and perhaps derive another purpose.

By way of example, my office collects statistics from roughly 250 government agencies on how they process FOI requests. We publish those statistics in an annual report, as required by the Australian Information Commissioner Act 2010 ss 30, 31. That is the sole purpose for which we collect the statistics, and we have no other use for them. But others may. So we make that raw data available on the data.gov web portal, and leave it to others to explore whether there is a different use or purpose. For example, a media organisation may wish to draw its own table of agency FOI performance; or a commercial firm may explore where openings exist to provide assistance or training to agencies.

We can multiply that example across government:

  • Should the unrefined survey information collected by the Australian Public Service Commission as part of the Annual State of the Service survey be publicly accessible in an anonymised form so that researchers can explore other dimensions of Australian government?
  • Should the case file information collected by the Australian Taxation Office or Centrelink be accessible also in an anonymised form so as to enable local government councils, business entities and community groups to reorganise the information by suburb or postcode to explore planning challenges, business opportunities or the need for new community services or recreational facilities?
  • Or, to highlight one of the more controversial examples in government, should the NAPLAN data published on the My School website be made available in a raw or database form to support different lines of inquiry into Australian educational patterns?[7]

Those are not rhetorical questions with a hidden 'yes' answer. They are complex, multi-faceted questions that are also politically dynamic. Making raw or unstructured data publicly accessible, downloadable and useable on open licensing terms raises difficult questions. What if people misconstrue the data or undermine the purpose for which it was collected? Will those who contributed the data object or refuse to supply it is made generally available? Is it fair that data collected by government at great expense can support a commercial profit-making enterprise that contributed nothing to the collection process?

Those questions raise difficult policy and cultural challenges, but they cannot be sidestepped by choosing to suppress valuable government data. They are questions we must address if we accept the FOI declaration that information held by government is a valuable national resource to be managed for public purposes.

There will be increasing pressure on Australian Government agencies to address those questions. Yesterday the Attorney-General announced that Australia will join 57 other countries as part of a global Open Government Partnership.[8] One of the commitments in the Open Government Declaration that OGP participants are required to implement is 'to proactively provide high-value information, including raw data, in a timely manner, in formats that the public can easily locate, understand and use, and in formats that facilitate reuse'.[9]

Thirdly, another feature of annual reports is that they are carefully and sensitively prepared so as not to reveal sensitive and confidential information. I have faced this challenge in preparing Commonwealth Ombudsman and OAIC annual reports, which contain illustrative case studies of the personal difficulties that people face in dealing with government. A case study must be meaningful without breaching the privacy of those who seek assistance from the office. An 'anonymised' case study may not conceal very much if excessive detail is given, as illustrated by the following two case studies taken from the annual report of an Ombudsman of another country:

Mr JM, a Forensic Laboratory Technician of the Chemistry/Toxicology Section at the Forensic Science Laboratory, wrote to the Senior Chief Executive Office of the Prime Minister's Office to complain about the rejection by the Officer in Charge of the Laboratory of his application for leave without pay to complete his final studies at the London School of Law during the period 28 March to 29 September 2007.

Widow PR, aged 64, is the owner of a plot of land of 10,679 sq metres at Highlands. She wanted to develop it but it was outside the limits of permitted development. She had so many problems in her life, financial and otherwise, that her life was completely shattered.

It is becoming increasingly difficult to anonymise or confidentialise data in a world of big data, data analytics and the world wide web. Some fear this may be an insurmountable task and that no mechanism will be fool-proof in the face of a determined attempt to penetrate de-identified data. An illustrative (though different) anecdote that bears out this difficulty concerns the uncovering of the secret relationship between CIA director David Petraeus and author Paula Broadwell. It is said that she took extensive precautions to hide her identity, never contacting him through her anonymous home e-mail service, and only using hotel and other public networks. Her identity was uncovered after the FBI correlated hotel registration data from several different hotels and found that hers was the common name![10]

Here again we face a challenge, but one that cannot be met by unreasonably clamping down on the sharing and distribution of government information. To quote again the recent Shakespeare Review in Britain, 'we will miss out on the enormous benefits of PSI' if we persist with a current unrealistic degree of expectation of any data controller to perfectly protect all data — an attitude that inhibits innovation'.[11]

The OAIC has taken the same balanced approach in a recent guideline containing practical advice on 'De-identification of data and information'.[12] The guideline explains that de-identifying information can maximise the utility and value of an agency's information assets without compromising privacy or confidentiality.

Fourthly, we can be reasonably certain that published, hard-copy annual reports become a permanent record of government. Copies can likely be found in departmental, university or public libraries, and increasingly on the web.

But there is a fresh danger. Many government reports are now born digital and made available only on the website of the agency that created them. As former Parliamentary Librarian and current ANU Librarian Roxanne Missingham has recently observed, online resources are much more fragile than hard-copy publications: 'When machinery of government changes occur, older material is particularly vulnerable to being 'cleaned out' and deleted. When government changes this risk is heightened even further.' [13]

Missingham's study of one agency's publications that were published online in 2001 revealed that by September 2012 — 28% are still available on the agency website, 24% on the National Library Pandora website, 19% on another website (eg, a university), and 29% are no longer available online. As she concludes, it may be that nearly a third are now 'digital dust'.

A similar incident encountered by the OAIC is that Australian Government Gazettes from a particular period were unavailable for a couple of months when the responsibility for making them available moved from one agency to another. Initially there was no notice on the website of either agency to explain this transfer or to advise on interim access arrangements. A particular worry in this incident is that Gazettes are one of the basic documents that record government transactions.

With a national election soon to be held in Australia, the OAIC has taken up with government the importance of ensuring that the manual on machinery of government changes contains appropriate guidance on preserving the records of government following any changes.

May I finish by commending of the IPAA and the participants in tonight's ceremony for celebrating the importance of annual reports. Completing and publishing an annual report is a highlight of the corporate year. The elation at completing this task was shared with me by a colleague, the Pakistani Insurance Ombudsman, in a covering letter conveying his first annual report: 'A matter of pleasure indeed it is for me to present to you our maiden Annual Report sprouted from the office founded just a year back'. Two years later he wrote, 'And now our third issue in a row knocks at your door to greet you'.

Well, it is now time to knock at the door of the 2013 award ceremony for Australian Government annual reports. I look forward to congratulating the winners.

Footnotes

[1] Report of the Royal Commission on Australian Government Administration (AGPS, 1976) 75.

[2] Freedom of Information Act 1982 s 3(3).

[3] Government 2.0 Taskforce, Engage: Getting on with Government 2.0 (2009) iii.

[4] Open data is the raw material of 'new industrial revolution' <www.guardian.co.uk/public-leaders-network/blog/2012/mar/19/open-data-raw-material-industrial>

[5] Secretary of State Hillary Rodham Clinton, 'Remarks at the Open Government Partnership Opening Session' (speech delivered at the conference of the 'Open Government Partnership', Brazil, 17 April 2012) <www.state.gov/secretary/rm/2012/04/188008.htm>

[6] See Shakespeare Review, An Independent Review of Public Sector Information (May 2013) 9.

[7] See Diamond and Australian Curriculum, Assessment and Reporting Authority [2013] AICmr 57.

[8] Attorney-General for Australia, 'Australia joins Open Government Partnership' (22 May 2013),

[9] See <www.opengovpartnership.org>

[10] Bruce Schneier, 'The Internet is a Surveillance State' <www.schneier.com/essay-418.html>

[11] Shakespeare Review, An Independent Review of Public Sector Information (May 2013) 15.

[12] See Information Policy Agency Resource 1 — De-identification of data and information - consultation draft April 2013 <www.oaic.gov.au/news/consultations/de-identification_guides/draft_de-identification_agency_resource.html>

[13] Roxanne Missingham, 'Government publishing – some issues and trends' (2012, unpublished paper – but see It's a grey, grey world: disappearing government information <http://greylitstrategies.info/presentations/its-grey-grey-world-disappearing-government-information>