AGD consultation paper – Use of automated decision making by government

Introduction

1. The Office of the Australian Information Commissioner (OAIC) welcomes the opportunity to comment on the Attorney General Department’s Consultation Paper on the Use of automated decision-making by government.
2. The OAIC is an independent Commonwealth regulator, established to bring together three functions: privacy functions (protecting the privacy of individuals under the Privacy Act 1988 (Cth) (Privacy Act) and other legislation), freedom of information functions (access to information held by the Commonwealth Government in accordance with the Freedom of Information Act 1982 (Cth)) (FOI Act), and information management functions (as set out in the Australian Information Commissioner Act 2010 (Cth)).
3. The Consultation Paper seeks views on transparency measures and safeguards that should be put in place to protect the rights of individuals under a new framework for automated decision-making (ADM) in government.
4. The context and purpose of ADM will also give rise to risks that may be latent for example a data breach^[1] and revealed only when an individual seeks to assert their right to access information or review an administrative decision. The resultant risks and harms can be mitigated by preserving the principles of good administrative decision-making - proper authorisation; appropriate procedures; appropriate assessment and adequate documentation.
5. The OAIC is highly supportive of the development of a consistent legal framework for the use of ADM. As the Consultation Paper recognises, while properly designed ADM systems can lead to increased efficiency and consistency in decision-making, there are also a number of risks which if not managed appropriately, have the potential to significantly disadvantage individuals and communities and undermine public trust and confidence in government. These include a loss of transparency in government decision-making as well as potential privacy risks resulting from the mishandling of personal information in relation to ADM systems. The increased use of emerging technologies such as Artificial Intelligence (AI) in connection with ADM systems has the potential to further amplify these risks.
6. A framework for the use of ADM must ensure that it considers and mitigates these risks and the potential impacts that they may have on individuals and the community. Additionally, given that many ADM systems are outsourced, the framework should also address the obligations and responsibilities of government agencies with respect to the use of contracted service providers, to ensure that the transparency and accountability of government decision-making is preserved.
7. This submission highlights the intersection of ADM reforms with key areas within the OAIC’s remit, particularly information access and privacy rights. It is important that a legal framework for ADM is underpinned by strong and consistent transparency and accountability obligations and safeguards, as well as effective regulatory oversight. The reforms should also align with existing protections and obligations in the FOI Act and Privacy Act, as well as with pending Privacy Act Review reforms^[2] and the Australian Government’s proposed mandatory guardrails for high-risk AI.
8. The OAIC supports the development of a clear and practical legal framework that assists government agencies to use ADM safely and consistently. The framework should apply broadly to the use of ADM systems both in support of decision-making processes and to fully automate government decisions. While the OAIC supports an approach which balances the risks and benefits of ADM, we note that there are potential risks that will need to be carefully considered and mitigated in the new framework, including the risk of rapid unmonitored technological advances and unintended or unpredictable outcomes. This submission endorses the principles of trust and transparency and makes a number of recommendations to AGD to enhance the proposals set out in the Consultation Paper.

Recommendation 1 . A legal framework for the use of ADM in government should take a broad approach to defining ADM that captures the use of ADM systems to both make decisions and support the decision-making process.

Recommendation 2 Introduce an express obligation for agencies to proactively publish information about the circumstances when ADM is in use, including an adequate description of the type of system in use, its purpose and how it operates (including the types of decisions for which it is used), the legislative basis for the decision and/or any policy relied upon, as well as the assurance processes in place to ensure that the system is being used lawfully.

Recommendation 3 . Ensure that a consistent legal framework applies to the release of information regarding ADM systems in government, by clarifying the right of access to information under the FOI Act (s 6C).

Recommendation 4. The Government consider publishing procurement advice for agencies that clarifies the information release and privacy obligations and expectations for contractors involved in the provision of ADM systems.

Recommendation 5 . Ensure the legal framework for the use of ADM aligns with obligations under the Privacy Act, including amendments in the Privacy and Other Legislation Amendment Act 2024 relating to the use of computer programs to make decisions.

Recommendation 6 . Ensure that consistent transparency obligations apply to the use of ADM whether or not it involves the handling of personal information.

Recommendation 7. Ensure the framework includes clear obligations for agencies to, upon request, provide a meaningful explanation of the operation of the ADM system that is sufficient to enable an individual to exercise their review rights.

Recommendation 8 . Consider establishing registers for the use of ADM and AI systems by Commonwealth agencies.

Recommendation 9 . Consider the development of a recording standard which guides agencies on providing clear, accessible and standardised information about the ADM systems they use to support a centralised register maintained by government to monitor in real time system deployment, use and notification of unintended or adverse consequences.

Recommendation 10 . To avoid regulatory duplication, consider how the safeguards in the framework may interact or overlap with existing Privacy Act and FOI Act obligations in respect of personal information, and how the introduction of further risk assessment obligations in respect of the use of ADM by government may align with the existing obligation for agencies to undertake a PIA for all high privacy risk projects.

Recommendation 11 . Consider obligations for agencies to identify and record the information input into an automated system, the provenance and accuracy of the inputs, and the impact of an automated system’s outputs on government decisions, as well as the procedures in place to ensure the accuracy of their outputs. Also consider obligations for agencies to proactively release information regarding data inputs relating to the use of ADM, in addition to proactive publication obligations discussed at Recommendation 2 .

Recommendation 12 . The Government consider opportunities to explore the use of ADM to improve timeliness and consistency in the processing of FOI requests by government agencies.

Importance of trust

9. The potential efficiency and opportunity benefits of ADM technology in government will only be fully enabled if the risks are appropriately mitigated and efforts are taken to build trust. In particular, transparency in the use of ADM is essential to promoting public trust and confidence in government decision-making, particularly given the significant impact that such decisions can have on the rights and interests of individuals, and the potential for ADM and other machine technology to produce large-scale systemic injustices.^[3]
10. Automated systems vary in complexity and function, but are likely to increasingly involve the use of emerging technologies such as AI, posing particular transparency challenges. There are also incidental powers exercised as precursors or inputs to the ultimate exercise of powers and functions by government agencies. These incidental inputs may be supported by ADM yet remain opaque to the technology employed to produce the ultimate outcome that may be captured under a new regulatory approach.
11. Additionally, the ‘black box’ problem posed by some AI technologies means that even AI developers may be unable to fully explain how a system came to generate an output.^[4] The opacity of these systems can pose significant challenges for government agencies seeking to provide a meaningful explanation about how automated decisions are made. Without a clear understanding of the way that ADM has been used in relation to a decision, it is also difficult for individuals to understand or exercise their right to seek review of the decision. As set out below, while the FOI Act contains proactive information publication obligations which will likely extend to information regarding the use of ADM systems in government, the OAIC considers additional steps may be needed to ensure that agencies consistently publish meaningful and detailed information about how they use these technologies.
12. There are also a number of privacy risks associated with the use of ADM which involves the handling of personal information, particularly when ADM systems use AI. The OAIC’s 2023 Australian Community Attitudes to Privacy Survey (ACAPS) showed that 89% of Australians believe they should have the right to know when their personal information is used in ADM if it could affect them, and only 21% were comfortable with government agencies using AI to make decisions about them.^[5]
13. The complexity of ADM systems can also make it difficult for individuals to identify when their personal information is used in the systems and to exercise their privacy and FOI rights, such as requesting the correction of or access to this information. There are also potential issues in relation to the accuracy of the output of such systems, which have the potential to replicate biases from their source data with discriminatory effects.^[6] It is therefore essential that these risks are appropriately protected within a legal framework for the use of ADM in government.
14. In light of these significant challenges, a legal framework for the use of ADM in government should take a broad approach to defining ADM. While ADM systems can be used to fully automate decision-making processes with minimal or no human intervention, they can also be used to support the decision-making process, such as by closing off options for a decision-maker, or providing analysis or prediction, such as risk scores.^[7] It is important that the use of ADM systems to contribute to government decision-making, even where a human is present at some points in the decision-making process or is responsible for making a final ‘decision’, is subject to transparency obligations and safeguards to ensure that the public can understand the basis on which decisions are made that affect them.

Transparency measures

15. The OAIC is broadly supportive of the transparency measures canvassed in the Consultation Paper, including proposed requirements for agencies to publish information about their use of ADM and make business rules and algorithms available for independent expert scrutiny.^[8] As the Consultation Paper recognises, there are existing transparency obligations under both the FOI Act and Privacy Act which apply to government use of ADM and which have the potential to overlap with any new measures. We therefore encourage the Department to carefully consider the scope and operation of these existing obligations, as well as pending reforms to the Privacy Act, to ensure that a new framework for ADM is underpinned by robust and consistent transparency measures while also preventing unnecessary regulatory overlap. Strengthening extant legislated rights and protections will provide certainty and clarity for both regulated entities and the community.  As recognised in the Consultation Paper, the reforms should also align with the Government’s proposed mandatory guardrails for the use of AI in high-risk settings.^[9]

FOI Act obligations

16. The FOI Act plays an important role in facilitating transparency of government decision-making, including in relation to the use of ADM. The FOI Act enshrines the right of access to information under Art. 19 of the Universal Declaration of Human Rights.
17. The objects of the FOI Act include requiring agencies to publish information; providing a right of access to documents; and increasing scrutiny, discussion, comment and review of the Government’s activities. These objects are essential to building and maintaining the openness, responsiveness and integrity of Australian Government agencies, which the OECD has found to be key drivers of trust in public institutions. When the public is more confident that government institutions act in the best interests of society and are responsive to feedback and evidence, they are more likely to have a higher degree of trust in the government.^[10]
18. The Information Publication Scheme (IPS) under Part II of the FOI Act requires agencies to publish certain information online. The IPS is an important element of the FOI framework which ensures that information held by Australian Government agencies is managed for public purposes and treated as a national resource, and facilitates public access to information promptly and at the lowest reasonable cost.^[11] The IPS requires the publication of (amongst other things) ‘operational information’, which is information that assists the agency to exercise its functions or powers in making decisions or recommendations that affect members of the public.^[12]
19. Operational information is likely to include information related to the use of ADM systems, including the rules and procedures that agencies apply in using ADM to make decisions that affect individuals. The publication of operational information ensures that individuals are not disadvantaged by a lack of awareness of the information used by agencies to make decisions.^[13] The OAIC considers that this could nonetheless be enhanced by an express obligation for agencies to proactively publish information about the circumstances when ADM is in use, to enhance community awareness of these systems. This should include an adequate description of the type of system in use, its purpose and how it operates (including the types of decisions for which it is used), as well as the assurance processes in place to ensure that the system is being used lawfully.
20. Additionally, the right of access to information provided for in the FOI Act extends to a range of documents relating to the use of ADM, including policy-related documents and those related to the administrative decision-making process. However, as highlighted below there are potential gaps in the information that may be captured by FOI Act obligations in the context of government use of ADM, which should be considered in establishing the ADM framework.

Outsourcing considerations

21. In particular, the OAIC encourages the Department to ensure that consistent transparency obligations and safeguards apply to the use of ADM systems in government including where aspects of the system may be outsourced. Currently, the FOI Act requires agencies to take contractual measures to ensure that they receive documents held by certain contractors or subcontractors if a person requests access to those documents.^[14] However, this requirement applies only to documents which relate to the performance of a ‘Commonwealth contract’, which is defined, relevantly, as a contract under which services are provided by another party, for or on behalf of an agency, to a person who is not the Commonwealth or an agency.^[15] In circumstances in which a contractor is providing automation in relation to elements of a decision (such as a risk rating that will be input to the decision), there is a risk that this arrangement may not fall within the definition of ‘Commonwealth contract’ and may therefore fall outside the scope of the FOI Act.
22. The OAIC notes that the Privacy Act takes a different approach to defining ‘Commonwealth contract’ – this is a contract, to which the Commonwealth or an agency is or was a party, under which services are to be, or were to be, provided to an agency.^[16] Under the Privacy Act, government agencies entering into a Commonwealth contract are required to take contractual measures to ensure that contracted service providers (including subcontractors) do not breach the APPs.^[17]
23. A consistent legal framework for the proactive and reactive release of information relating to the use of ADM in government, which applies regardless of whether aspects of the ADM system have been outsourced, would provide greater certainty for the community and government agencies regarding information access. This could be supplemented by the use of standard contractual requirements and the publication of procurement advice that clarifies the information release and privacy obligations and expectations for contractors involved in the provision of ADM systems.
24. Addressing the distinction between ‘services’ and decisions or decision inputs provided under government contracts will also inject certainty in the preservation of community rights, elevate industry standards and ensure consistency in the granting and management of government contracts.
25. Monitoring and reporting obligations, such as requirements for contracted service providers to monitor and review outputs, preserve access to ADM systems, maintain audit logs and notify of adverse incidents, will also ensure that agencies are made aware of any issues affecting the operation and outputs of outsourced ADM systems and help to mitigate the potential risks of unintended or unpredictable decision outcomes.
26. As discussed in the Consultation Paper, the FOI Act also contains exemptions for the protection of information that is commercially valuable or would disclose trade secrets.^[18] This may potentially restrict access to information such as business rules, algorithms or source codes developed and used in relation to a particular automated system.
27. It is therefore important to ensure that the protection of commercially sensitive information does not restrict an individual’s access to important information about the way that ADM systems are used in government decision-making that affect them.

Privacy Act obligations

28. Agencies also have transparency and notice obligations under the Privacy Act and Australian Privacy Principles (APPs) when their use of ADM involves the handling of personal information, which may interact with the transparency and notification safeguards discussed in the Consultation Paper. In particular:
    1. APP 1 requires agencies to take reasonable steps to implement practices, procedures and systems to ensure they comply with the APPs, and to have a clearly expressed and up-to-date Privacy Policy. APP 1 is intended to ensure that APP entities manage personal information in an open and transparent way, enhancing organisational accountability and building community trust and confidence in an entity’s information handling practices.^[19]
    2. APP 5 requires agencies that collect personal information about an individual to take reasonable steps either to notify the individual of certain matters or to ensure the individual is aware of those matters. This includes the purposes for which information is being collected and the agency’s usual disclosures of personal information of the kind being collected.
29. The OAIC notes there is interaction between Privacy Act protections and the FOI Act, which provides individuals with the right to apply to government agencies to access their personal information and request amendments or annotations to their personal records, further bolstering the protections for personal information.
30. The Privacy and Other Legislation Amendment Act 2024, which passed both Houses of Parliament on 29 November 2024, will amend APP 1 to require APP entities to ensure their Privacy Policies contain specified information regarding the use of computer programs to make decisions which could reasonably be expected to significantly affect the rights or interests of an individual. Privacy Policies must include information about the kinds of personal information used in the operation of computer programs and the kinds of decisions either made solely by the operation of such programs or for which a thing, substantially and directly related to making the decision, is done by the operation of such programs.^[20]
31. The amendment was proposed as part of the Privacy Act Review, which also recommended the introduction of a right for individuals to request meaningful information about how substantially automated decisions with legal or similarly significant effect are made.^[21] The Final Report of the Privacy Act Review recognised that providing individuals with meaningful information on automated decisions with legal or similarly significant effect ensures that individuals have sufficient understanding about the rationale for automated decisions to enable them to exercise other rights, both under privacy law as well as other frameworks such as administrative or discrimination law.^[22] It is important that a legal framework for the use of ADM include clear obligations for agencies to, upon request, provide a meaningful explanation of the operation of the ADM system that is sufficient to enable an individual to exercise their review rights.
32. A consistent framework for the use of ADM should align with both existing Privacy Act obligations as well as with these forthcoming amendments. Additionally, given that Privacy Act obligations apply only where personal information is used in the operation of the ADM system, we encourage the Department to consider any inconsistencies in relation to the obligations that apply to the use of ADM that does not involve personal information. ADM systems may be used across a wide range of settings including the adjudication of rights and entitlements, resource allocation and planning, enabling compliance and supporting public service operations,^[23] and with varying types of information, from sensitive information such as health information or criminal record information to non-personal information such as aggregated data or operational information.  To ensure community trust in government decision-making, it is important that strong transparency obligations apply to the use of ADM even in circumstances where personal information is not used in the ADM system.

ADM register

33. In addition to the measures discussed above, we note that a NSW parliamentary committee examining the use of AI has recently recommended that the NSW Government consider maintaining a publicly available register of ADM systems available within government agencies and when they are applied.^[24] The Committee noted the importance of ensuring the availability and use of ADM systems by government is monitored to ensure they are treated appropriately and used responsibly.^[25] This approach has been taken in other jurisdictions, including Scotland which has established an AI Register to provide information on the AI systems in use or in development within the Scottish public sector.^[26]
34. The model operating within the Scottish public sector adopts a public facing approach. A centralised repository that provides government with real time notification of use by which agency under what conditions and that serves as an adverse incident notification system may also enhance governments’ ability to manage ADM systems.
35. The OAIC considers that establishing a register for the use of ADM and AI by Commonwealth agencies would complement the safeguards discussed in the Consultation Paper and strengthen regulatory oversight by improving awareness of the types of ADM technologies in use, how they are being designed and the way that they are being used (including whether they are used to support or replace human decision-making), and assist with the identification of unintended harms which may be associated with particular technologies or use cases. Greater transparency through the use of an ADM register can also promote knowledge-sharing across agencies in relation to the different types of ADM systems, appropriate use cases and the associated risks and benefits.^[27]
36. A register could be accompanied by the development of a recording standard which guides agencies on providing clear, accessible and standardised information about the ADM systems they use. This approach has been taken by the UK Government, which has developed an Algorithmic Transparency Recording Standard to support public sector bodies to publish information about the algorithmic tools they use in decision-making processes that affect members of the public.^[28] An ADM recording standard would further support agencies to provide appropriate transparency in their use of ADM and promote enhanced government accountability.

Other safeguards

37. The Consultation Paper considers a range of further safeguards to accompany a legal framework for ADM, including safeguards which would apply during the pre-implementation phase, as well as system-level and decision-level safeguards.
38. The OAIC strongly supports the establishment of robust safeguards both prior to the implementation of an ADM system and throughout its lifecycle. We note that to ensure the effectiveness of a legal framework for ADM, it is important that the reforms are supported by mature information governance frameworks that provide for consistent storage and retrieval of information together with systems to assure data provenance. The OAIC supports safeguards discussed in the Consultation Paper regarding audit and compliance mechanisms, including the need for good record-keeping practices.^[29] This could include obligations for agencies to identify and record the information input into an automated system and the impact of an automated system’s outputs on government decisions.
39. The introduction of stewardship as an APS value has confirmed the obligations of the APS in its custodianship of government information and recognition of its strategic importance as a national resource. This newly confirmed alignment gives rise to expansion of related obligations such as the creation of, and access to information together with sound governance throughout the life cycle of government information. These obligations transcend all manifestations of government information and inspire community expectations.
40. Whole of government consideration of the impact of ADM in the context of an application under the FOI Act is required to both preserve rights and obligations but also to meet community expectations. The administrative burden of searching for information in response to an application for information relevant to the use of ADM will be relieved through ensuring the provenance of information inputs, the adoption of conventions and the proactive release of information regarding data inputs together with the use of ADM.
41. The Consultation Paper also discusses the potential for the framework to clarify the duty of agencies to take reasonable steps to ensure automated decisions meet administrative law standards.^[30] The OAIC supports this suggestion as an important means of ensuring that the design and implementation of ADM systems in government is informed by and preserves administrative law principles, and that government decision-making which relies on automated systems remains properly authorised by law.
42. We also note that some of the safeguards discussed in the Consultation Paper are likely to interact with existing Privacy Act and FOI Act obligations for agencies. For example, the paper discusses the introduction of a risk assessment requirement as a possible pre-implementation safeguard, to allow the risks of an ADM system to be considered, identified and mitigated prior to deployment.^[31] Such an obligation would likely overlap with an existing obligation for agencies to conduct a privacy impact assessment (PIA) for all high privacy risk projects.^[32] A PIA is intended to assist agencies to identify the impact of a project (such as a new use of ADM) on the privacy of individuals and identify ways to manage, minimise or eliminate those impacts.^[33] We encourage the Department to consider how the introduction of further risk assessment obligations may align with the existing PIA requirements to avoid regulatory duplication.
43. The Consultation Paper also canvasses mechanisms to ensure that a particular automated system is appropriately governed and operated and that individual decisions are fair, lawful and accurate. This includes, for example, requirements to update the ADM system throughout its lifecycle and pathways for human intervention to ensure meaningful human oversight of the outputs of ADM systems and correction of inaccuracies.
44. The OAIC supports the inclusion of these safeguards and considers that establishing appropriate human oversight of the outputs of automated systems will generally be a critical means of ensuring that agencies comply with their accuracy obligations under APP 10 when using ADM, as well as their obligations when responding to requests for correction or amendment of personal information under the FOI Act or APP 13.^[34]
45. APP 10 requires agencies to take reasonable steps to ensure that the personal information they use and disclose is, having regard to the purpose of the use or disclosure, accurate, up-to-date, complete and relevant. The reasonable steps that an agency should take for the purposes of APP 10 will depend on circumstances including the sensitivity of the personal information and the possible adverse consequences for an individual if the quality of personal information is not ensured.^[35] Given the significant potential accuracy risks associated with ADM systems, particularly those which use AI, it is important that agencies seeking to use automated systems in decision-making carefully consider whether they can do so in way that complies with their privacy obligations in respect of accuracy.^[36]

Use of ADM to support FOI processes

46. Finally, the OAIC recognises that the use of ADM in government has the potential to deliver benefits by reducing administrative burdens and increasing efficiency of certain government processes. We consider that there may be opportunities to explore the use of ADM to assist with the processing of FOI requests by government agencies, with a view to improving the performance of agencies in terms of timeliness and consistency of responding to such requests.
47. The OAIC suggests that this could be pursued as a whole-of-government initiative rather than on an agency-by-agency basis to ensure a consistent approach which is supported by strong safeguards. Subject to appropriate resourcing, the OAIC would be well placed to take a leading role in progressing such an initiative.