Emerging Issues Brief: Gen AI use and the public service

As Generative A.I (GenAI) technologies are increasingly adopted within public service organisations, including within police jurisdictions, some anticipated risks have materialised while unforeseen and novel ones have also emerged. Key among the latter are employee ‘shadow’ uses of GenAI, third-party disclosure of sensitive information to commercial GenAI platforms, and third-party production of work containing undisclosed GenAI inputs. This Emerging Issues Brief (EIB) discusses each threat and offers some strategic considerations for police organisations seeking to respond.  

Anticipated and realised threats 

As Generative A.I (GenAI) technologies are increasingly adopted by public service organisations, several anticipated risks have already been realised. Employee disclosure of private information to commercial GenAI platforms is one such risk. For example, in 2024 the Victorian Information Commissioner (OVIC) investigated the disclosure of highly sensitive information to ChatGPT by a Child Protection worker. The investigation found that the employee had used ChatGPT to draft a Protection Application Report for a child sex offence case, causing both the disclosure and the inclusion of several significant inaccuracies in the resulting report. To address these risks, many public service organisations, including police, have implemented guidelines and policies governing the use of GenAI.

Unforeseen and novel threats

Yet unforeseen and novel challenges posed by GenAI are emerging with increased frequency. This is a likely outcome of rapid advancements in the technology, with its adoption also outpacing governance. This EIB covers three emerging threats which could impact police organisations. These are:

• Shadow GenAI use.
• Third party disclosure. 
• Third party production

Shadow GenAI use 

Undisclosed employee use of GenAI, which bypasses organisational security protocols, is referred to as a ‘shadow’ use of GenAI. In their 2025 survey of 500 professionals across Australia and New Zealand, analytics firm Josys found that approximately 1/3 respondents admitted to regularly uploading sensitive information to various GenAI platforms without formal oversight. Sensitive information included strategy documents, financial material, and personally identifiable data. While the survey did not include police staff, it is reasonable to assume some level of shadow use is occurring within police organisations, likely due to the high burden of administrative workloads, some of which can be managed with GenAI. Even a relatively small number of cases would be concerning given the importance of procedure and disclosure in policing practice, and the public’s perception of both.  

Detecting shadow use is extremely challenging. Internet traffic can be monitored, and work devices audited. However, these approaches may only indicate some level of use and can’t capture information or prompt input, and subsequent outputs. Meanwhile, blocking access entirely does not prevent the use of GenAI via private devices and the subsequent integration of outputs into work items. Detecting possible use by scrutinising work artefacts is equally challenging as it is difficult to ascertain whether inappropriate or unsuitable content is the result of GenAI use, or evidence of staff skill/practice deficiencies.   

Third party disclosure

Even if public sector employees were never to engage in shadow GenAI use, the public sector is not immune from private sector misuse of GenAI given that private companies are frequently contracted to provide a range of services to public sector organisations. Such misuse can stem from relaxed attitudes toward GenAI use, as demonstrated by a partner at consulting firm KPMG caught using GenAI to answer questions on an internal exam about appropriate A.I use. Such attitudes are particularly concerning as they can minimise the importance of disclosure practices. In mid-2025, the NSW Reconstruction Authority (RA) revealed a serious breach by a contractor handling data for the Northern Rivers Resilient Homes Program. 2031 people had their data uploaded to ChatGPT, including names, contact details, addresses, ‘limited financial commentary,’ as well as sensitive personal information. 

While police organisations are less likely to be outsourcing the handling of certain types of data, such as personal information provided by victims of crime, third party providers such as I.T or insurance services may handle police employee information. However, there is some risk that due in part to relaxed attitudes toward GenAI use, these providers may disclose police employee information to GenAI platforms.

The best-case outcome of third-party disclosure is the data remaining inaccessible to anyone without authorised system access. However, GenAI platform have vulnerabilities. In mid-November 2025 an attacker breached Mixpanel (an OpenAI contractor), managing to export data containing some identifiable OpenAI user information. Malicious actors may also gain access to disclosed data without breaching systems. GenAI models can output disclosed data as user inputs can be included in training data. If prompted in a certain way, this data may be included in model outputs. While some providers have safeguards against private information being obtained in this way, the provider still needs to be informed of the disclosure to remove the information. Malicious actors seeking disclosed data through prompting do face the challenge of delineating between legitimately disclosed data, and mere hallucinations. However, outputs can be cross-referenced with publicly available sources or with data obtained through other breaches and subsequently sold on the Darknet.   

Besides threatening the security of police employee data, third-party disclosures may also cause public concerns about the safety and security of police data holdings. Public surveys have captured existing worries about data privacy in Australia. 62% of respondents to the 2023 Australian Community Attitudes to Privacy Survey for instance, claimed data privacy was a ‘major concern’ in their life. Fewer than half trusted organisations to handle their data appropriately. Even a limited breach of police employee data may amplify these fears. 

Third party production

Public service organisations also contract private companies for advice, research, and evaluations. The same drivers of third-party disclosure can result in contracted work artefacts including undisclosed GenAI inputs. This poses a credibility risk to public sector organisations. In October 2025, consultancy firm Deloitte produced a report for the commonwealth Department of Employment and Workplace Relations (DEWR) with a litany of errors caused by the (non-disclosed) use of GenAI. These errors included non-existent references, and a fabricated (i.e. hallucinated) quote purportedly from a Federal Court judgement. Deloitte subsequently retracted the report and issued a revision with amended footnotes and a rewritten reference list. As highlighted by one of the academics misquoted in the original report however, the initial misuse undermined the credibility of the final report because its foundation were still “built on a flawed, originally undisclosed, and non-expert methodology.” 

Public trust and confidence could be undermined if a similar case occurred involving a police jurisdiction. Public trust and confidence in policing is partly influenced by perceptions of organisational fairness, and effectiveness. A belief that practice is being informed by possibly hallucinated data may undercut both perceptions. Fairness may be called into question because the direction hallucinated data may point practice toward would be arbitrary, with no principled basis behind its guidance. Effectiveness may be similarly questioned as hallucinated data has no grounding in reality and therefore no reliable relationship to best practice. 

Finally, detecting third-party production is challenging. Public scrutiny is an effective means of identifying hallucinations or fabricated content as demonstrated in the Deloitte case. Yet many reports prepared for police may not be suitable for public release. Any GenAI errors may therefore go undetected, at least until a report is disclosed through some legitimate means like a Freedom of Information request or a Commission of Inquiry. Where reports are made public and undisclosed GenAI use detected, this may threaten public perceptions of trust and confidence well beyond the report in question. The risk of wider trust contagion could be mitigated by a constructive framing of public releases as a quality assurance mechanism. However, this does not mitigate the first challenge, namely that public oversight and scrutiny cannot always be the remedy for detecting errors in material containing sensitive information.  

Strategic Considerations 

How might Australian and New Zealand police organisations address the risks outlined above? Operationalising ANZPAA’s Responsible and Ethical Artificial Intelligence Framework is a necessary first step to outline appropriate employee uses.

The procurement of existing GenAI solutions with agency specific security policies may also blunt incentives to engage in shadow use by providing employees with a secure, alternative to commercial models. This approach has been trialled by government agencies in Australia. However these commercial models may lack the ability to track and log user activity, both of which are necessary for auditing and oversight of employee inputs and model outputs.

An alternative is the development of bespoke, in-house GenAI platforms. For example, the U.S Department of Homeland Security launched DHSChat, a secure GenAI platform built entirely by a newly established departmental AI corps for the DHS’ 19,000 employees.   

However, building custom GenAI models is costly if done by individual organisations. This has led to the implementation of inter-departmental GenAI platforms, such as the Commonwealth’s ‘GovAI Chat’ for the Australian Public Service (APS). Given the importance of information security in policing however, it is unlikely that jurisdictions would realistically be able to leverage such an option even at the state/territory level. Moreover, some employees may have already become accustomed to using their own personalised GenAI models on private devices, given the widespread availability of the technology and its low barrier to entry. The existence of in-house solutions may not be able to overcome such use.       

Perhaps most importantly, creating and encouraging the use of in-house GenAI models requires a broader strategic conversation at the cross-jurisdictional level about striking the right balance of GenAI use. GenAI can clearly generate efficiency gains for policing, subsequently freeing up staff hours. However, emerging evidence also suggests that overreliance on GenAI can effectively outsource critical thinking skills and may degrade cognitive capabilities in the long term. If the balance tips too heavily in favour of widespread GenAI use, staff may indeed be able to reclaim work hours but at the cost of being less cognitively capable of making effective use of these hours.

Addressing the risks of third-party disclosure and production likely requires updating legal instruments such as procurement and tender/approach to market policies, as well as contracts with preferred service providers. Where not already updated, these instruments may require specific wording adjustments requiring adherence to organisational policy on AI use more broadly, as well as the disclosure of GenAI use in artefacts produced for policing.   

Finally, there may be value in conducting organisational GenAI security posture assessments. Security posture assessments are used in cybersecurity practice, where ‘posture’ represents an organisation’s overall preparedness to defend against and recover from cyber threats. In the context of GenAI misuse, this kind of approach could evaluate threats like shadow use, third-party disclosure and third-party production according to their possible level of organisational impact. Gap analyses can then be conducted of organisational capability and capacity in mitigating and responding to any identified risks.

BACK TO BLOG