A six-month evaluation of Microsoft 365 Copilot in the Australian public sector has shown that the AI assistant can save employees up to an hour per day, though its implementation faces significant hurdles.
The Australian Digital Transformation Agency (DTA) conducted the trial across 56 government agencies, issuing nearly 6,000 licenses and gathering feedback from more than 2,000 participants. Responses to individual questions were sometimes only in the triple digits, which seems to be an overall weakness of the study.
Despite the generally positive attitude toward the technology expressed by 77 percent of participants, only one-third of users engaged with the AI assistant daily. More than 80 percent (n=827) said they would like to continue using Microsoft 365 Copilot after the trial, which is good news for Microsoft.
69 percent of employees reported getting work done faster, and only 7 percent reported losing time due to fact-checking and content review. The trial found that Copilot worked best for specific tasks like summarizing information, creating initial document drafts, and information retrieval. Microsoft CEO Satya Nadella recently described summarization as "a big deal" for generative AI.
Middle management and IT staff saw the greatest productivity gains, with around 40 percent of users reporting that they used the time they saved for higher-value work, such as management tasks and strategic planning. 61% of participants agreed that Copilot improved the quality of their work.
Implementation is challenging
The DTA identified several major obstacles during the trial. Technical integration issues, insufficient training in prompt engineering, and unclear legal responsibilities hampered adoption.
The agency also noted worries about data security, information management, and potential bias in AI outputs. 61 percent of managers couldn't reliably distinguish between AI-generated and human-created content.
Some participants expressed concern about the loss of basic skills, such as summarizing text, and the environmental impact of using AI.
The evaluation highlighted potential workplace equity issues, noting that increased AI adoption could disproportionately affect administrative positions - roles that traditionally serve as entry points for women and marginalized groups.
Based on these findings, the DTA recommends that government agencies develop comprehensive implementation plans that include specific training programs, clear usage guidelines, and active risk management strategies. The agency also suggests that departments analyze their workflows to identify additional beneficial applications for the technology.
