Was the Department of Justice using experimental chatbots to give immigration advice?

Department of Justice chatbots to answer questions about asylum and citizenship were developed via a “pre-commercial initiative” that was “aimed at supporting the early stages of developing new technologies and products”.

That’s according to Enterprise Ireland, in response to a request under the Freedom of Information Act by Kris Shrishak, a technology fellow at the Irish Council for Civil Liberties (ICCL).

Chatbot Erin took questions about the asylum process. Chatbot Tara answered citizenship and non-asylum immigration questions. 

Up until this summer, the International Protection Office’s (IPO’s) automated email response encouraged people seeking asylum to turn to Erin for “urgent” questions. 

The chatbot’s answers were sometimes wrong or misleading. On 4 November, the Minister for Justice, Fianna Fáil TD Jim O’Callaghan, told the Dáil that the department has now retired Erin.

It’s unclear whether Erin or Tara were the prototypes developed via the Small Business Innovation Research (SBIR) scheme, or final products rolled out later. A spokesperson for the Department of Justice did not directly address the query. 

But during an Oireachtas committee debate recently, independent Senator Lynn Ruane raised concerns about deploying these chatbots, if they were “probably not fully tested”.

Sinéad Gibney, a Social Democrats TD, who had also asked about the chatbots at the Dáil, said in an email last week that half-baked products shouldn’t be tested on vulnerable people. 

“The public should not be used as guinea pigs, particularly vulnerable groups in a legal process which could be impacted by a chatbot giving an incorrect answer,” she said.

Neither Erin nor the old version of Tara was AI-powered, O’Callaghan told the Dáil on 4 November. They relied on scripted answers, he said.

But Tara has been upgraded. The Department of Justice has used Microsoft Copilot’s AI to embed a new smart chatbot in its “Digital Contact Centre for its immigration customers”, said a spokesperson. 

A review of answers from the new AI-powered Tara, by immigration solicitor Wendy Lyon, found that it can also offer incorrect or misleading advice. Though most of them had been marked as correct by officials during a test run, documents suggest. 

Is this a test run?

Shrishak became interested in the Department of Justice’s use of chatbots and generative AI after reading the story about the IPO chatbot Erin, and set out to find more.

On 26 May, the Department of Justice sent Shrishak a decision letter in response to a Freedom of Information request.

In refusing access to some documents, it explained the funding scheme for its Tara chatbot. Shrishak had asked to see the tendering and contract documents.

The decision letter said Enterprise Ireland and the Office of Government Chief Information Officer had funded the development of the chatbot as part of the Small SBIR initiative. So it doesn’t have any records of its contracts and tendering process.

It hired a vendor for its maintenance, though, and information about the firm and the cost is commercially sensitive, so it won’t release it, the letter said. 

Although, earlier this month at the Dáil, O’Callaghan, the Minister for Justice, named Ludex Ltd, a small Dublin company, as having been paid €229,454 to support and maintain the chatbots Erin and Tara.

When Shrishak, the ICCL technology fellow, turned to Enterprise Ireland, a staffer said it didn’t hold any tender or contract documents for “Tara or Erin” chatbots.

Its SBIR scheme was a “pre-commercial” endeavour “aimed at supporting the early stages of developing new technologies and products”, said the Enterprise Ireland letter.

A full tendering process would’ve been needed to “procure any solutions that arose from the prototypes that may have been created during the program”, the letter says.

Both Erin and Tara were designed to make the information on the IPO and Irish Immigration websites more accessible, said the Department of Justice spokesperson.

They weren’t offering “detailed personal or legal advice”, they said.

A representative for Ludex has not yet addressed queries sent on Saturday including one  asking if it was maintaining the chatbots or rolled out final products for national use.

In search of transparency 

What Shrishak learned about the chatbots cropped up during an Oíreachtas committee debate on artificial intelligence use last month when Ruane, the independent senator, flagged worries about pre-commercial nature of the SBIR project, which didn’t trigger a full tendering process. 

In response, Barry Lowry – the government’s Chief Information Officer (CIO) – said the idea is that these companies cut their teeth during the pilot and emerge as strong contenders for the procurement bidding.

“We want Irish companies to be competitive and successful,” Lowry said.

He said once the SBIR scheme ran its course, the government had to pivot to procurement to offer a product for national use. Still, he doesn’t know much about the Department of Justice’s chatbots, said Lowry. 

A spokesperson for Enterprise Ireland reiterated the point of the SBIR project, as outlined to Shrishak of ICCL, and said it had shortlisted two companies to participate, Ludex Ltd and Derelinx Ltd.

It stressed again that a full tendering process would have been needed to procure solutions that emerged from the scheme’s prototypes. 

Ludex Ltd arose both as the company selected for the chatbot Tara under the SBIR programme and as the firm that stayed on to maintain the Department of Justice’s asylum and immigration chatbots.

A December 2020 interim report for the SBIR scheme suggests that Ludex was proposing use cases for the chatbot Tara, and researching different aspects of the citizenship process on the government website.

Lightening the workload for staff who often had to answer questions that had been asked before was part of its pitch. 

In an email last week, Ruane, the independent senator, said people being asked to use these chatbots are among the most vulnerable and in profound need of information about their rights.

She is concerned, she said, that people turn to these tools out of desperation, and that no one will be held accountable if they end up with the wrong advice, referencing the IPO chatbot that came with a disclaimer which shrugged off responsibility. 

Shrishak, of ICCL, said he’d also asked for a cost-benefit analysis and environmental assessment reports for the chatbot project, but the Department of Justice didn’t have them. 

A briefing document on deploying the AI chatbot says under the heading “environmental and societal well-being” that since it’s designed to “surface” relevant information to users, it is “in line with the UN’s Sustainable Development Goals of building an inclusive society for all”.

If AI ruled the world

The spokesperson for the Department of Justice said that at the moment, people seeking asylum can email their questions to info@ipo.gov.ie.

It’s also in the process of revamping its website, and does its best to update information on the current website, too, they said.

The Department of Justice’s new Tara chatbot relies on Microsoft Copilot’s AI. During a trial run, the Department of Justice had internally assessed almost all of its responses to immigration queries as correct, documents released to Shrishak show.

But when Shrishak asked Lyon, the immigration solicitor, to review the AI chatbot’s responses, there were suggestions it could be unreliable. 

Large language models (LLMs) often don’t give the same answer twice because they operate under a notion of probability, which can make their responses unpredictable as they choose each word in their reply from an array of possible suitable next words. 

A briefing document says “an assigned team will be responsible for the chatbot's content accuracy and updates”. Users can also give feedback to improve the model, it says.

Nevertheless, in Tara's responses during a test run, for citizenship eligibility, it said any “adult who has legally lived in Ireland” for five years can apply.

That’s not true, though. Some immigrants can live on student stamps for nearly a decade, without any of those years counting towards citizenship. The law excludes that immigration stamp, but that doesn’t mean the holder hasn’t “legally” lived here.

In response to a question about rights to stay here as a recent EU citizen who used to hold a long-term residency stamp 4, the answer is just wrong, even though it’s marked as correct by officials. 

It says, “You can apply for EU Treaty Rights as the family member of an EU/EEA or Swiss citizen.”

“If the person now has EU citizenship, they probably don't need to apply for anything at all,” says Lyon’s review response. 

A spokesperson for the Department of Justice did not say whether a lawyer had reviewed those responses. 

They said that in its roll-out, it has “leveraged AI in a low-risk way to offer enhanced digital services”. That also includes aid with appointment booking and status inquiries to both immigrants and staff.

It has trained staff on how to use it, too, the spokesperson said.

In listing challenges of using the AI-powered chatbot, officials had noted difficulty “to scale personalized service” and the toll of resolving multiple queries at the same time on the “agent”.

Shrishak, of ICCL, who had obtained documents on the old chatbots and the new AI-fueled Tara under the FOI Act, says he’s concerned about their use when they can be wrong or throw up unhelpful responses.

People might ask life-altering questions, full of nuances and in need of a skilled staffer’s response, he said.

“The chatbot is not the answer,” Shrishak said.