Three test cases expose the limits of AI financial advice

Free AI tools answer money questions in seconds, and researchers wanted to know how well they handle people who cannot afford a wrong turn. They put three vulnerable profiles to ChatGPT, Claude and Perplexity: a graduate saving during a cost-of-living squeeze, a pregnant woman whose partner keeps separate finances, and a single parent nudged towards crypto. The answers were structured and practical, but the models overlooked risks stated in the prompts and fell back on stereotypes. For South Africans, where advice fees put a planner out of reach for many and crypto pitches circulate widely, that gap matters..By Bomikazi Zeka, and Raechel Johns*.Ever used or thought of using artificial intelligence (AI) for financial advice? Is it a good idea?Anyone with the internet, even on their phone, can access AI. Tools like ChatGPT, Claude and Perplexity are free, instant, and easy to use. They can break down complex financial concepts and provide financial solutions in seconds. But can they safely guide a user through a nuanced, high-stakes financial crisis?Our recent research explored this question, drawing on our financial planning and consumer behaviour expertise. We evaluated how three open-access AI models handled the financial queries of users whose personal situations put them at risk of harm.To do this we created hypothetical scenarios representing different life stages, economic challenges and socioeconomic vulnerabilities. We then ran these scenarios through OpenAI’s ChatGPT, Anthropic’s Claude and Perplexity AI:a 22-year-old university graduate wanting to save for a home deposit during a cost-of-living crisisa pregnant woman seeking financial advice on planning for maternity leave, with a partner who doesn’t share moneya single parent of two children, with a modest income, who was told by their cousin to invest in cryptocurrency..In our analysis we found that AI offers highly structured and practical advice. But it has blind spots. We provided the models with explicit information about how the three cases were vulnerable, but the models didn’t respond appropriately to that. They even offered advice that could make financial harm worse.We concluded from our research that AI is a powerful tool for financial fact-finding and brainstorming. If you are already financially literate, and know how to cross-check data, AI can be a useful financial aid. But, for now, AI cannot replace the human element.Running scenariosWe chose Claude for its versatility and conservative approach, ChatGPT for being an all-round assistant, and Perplexity for its contextual understanding.We developed five hypothetical scenarios and simultaneously ran each scenario five times to test for consistency in the output. Each scenario was run in a different web browser, in incognito mode, with cleared cache and cookies to eliminate any retained information. This was also to ensure that the output generated was not influenced by retained data.To reduce the risk of AI-generated bias, we removed identifying attributes that could influence model outputs, such as names, locations, race and income. We then assessed how the AI models considered the user’s vulnerability, area of financial need, and goal or desired outcome.The models distinctly addressed the financial query, yet revealed gaps in how they balanced advice with considerations of vulnerability.At first glance, the AI models looked like they were giving sound financial advice. But our analysis showed that they handled vulnerability in vastly different ways.ChatGPT was highly detailed and practical, but did not recognise the vulnerability embedded within the prompts. Instead, it relied on the information that was explicitly stated. It didn’t consider whether the user’s situation suggested a need for additional support or tailored guidance.Perplexity emerged as the most conservative and risk averse as it most frequently urged users to seek professional financial advice. But it produced the least detailed responses.Claude offered comprehensive recommendations. But it leaned heavily towards self-guided financial planning.The vulnerability blind spotAI is trained on massive datasets designed by humans who are biased. For instance, when models are trained on historical records that reflect systemic discrimination, the tools can fall back on social stereotypes and make biased assumptions in their output.Research shows, too, that when AI models explain their recommendations, humans are far more likely to trust the advice blindly, ignoring whether it is actually correct.The most alarming finding from our research was the technology’s failure to consistently recognise and address the needs or concerns of vulnerable users.For instance, in the case of the graduate, the recommendations focused on saving for a deposit but ignored how the high cost of living would make that harder. This was even when explicit details about the graduate’s financial circumstances had been provided in the prompt.The response didn’t consider how achieving the long term goal would affect the current lifestyle..Read more:.Massive Cloudflare glitch hits ChatGPT, X, and major websites worldwide.In the case of the pregnant mother, Perplexity and ChatGPT’s output assumed the partner would assist with household expenses after the birth, even though the prompt explicitly stated that the partners did not share their finances. The models did not consistently use the detail provided in the prompt and instead generated a response based on a more common assumption about household financial arrangements.We saw the output reflecting social stereotypes where mothers are strongly associated with parenting while fathers are strongly associated as material providers.This shows that even as AI evolves, it still replicates biases.Take the single parent asking for cryptocurrency recommendations. Even though the AI models advised caution, ChatGPT described in detail how to get into cryptocurrency investments and recommended cryptocurrencies for beginners.A human financial advisor would immediately flag these areas for consideration: a modest income, children, a high-risk investment, and anecdotal advice from the cousin. A human advisor would first gauge the user’s overall financial position, time horizon, risk appetite and investment objectives.Our research shows that AI models are limited in providing tailored financial advice, unless the user provides additional explicit input.Where do we go from here?Financial planning isn’t just about numbers; it’s about advice based on human values, emotional anxieties, family dynamics, personal experiences and risk tolerance. AI has opened the door to instant financial information for the masses. But until these models can truly comprehend the complex, vulnerable and emotional realities of humans, the most valuable financial skill remains critical thinking and asking yourself: “does this make sense for me?”.If AI tools are to become an avenue for financial guidance, there must be safeguards. Policymakers and financial regulators must build clear frameworks around AI generated advice. What also needs to be considered is what AI models do with the financial information provided. Strict transparency standards are needed to regulate how these models handle consumer data.Financial information is deeply sensitive and consumers must have absolute clarity on how their inputs are stored, whether the AI retains a “memory” of their finances, and who has access to that data..*Bomikazi Zeka Associate Professor in Finance, University of Canberra.*Raechel Johns Professor of Marketing; Co-Director, Centre for Intergenerational Digital and Financial Wellbeing (CIDFW); Canberra Business School, University of Canberra.This article was first published by The Conversation and is republished with permission..Sign up for your early morning brew of the BizNews Insider to keep you up to speed with the content that matters. Register here.Support South Africa’s bastion of independent journalism, offering balanced insights on investments, business, and the political economy, by joining BizNews Premium. Register here.If you prefer WhatsApp for updates, sign up to the BizNews channel here.

Three test cases expose the limits of AI financial advice

Related Stories

FT: Brin tightens grip on Google's AI push as Hassabis steps back from DeepMind

The Economist: OpenAI’s Hugging Face hack - why the AI containment problem just got a lot scarier

What fruit flies can teach us about the human brain

Priced for perfection: A retired fund manager's warning on the AI boom