Opportunities, Risks, and the Question of Professional Judgment

An analysis of how large language models are reshaping finance workflows, and what responsible adoption requires from practitioners:

The context of these changing workflows was recently brought into sharp focus during a dedicated event organized by Rotary Milano Digital. The guest speaker for the evening was Gianluca Garofalo, Manager of Packaged App Development at Accenture, a seasoned technology professional with around 30 years of IT consulting experience specializing in financial and insurance enterprise infrastructure. Drawing heavily from his presentation, this analysis combines fundamental structural themes with direct industry insights from the event regarding the technical reality of Large Language Models and their operational impacts.

Artificial intelligence has been deeply rooted in everyday financial workflows through client reports, document screening, and earnings summaries. This shift is driven by Large Language Models. In finance, inaccuracies can have significant regulatory and financial consequences. Therefore, understanding the functioning of these systems is essential for modern practitioners.

A Large Language Model is a statistical system trained to predict the next element in a series of text. It generates language by pattern recognition learned across numerous high volume training data. The output created is not from a secured database and lacks structural comprehension. As Mr. Garofalo put it, “the model does not ‘know‘ things, it predicts them.” High fluency and sophistication do not guarantee risk-free outcomes in a professional world.

Modern models are built on transformer architectures. Rather than processing language sequentially, word by word, transformers use attention mechanisms and examine relationships between words simultaneously. The model identifies the most relevant parts of text. First the raw text is converted into tokens. The small units may represent whole words, sub-word units, punctuation marks, or individual characters. But tokenizing niche acronyms or unusual ticker symbols can be incomplete and unreliable.

Concern over AI tools mishandling financial data isn’t rare, research confirms that standard LLMs frequently hallucinate when handling financial tasks like explaining concepts or retrieving stock prices, and even when given actual financial documents, AI can distort the facts. For example: if a report mentions a 6-to-1 stock split, a poorly grounded AI might incorrectly state it was a 10-to-1 split simply because its prediction went off track. The privacy angle matters too, major banks including JPMorgan Chase, Wells Fargo, and Goldman Sachs banned internal use of ChatGPT-style tools specifically over fears that proprietary client data could be transmitted to external servers.

Sometimes the AI algorithms produce outputs that are not based on training data and do not follow any identifiable pattern. In other words the response is “hallucinated“, meaning that the interpretations happen due to various factors, including overfitting, training data bias and high model complexity. Documented cases already exist: non-existent court rulings cited by lawyers, invented bibliographic references, and fabricated market data, all delivered in a confident, professional tone. While many of these issues have been addressed and resolved the use of AI tools can have unforeseen and undesirable consequences.

The Three Development Stages of AI and Their Implications for Financial Services

Artificial intelligence has evolved through three distinct stages within financial services. Each stage carries different operational capabilities and distinct governance challenges. Largest financial institutions run older systems built on classical AI while deploying generative tools and experimenting with agentic systems.

The first stage, dominant until roughly 2022, includes rule-based systems and statistical models embedded in financial infrastructure for past decades. Banks have well-adopted machine learning methods for fraud detection, credit underwriting, and portfolio risk management. Algorithms like Random Forests and Support Vector Machines are standard tools for anomaly detection and earnings classification. They only follow fixed and predetermined rules with precision; they are not able to generate texts or perform tasks they were not trained for.

The second stage, generative AI (mainly employed from 2023 onwards) represents a qualitative departure from traditional models. Rather than predicting a numerical output from structured data, generative models produce language and analysis from unstructured inputs. In finance, firms utilize generative tools to give advisors instant access to massive research databases and to summarize lengthy client meetings. Despite generative AI creating text-based and analytical outputs, the responsibility for reviewing and validating the creation remains entirely with the human professionals.

The newest category of AI, agentic AI, moves beyond simply generating to executing. Agentic systems are capable of pursuing a goal in multiple steps by coordinating digital tools and accessing external data. They plan and execute actions in a sequence and make their own decisions with limited human input. Examples of agentic AI are currently being used in applications like ongoing credit checks using live transaction data and automated compliance processes.

The implications for governance are particularly complex. When agents act independently to complete multiple actions, it will be increasingly difficult to create clear human oversight of the decisions made by the AI.

Cognitive Delegation and Its Professional Consequences

During the event, Gianluca Garofalo framed this shift in historical terms: for centuries, technology extended our physical capacities, lifting, accelerating, multiplying force. Today, for the first time, we are delegating parts of the cognitive process itself: synthesis, analysis, writing, and the first forms of judgment. He called this “cognitive delegation“, an invisible change that makes no noise, but deeply modifies how we reason and make decisions.

As generative artificial intelligence continues to be integrated into finance, it is reshaping the way that we complete important tasks associated with synthesis and analysis. Tasks such as summarizing regulatory filings or preparing compliance documentation are increasingly being outsourced (delegated) to generative AI systems. While this represents an increase in operational efficiency in the short run, it raises serious issues about whether the expertise present in the professional workforce will remain intact. Many of the analytical activities traditionally performed by personnel that have supported their development of critical skills are being replaced (or at least augmented) by advanced technology. The most pronounced transformations are occurring for early career professionals.

Because generative AI platforms have begun to minimize the performance gap between early career and late career professionals on complex cognitive tasks, early career professionals gain increased levels of baseline productivity, but as well, every additional “unit” of output may reduce their opportunities for development through the traditional learning process at a financial institution. Early career professionals are benefiting from reduced costs of research; however, there is a risk that they may develop a less complete understanding than previous generations of professionals due to an over-reliance on these advanced tools. Hence, many finance professionals are spending less time developing original reports and more time scrutinising the outputs from generative AI systems.

Generative AI is becoming more common in financial services; as a result, the way professionals create synthesis and conduct analysis is changing. Many of the tasks performed by junior professionals (such as summarizing regulatory filings and drafting compliance documents) are now being carried out by AI systems. While operationally, that may provide short-term efficiencies, the larger implication is whether the expertise developed by professional staff will continue to be sustained. Many of the analytical functions that enabled junior staff to build critical skills will now be partially automated. This is most noticeable in the case of junior professionals.

The shift presents new challenges in the definition of professional responsibility. AI-generated recommendations increasingly factor into financial decisions made by professionals. However, when professionals depend solely on these automated recommendations and do not verify the underlying assumptions that have led to these conclusions, the consequences of relying on AI-generated recommendations becomes more problematic. Financial professionals must continue to possess the skills necessary to critically evaluate the accuracy of AI-generated outputs. The dangers lie in systemically delegating tasks to machines without verification. As time passes, this could result in a polarization of professional labour where professionals who provide oversight of AI systems separate from those who perform routine operational functions.

The Formation of Professional Judgment in an AI-Assisted Environment

Junior analysts typically acquire analytical proficiency through repetitively performing challenging, analytical tasks. Drafting of market overviews and performing manual analysis have historically been key to providing junior staff with the experience necessary to develop critical thinking. The basis for a professional’s judgement in finance is the result of having been exposed to complex information, analytical errors, and repeated analysis of the information being analysed. These intensive processes teach junior employees how to interpret uncertainty, evaluate assumptions, and distinguish reliable analysis from superficial fluency.

The development of large language models changes how the growth of professionals in analytical roles occurs. Generative AI systems can generate a first draft report and summarize earnings calls people who work in these roles create in seconds. As generative AI systems can generate first draft, the speed at which a person’s operational ability operates creates new tension between a new high level of speed of operation and the amount of cognitive friction necessary to gain true expertise.

True analytical competence rarely develops through passive observation. Rather, activities such as reconciling conflicting or differing source documents, identifying inconsistencies, and so forth require junior professionals to build true analytical competence.

The ability to present polished automated outputs will create conditions in which professionals produce “results” without truly understanding the economic processes producing the results. This is what Garofalo described as “AI creating experts without experience.” The output text exists, but there is no reasoning path associated with its construction. Professionals with no cognitive friction will never experience competence as a professional. The lack of cognitive friction is also likely to result in professionals who are not able to build their own professional judgment.

During the period where AI performs the first draft of the cognitive work, junior professionals may move from being active producers of analysis to being passive reviewers of AI-generated reports. The cognitive demands of reviewing an AI-generated report will not be the same as producing an analysis by yourself. As a consequence, a junior professional may develop an overreliance on AI-generated results even though they are not accurate in the given context. A phenomenon occurring in high-stakes financial environments where overreliance on AI-generated results will create long-term risks concerning developing independent patterns of professional judgment.

Accountability Structures Under the EU AI Act: The Provider-Deployer Distinction

In 2024–2027, when the EU AI Act takes effect, the question of who is liable for a financial decision made with AI tools will shift from a theoretical discussion to the reality of everyday practice for finance professionals. The world’s most comprehensive AI regulation (the EU AI Act) will create a clear line of demarcation between financial institutions acting as AI tool providers (such as OpenAI, Google, and Anthropic) and acting as AI tool deployers in their own right. For example, a financial institution that develops its own AI tools in-house will act as both a provider and deployer simultaneously, whereas a financial institution that uses a third party to supply the AI tools will only be acting as a deployer.

For most financial institutions, the deployer status will be the primary focus for operational purposes. Deployers, according to Article 26 of the EU AI Act, must operate AI tools according to provider specifications. Deployers must have adequately trained personnel authorized to perform human oversight for AI tools. Deployers must keep logs generated by AI tools for at least six months, and the financial institution that uses the AI tool bears the burden of documenting its AI-assisted decision-making. Furthermore, even if a financial institution has an agreement with a provider that limits the liability of the provider in case of failure of an AI tool, this will not change the legal status of the financial institution as a deployer for purposes of any decisions made using an AI tool.

This framework is highly demanding for finance practitioners. Assigning a person to oversee an AI system without giving them practical mechanisms to halt it does not satisfy regulatory requirements, a supervisor in name only does not meet the bar. Oversight roles that carry no operational authority represent a common failure mode in AI governance. The person assigned to supervise an AI-driven credit tool must have the technical literacy and institutional authority to override the system.

Annex III of the EU AI Act classifies AI systems used for evaluating creditworthiness or risk pricing in life and health insurance as high-risk. For these applications, the enforcement framework sets penalties for non-compliance at up to 3% of global annual turnover, rising to 7% for prohibited practices. Institutions should treat December 2, 2027, the new deadline set by the recently approved digital omnibus, which deferred the original August 2026 date, as the operative enforcement date for these high-risk systems. Under individual accountability frameworks, responsibility flows upward to named senior managers. Regulators do not accept vendor errors as a valid defense for compliance breaches.

Toward a Framework for Responsible AI Adoption in Finance

AI systems have the technical specifications to perform tasks efficiently and have begun commercializing in multiple industries; however, the question of personal accountability for unaccountable AI judgement will be a more challenging task to navigate. AI literacy is not just about using AI tools; rather, it is about analyzing the output of AI tools and knowing when not to use AI.

In this way, a practical framework for adopting and developing capabilities around using AI to perform workflows will require a combination of three distinct elements: use, evaluate and stop. If you can identify multiple ways to use AI to automate the same workflow, then you might be considered competent. Ultimately, when you are able to use AI at a minimum level, your competencies for using AI will move beyond efficiency into comprehending how tasks can be automated, as well as determining which tasks require human judgment due to the complexity of context. For example, in the finance industry, being able to automate document sorting and synthesizing documents to establish an initial understanding of the data is appropriate, while advising clients about an appropriate loan or making credit decisions will not be automated.

The evaluation phase is where a professional develops their true competency through the evaluation of AI’s performance. The output generated from AI has a greater concern for language-based coherence than fact-based validity. Evaluation is much more than just verifying the accuracy of fact; rather, it requires the determination of whether the premise upon which the model is based is justified and whether the framework through which the data was analyzed produced an unfair or biased conclusion. The only way for a professional to develop their evaluation skills using AI is using their analytic skills built upon traditional practices, such as reading a company’s financial statements.

The last piece of judgement is knowing when you need to retain the judgement for yourself. This is because your human judgement will have the greatest value in areas that AI will probably produce misleading output. The most obvious instances would be when the output would materially impact a client or create substantial regulatory liability. The person who certifies AI-assistance has operational responsibility for it. Cognitive delegation signifies an active decision, not an inevitable outcome. The technology does not have the ability to exercise independent judgement so that the entire framework of responsible implementation is predicated on the professional.

Cognitive Delegation and Professional Responsibility

The most lasting takeaway from the evening was not technical at all. During the Rotary Club Milano Digital event, Garofalo emphasized a profound macro-historical shift that puts these professional and operational risks into a broader context. He reminded the audience that for over two centuries following the Industrial Revolution, technology was designed to extend the human body, automating physical work to increase manual speed, force, and output. Today, artificial intelligence represents a qualitative break because it directly enters our cognitive processes, supporting and intersecting with human thought and it is this break that gives rise to what he called “cognitive delegation.”

Stepping into a professional world that is adopting an entirely new set of practices is challenging. Our generation is first to face the changes that AI brings into the workplace. While these tools may seem to simplify technical taste, they do not replace the human responsibility and critical judgement required of us. The hours spent on technical work may feel obsolete but that friction is not an inefficiency to be eliminated. Staying up to date with these technological advancements will keep us competitive and enable us to deliver the highest level of efficiency. While leaning on machine learning is intuitive, professionals must remain sharp in monitoring and analyzing AI outputs. True expertise and deep knowledge will always be crucial to shaping our industries for better.

Written by:

• Domenico Agostino
• Keti Aznaurashvili
• Nicolò Ballabio
• Claudia Cristofolini
• Giorgio Signorile
• Matilde Sommese