The advent of Generative AI

There has been plenty of discussions in social media regarding the pros and cons of using Generative AI to boost productivity. For text-based content, Generative AI responds to prompts with a pre-trained model that is tailored to the specificity of different domains of discourse.

When the response is mis-guided, it is labeled as hallucination.

While some argued that such behavior as a sign of emerging or super intelligence, others called it a form of fabrication.

Some suggested that hallucination could be avoided if the context being inquired is submitted as part of the prompt. Doing so would confine the generated response within the relevant context, thus avoiding "out of context" response or hallucinations. Unfortunately, even with this confinement, there is still plenty of room for error. When the response to a prompt is well articulated, the meaning can be wrong, illogical or "out of context". This happens when the underlying technology is a Large Language Model (LLM). It is not derived from sentential logic. Instead, it is based on statistical patterns found between tokens that are derived from words. Human's understanding of natural language resembles the way context and semantics that are expressed in sentential logic.

What does it mean when response from LLM sounds right but does not correspond to the input prompt or it is wrong or illogical?

The obvious answer is that users find it credible since the response is articulated with authoritative tone and sounds right. If the user is misinformed and there is no tool available for "fact check", the user bears the consequence. Any strategy, decision or action derived from such misunderstanding will yield adverse result. Generative AI accuracy concern became a major topic of discussion at the 2024 World Economic Forum Annual Meeting.

To demonstrate this issue, we conducted a session with Generative AI

The content of an article published by CNBC, "IMF upgrades global growth forecast, citing U.S. resilience and policy support in China" was submitted as part of the prompt to GPT, a Generative AI system from OpenAI. The following is a copy of the GPT session

The question to GPT is "Based on the above corpus, what is the cause for inflation to come down so fast?"

What was GPT's response?

GPT pointed to four items in its response. Out of the four factors, two of them, namely, "Resilient Global Economy" and "Strong Labor Markets" are actually signs for inflation to go up, not down. However, due to the style and verbage rendered by GPT, it sounds convincing and authoritative. A user without adequate knowledge of general economic trends on inflation would be misled by the response.

In fact, all these four points given by GPT as reasons for inflation to come down so fast is taken out of context.

Fact Check GPT response

Contrary to GPT's response, economic growth is generally the reason inflation goes up. A strong job market is also a reason for inflation to increase.

"resilient global economy" is normally a precursor of higher inflation. IMF expressed worry that Central Banks would be reluctant to lower interest rate, which would slow down the decline of inflation as evidenced in the following excerpt:

"What we've seen is a very resilient global economy in the second half of last year, and that's going to carry over into 2024," IMF's chief economist, Pierre-Olivier Gourinchas, told CNBC's Karen Two on Tuesday.

"This is a combination of strong demand in some of these countries, private consumption, government spending. But also, and this is quite important in the current context, a supply component as well. ... So very strong labor markets, supply chain frictions that have been easing, and the decline in energy and commodity prices."

"While central banks must not ease too early, there is also a risk coming into sight of policy remaining too tight for too long which would slow growth and bring inflation below 2% in advanced economies, Gourinchas added."

An explanation by a U.S. Bank also stated strong labor market is a reason for inflation to stay high:

"One consideration that could affect any decisions by the Fed is job market's ongoing strength, which may continue to provide consumers the wherewithal to maintain higher spending levels. That heightens the risk that inflation could persist or even trend higher."

We asked GPT a follow-up question to verify the finding against its model, "is your finding consistent with conventional thinking?".

GPT simply regurgitates what was in its first response.

Can a propositional logic processor help to fact check GPT response?

SiteFocus offers a solution that performs logical audit on corpus. That solution is called ELAINE, a symbolic logic platform. Submitting the same CNBC article to ELAINE yields a different finding.

ELAINE found the following excerpt from the corpus that corresponds to the CNBC article offering "restrictive monetary policy" as the main reason that led to inflation falling faster than expected.

"But restrictive monetary policy has led to inflation falling faster than expected in most regions"

Excerpt from ELAINE's analysis on the original article showed the following from its logical analysis:

Risk to Financial Industry is real as Bloomberg rolls out its GAI

More and more businesses are making bold moves to incorporate Generative AI in products, from AI hiring to AI in Microsoft Excel.

Bloomberg recently announced the offering of Generative AI Tool for clients to inquire about Earnings Transcripts.

For business users of product like this, the question of accuracy will remain an unresolved issue. Naturally there are compelling incentives for solving this "accuracy" issue as the Generative AI market continues to grow.

Generative AI Audit - ELAINE

At SiteFocus, we believe that a solution to mitigate this risk is at hand. ELAINE, a solution based on symbolic logic will offer clarity to any natural language narratives, SiteFocus believes that "logical audit" on prompts and responses from Generative AI is the ultimate solution. By auditing logic found in input prompt and output response, users can make an unbiased assessment of the quality generated by Generative AI.

ELAINE Community is a Web-based program that is open to the public. By leveraging the ELAINE platform in logic audit, anyone in the knowledge business can engage opportunities around the adoption of Generative AI. "Accuracy" is becoming the last mile for business to adopt Generative AI in their business.