When Generative AI runs into a deadend, ELAINE opens new doors.
Many knowledge experts on vertical knowledge domains are puzzled by the arrival of Generate AI. Some feel threaten by it.
Problem: When Generative AI response is inaccurate, user suffers
Generative AI is an implicit knowledge system that is built with human artifacts across different knowledge domains and it promises to provide responses on various domain of discourse.
Implicit knowledge system provides response to a prompt that regurgitates human artifacts that the user already familiar with.
The knowledge system intrinsically has no understanding of the words that it generates other than the fact that generated text would be similar to a response from a knowledgeable person when confronted with the prompt. As a result, the content generated is not audited nor verified for accuracy.
When a response sounds logical, authoritative, and is articulated with the correct sound bites, it can easily be accepted by the user as trustworthy.
However, users ultimately are the ones bearing the consequence when trusting unverified information from Generative AI.
This problem is not unique to Generative AI.
Solution: Generative AI audit - ELAINE
ELAINE is a a propositional logic processor that can express natural language text in propositional logic and provide a logical version of the text for audit.
How to fact check Generative AI responses?
For demonstration, we take the content of an article published by CNBC "IMF upgrades global growth forecast, citing U.S. resilience and policy support in China" discussing "the cause for inflation to come down so fast" and perform the following test:
1. Submit the text of the CNBC article to GPT from OpenAI and then asked GPT "what is the cause for inflation to come down so fast?"
2. Submit the text from the same article to ELAINE for analysis to fact check GPT's response.
3. Compare the results for accuracy
Result from GPT:
GPT pointed to four items in its response attributes to rapid decline in global inflation: Resilient Global Economy, Strong Labor Markets, Easing Supply Chain Friction, and Decline in Energy and Commodity Prices
A followup prompt asking GPT to verify the finding against its model, and GPT regurgitated what was stated in its first response.
Result from ELAINE analysis
"But restrictive monetary policy has led to inflation falling faster than expected in most regions"
Comparing the two results
Contrary to GPT's response, economic growth is generally the reason for inflation to go up. A strong job market is also a reason for inflation to increase.
An article by a U.S. Bank also stated that strong labor market is a reason for inflation to stay high:
"One consideration that could affect any decisions by the Fed is job market's ongoing strength, which may continue to provide consumers the wherewithal to maintain higher spending levels. That heightens the risk that inflation could persist or even trend higher."
ELAINE correctly analyzed and abstracted the context from the CNBC article "But restrictive monetary policy has led to inflation falling faster than expected in most regions, which Gourinchas called the 'other piece of good news' in Tuesday's report."
The test above shows that GPT's response is not just lacking accuracy, it is blatantly wrong. Adopting it in business strategy will result in undesirable consequences.
With finding from ELAINE, even a layman (without expertise in the domain of discourse) would take note on such flaw. This process will introduce opportunities for domain experts to fill the void.
Click to learn more and view details of the above demonstration