TikTok took nine months to reach 100 million users, while OpenAI's chatGPT achieved the same feat in just two months. This breakthrough in Large Language Models (LLMs) has sparked a gold rush, with companies, governments, and academics eager to invest millions, if not trillions of dollars. However, it is crucial to evaluate LLMs properly and choose the right model for the right use case.

To understand LLMs (Large Language Model), it is important to look into them through the following lens of natural language processing domains:

• Natural Language Understanding
• Natural Language Generation
• Knowledge Intensive Task
• Logic Reasoning Capability

To illustrate these concept, let's say you want to use copilot to write you a python program to calculate the risk of cancer given a dataset.

To accomplish this task, LLMs need to understand your prompt (Natural Language Understanding) and generate appropriate python program (Natural Language Generation). Further, its need to the domain knowledge on cancer detection (Knowledge Intensive Task). If you ask "hey why do you write the program this", the LLMs ideally can defend its logic with reasoning(Logic Reasoning Capability).

If this article, I will mainly talk about NLU, NLG and Knowledge Intensive Task. Even though the LLMs are gradually shifting from open-source (source code available to public) to closed-source, their performance in downstream tasks can still be evaluated.

Let’s get a few definitions out of the way:

• Fine-Tuned Models: Downstream task specific fine-tuned ML models
• LLM: Large language model with pre-trained data with billions of data parameters

Use case #1: Chatbot Natural Language Understanding

In this case, deploying a LLMs might be an overkill. The intent is clearly defined, given available training data, the traditional fine-tuned model should serve the users really well.

However, the following scenarios, LLMs might be a better candidate:

1. Fine-tuning data is not available due to regulation
2. Enable Chatbot to handle general requests, from booking tickets to escalate user complaints

It is important to be aware to rely on LLMs to handle general requests is pretty dangerous, given pre-trained data maybe counterfactual and knowledge in pre-trained data may counter the actual knowledge in the use case.

Use case #2: LLMs for Natural Language Generation

Yes! LLMs excel in generation of language. Use case such as github copilot, available to consumer, is a great example to showcase this capability. In paper “Language Models are Few-Shot Learners” , the researchers suggest ”news articles generated by LLMs are almost indistinguishable from real news articles by humans.”

Use case #3: Using LLMs on Knowledge Specific Task

The concerns are LLMs may not have the specific domain knowledge and its pre-trained data can be counterfactual. However, even though the fine-tuned models excels in some knowledge intensive task, which are strictly bounded by its fine-tuned data, it is typically hard to scale for enterprise users.

A solution to allow LLMs to have knowledge you need to solve a task is to augment LLMs with information retrieval ability. Let’s say if we create an embedding of a text corpus( for example, a prep book for LSAT) in the memory of LLMs, asking LLMs to take an LSAT is not difficult at all. How difficult it is to take an open book exam? Given OpenAI has not released anything practical on GPT4, this could be their secrete sauce.

What is My Favorite Use Case?

Data annotation has been a great challenge in ML development. It is painful to annotate or label data. Startup like Scale AI that provides such solution and they have been doing pretty well to supply highly skilled data annotators. But this is costly and human makes mistakes as well.

What is an example of data annotation? If you have used chatGPT and you have clicked 👍👎 on the response? Yes, you are working as a free data annotator!

LLMs can be the next candidate in line to solve this problem and further accelerate the industry.

Consideration of Risks?

Without a doubt, GPT model can be one of the most exciting technology breakthroughs of the century, we need to be cautious on its potential risks.

Besides, the AI taking over jobs theories, which is very unlikely to happen. This development will fundamentally alter the status quo from how society interact with AI/ML to how the job are performed. It will bring the restructuring of Economy, some jobs will be eliminated, but more jobs will be created.

Further, the electricity consumption is also quite alarming. To train GPT3 of 175 Billion Parameter, it costs 4.6 millions. To train transformer model with 6B parameters to completion is estimated to be around 103.5 MWh!

Enterprise and personal user might consider differently for their model requirement on top of costs and environment consideration. As remarkable as GPT models are, mostly like the majority of use cases will be remain in classic fine-tuned ML models for the next 5- 10 years.