There are two types of knowledge: knowledge of ‘that’ and knowledge of ‘how’. The first is called declarative knowledge, which is concerned with facts. The latter is called procedural knowledge, which is concerned with how to perform a task. Knowing what the capital of Australia is declarative knowledge and knowing how to coding is procedural knowledge.

We acquire knowledge by learning, doing, and reflecting. We read books, listen to podcasts, watch YouTube and attend conferences. We learn riding bycile and coding by doing it. We journal, meditate and reflect.

Acquired knowledge will be stored in memory1. But Most people have poor memories. So we take notes. Draw diagrams and mind-maps. But more often than not this only create false sense of understanding. A few months later you forget most of what your draw. There is widely accepted belief of 10,000-Hour rule with deliebreate practice to gain true mastery.

We are limited by time and energy in learning. We are also constrained by our biology in retaining and applying knowledge.

Is there a better way?

Large Language Model

LLM learn similar to human. We feed LLM massive data with the goal to minize the error when its doing the next word prediction.

LLM is not set out to model the whole reality. It had a moderate ambition of doing sequence to sequence transformation initially. But it turned out, by understanding the language purely statistically, it ended up being able to do amazing things. It knows the perfect grammar. It can do perfect summarization. It’s aware of the sentiment of the text. With the model scale, it shows lots of emergent capability. Geoffrey Hinton worried that an increasingly capable model will be able to draw an analogy from seemingly unrelated domains and thus outsmart humans.

In order to get most out of LLM, you will need to know how to ask2, or prompt effectively. However, lots of prompting technincal (e.g few shot chain of thoughts) will become unnecessary 3 with time pass by. It will become smarter and more accurate when the model scales4 or with new architecture that will beats the transformer.

It is almost beyond doubt that LLM will exponentially surpass humans in many tasks.

LLM Limitations

LLM, after being trained, comes with stale knowledge with a cut-off date. Unlike humans, it won’t take action, reflect, and learn. To inject new knowledge, you can use retraining, fine-tuning, or the Retrieval Augmented Generation (RAG) method. However, it is cost-prohibitive for most users to retrain a base foundation model.

LLM can make things up. Unlike people who lies, LLM are not intentional. It just follows statistical information and generates the most plausible sequence of text. One technique to fix this is called grounding. One way of doing this is to provide additional information along with the query and ask LLM to provide a citation. This technique is the main structure of the RAG5.

LLM is only as good as the data it is trained on. Therefore, LLM will have biases, just as humans do. These biases will radiate to the users and society. The problem will exacerbate if we rely on only one model and increasingly view the LLM as the source of knowledge. So, it is important and inevitable, due to both economical and geopolitical reasons, to have more than one LLM competing in the future. There should be no LLM monopoly. Open source is key.

LLM is not great at reasoning. Reasoning is based on logical, conceptual, or theoretical principles. These are not LLM’s strong suits. Techniques such as chain of thoughts aim to guide LLM to break a complex problem into smaller steps, sometimes providing some examples (called few-shot prompting). Reasoning ability is key to implementing an intelligent agent - who is capable of independently achieving the goal, often by repeating reasoning, action, and observing the impact made through action.

LLM is not great at tasks that require determinism or interacting with the external world. Function calling is a new capability that OpenAI provides to augment the capability of LLM. Users first register these functions or tools with LLM. When users pass questions or send commands to LLM, it parses the command, selects the right one, provides the command with the correct parameters, and the agent will invoke the tools to take actions. LLM acts as a tool orchestrator. In other words, it is an agent or a bot capable of taking natural language as input and executing tasks to achieve the goal. Language agent is one of the hot research topics. Popular frameworks for building agents include Langchain Agent and Microsoft Semantic Kernel.

Take advatange of LLM. It is still a tool, so far.

  1. Declarative knowledge in semantic memory, procedural knowledge in procedural memory and, your personal expericen in the episodic memory which you can still feel vivid after many years. ↩

  2. Humans long emphasized the importance of asking questions (compared with talking). But most people suck. People in general talk too much than they should, out of ego, insecurity or loneliess. ↩

  3. Nowadays, ChatGPT can solve the examples used by the Chain of Thought technique without any chain of thought prompting techniques. It has either been intergrated into the model it self or through the API you are intercting with. ↩

  4. The scaling laws of LLM indicate that the performance of LLM will increase with the size of the model and the size of the training data. ↩

  5. Check out my toy project that can setup a local rag dev enviroment ↩