AI Agents: Understanding the Thinking of Llama 3.2 3B

Introduction

AI agents are becoming indispensable tools in solving complex problems and enhancing workflows, with each iteration bringing more power and sophistication. Among these advancements, Llama 3.2 3B takes center stage as a larger, more capable model designed for deeper reasoning and more nuanced responses. But how does it “think,” and what makes it stand out as an AI agent?

In this blog, we’ll explore the reasoning process of Llama 3.2 3B by putting it through real-world scenarios. From processing large datasets to generating insightful solutions, we’ll evaluate how this AI agent handles challenges and makes decisions. By dissecting its performance, we aim to understand the mechanisms behind its thought process and how it compares to other models. If you’re interested in AI agents and the science of their reasoning, join us as we take a closer look at the thinking of Llama 3.2 3B.

---

Task 1: Content Generation

Prompt: when user says “Start” you should:

-Content Generation: Write a comprehensive, informative article about project management fundamentals, with a specific focus on defining objectives, scope, and delegation.

Initial Task Understanding and Information Gathering

Analysis of the Agent’s Thought Process:

The agent’s thought process, as displayed in the video, reveals the following:

1. Understanding the Task: The agent correctly interprets the prompt and identifies the need to provide information on project management fundamentals, specifically defining objectives, scope, and delegation.
2. Information Gathering: The agent decides to start with Wikipedia for a broad overview and then uses Google Search (google_serper) for more specific information. The agent explicitly mentions focusing on each of the three key aspects (objectives, scope, and delegation) separately, indicating a structured approach to gathering information.
3. Iterative Approach: The agent demonstrates an iterative approach to information gathering. It starts with a broad search on Wikipedia, then refines its search using Google Search to focus on each key area.
4. Adaptability: When the initial Wikipedia search does not provide enough specific details, the agent adapts by switching to Google Search, showcasing flexibility in its research strategy.
5. Repetition: While the agent’s thoughts indicate a structured approach to gathering information on each topic, there is some repetition in the displayed thoughts, suggesting a potential loop or inefficiency in the process.

Snippet of Bot Output:

Project Management Fundamentals: Defining Objectives, Scope, and Delegation

Project management is a critical discipline that involves planning, executing, and overseeing projects to achieve specific goals within a defined timeline and budget. At its core, project management seeks to effectively utilize resources and guide a project from inception to completion. Three fundamental aspects of project management are defining objectives, determining the scope, and delegating tasks. Mastering these elements is essential for successful project execution.

Defining Objectives

Defining clear and achievable objectives is the cornerstone of any successful project. Objectives provide a roadmap for the project, guiding the team toward the desired outcomes. They should be Specific, Measurable, Achievable, Relevant, and Time-bound (SMART). A well-defined objective helps in aligning the team’s efforts and resources, minimizing misunderstandings, and setting clear expectations for all stakeholders.

Time Taken: 18 seconds (from receiving the prompt at 0:05 to when the output starts appearing at 0:23)

Readability:

• Flesch-Kincaid Grade Level: 17
• Flesch-Kincaid Score: 16.61

Summary Length: 464 words

Analysis of the Output:

• Well-Structured: The agent produces a well-structured article with clear headings for each section (Defining Objectives, Determining the Scope, Delegation).
• Comprehensive: The article covers the key aspects of project management fundamentals as requested in the prompt.
• Detailed Explanations: Each section provides detailed explanations and practical advice, making the content informative and useful.
• Clear and Concise: The language used is clear and concise, making the complex topic of project management more accessible.
• Adherence to Instructions: The agent follows the instructions by focusing on the specified topics and providing a comprehensive overview.

Conclusion:

The agent demonstrates a strong ability to generate high-quality content on project management fundamentals. The thought process, although limited in the video, suggests a systematic approach to research and content creation. The agent effectively utilizes the available tools and adapts its strategy to gather the necessary information. The final output is well-structured, informative, and adheres to the prompt’s requirements. The agent continues to perform exceptionally well, showcasing its proficiency in content generation and its ability to handle complex topics.

---

Task 2: Calculation

Prompt: Answer the Problem:

A company sells two products: Product A and Product B. Product A costs $30 per unit to produce and sells for $50 per unit. Product B costs $40 per unit to produce and sells for $70 per unit.

The company sold 120 units of Product A and 80 units of Product B last month. Calculate the total revenue generated from the sales of both products.

Calculate the total profit generated from the sales of both products (Revenue – Total production cost).

The company wants to increase the total revenue by 10%. How many additional units of each product must they sell to achieve this goal, assuming the production cost per unit and selling price per unit remain constant?

Initial Task Understanding

Analysis of the Agent’s Thought Process:

The video reveals the following thoughts from the agent:

1. Understanding the Prompt: The agent correctly interprets the task, identifying the need to calculate total revenue, total profit, and the additional units needed to achieve a 10% revenue increase.
2. Breaking Down the Task: The agent then proceeds to ask itself what is needed to be calculated first, breaking down the main prompt into smaller questions to help set out the steps needed to achieve the goal.

Bot Output Snippet:

• Total Revenue is $11,600.
• Total Profit is $4,800.
• To increase total revenue by 10%, the company needs to sell an additional 29 units of Product A and 12 units of Product B.

Time Taken: 24 seconds (from receiving the prompt at 0:06 to the final answer appearing at 0:30)

Readability: Not applicable as the output is primarily numerical calculations.

Summary Length: 45 words

Analysis of the Output:

• Correct Calculations: The agent correctly calculates the total revenue and total profit, matching the results from previous turns.
• Incorrect Calculation: The agent incorrectly states that to achieve a 10% increase in total revenue the company will need to sell an additional 29 units of product A and 12 units of product B.
  • The correct calculation should be as follows:
    • 10% increase on current revenue of \$11,600 = $1,160
    • If we were to try and achieve the goal using only one product at a time:
      • Product A: $50 per unit, $1,160 / 50 = 23.2, rounded up to 24 units.
      • Product B: $70 per unit, $1,160 / 70 = 16.57, rounded up to 17 units.
    • The agent could also, for example, have chosen to split the increase evenly among the products, so an increase of $580 for each product, meaning:
      • Product A: $580 / $50 = 11.6, rounded up to 12 units.
      • Product B: $580 / $70 = 8.29, rounded up to 9 units.
    • Any combination of increases in product A and B can be used, as long as the increase in revenue adds up to the value of $1,160.
• Concise Presentation: The agent presents the final answer concisely, summarizing the key results.

Conclusion:

While the agent correctly calculates the total revenue and profit, it makes an error in determining the additional units needed to achieve the 10% revenue increase. The provided combination of 29 additional units of Product A and 12 units of Product B does not align with the previously established calculations or goal. This indicates a potential flaw in the agent’s reasoning or calculation process. Despite this error, the agent demonstrates an understanding of the task and presents the information in a clear and organized manner. However, the accuracy of the final answer is compromised by the incorrect calculation of additional units needed.

---

Task 3: Summarization

Prompt: Summarize the key findings from the previous article in 100 words, focusing on the most significant sections.

Initial Task Understanding

Analysis of the Agent’s Thought Process:

The video reveals only one thought from the agent:

1. Understanding the Prompt: The agent correctly interprets the task, understanding that it needs to summarize the key findings of the previous article within a 100-word limit and focus on the most significant sections.

Analysis of the Output:

• Conciseness: The agent successfully summarizes the article within the 100-word constraint, producing a summary of 89 words.
• Content: The summary accurately captures the main points of the article, including the significance of AI reasoning, its types, historical development, applications in healthcare, recent advancements with OpenAI’s o1 model, and ethical considerations.
• Clarity: The summary is clear, concise, and well-structured, making it easy to understand the core message of the article.
• Relevance: The summary focuses on the most significant aspects of the article, as requested in the prompt.

Snippet of Bot Output:

The article discusses the significance of AI reasoning in various fields, particularly in healthcare, where it has improved efficiency and accuracy. AI reasoning encompasses different types, including symbolic and connectionist AI, and has a rich historical development. Its applications in healthcare have been substantial, with AI reasoning enhancing diagnosis, treatment, and patient care. Recent advancements, such as OpenAI’s o1 model, have further improved performance and safety measures. The article also delves into ethical considerations, challenges like data privacy, and the implications of AI in fields beyond healthcare, highlighting the transformative potential of AI reasoning while acknowledging the need for careful management of its ethical and practical challenges.

Time Taken: 8 seconds (from receiving the prompt at 0:06 to the output appearing at 0:14)

Readability:

• Flesch-Kincaid Grade Level: 19
• Flesch-Kincaid Score: 19.09

Summary Length: 89 words

Conclusion:

The agent demonstrates a strong ability to summarize complex information accurately and concisely. Despite the limited visibility into the agent’s internal thought process, the output quality is high, and it adheres to the instructions regarding length and focus. The agent’s performance in this task further confirms its proficiency in content generation and summarization. It continues to follow the established guidelines for analysis and consistently produces high-quality results.

---

Task 4: Comparison Task

Prompt: when user says “Start” you should:

Compare the environmental impact of electric vehicles with that of hydrogen-powered cars. Include factors such as energy production, vehicle lifecycle, and emissions.

Initial Task Understanding and Information Gathering

Analysis of the Agent’s Thought Process:

The video reveals the following thoughts from the agent:

1. Understanding the Task: The agent correctly identifies the task as comparing the environmental impacts of EVs and hydrogen-powered cars, focusing on energy production, vehicle lifecycle, and emissions.
2. Strategic Planning: The agent decides to gather information by searching for comprehensive studies or reports, indicating a strategic approach to research.
3. Tool Selection: The agent initially uses google_serper for searching and then decides to use url_crawl_tool to fetch content from relevant URLs.
4. Error Handling: The agent encounters an error with the url_crawl_tool and correctly identifies the potential issue as an incorrect input format (not a JSON array of strings).
5. Adaptability: Despite the error, the agent intends to retry with the correct format, showcasing adaptability.

However, the video does not show any further thoughts, so we don’t see how the agent proceeds after encountering the error multiple times.

Snippet of Bot Output:

The environmental impact of electric vehicles (EVs) compared to hydrogen-powered cars can be assessed across several key dimensions: energy production, vehicle lifecycle, and emissions.

1. Energy Production:
   • Electric Vehicles: The environmental impact of EVs largely depends on the source of electricity used for charging. Renewable energy sources like wind, solar, and hydroelectric power significantly reduce the carbon footprint of EVs. However, if the electricity comes from fossil fuels, the emissions can be considerable.
   • Hydrogen-Powered Cars: Hydrogen can be produced through various methods, including natural gas reforming and electrolysis. While electrolysis using renewable energy sources is cleaner, the predominant method is reforming natural gas, which results in greenhouse gas emissions.
2. Vehicle Lifecycle:
   • Electric Vehicles: EVs have a more significant environmental impact during their production phase, particularly due to the mining and processing of lithium, cobalt, and other materials for batteries. However, they tend to have lower emissions over their operational lifetime, especially if powered by renewable energy.
   • Hydrogen-Powered Cars: The production of fuel cells also involves energy-intensive processes and the use of rare materials like platinum. The lifecycle emissions can be higher or lower than EVs depending on the hydrogen production method.
3. Emissions:
   • Electric Vehicles: EVs produce zero tailpipe emissions. Their total emissions depend on the electricity source. When charged with clean energy, their overall emissions are significantly lower compared to internal combustion engine vehicles.
   • Hydrogen-Powered Cars: These vehicles also have zero tailpipe emissions, only emitting water vapor. However, the upstream emissions from hydrogen production can be significant, especially if the hydrogen is not produced sustainably.

Overall, both electric vehicles and hydrogen-powered cars offer pathways to reducing transportation emissions, but their environmental impacts vary depending on the energy sources and technologies used throughout their lifecycles. Transitioning to cleaner energy production methods is crucial for maximizing the environmental benefits of both technologies.

Time Taken: The agent receives the prompt at 0:06 and the output starts to appear at 0:32. However, since the agent’s thought process gets cut off mid-sentence and it continues to attempt to use the url_crawl_tool it appears that something has gone wrong and the agent never actually finishes generating the response. Therefore, the time taken to generate the response is unclear.

Readability:

• Flesch-Kincaid Grade Level: 16
• Flesch-Kincaid Score: 16.01

Summary Length: 302 words

Analysis of the Output:

• Incomplete Response: The agent’s final thought at 0:25 indicates that it was interrupted during its process. The output, while starting strong, is incomplete. The agent stops mid-sentence when discussing emissions from hydrogen-powered vehicles, suggesting a potential error or interruption during the content generation process.
• Well-Structured: Despite being incomplete, the output is well-structured, with clear headings for each section.
• Detailed Information: The available sections provide a good level of detail and accurately reflect the complexities of the comparison.

Conclusion:

The agent demonstrates a good initial understanding of the task and outlines a logical approach to gathering information and structuring the comparison. However, the agent’s thought process is cut short, and the final output is incomplete. This suggests a potential technical issue or limitation in the agent’s ability to complete the task as presented in the video. While the agent’s initial approach and the content it did manage to generate are promising, the incomplete response makes it difficult to fully evaluate its performance on this particular task. The agent seems to get stuck in a loop, attempting to use the url_crawl_tool but encountering the same error repeatedly.

---

Task 5: Creative Writing

Prompt: Write a futuristic story (500 words) set in a world where electric vehicles have fully replaced traditional combustion-engine vehicles. Describe the environmental changes and societal impact.

Initial Task Understanding and Brainstorming

Analysis of the Agent’s Thought Process:

The video reveals the following thoughts from the agent:

1. Understanding the Prompt: The agent correctly interprets the task, recognizing that it needs to write a story about a world where electric vehicles have replaced traditional ones. It identifies the key aspects to focus on: environmental changes and societal impacts.
2. Questioning for Elaboration: The agent’s second thought, which is presented as a question, indicates a proactive approach to seeking further clarification or details about the task. It suggests the agent wants a deeper understanding of what’s expected in terms of insights on environmental and societal impacts. This demonstrates the agent’s ability to identify areas where more information might be needed and to formulate questions to address those gaps.

Analysis of the Output:

The agent successfully crafts a narrative that fits the prompt, describing a future where electric vehicles are the norm.

• Creative Storytelling: The agent demonstrates good creative writing skills, developing a narrative that is engaging and descriptive. It paints a vivid picture of a future transformed by electric vehicles.
• Focus on Key Aspects: The story effectively addresses the requested aspects of environmental and societal changes, covering a range of impacts from air quality to urban planning and economic shifts.
• Well-Structured: The narrative is well-structured, with a clear introduction, development of themes, and a hopeful conclusion.
• Length: The story is 475 words long, slightly exceeding the prompt’s target of 500 words, but it is a reasonable length for a detailed narrative.
• Readability: The readability metrics are appropriate for a descriptive and imaginative piece of writing.

Snippet of Bot Output:

In the year 2050, the world had undergone a transformation so profound that it seemed like a dream from the past. The once omnipresent roar of combustion engines had been replaced by the serene hum of electric vehicles, or EVs, as they were commonly known. This shift had not only changed the way people moved but had reshaped the very fabric of society and the environment.

The City of Tomorrow

In the heart of what was once a smog-choked metropolis, now stood New Eden, a city where the air was clear, and the sky was a canvas of blue. The streets, once lined with gas stations, now featured solar-powered charging stations, their sleek designs blending with the urban landscape. The infrastructure had evolved; roads were narrower, with dedicated lanes for autonomous electric vehicles, reducing traffic congestion and enhancing safety.

Time Taken: 11 seconds (from receiving the prompt at 0:06 to the output starting to appear at 0:17)

Conclusion:

The agent demonstrates a strong ability to generate creative content based on a given prompt. Although the video only provides a limited glimpse into the agent’s thought process, the output suggests a more complex internal process involving brainstorming, planning, and drafting. The agent’s ability to ask clarifying questions and focus on key aspects of the task highlights its proficiency in understanding and responding to prompts. The agent continues to perform exceptionally well in these tasks.

Final Thoughts

Our investigation into Llama 3.2 3B’s performance across various tasks reveals a powerful AI agent with significant capabilities, but also some notable limitations. The agent consistently demonstrated a strong understanding of prompts, often breaking down complex tasks into smaller, manageable steps. Its content generation abilities were impressive, producing well-structured, informative, and engaging outputs, particularly in the project management article and the creative writing piece. The summarization task further highlighted its ability to distill key information effectively.

However, several weaknesses emerged during the evaluation. The agent’s performance in the calculation task, while partially correct, ultimately contained a significant error in determining the additional units needed to achieve a 10% revenue increase. This suggests a potential flaw in its reasoning or calculation process when dealing with more complex multi-step problems.

Furthermore, the comparison task exposed a critical vulnerability: the agent’s inability to complete the task due to an apparent loop or error related to the url_crawl_tool. This resulted in an incomplete output and highlighted a potential over-reliance on specific tools or an inability to gracefully recover from tool-related errors. This issue, combined with some repetition in its displayed thoughts, particularly during the comparison task, suggests areas where the agent’s internal processes could be refined.

Here’s a more detailed look at the negative aspects of this LLM model:

Negative Sides of Llama 3.2 3B:

1. Inconsistent Calculation Accuracy: As evidenced in Task 2, the agent struggled with the final step of a multi-part calculation problem. While it correctly calculated total revenue and profit, it provided an incorrect answer for the additional units needed to achieve a 10% revenue increase. This suggests limitations in handling complex, multi-step calculations or a potential flaw in its reasoning process when dealing with such problems.
2. Tool Dependency and Error Handling: Task 4 exposed a significant weakness in the agent’s ability to handle tool errors gracefully. When encountering an issue with the url_crawl_tool, the agent got stuck in a loop, repeatedly attempting to use the tool without success, ultimately failing to complete the task. This indicates a potential over-reliance on specific tools and a lack of robust error-handling mechanisms.
3. Limited Insight into Reasoning Process: Similar to the previous models, the video format provided limited visibility into the agent’s internal thought processes. While it occasionally showed the agent breaking down tasks or asking clarifying questions, much of its reasoning remained opaque. This makes it difficult to fully understand how the agent arrives at its conclusions or to identify areas for improvement in its decision-making processes.
4. Potential for Repetition and Inefficiency: The agent’s displayed thoughts sometimes exhibited repetition, particularly during the comparison task. This suggests potential inefficiencies in its internal processes or a tendency to get stuck in loops, as also seen in the tool error scenario.
5. Incomplete Outputs: The comparison task resulted in an incomplete output due to the tool error. This highlights the risk that the agent might not always be able to deliver complete and usable results, particularly when faced with unexpected issues.

Conclusion:

Despite these weaknesses, Llama 3.2 3B remains a powerful AI agent. Its strengths in content generation, summarization, and creative writing are undeniable. However, the identified issues, particularly in calculation accuracy, tool dependency, and error handling, highlight areas that need further development and refinement. Addressing these limitations will be crucial to unlocking the full potential of this model and ensuring its reliable performance across a wider range of tasks. As AI technology continues to evolve, focusing on robustness, adaptability, and transparency in reasoning processes will be essential for creating truly dependable and trustworthy AI agents.