{"id":9404,"date":"2024-10-11T17:26:04","date_gmt":"2024-10-11T17:26:04","guid":{"rendered":"https:\/\/www.webhi.com\/how-to\/?p=9404"},"modified":"2024-10-14T16:44:57","modified_gmt":"2024-10-14T16:44:57","slug":"simplified-tutorial-on-running-llms-llama-3-locally-with-llama-cpp","status":"publish","type":"post","link":"https:\/\/www.webhi.com\/how-to\/simplified-tutorial-on-running-llms-llama-3-locally-with-llama-cpp\/","title":{"rendered":"Simplified Tutorial on Running LLMs (Llama 3) Locally with llama.cpp"},"content":{"rendered":"\n<figure class=\"wp-block-image size-full\"><img decoding=\"async\" width=\"1200\" height=\"675\" src=\"https:\/\/www.webhi.com\/how-to\/gilrogre\/2024\/10\/llama_cpp_en.jpg\" alt=\"Run LLMs Locally llama.cpp Llama 3 Windows MAC OS\" class=\"wp-image-9416\" srcset=\"https:\/\/www.webhi.com\/how-to\/gilrogre\/2024\/10\/llama_cpp_en.jpg 1200w, https:\/\/www.webhi.com\/how-to\/gilrogre\/2024\/10\/llama_cpp_en-300x169.jpg 300w, https:\/\/www.webhi.com\/how-to\/gilrogre\/2024\/10\/llama_cpp_en-1024x576.jpg 1024w, https:\/\/www.webhi.com\/how-to\/gilrogre\/2024\/10\/llama_cpp_en-768x432.jpg 768w, https:\/\/www.webhi.com\/how-to\/gilrogre\/2024\/10\/llama_cpp_en-150x84.jpg 150w\" sizes=\"(max-width: 1200px) 100vw, 1200px\" \/><\/figure>\n\n\n\n<p>With the rise of open-source <strong>large language models (LLMs)<\/strong>, the ability to run them efficiently on local devices is becoming a game-changer. In this guide, we&#8217;ll dive into using&nbsp;<strong>llama.cpp<\/strong>, an open-source C++ library that allows you to run LLMs like&nbsp;<strong>Llama 3<\/strong>&nbsp;locally. Whether you&#8217;re a developer or a machine learning enthusiast, this step-by-step tutorial will help you get started with llama.cpp, an easy-to-install library that optimizes LLM inference on your hardware, whether it&#8217;s a desktop computer or cloud-based infrastructure.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\" id=\"what-you-need-to-get-started\"><strong>What You Need to Get Started<\/strong><\/h2>\n\n\n\n<p>Before diving into the specifics, it&#8217;s important to ensure your hardware is ready. To follow this guide, having at least 8 GB of VRAM is recommended. However, llama.cpp offers flexibility with optimizations, especially when it comes to model quantization, which we&#8217;ll cover in a bit.<\/p>\n\n\n\n<p>This tutorial works with models like&nbsp;<strong>Llama-3\u20138B-Instruct<\/strong>, but you can choose other models available from Hugging Face.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\" id=\"understanding-llamacpp\"><strong>Understanding llama.cpp<\/strong><\/h2>\n\n\n\n<p>So, what is&nbsp;<strong>llama.cpp<\/strong>? Essentially, it&#8217;s a lightweight C++ library designed to simplify the process of running LLMs locally. It allows for efficient inference on different hardware setups, from basic desktops to high-performance cloud servers.<\/p>\n\n\n\n<p>With llama.cpp, you\u2019ll benefit from several features:<\/p>\n\n\n\n<ul>\n<li><strong>Top Performance<\/strong>: It provides cutting-edge inference for large models.<\/li>\n\n\n\n<li><strong>Easy Setup<\/strong>: Installing llama.cpp is straightforward, requiring no external dependencies.<\/li>\n\n\n\n<li><strong>Quantization Support<\/strong>: Reduces model size by converting them to lower precision, helping run models on devices with limited memory.<\/li>\n\n\n\n<li><strong>Multi-platform Compatibility<\/strong>: It works on MacOS, Windows, and Linux, with support for Docker and FreeBSD.<\/li>\n\n\n\n<li><strong>Efficient Resource Use<\/strong>: llama.cpp takes full advantage of both CPU and GPU resources, making hybrid inference possible.<\/li>\n<\/ul>\n\n\n\n<h3 class=\"wp-block-heading\" id=\"challenges-you-might-face\"><strong>Challenges You Might Face<\/strong><\/h3>\n\n\n\n<p>While llama.cpp is an incredible tool, it\u2019s not without its challenges:<\/p>\n\n\n\n<ul>\n<li><strong>Sparse Documentation<\/strong>: Because it\u2019s open-source, the available documentation can sometimes be lacking. However, the active community is a great resource.<\/li>\n\n\n\n<li><strong>Technical Expertise<\/strong>: Setting up and running models, particularly with custom configurations, can be a bit complex, especially for users new to C++ or machine learning infrastructure.<\/li>\n<\/ul>\n\n\n\n<h2 class=\"wp-block-heading\" id=\"step-by-step-setup\"><strong>Step-by-Step Setup<\/strong><\/h2>\n\n\n\n<p>Let\u2019s go through the installation and setup process so you can get llama.cpp up and running with the&nbsp;<strong>Llama 3<\/strong>&nbsp;model.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\" id=\"1-cloning-the-repository\"><strong>1. Cloning the Repository<\/strong><\/h3>\n\n\n\n<p>First, you\u2019ll need to download the llama.cpp repository from GitHub. You can do this with the following commands:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code lang=\"bash\" class=\"language-bash\">$ git clone https:\/\/github.com\/ggerganov\/llama.cpp\n$ cd llama.cpp<\/code><\/pre>\n\n\n\n<h3 class=\"wp-block-heading\" id=\"2-building-llamacpp\"><strong>2. Building llama.cpp<\/strong><\/h3>\n\n\n\n<p>Next, you\u2019ll need to build the project. The build process will vary slightly depending on whether you&#8217;re using MacOS or Windows.<\/p>\n\n\n\n<p>For&nbsp;<strong>MacOS<\/strong>, with Metal support, you can run:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code lang=\"bash\" class=\"language-bash\">$ make llama-server<\/code><\/pre>\n\n\n\n<p>Alternatively, you can use CMake:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code lang=\"bash\" class=\"language-bash\">$ cmake -B build\n$ cmake --build build --config Release<\/code><\/pre>\n\n\n\n<p>For&nbsp;<strong>Windows<\/strong>, with CUDA support, the commands are slightly different:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code lang=\"bash\" class=\"language-bash\">C:\\Users\\Bob&gt; make llama-server LLAMA_CUDA=1<\/code><\/pre>\n\n\n\n<p>Or using CMake:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code lang=\"bash\" class=\"language-bash\">C:\\Users\\Bob&gt; cmake -B build -DLLAMA_CUDA=ON\nC:\\Users\\Bob&gt; cmake --build build --config Release<\/code><\/pre>\n\n\n\n<p>Or <strong>Download <\/strong>prebuilt release from the official repo:<\/p>\n\n\n\n<p><a href=\"https:\/\/github.com\/ggerganov\/llama.cpp\/releases\" target=\"_blank\" rel=\"noreferrer noopener\"><em>LLAMA CPP Releases Page<\/em><\/a><\/p>\n\n\n\n<p>Once completed, you\u2019re ready to start using the model.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\" id=\"downloading-and-preparing-the-model\"><strong>Downloading and Preparing the Model<\/strong><\/h2>\n\n\n\n<p>In this example, we\u2019ll use the&nbsp;<strong>Meta-Llama-3\u20138B-Instruct<\/strong>&nbsp;model, though you can adapt this for any model you prefer.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\" id=\"1-install-hugging-face-cli\"><strong>1. Install Hugging Face CLI<\/strong><\/h3>\n\n\n\n<p>To begin, install the Hugging Face command line interface:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code lang=\"bash\" class=\"language-bash\">$ pip install -U \"huggingface_hub[cli]\"<\/code><\/pre>\n\n\n\n<p>Create an account on Hugging Face if you don\u2019t have one already, and generate your access token from their settings page. You\u2019ll need this to access the models.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\" id=\"2-login-to-hugging-face\"><strong>2. Login to Hugging Face<\/strong><\/h3>\n\n\n\n<p>Log in with your Hugging Face credentials:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code lang=\"bash\" class=\"language-bash\">$ huggingface-cli login<\/code><\/pre>\n\n\n\n<p>Once you&#8217;re logged in, accept the terms for the&nbsp;<strong>Llama-3\u20138B-Instruct<\/strong>&nbsp;model and wait for access approval.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\" id=\"3-downloading-the-model\"><strong>3. Downloading the Model<\/strong><\/h3>\n\n\n\n<p>You have two main options when downloading the model:&nbsp;<strong>non-quantized<\/strong>&nbsp;or&nbsp;<strong>GGUF quantized<\/strong>.<\/p>\n\n\n\n<ul>\n<li><strong>Option 1: Non-quantized Model<\/strong><\/li>\n<\/ul>\n\n\n\n<p>To download the non-quantized model:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code lang=\"bash\" class=\"language-bash\">$ huggingface-cli download meta-llama\/Meta-Llama-3-8B-Instruct --exclude \"original\/*\"  --local-dir models\/Meta-Llama-3-8B-Instruct<\/code><\/pre>\n\n\n\n<p>After downloading, you&#8217;ll need to install Python dependencies and convert the model to GGUF format:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code lang=\"bash\" class=\"language-bash\">$ python -m pip install -r requirements.txt\n$ python convert-hf-to-gguf.py models\/Meta-Llama-3-8B-Instruct<\/code><\/pre>\n\n\n\n<ul>\n<li><strong>Option 2: Quantized Model<\/strong><\/li>\n<\/ul>\n\n\n\n<p>If you\u2019re working with hardware constraints, you can download a quantized version directly:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code lang=\"bash\" class=\"language-bash\">$ huggingface-cli download path_to_gguf_model --exclude \"original\/*\" --local-dir models\/Meta-Llama-3-8B-Instruct<\/code><\/pre>\n\n\n\n<h2 class=\"wp-block-heading\" id=\"using-quantization-for-hardware-optimization\"><strong>Using Quantization for Hardware Optimization<\/strong><\/h2>\n\n\n\n<p>Quantization allows you to run models on devices with lower memory, such as systems with less than 16 GB of VRAM. By reducing the precision of model weights from 16-bit to 4-bit, you can save a lot of memory without sacrificing too much performance.<\/p>\n\n\n\n<p>To quantize the model:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code lang=\"bash\" class=\"language-bash\">$ .\/llama-quantize .\/models\/Meta-Llama-3-8B-Instruct\/ggml-model-f16.gguf .\/models\/Meta-Llama-3-8B-Instruct\/ggml-model-Q4_K_M.gguf Q4_K_M<\/code><\/pre>\n\n\n\n<p>This will produce a quantized model that is ready for local inference.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\" id=\"running-llama-server\"><strong>Running llama-server<\/strong><\/h2>\n\n\n\n<p>Once your model is set up, you can launch the&nbsp;<strong>llama-server<\/strong>&nbsp;to handle HTTP requests, allowing you to interact with the model using standard APIs.<\/p>\n\n\n\n<p>For MacOS:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code lang=\"bash\" class=\"language-bash\">$ .\/llama-server -m models\/Meta-Llama-3-8B-Instruct\/ggml-model-Q4_K_M.gguf -c 2048<\/code><\/pre>\n\n\n\n<p>On Windows, use:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code lang=\"bash\" class=\"language-bash\">C:\\Users\\Bob&gt; llama-server.exe -m models\\\\Meta-Llama-3-8B-Instruct\\\\ggml-model-Q4_K_M.gguf -c 2048<\/code><\/pre>\n\n\n\n<p>Now, you can start sending requests to&nbsp;<code>http:\/\/localhost:8080<\/code>.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\" id=\"building-a-python-chatbot\"><strong>Building a Python Chatbot<\/strong><\/h2>\n\n\n\n<p>You can also create a simple Python chatbot to interact with your model. Here\u2019s a basic script to send requests to the llama-server:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code lang=\"python\" class=\"language-python\">import requests\nimport json\n\ndef get_response(server_url, messages, temperature=0.7, max_tokens=4096, stream=True):\n    data = {\n        \"messages\": messages,\n        \"temperature\": temperature,\n        \"max_tokens\": max_tokens,\n        \"stream\": stream,\n    }\n    response = requests.post(f\"{server_url}\/v1\/chat\/completions\", json=data)\n    return response.json()\n\ndef chatbot(server_url):\n    messages = [{\"role\": \"system\", \"content\": \"You are a helpful assistant.\"}]\n    while True:\n        user_input = input(\"You: \")\n        if user_input.lower() == \"exit\":\n            break\n        messages.append({\"role\": \"user\", \"content\": user_input})\n        print(\"Assistant: \", get_response(server_url, messages))\n\nif __name__ == \"__main__\":\n    chatbot(\"http:\/\/localhost:8080\")<\/code><\/pre>\n\n\n\n<p>This simple script creates a chatbot that communicates with the llama-server to generate responses.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\" id=\"conclusion\"><strong>Conclusion:<\/strong><\/h2>\n\n\n\n<p>Using&nbsp;<strong>llama.cpp<\/strong>&nbsp;to run large language models like&nbsp;<strong>Llama 3<\/strong>&nbsp;locally can be a powerful and efficient solution, especially when high-performance inference is required. Despite its complexity, llama.cpp provides flexibility with multi-platform support, quantization, and hardware acceleration. Although it may be challenging for newcomers, its growing community and extensive features make it a top choice for developers and researchers alike.<\/p>\n\n\n\n<p>For those willing to dive deep, llama.cpp offers endless possibilities for local and cloud-based LLM inference.<\/p>\n","protected":false},"excerpt":{"rendered":"<p>With the rise of open-source large language models (LLMs), the ability to run them efficiently on local devices is becoming a game-changer. In this guide, we&#8217;ll dive into using&nbsp;llama.cpp, an open-source C++ library that allows you to run LLMs like&nbsp;Llama 3&nbsp;locally. Whether you&#8217;re a developer or a machine learning enthusiast, this step-by-step tutorial will help ..<\/p>\n<div class=\"clear-fix\"><\/div>\n<p><a href=\"https:\/\/www.webhi.com\/how-to\/simplified-tutorial-on-running-llms-llama-3-locally-with-llama-cpp\/\" title=\"read more...\">Read more<\/a><\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":[],"categories":[281,53],"tags":[],"yoast_head":"\n<title>Simplified Tutorial on Running LLMs (Llama 3) Locally with llama.cpp - WebHi Tutorials &amp; Documentations<\/title>\n<meta name=\"description\" content=\"Learn how to run LLMs like Llama 3 locally with llama.cpp, offering efficient on-device inference for top-notch performance and minimal setup.\" \/>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/www.webhi.com\/how-to\/simplified-tutorial-on-running-llms-llama-3-locally-with-llama-cpp\/\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"Simplified Tutorial on Running LLMs (Llama 3) Locally with llama.cpp - WebHi Tutorials &amp; Documentations\" \/>\n<meta property=\"og:description\" content=\"Learn how to run LLMs like Llama 3 locally with llama.cpp, offering efficient on-device inference for top-notch performance and minimal setup.\" \/>\n<meta property=\"og:url\" content=\"https:\/\/www.webhi.com\/how-to\/simplified-tutorial-on-running-llms-llama-3-locally-with-llama-cpp\/\" \/>\n<meta property=\"og:site_name\" content=\"WebHi Tutorials &amp; Documentations\" \/>\n<meta property=\"article:publisher\" content=\"https:\/\/www.facebook.com\/webhi.technology\" \/>\n<meta property=\"article:published_time\" content=\"2024-10-11T17:26:04+00:00\" \/>\n<meta property=\"article:modified_time\" content=\"2024-10-14T16:44:57+00:00\" \/>\n<meta property=\"og:image\" content=\"https:\/\/www.webhi.com\/how-to\/gilrogre\/2024\/10\/llama_cpp_en.jpg\" \/>\n<meta name=\"author\" content=\"webhi\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:creator\" content=\"@WebHiTechnology\" \/>\n<meta name=\"twitter:site\" content=\"@WebHiTechnology\" \/>\n<meta name=\"twitter:label1\" content=\"Written by\" \/>\n\t<meta name=\"twitter:data1\" content=\"webhi\" \/>\n\t<meta name=\"twitter:label2\" content=\"Est. reading time\" \/>\n\t<meta name=\"twitter:data2\" content=\"5 minutes\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\/\/schema.org\",\"@graph\":[{\"@type\":\"TechArticle\",\"@id\":\"https:\/\/www.webhi.com\/how-to\/simplified-tutorial-on-running-llms-llama-3-locally-with-llama-cpp\/#article\",\"isPartOf\":{\"@id\":\"https:\/\/www.webhi.com\/how-to\/simplified-tutorial-on-running-llms-llama-3-locally-with-llama-cpp\/\"},\"author\":{\"name\":\"webhi\",\"@id\":\"https:\/\/www.webhi.com\/how-to\/#\/schema\/person\/b31e76e2311cdc0bb90f5e2733059a54\"},\"headline\":\"Simplified Tutorial on Running LLMs (Llama 3) Locally with llama.cpp\",\"datePublished\":\"2024-10-11T17:26:04+00:00\",\"dateModified\":\"2024-10-14T16:44:57+00:00\",\"mainEntityOfPage\":{\"@id\":\"https:\/\/www.webhi.com\/how-to\/simplified-tutorial-on-running-llms-llama-3-locally-with-llama-cpp\/\"},\"wordCount\":864,\"commentCount\":0,\"publisher\":{\"@id\":\"https:\/\/www.webhi.com\/how-to\/#organization\"},\"articleSection\":[\"Artificial Intelligence\",\"Windows system administration\"],\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"CommentAction\",\"name\":\"Comment\",\"target\":[\"https:\/\/www.webhi.com\/how-to\/simplified-tutorial-on-running-llms-llama-3-locally-with-llama-cpp\/#respond\"]}]},{\"@type\":\"WebPage\",\"@id\":\"https:\/\/www.webhi.com\/how-to\/simplified-tutorial-on-running-llms-llama-3-locally-with-llama-cpp\/\",\"url\":\"https:\/\/www.webhi.com\/how-to\/simplified-tutorial-on-running-llms-llama-3-locally-with-llama-cpp\/\",\"name\":\"Simplified Tutorial on Running LLMs (Llama 3) Locally with llama.cpp - WebHi Tutorials &amp; Documentations\",\"isPartOf\":{\"@id\":\"https:\/\/www.webhi.com\/how-to\/#website\"},\"datePublished\":\"2024-10-11T17:26:04+00:00\",\"dateModified\":\"2024-10-14T16:44:57+00:00\",\"description\":\"Learn how to run LLMs like Llama 3 locally with llama.cpp, offering efficient on-device inference for top-notch performance and minimal setup.\",\"breadcrumb\":{\"@id\":\"https:\/\/www.webhi.com\/how-to\/simplified-tutorial-on-running-llms-llama-3-locally-with-llama-cpp\/#breadcrumb\"},\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\/\/www.webhi.com\/how-to\/simplified-tutorial-on-running-llms-llama-3-locally-with-llama-cpp\/\"]}]},{\"@type\":\"BreadcrumbList\",\"@id\":\"https:\/\/www.webhi.com\/how-to\/simplified-tutorial-on-running-llms-llama-3-locally-with-llama-cpp\/#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"Home\",\"item\":\"https:\/\/www.webhi.com\/how-to\/\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"Simplified Tutorial on Running LLMs (Llama 3) Locally with llama.cpp\"}]},{\"@type\":\"WebSite\",\"@id\":\"https:\/\/www.webhi.com\/how-to\/#website\",\"url\":\"https:\/\/www.webhi.com\/how-to\/\",\"name\":\"WebHi Tutorials &amp; Documentations\",\"description\":\"System administration and knowledge base\",\"publisher\":{\"@id\":\"https:\/\/www.webhi.com\/how-to\/#organization\"},\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":{\"@type\":\"EntryPoint\",\"urlTemplate\":\"https:\/\/www.webhi.com\/how-to\/?s={search_term_string}\"},\"query-input\":\"required name=search_term_string\"}],\"inLanguage\":\"en-US\"},{\"@type\":\"Organization\",\"@id\":\"https:\/\/www.webhi.com\/how-to\/#organization\",\"name\":\"WebHi Technology\",\"url\":\"https:\/\/www.webhi.com\/how-to\/\",\"logo\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\/\/www.webhi.com\/how-to\/#\/schema\/logo\/image\/\",\"url\":\"https:\/\/www.webhi.com\/how-to\/gilrogre\/2022\/07\/logo.png\",\"contentUrl\":\"https:\/\/www.webhi.com\/how-to\/gilrogre\/2022\/07\/logo.png\",\"width\":288,\"height\":95,\"caption\":\"WebHi Technology\"},\"image\":{\"@id\":\"https:\/\/www.webhi.com\/how-to\/#\/schema\/logo\/image\/\"},\"sameAs\":[\"https:\/\/www.facebook.com\/webhi.technology\",\"https:\/\/twitter.com\/WebHiTechnology\"]},{\"@type\":\"Person\",\"@id\":\"https:\/\/www.webhi.com\/how-to\/#\/schema\/person\/b31e76e2311cdc0bb90f5e2733059a54\",\"name\":\"webhi\",\"image\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\/\/www.webhi.com\/how-to\/#\/schema\/person\/image\/\",\"url\":\"https:\/\/www.webhi.com\/how-to\/ahuphiph\/litespeed\/avatar\/e20da107d0f4c765ead2eef88ad019d8.jpg?ver=1783634435\",\"contentUrl\":\"https:\/\/www.webhi.com\/how-to\/ahuphiph\/litespeed\/avatar\/e20da107d0f4c765ead2eef88ad019d8.jpg?ver=1783634435\",\"caption\":\"webhi\"},\"sameAs\":[\"https:\/\/www.webhi.com\/how-to\"],\"url\":\"https:\/\/www.webhi.com\/how-to\/author\/webhi\/\"}]}<\/script>\n","yoast_head_json":{"title":"Simplified Tutorial on Running LLMs (Llama 3) Locally with llama.cpp - WebHi Tutorials &amp; Documentations","description":"Learn how to run LLMs like Llama 3 locally with llama.cpp, offering efficient on-device inference for top-notch performance and minimal setup.","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/www.webhi.com\/how-to\/simplified-tutorial-on-running-llms-llama-3-locally-with-llama-cpp\/","og_locale":"en_US","og_type":"article","og_title":"Simplified Tutorial on Running LLMs (Llama 3) Locally with llama.cpp - WebHi Tutorials &amp; Documentations","og_description":"Learn how to run LLMs like Llama 3 locally with llama.cpp, offering efficient on-device inference for top-notch performance and minimal setup.","og_url":"https:\/\/www.webhi.com\/how-to\/simplified-tutorial-on-running-llms-llama-3-locally-with-llama-cpp\/","og_site_name":"WebHi Tutorials &amp; Documentations","article_publisher":"https:\/\/www.facebook.com\/webhi.technology","article_published_time":"2024-10-11T17:26:04+00:00","article_modified_time":"2024-10-14T16:44:57+00:00","og_image":[{"url":"https:\/\/www.webhi.com\/how-to\/gilrogre\/2024\/10\/llama_cpp_en.jpg"}],"author":"webhi","twitter_card":"summary_large_image","twitter_creator":"@WebHiTechnology","twitter_site":"@WebHiTechnology","twitter_misc":{"Written by":"webhi","Est. reading time":"5 minutes"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"TechArticle","@id":"https:\/\/www.webhi.com\/how-to\/simplified-tutorial-on-running-llms-llama-3-locally-with-llama-cpp\/#article","isPartOf":{"@id":"https:\/\/www.webhi.com\/how-to\/simplified-tutorial-on-running-llms-llama-3-locally-with-llama-cpp\/"},"author":{"name":"webhi","@id":"https:\/\/www.webhi.com\/how-to\/#\/schema\/person\/b31e76e2311cdc0bb90f5e2733059a54"},"headline":"Simplified Tutorial on Running LLMs (Llama 3) Locally with llama.cpp","datePublished":"2024-10-11T17:26:04+00:00","dateModified":"2024-10-14T16:44:57+00:00","mainEntityOfPage":{"@id":"https:\/\/www.webhi.com\/how-to\/simplified-tutorial-on-running-llms-llama-3-locally-with-llama-cpp\/"},"wordCount":864,"commentCount":0,"publisher":{"@id":"https:\/\/www.webhi.com\/how-to\/#organization"},"articleSection":["Artificial Intelligence","Windows system administration"],"inLanguage":"en-US","potentialAction":[{"@type":"CommentAction","name":"Comment","target":["https:\/\/www.webhi.com\/how-to\/simplified-tutorial-on-running-llms-llama-3-locally-with-llama-cpp\/#respond"]}]},{"@type":"WebPage","@id":"https:\/\/www.webhi.com\/how-to\/simplified-tutorial-on-running-llms-llama-3-locally-with-llama-cpp\/","url":"https:\/\/www.webhi.com\/how-to\/simplified-tutorial-on-running-llms-llama-3-locally-with-llama-cpp\/","name":"Simplified Tutorial on Running LLMs (Llama 3) Locally with llama.cpp - WebHi Tutorials &amp; Documentations","isPartOf":{"@id":"https:\/\/www.webhi.com\/how-to\/#website"},"datePublished":"2024-10-11T17:26:04+00:00","dateModified":"2024-10-14T16:44:57+00:00","description":"Learn how to run LLMs like Llama 3 locally with llama.cpp, offering efficient on-device inference for top-notch performance and minimal setup.","breadcrumb":{"@id":"https:\/\/www.webhi.com\/how-to\/simplified-tutorial-on-running-llms-llama-3-locally-with-llama-cpp\/#breadcrumb"},"inLanguage":"en-US","potentialAction":[{"@type":"ReadAction","target":["https:\/\/www.webhi.com\/how-to\/simplified-tutorial-on-running-llms-llama-3-locally-with-llama-cpp\/"]}]},{"@type":"BreadcrumbList","@id":"https:\/\/www.webhi.com\/how-to\/simplified-tutorial-on-running-llms-llama-3-locally-with-llama-cpp\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https:\/\/www.webhi.com\/how-to\/"},{"@type":"ListItem","position":2,"name":"Simplified Tutorial on Running LLMs (Llama 3) Locally with llama.cpp"}]},{"@type":"WebSite","@id":"https:\/\/www.webhi.com\/how-to\/#website","url":"https:\/\/www.webhi.com\/how-to\/","name":"WebHi Tutorials &amp; Documentations","description":"System administration and knowledge base","publisher":{"@id":"https:\/\/www.webhi.com\/how-to\/#organization"},"potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/www.webhi.com\/how-to\/?s={search_term_string}"},"query-input":"required name=search_term_string"}],"inLanguage":"en-US"},{"@type":"Organization","@id":"https:\/\/www.webhi.com\/how-to\/#organization","name":"WebHi Technology","url":"https:\/\/www.webhi.com\/how-to\/","logo":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/www.webhi.com\/how-to\/#\/schema\/logo\/image\/","url":"https:\/\/www.webhi.com\/how-to\/gilrogre\/2022\/07\/logo.png","contentUrl":"https:\/\/www.webhi.com\/how-to\/gilrogre\/2022\/07\/logo.png","width":288,"height":95,"caption":"WebHi Technology"},"image":{"@id":"https:\/\/www.webhi.com\/how-to\/#\/schema\/logo\/image\/"},"sameAs":["https:\/\/www.facebook.com\/webhi.technology","https:\/\/twitter.com\/WebHiTechnology"]},{"@type":"Person","@id":"https:\/\/www.webhi.com\/how-to\/#\/schema\/person\/b31e76e2311cdc0bb90f5e2733059a54","name":"webhi","image":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/www.webhi.com\/how-to\/#\/schema\/person\/image\/","url":"https:\/\/www.webhi.com\/how-to\/ahuphiph\/litespeed\/avatar\/e20da107d0f4c765ead2eef88ad019d8.jpg?ver=1783634435","contentUrl":"https:\/\/www.webhi.com\/how-to\/ahuphiph\/litespeed\/avatar\/e20da107d0f4c765ead2eef88ad019d8.jpg?ver=1783634435","caption":"webhi"},"sameAs":["https:\/\/www.webhi.com\/how-to"],"url":"https:\/\/www.webhi.com\/how-to\/author\/webhi\/"}]}},"_links":{"self":[{"href":"https:\/\/www.webhi.com\/how-to\/wp-json\/wp\/v2\/posts\/9404"}],"collection":[{"href":"https:\/\/www.webhi.com\/how-to\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.webhi.com\/how-to\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.webhi.com\/how-to\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/www.webhi.com\/how-to\/wp-json\/wp\/v2\/comments?post=9404"}],"version-history":[{"count":4,"href":"https:\/\/www.webhi.com\/how-to\/wp-json\/wp\/v2\/posts\/9404\/revisions"}],"predecessor-version":[{"id":9480,"href":"https:\/\/www.webhi.com\/how-to\/wp-json\/wp\/v2\/posts\/9404\/revisions\/9480"}],"wp:attachment":[{"href":"https:\/\/www.webhi.com\/how-to\/wp-json\/wp\/v2\/media?parent=9404"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.webhi.com\/how-to\/wp-json\/wp\/v2\/categories?post=9404"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.webhi.com\/how-to\/wp-json\/wp\/v2\/tags?post=9404"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}