How to Select SBC (Lattepanda/Raspberry Pi) for Local LLM (LLaMA, LLaMA2, Phi-2, Mixtral-MOE, etc.)

When selecting a single-board computer (SBC) for a local large language model (LLM), several factors must be considered, including performance, resource requirements, hardware features, and budget. This article will introduce how to choose between Lattepanda 3 Delta, Lattepanda Sigma, Raspberry Pi 4, and Raspberry Pi 5 to meet your LLM application needs.

Hardware Comparison

Lattepanda 3 Delta and Lattepanda Sigma utilize Intel processors based on the x86 architecture, offering higher performance. Raspberry Pi 4 and Raspberry Pi 5 use processors based on the ARM architecture, providing lower performance suitable for lightweight LLM tasks.

1. Lattepanda 3 Delta is a powerful single-board computer equipped with an x86 architecture Intel processor:

• Supports Windows 10 & 11, Linux, Android x86 OS systems
• Compatible with 200+ sensors and actuators

2. Lattepanda Sigma features a more robust hardware configuration capable of handling more complex tasks and applications:

• Supports Windows 10 & 11, Linux
• Similar to Lattepanda 3 Delta in terms of software and hardware ecosystem, with community and developer support, and abundant tutorials and third-party software libraries available.

3. Raspberry Pi 4B is a popular ARM architecture single-board computer:

• Supports Raspberry Pi OS, Ubuntu
• Offers a wide range of software and tools from the Raspberry Pi official software repository and other third-party software sources
• Many sensors and hardware manufacturers have already adapted their products for its hardware.

4. Raspberry Pi 5:

• Supports Raspberry Pi OS, Ubuntu
• Compared to Raspberry Pi 4, the hardware ecosystem of Raspberry Pi 5 is less mature; however, manufacturers, communities, and developers will provide longer-term software and hardware updates and support for Raspberry Pi 5.

Resource and Memory Constraints

LLM usually requires a large amount of memory to store model parameters and intermediate calculation results. For example, a model with billions of parameters may require tens of gigabytes or more of memory. On limited system memory, you can consider using model compression techniques, and quantization to reduce the size of the model, and ensure that the selected SBC has sufficient memory capacity to be able to load and run the LLM model.

A powerful CPU is crucial to handle the inference and training of LLM, and hardware accelerators such as GPU can significantly improve the training and inference speed of LLM. Although it is possible to run LLM with only SBC's CPU, its performance may not be comparable to that of a GPU or dedicated acceleration hardware. Therefore, when choosing between an SBC and an LLM, make sure its CPU performance is powerful enough to handle the computational load required by the LLM.

SBC with smaller memory may not be able to simultaneously store the LLM model and its parameters, conversation history, input data, and intermediate results during inference. After multiple rounds of dialogue, the memory has been exhausted, which may cause the LLM program to crash.

Lower resource and memory constraints may cause LLM to perform slower inference when processing long texts. CPU with smaller memory faces multiple performance bottlenecks when processing LLMs, and these bottlenecks work together to slow down token processing.

Considering the memory and storage requirements of LLMs, Lattepanda Sigma typically offers larger memory and storage capacities, making it better suited to support LLM operations. Raspberry Pi 4 and Raspberry Pi 5, on the other hand, have relatively smaller memory and storage capacities, requiring adaptation for LLMs with smaller memory, such as phi-2.

Deployment Discrepancies

We utilize the LLaMA.cpp and CPU for LLM inference. For LLM original model files in .pth format, they need to be quantized into GGUF format before running on the CPU. Considering the memory constraints of SBCs, we quantize GGUF model files into int4 format. For phi-2, an original model of approximately 6GB is reduced to only 1.6GB after Q4 quantization.

For LattePanda 3 Delta, Raspberry Pi 4, and Raspberry Pi 5, once you have selected a suitable LLM, due to memory limitations, you need to first download the LLM original model on another Linux PC and perform quantization before copying the quantized model to the SBC for LLM execution. However, on Lattepanda Sigma, you can directly download the LLM model and perform quantization.

LLM token speed

Real-time Performance:

Assessing the SBC's performance in handling real-time language tasks, including response time and processing latency.

Benchmark for LP 3 delta， LP Sigma， Raspberry Pi 4B， Raspberry Pi 5

Comparing SBCs with the same LLM model as the standard, it is evident from the above table that Lattepanda Sigma is significantly better than the other SBC.

Test for Raspberry Pi 5 (8GB) & LLM

For specific deployment steps, please refer to the following:

Deploy and run LLM on Raspberry Pi 5 vs Raspberry Pi 4B (LLaMA, LLaMA2, Phi-2, Mixtral-MOE, mamba-gp

Test for Raspberry Pi 4B (8GB) & LLM

For specific deployment steps, please refer to the following:

Deploy and run LLM on Raspberry Pi 4B (LLaMA, Alpaca, LLaMA2, ChatGLM)

Test for Lattepanda 3 Delta 864 (8GB) & LLM

For specific deployment steps, please refer to the following:

Deploy and run LLM on Lattepanda 3 Delta 864 (LLaMA, LLaMA2, Phi-2, ChatGLM2)

Test for Lattepanda Sigma (32GB) CPU & LLM

For specific deployment steps, please refer to the following:

Deploy and run LLM on LattePanda Sigma (LLaMA, Alpaca, LLaMA2, ChatGLM)

Summary

In summary, when selecting a SBC suitable for local LLMs, several factors need consideration.

In terms of hardware configuration, Lattepanda 3 Delta is equipped with an Intel Celeron N5105 processor, 8GB memory, and 64GB eMMC storage. Lattepanda Sigma features a more potent Intel Core i5-1340P processor, supporting up to 32GB LPDDR5 memory and M.2 NVMe/SATA SSD storage. Raspberry Pi 4B and Raspberry Pi 5 are equipped with Broadcom BCM2711 and BCM2712 processors, offering relatively smaller memory and storage capacities.

Considering resource and memory constraints, LLMs typically demand substantial memory to store model parameters and intermediate computation results. Therefore, it's crucial to ensure that the chosen SBC has a sufficiently large memory capacity. Due to its expandable memory and storage capacity, Lattepanda Sigma can better support LLM operations compared to Raspberry Pi 4B and Raspberry Pi 5, which may require adaptation for LLM models with smaller memory footprints.

Regarding deployment, Lattepanda Sigma allows direct download and quantization of LLM models. However, for Lattepanda 3 Delta, Raspberry Pi 4, and Raspberry Pi 5, you need to download the LLM original model on another Linux PC and perform quantization before transferring the quantized model to the SBC for execution.

Finally, model inference speed is a crucial performance metric for assessing SBCs. According to test results, Raspberry Pi 5 exhibits a significant improvement in processing speed compared to Raspberry Pi 4B. Particularly noteworthy is the outstanding performance of phi-2-Q4 on Raspberry Pi 5, with an evaluation time speed of 5.13 tokens/s. However, due to RAM capacity limitations, both Raspberry Pi 5 and Raspberry Pi 4B may still encounter constraints when processing large-scale LLMs. Lattepanda 3 Delta demonstrates slightly better LLM performance than Raspberry Pi 4B and Raspberry Pi 5. Nonetheless, Lattepanda Sigma provides higher performance, achieving speeds of up to 6 tokens/s when running llama2-7b-Q4, meeting the requirements of applications demanding more from LLMs, albeit at a higher price, making it suitable for those with larger budgets.

Running LLMs can lead to high CPU loads, generating considerable heat. It's essential to ensure that the SBC has adequate cooling measures in place to prevent overheating and maintain system stability. Lattepanda 3 Delta includes an active cooling system with a small fan to help dissipate heat, while users may need to ensure sufficient ventilation space to prevent overheating. Lattepanda Sigma comes equipped with a cooling fan to maintain proper processor temperature under load. Raspberry Pi 4B utilizes the Broadcom BCM2711 SoC and, although lacking a built-in fan, users can purchase official or third-party heat sinks, especially when running high-load applications or operating in high-temperature environments.