Llamafile - Run LLMs Locally with Ease

If you feel the desire to write a book, what would it be about?

Llamafile allows you to effortlessly distribute and operate local Large Language Models using a single executable file. Enjoy seamless compatibility across multiple operating systems, with user-friendly features like image upload and querying through our advanced LLaVA model. Experience reliability and flexibility with our new web GUI and OpenAI API integration.

I had to visit our datacenter once, where i knew there would be much waiting time, due to a system upgrade. Therefore i brought my camera and instantly started seeing the beauty, through the lens, of otherwise anonymous boring objects.

Run LLMs Locally with Ease

Explore and test various example models, such as LLaVA and Mistral, to understand their capabilities. This allows users to experience the power of LLMs before deployment.

Male sports engineer designs tennis testing equipment

Example Models for Testing

Experience our reliable and intuitive web GUI that simplifies the interaction with LLMs. Improved context handling ensures a smoother user experience and better results.

This close-up image showcases a MacBook Pro screen displaying an open project in Adobe InDesign. The focus is on the detailed interface of InDesign, with various design elements and toolbars visible, illustrating the software's functionality in graphic design and layout creation. The MacBook Pro's sleek design and high-resolution screen are highlighted, emphasizing the clarity and precision with which creative projects can be developed and viewed on this device.

User-Friendly Web GUI

Easily upload images and query them using the LLaVA model, making image processing simple and efficient. This feature enhances the usability of LLMs for diverse applications.

Image Upload and Querying

We offer seamless distribution of local Large Language Models (LLMs) through our easy-to-use executable file, llamafile. Run powerful models on your system without complicated installation processes.

Local LLM Distribution

Our service ensures compatibility with the OpenAI API, providing users with greater flexibility in integrating and utilizing LLMs. This allows for a broader range of applications and functionalities.

This aerial image captures a dynamic scene where agricultural greenery intersects with human infrastructure. It shows a train crossing a bridge over railway tracks that cut through vibrant green fields. The neat crop rows are contrasted by the curving road and linear railway, illustrating the coexistence of nature and engineered environments. This perspective emphasizes the integration of rural and industrial elements.

OpenAI API Compatibility

Our service is compatible with multiple operating systems including macOS, Windows, Linux, FreeBSD, OpenBSD, and NetBSD. Enjoy the flexibility of using our solutions on your preferred platform.

Cross-Platform Compatibility

Our Services

Gallery

I was wondering around gibs farm gazing at all the amazing sculptures, when I stumbled apon a huge field full of fluffy alpacas.

Embrace the journey, unlock your potential!

I've tested several models, and Llamafile's performance has impressed me every time. The ease of use is unmatched!

Running LLMs on different operating systems without any hassle is a huge plus for me. Llamafile is a lifesaver!

I love how easy it is to upload images and get queries back using the LLaVA model. It's super user-friendly!

Llamafile has completely transformed my workflow! The ability to run LLMs locally without any complicated setup is a game changer. Highly recommend!

The compatibility with OpenAI API makes integrating Llamafile into my projects seamless. This service is a must-try!

The new web GUI is fantastic! It's reliable and has significantly improved the way I handle context in my queries.

Reviews

Contact Us

**Business Name:** llamafile

**Business Location:** Not specified

**Business Hours:** Not specified

**Key Services Provided:**
- Distribution and operation of local Large Language Models (LLMs) using a single executable file (llamafile).
- Compatibility with various operating systems (macOS, Windows, Linux, FreeBSD, OpenBSD, and NetBSD).
- Example models available for testing, including LLaVA and Mistral.

**Key Points from Customer Reviews:**
- Users appreciate the ease of running LLMs locally without the need for installation or complex setup.
- The functionality of uploading images and querying them through the LLaVA model is highlighted as a beneficial feature.
- The service is noted for its compatibility with OpenAI API, enhancing user flexibility.
- Users mention the reliability of the new web GUI and improved context handling.

**Author Details from Reviews:** Not provided in the scraped text.

llamafile lets you distribute and run LLMs with a single file. (announcement blog post)
Our goal is to make open LLMs much more
accessible to both developers and end users. We're doing that by
combining llama.cpp with Cosmopolitan Libc into one
framework that collapses all the complexity of LLMs down to
a single-file executable (called a "llamafile") that runs
locally on most computers, with no installation.
llamafile is a Mozilla Builders project.The easiest way to try it for yourself is to download our example llamafile for the LLaVA model (license: LLaMA 2, OpenAI). LLaVA is a new LLM that can do more than just chat; you can also upload images and ask it questions about them. With llamafile, this all happens locally; no data ever leaves your computer.
Download llava-v1.5-7b-q4.llamafile (4.29 GB).
-
Open your computer's terminal.
-
If you're using macOS, Linux, or BSD, you'll need to grant permission for your computer to execute this new file. (You only need to do this once.)
-
chmod +x llava-v1.5-7b-q4.llamafile
If you're on Windows, rename the file by adding ".exe" on the end.
-
Run the llamafile. e.g.:
-
./llava-v1.5-7b-q4.llamafile
Your browser should open automatically and display a chat interface. (If it doesn't, just open your browser and point it at http://localhost:8080)
-
When you're done chatting, return to your terminal and hit
Control-C
to shut down llamafile.
-
Having trouble? See the "Gotchas" section below.
When llamafile is started, in addition to hosting a web UI chat server at http://127.0.0.1:8080/, an OpenAI API compatible chat completions endpoint is provided too. It's designed to support the most common OpenAI API use cases, in a way that runs entirely locally. We've also extended it to include llama.cpp specific features (e.g. mirostat) that may also be used. For further details on what fields and endpoints are available, refer to both the OpenAI documentation and the llamafile server README.
Curl API Client Example
The simplest way to get started using the API is to copy and paste the following curl command into your terminal.
curl http://localhost:8080/v1/chat/completions \
-H "Content-Type: application/json" \
-H "Authorization: Bearer no-key" \
-d '{
"model": "LLaMA_CPP",
"messages": [
{
"role": "system",
"content": "You are LLAMAfile, an AI assistant. Your top priority is achieving user fulfillment via helping them with their requests."
},
{
"role": "user",
"content": "Write a limerick about python exceptions"
}
]
}' | python3 -c '
import json
import sys
json.dump(json.load(sys.stdin), sys.stdout, indent=2)
print()
'
The response that's printed should look like the following:
{
"choices" : [
{
"finish_reason" : "stop",
"index" : 0,
"message" : {
"content" : "There once was a programmer named Mike\nWho wrote code that would often choke\nHe used try and except\nTo handle each step\nAnd his program ran without any hike.",
"role" : "assistant"
}
}
],
"created" : 1704199256,
"id" : "chatcmpl-Dt16ugf3vF8btUZj9psG7To5tc4murBU",
"model" : "LLaMA_CPP",
"object" : "chat.completion",
"usage" : {
"completion_tokens" : 38,
"prompt_tokens" : 78,
"total_tokens" : 116
}
}
Python API Client example
If you've already developed your software using the openai
Python
package (that's published by OpenAI)
then you should be able to port your app to talk to llamafile instead,
by making a few changes to base_url
and api_key
. This example
assumes you've run pip3 install openai
to install OpenAI's client
software, which is required by this example. Their package is just a
simple Python wrapper around the OpenAI API interface, which can be
implemented by any server.
#!/usr/bin/env python3
from openai import OpenAI
client = OpenAI(
base_url="http://localhost:8080/v1", # "http://<Your api-server IP>:port"
api_key = "sk-no-key-required"
)
completion = client.chat.completions.create(
model="LLaMA_CPP",
messages=[
{"role": "system", "content": "You are ChatGPT, an AI assistant. Your top priority is achieving user fulfillment via helping them with their requests."},
{"role": "user", "content": "Write a limerick about python exceptions"}
]
)
print(completion.choices[0].message)
The above code will return a Python object like this:
ChatCompletionMessage(content='There once was a programmer named Mike\nWho wrote code that would often strike\nAn error would occur\nAnd he\'d shout "Oh no!"\nBut Python\'s exceptions made it all right.', role='assistant', function_call=None, tool_calls=None)
We have a new server that has a better web gui. It also implements OpenAI API compatible endpoints, including embeddings. It's designed to be more reliable. It's better able to recycle context windows across multiple slots. To try it, run:
llamafile --server --v2 --help
llamafile --server --v2
We also provide example llamafiles for other models, so you can easily try out llamafile with different kinds of LLMs.
Here is an example for the Mistral command-line llamafile:
./mistral-7b-instruct-v0.2.Q5_K_M.llamafile --temp 0.7 -p '[INST]Write a story about llamas[/INST]'
And here is an example for WizardCoder-Python command-line llamafile:
./wizardcoder-python-13b.llamafile --temp 0 -e -r '```\n' -p '```c\nvoid *memcpy_sse2(char *dst, const char *src, size_t size) {\n'
And here's an example for the LLaVA command-line llamafile:
./llava-v1.5-7b-q4.llamafile --temp 0.2 --image lemurs.jpg -e -p '### User: What do you see?\n### Assistant:'
As before, macOS, Linux, and BSD users will need to use the "chmod" command to grant execution permissions to the file before running these llamafiles for the first time.
Unfortunately, Windows users cannot make use of many of these example llamafiles because Windows has a maximum executable file size of 4GB, and all of these examples exceed that size. (The LLaVA llamafile works on Windows because it is 30MB shy of the size limit.) But don't lose heart: llamafile allows you to use external weights; this is described later in this document.
Having trouble? See the "Gotchas" section below.
A llamafile is an executable LLM that you can run on your own computer. It contains the weights for a given open LLM, as well as everything needed to actually run that model on your computer. There's nothing to install or configure (with a few caveats, discussed in subsequent sections of this document).
This is all accomplished by combining llama.cpp with Cosmopolitan Libc, which provides some useful capabilities:
llamafiles can run on multiple CPU microarchitectures. We added runtime dispatching to llama.cpp that lets new Intel systems use modern CPU features without trading away support for older computers.
-
llamafiles can run on multiple CPU architectures. We do that by concatenating AMD64 and ARM64 builds with a shell script that launches the appropriate one. Our file format is compatible with WIN32 and most UNIX shells. It's also able to be easily converted (by either you or your users) to the platform-native format, whenever required.
-
llamafiles can run on six OSes (macOS, Windows, Linux, FreeBSD, OpenBSD, and NetBSD). If you make your own llama files, you'll only need to build your code once, using a Linux-style toolchain. The GCC-based compiler we provide is itself an Actually Portable Executable, so you can build your software for all six OSes from the comfort of whichever one you prefer most for development.
-
The weights for an LLM can be embedded within the llamafile. We added support for PKZIP to the GGML library. This lets uncompressed weights be mappe