> ## Documentation Index
> Fetch the complete documentation index at: https://docs.liquid.ai/llms.txt
> Use this file to discover all available pages before exploring further.
# Messages & Content
> ChatMessage, ChatMessageContent, audio format requirements — same shape on every platform.
`ChatMessage` and `ChatMessageContent` mirror the OpenAI chat-completions message schema. Both are declared once in `commonMain` (`data class ChatMessage`, `sealed class ChatMessageContent`) and Kotlin/Native + SKIE bridge the Kotlin types into Swift — there are no separate "native" Swift declarations.
## `ChatMessage`
The Swift class is generated from the Kotlin `data class`. Kotlin parameter defaults don't propagate, so the primary init requires all four arguments explicitly:
```swift theme={"theme":{"light":"github-light","dark":"github-dark"}}
public class ChatMessage {
public var role: ChatMessage.Role
public var content: [ChatMessageContent]
public var reasoningContent: String?
public var functionCalls: [LeapFunctionCall]?
// Primary init — pass `reasoningContent: nil, functionCalls: nil` for ordinary messages.
public init(
role: ChatMessage.Role,
content: [ChatMessageContent],
reasoningContent: String?,
functionCalls: [LeapFunctionCall]?
)
// Secondary inits (from Kotlin secondary constructors):
public init(role: ChatMessage.Role, content: ChatMessageContent) // single content
public init(role: ChatMessage.Role, textContent: String) // plain text
public enum Role {
case system, user, assistant, tool
}
}
```
```kotlin theme={"theme":{"light":"github-light","dark":"github-dark"}}
@Serializable(with = ChatMessageJsonSerializer::class)
data class ChatMessage(
val role: Role,
val content: List,
@SerialName("reasoning_content") val reasoningContent: String? = null,
@SerialName("tool_calls") val functionCalls: List? = null,
) {
// Single-content secondary ctor (wraps the part in a list, drops defaults).
constructor(role: Role, content: ChatMessageContent)
// Plain-text secondary ctor (parameter name is `textContent`).
constructor(role: Role, textContent: String)
enum class Role(val type: String) {
SYSTEM("system"),
USER("user"),
ASSISTANT("assistant"),
TOOL("tool");
companion object {
fun fromTypeString(type: String): Role // throws LeapSerializationException on unknown values
}
}
}
```
`ChatMessage` is `@Serializable` via the dedicated `ChatMessageJsonSerializer` — encode/decode through `kotlinx.serialization.json.Json` rather than ad-hoc `JSONObject` helpers. See [Utilities → Serialization](./utilities#serialization).
### Fields
* **`role`** — the speaker (`user`, `system`, `assistant`, or `tool`). Use `tool` when appending function-call results back into the history.
* **`content`** — ordered fragments. Supported part types: `Text`, `Image` (JPEG bytes wrapped in a data URL), `Audio` (WAV bytes or `input_audio` payload), and on Kotlin `AudioPcmF32` for raw float samples.
* **`reasoningContent`** — text emitted by reasoning models inside `` / `` tags. `null` for non-reasoning responses.
* **`functionCalls`** — calls returned by `MessageResponse.FunctionCalls` on the previous turn, included when appending tool-call results to history.
### Serialization
Round-trip the message through `kotlinx.serialization` — there is no separate "from `[String: Any]`" initializer on either platform.
Encode with `LeapJson.encodeToString` (or your own `JSONEncoder` against the OpenAI shape) and decode with the matching Kotlin serializer. See [Utilities → Serialization](./utilities#serialization) for examples that route through `LeapJson`.
`ChatMessage` is `@Serializable`. Encode with `Json.encodeToString(message)` and decode with `Json.decodeFromString(jsonString)` — see [Utilities → Serialization](./utilities#serialization). On error, expect a `LeapSerializationException` (not `LeapSerializationError`).
## `ChatMessageContent`
`ChatMessageContent` is the Kotlin `sealed class` bridged to Swift — switch on its subclasses with SKIE's `onEnum(of:)` helper. There is no native Swift `enum`, no positional `.image(_:)` / `.audio(_:)` factory, and no `init(from json:)`. Use the static factories on the Swift overlay:
```swift theme={"theme":{"light":"github-light","dark":"github-dark"}}
// Text (cross-platform):
ChatMessageContent.text(_ text: String) -> ChatMessageContent
// Image:
ChatMessageContent.fromJPEGData(_ jpegData: Data) -> ChatMessageContent.Image
ChatMessageContent.image(url: String) -> ChatMessageContent.Image // data URL or remote URL
// Audio:
ChatMessageContent.fromWAVData(_ wavData: Data) -> ChatMessageContent.Audio
ChatMessageContent.audio(data: Data, format: String = "wav") -> ChatMessageContent.Audio
ChatMessageContent.fromFloatSamples(_ samples: [Float], sampleRate: Int, channelCount: Int = 1)
-> ChatMessageContent.Audio
// iOS only — UIKit:
public static func fromUIImage(_ image: UIImage) throws -> ChatMessageContent
// (JPEG quality is fixed at 0.85; no compressionQuality parameter is exposed.)
```
`fromUIImage` is iOS-only and takes only the image — JPEG compression quality is hard-coded to `0.85` in the overlay (`leap-sdk/src/iosMain/.../ChatMessageContentExtensionsIos.kt`). There is no `fromNSImage` factory; on macOS, convert your `NSImage` to JPEG `Data` yourself and pass it through `fromJPEGData(_:)`.
On the wire, image parts are encoded as OpenAI-style `image_url` payloads (with a `data:image/jpeg;base64,...` URL) and audio parts as `input_audio` arrays with Base64 data.
```kotlin theme={"theme":{"light":"github-light","dark":"github-dark"}}
sealed class ChatMessageContent {
data class Text(val text: String) : ChatMessageContent()
data class Image(val imageUrl: ImageUrl) : ChatMessageContent() {
// Convenience secondary ctor — wraps the bytes in a data: URL.
constructor(jpegByteArray: ByteArray)
val jpegByteArray: ByteArray // derived property: decodes the data: URL
// Nested wrapper for the OpenAI `image_url` wire shape.
data class ImageUrl(val url: String)
}
data class Audio(val inputAudio: InputAudio) : ChatMessageContent() {
// Convenience secondary ctor — wraps the bytes in an InputAudio.
constructor(data: ByteArray)
val data: ByteArray // derived property: decodes the base64 InputAudio payload
data class InputAudio(val data: String, val format: String) // base64-encoded `data`
}
// Convenience helpers (declared on the sealed class) wrap raw PCM into Audio:
fun toWavBytes(): ByteArray // on AudioPcmF32 — encodes float samples as 16-bit PCM WAV
fun toAudio(): Audio // on AudioPcmF32 — same bytes wrapped as ChatMessageContent.Audio
data class AudioPcmF32(val samples: FloatArray, val sampleRate: Int) : ChatMessageContent()
}
```
Serialize via `kotlinx.serialization` (every variant is `@Serializable`).
Android-specific helper: `ImageUtils.fromBitmap(bitmap, compressionQuality = 85)` (in `ai.liquid.leap.message`) re-encodes an Android `Bitmap` to JPEG and returns a `ChatMessageContent.Image`. It's a `suspend` function — call it from a coroutine.
```kotlin theme={"theme":{"light":"github-light","dark":"github-dark"}}
import ai.liquid.leap.message.ImageUtils
val image: ChatMessageContent.Image = ImageUtils.fromBitmap(bitmap, compressionQuality = 85)
```
* **`Text`** — plain text fragment.
* **`Image`** — JPEG-encoded image bytes. Only vision-capable models can interpret image parts.
* **`Audio`** — WAV-encoded audio bytes (see [audio format requirements](#audio-format-requirements) below).
* **`AudioPcmF32`** (Kotlin) — raw float32 mono PCM in memory. Avoids the WAV encoding step when you already have samples; the engine handles framing internally. Kotlin-only.
* **`fromFloatSamples(...)` (Swift)** — convenience that wraps `[Float]` samples into a `ChatMessageContent.Audio` WAV blob (via `FloatAudioBuffer.makeAudioContent()`). Different from Kotlin's `AudioPcmF32`: this one DOES re-encode through WAV. There is no Swift surface for raw `AudioPcmF32` today.
## Audio format requirements
The LEAP inference engine expects WAV-encoded audio with these specifications:
| Property | Required value | Notes |
| --------------- | -------------------- | ------------------------------------ |
| **Container** | WAV (RIFF) | Only WAV is supported |
| **Sample rate** | 16000 Hz recommended | Other rates auto-resampled to 16 kHz |
| **Encoding** | PCM | Float32, Int16, Int24, or Int32 |
| **Channels** | Mono (1) | Stereo is **rejected** |
| **Byte order** | Little-endian | Standard WAV |
**Supported PCM encodings**
* **Float32** — 32-bit floating point, normalized to \[-1.0, 1.0]
* **Int16** — 16-bit signed integer (recommended)
* **Int24** — 24-bit signed integer
* **Int32** — 32-bit signed integer
The engine **only accepts WAV**. M4A, MP3, AAC, OGG, and other compressed formats are rejected. Convert to WAV before sending.
**Mono required.** Stereo or multi-channel WAVs are rejected with an error. Downmix to mono first.
**Automatic resampling.** The engine resamples to 16 kHz when needed, but providing 16 kHz audio directly avoids the resampling overhead. For best quality, record at 16 kHz mono.
## Creating audio content
### From a WAV file
```swift theme={"theme":{"light":"github-light","dark":"github-dark"}}
let wavURL = Bundle.main.url(forResource: "audio", withExtension: "wav")!
let wavData = try Data(contentsOf: wavURL)
let message = ChatMessage(
role: .user,
content: [
.text("What is being said in this audio?"),
ChatMessageContent.fromWAVData(wavData)
],
reasoningContent: nil,
functionCalls: nil
)
```
```kotlin theme={"theme":{"light":"github-light","dark":"github-dark"}}
val wavBytes = File("/path/to/audio.wav").readBytes()
val audio = ChatMessageContent.Audio(wavBytes)
val message = ChatMessage(
role = ChatMessage.Role.USER,
content = listOf(
ChatMessageContent.Text("What is being said in this audio?"),
audio
)
)
```
### From raw PCM samples
```swift theme={"theme":{"light":"github-light","dark":"github-dark"}}
// Float samples normalized to [-1.0, 1.0]
let samples: [Float] = [0.1, 0.2, 0.15, -0.3 /* ... */]
let audioContent = ChatMessageContent.fromFloatSamples(
samples,
sampleRate: 16000,
channelCount: 1
)
let message = ChatMessage(
role: .user,
content: [.text("Transcribe this audio"), audioContent],
reasoningContent: nil,
functionCalls: nil
)
```
```kotlin theme={"theme":{"light":"github-light","dark":"github-dark"}}
import ai.liquid.leap.audio.FloatAudioBuffer
val audioBuffer = FloatAudioBuffer(sampleRate = 16000)
audioBuffer.add(floatArrayOf(0.1f, 0.2f, 0.15f /* ... */))
audioBuffer.add(floatArrayOf(0.3f, 0.25f /* ... */))
val wavBytes = audioBuffer.createWavBytes()
val audio = ChatMessageContent.Audio(wavBytes)
```
Or skip the WAV encoding entirely with `ChatMessageContent.AudioPcmF32(samples, sampleRate)` — the engine handles the framing internally and you save the WAV header overhead.
### Recording from the microphone
Configure `AVAudioRecorder` with WAV-compatible settings:
```swift theme={"theme":{"light":"github-light","dark":"github-dark"}}
import AVFoundation
let audioURL = FileManager.default.temporaryDirectory
.appendingPathComponent("recording.wav")
let settings: [String: Any] = [
AVFormatIDKey: kAudioFormatLinearPCM,
AVSampleRateKey: 16000.0, // 16 kHz
AVNumberOfChannelsKey: 1, // Mono
AVLinearPCMBitDepthKey: 16, // 16-bit
AVLinearPCMIsFloatKey: false,
AVLinearPCMIsBigEndianKey: false
]
let recorder = try AVAudioRecorder(url: audioURL, settings: settings)
recorder.record()
// ...
recorder.stop()
let wavData = try Data(contentsOf: audioURL)
let audioContent: ChatMessageContent = ChatMessageContent.fromWAVData(wavData)
```
Use `android.media.AudioRecord` or a library like `WaveRecorder`:
```kotlin theme={"theme":{"light":"github-light","dark":"github-dark"}}
import com.github.squti.androidwaverecorder.WaveRecorder
val recorder = WaveRecorder(outputFilePath)
recorder.configureWaveSettings {
sampleRate = 16000
channels = android.media.AudioFormat.CHANNEL_IN_MONO
audioEncoding = android.media.AudioFormat.ENCODING_PCM_16BIT
}
recorder.startRecording()
// ...
recorder.stopRecording()
val wavBytes = File(outputFilePath).readBytes()
val audioContent = ChatMessageContent.Audio(wavBytes)
```
Use `javax.sound.sampled.TargetDataLine`:
```kotlin theme={"theme":{"light":"github-light","dark":"github-dark"}}
import javax.sound.sampled.AudioFormat
import javax.sound.sampled.AudioSystem
import javax.sound.sampled.TargetDataLine
val format = AudioFormat(16000f, 16, 1, true, false)
val line = AudioSystem.getTargetDataLine(format)
line.open(format)
line.start()
// ... read into a byte buffer, wrap in WAV header ...
line.stop(); line.close()
val audioContent = ChatMessageContent.Audio(wavBytes)
```
For simpler cases, JVM apps that already have a `FloatAudioBuffer`-equivalent in their codebase can use `ChatMessageContent.AudioPcmF32(samples, sampleRate)` directly.
### Audio duration
* **Minimum** — at least 1 second of audio for reliable speech recognition.
* **Maximum** — bounded by the model's context window (typically several minutes).
* **Silence** — trim excessive silence from the start and end for better results.
## Audio output from models
Audio-capable models like `LFM2.5-Audio-1.5B` emit float32 PCM frames via `MessageResponse.AudioSample`. Output sample rate is typically **24 kHz** (vs. 16 kHz for input).
```swift theme={"theme":{"light":"github-light","dark":"github-dark"}}
for try await response in conversation.generateResponse(message: userMessage) {
if case .audioSample(let audio) = onEnum(of: response) {
// audio.samples: [Float] in [-1.0, 1.0]
// audio.sampleRate: Int (typically 24000 for audio-gen models)
audioPlayer.enqueue(samples: audio.samples, sampleRate: Int(audio.sampleRate))
}
}
```
```kotlin theme={"theme":{"light":"github-light","dark":"github-dark"}}
conversation.generateResponse(userMessage)
.onEach { response ->
if (response is MessageResponse.AudioSample) {
// response.samples: FloatArray in [-1.0, 1.0]
// response.sampleRate: Int (typically 24000)
audioBuffer.add(response.samples)
}
}
.collect()
```
Audio **input** should be 16 kHz; audio **output** from generation models is typically **24 kHz**. Configure your playback pipeline accordingly.