[WIP] Agent API integration

[pblazej]

• Extracts many parts of AppViewModel into Conversation
  • Simplifies internal dependencies, etc. so that the app is mostly a wrapper around (extracted) Conversation - ready to be consumed by UI
• Extracts some first UI components like ChatScrollView

Minimal app example (voice agent)

import LiveKitComponents

@main
struct VoiceAgentApp: App {
    @StateObject private var conversation = Conversation(credentials: SandboxTokenServer(id: sandboxId))

    var body: some Scene {
        WindowGroup {
            MinimalExample()
                .environmentObject(conversation)
                .environmentObject(LocalMedia(conversation: conversation))
                .environment(\.agentName, "blaze") // optional, for multiple agents
        }
    }
}

struct MinimalExample: View {
    @LKConversation private var conversation
    @LKAgent private var agent
    // or filter by name for a specific agent
    // @LKAgent(named: "blaze") private var agentBlaze
    @LKLocalMedia private var localMedia
    
    var body: some View {
        if conversation.isReady {
            VStack {
                if let agent {
                    BarAudioVisualizer(agent: agent)
                }
                // Optionally, add chat
                ChatScrollView { message in
                    switch message.content {
                    case .agentTranscript(let text): Text(text).foregroundStyle(Color.red)
                    case .userTranscript(let text), .userInput(let text): Text(text)
                    }
                }
                AsyncButton {
                     await localMedia.toggleMicrophone()
                } label: {
                    Text(localMedia.isMicrophoneEnabled ? "Mute" : "Unmute")
                }
                AsyncButton {
                    await conversation.end()
                    conversation.restoreMessageHistory([])
                } label: {
                    Text("End")
                }
            }
        } else {
            AsyncButton {
                await conversation.start()
            } label: {
                Text("Start")
            }
        }
    }
}

[pblazej]

As for known issues: the visualizer for avatars probably won't work in the "minimal" case, due to wrong order of fallbacks, will fix this soon - let's focus on the public APIs.

[1egoman]

Left some notes - I think some of the discussion on the web / react pull request also applies here too, and we just need to come to some consistent decisions both implementations can adopt.

[1egoman]

Naming note: In the react / web implementation, I called this isAvailable. We probably should standardize on one name in both implementations - I don't have a strong preference which one though. Are you attached to either one?

[pblazej]

Yeah, I changed that in one of the last commits - still pretty confusing with the preconnect thing, I'm also thinking of renaming connectionState → roomConnectionState.

[pblazej]

@1egoman thank you for digging into that (again) 😄 IMO the areas of improvement are still the same (not that many of them). Let it serve as our bullet points for the meeting.

[pblazej]

@lukasIO @1egoman recent improvements and notable mentions:

• moved Message creation, split userInput from userTranscript
• removed Features in favor of app-level FeatureSwitches
• exposed _room
• added CredentialsProvider, let's try to finalize that in the web pr first 🤞
• moved all media/track related stuff to what's called DeviceSwitcher for now (better name LocalMedia ?)
• added agentName "handling" - that should be enough to "wait for this particular agent"

[pblazej]

@lukasIO re: separate "media/devices hook" - I updated the minimal example above 🫴

  MinimalExample()
                .environmentObject(DeviceSwitcher(room: room))
                .environmentObject(AgentSession(credentials: Sandbox(id: "your-id"), room: room))

The key is both must point to the same Room (created externally, no way to make it implicit). Also, you gotta think about initialization (cannot instantiate prop with another prop basically).

The only solution it to make them coupled, so that the Room is passed implicitly like that d8a2e04. I think it's fairly easy to understand and harder to break.

[bcherry]

this is a really nice simplification

[bcherry]

you should include some notes about alternatives and also update the readme instructions to match the changes

[pblazej]

yep, let's try to merge the pyramid of PRs first, I need to bump sdk+components anyway here 👍