AI ML: Script AI Clients

AI-first map for choosing the built-in EyeAuras AI SDK/runtime first and using direct provider SDKs only when intentionally bypassing it.

Script AI Clients Discovery Map

Reference map for script mini-apps that need AI chat, vision, tool-calling, or agent workflows. Start with the built-in EyeAuras.AI runtime and add EyeAuras.AI.Desktop for desktop integration. Direct provider clients are an explicit alternative when you need to own the request pipeline.

User Intents

  • Use the built-in EyeAuras AI SDK/runtime from a script or mini-app.
  • Call a chat/vision/OCR model through configured EyeAuras AI profiles.
  • Send a captured image to an AI model.
  • Register AI-callable tools through the existing Semantic Kernel-based infrastructure.
  • Decide when a script should intentionally bypass the built-in AI SDK with a direct provider NuGet.
  • Keep API keys and model configuration outside hardcoded source when possible.

Concept Model

  • EyeAuras.AI is the non-desktop runtime project; EyeAuras.AI.Desktop adds Windows integration. EyeAuras.AISdk references both as a compatibility aggregate. Project names do not establish published NuGet availability.
  • Built-in EyeAuras AI runtime is the app subsystem for chat sessions, profiles, retrieval, Semantic Kernel plugins, tool approval, MCP/Codex surfaces, and reusable chat UI.
  • Use the built-in runtime as the default path for app-integrated AI. It already owns profile selection, key resolution, tool policy, session state, retrieval, and UI/event plumbing.
  • External AI client NuGets are ordinary script dependencies used directly by a mini-app when the script deliberately bypasses the built-in runtime.
  • External AI client assemblies are not default host/script references. Add them explicitly with #r "nuget:..." or project references when a script wants to call a provider SDK directly.
  • Direct clients are an escape hatch for provider-specific features, experiments, or standalone calls that should not participate in EyeAuras AI profiles, plugins, retrieval, or chat UI.
  • Captured screen/window images usually come from CV/image APIs before being attached to an AI session or converted to provider-specific image input.

Invoke a built-in tool locally

This top-level EyeAuras script assumes compatible EyeAuras.AI and Semantic Kernel references have already been prepared. The host supplies Log and cancellationToken:

csharp
using EyeAuras.AI;
using EyeAuras.AI.PluginHost;
using Microsoft.SemanticKernel;

var host = new AiPluginHost();
host.AddOrUpdatePlugin("hello", new AiHelloWorldPlugin());
try
{
    var result = await host.InvokeToolAsync(
        "hello", "echo_with_time",
        new KernelArguments { ["request"] = "Hello" },
        cancellationToken: cancellationToken);
    Log.Info(result.Succeeded ? $"Response: {result.Value}" : result.Error);
}
finally
{
    host.RemovePlugin("hello");
}

The log receives the echoed text and host time. No provider key is needed; this is not a model conversation and does not apply session approval policy. For a real assistant, follow the session creation operation, including AddAiCoreServices, configuration, engine injection and engine-owned disposal. Editor factories/configurators come from the editor context, not arbitrary script globals.

Common Flow: Built-in AI SDK

  1. Prepare compatible host/project references to EyeAuras.AI and the desktop layer when required; verify the actual assemblies and dependencies.
  2. Resolve AI services through the script/project host surface, usually AiEngine, IAiProfileRepository, session configurators, or UI factories.
  3. Create an IAiSession / IAiChatSession through AiEngine.
  4. Use configured AiModelProfile entries instead of hardcoded provider clients and keys.
  5. Attach AiKernelPlugin tools, retrieval plugins, or context snapshots when the workflow needs them.
  6. Send turns through the session and render events with AiChatViewModel / AiChatComponent when UI is needed.
  7. Dispose session handles and subscriptions with script lifetime.

Common Flow: Direct Provider Escape Hatch

  1. Confirm the built-in AI SDK/runtime does not cover the required provider feature or experiment.
  2. Add the provider client explicitly with #r "nuget:..." or a project reference.
  3. Keep provider API keys in variables, config providers, user settings, or secret storage.
  4. Capture or receive inputs from EyeAuras APIs, then convert them to the provider-specific request format.
  5. Dispose provider clients, subscriptions, and buffers with script lifetime.

Prefer

  • Prefer built-in EyeAuras.AISdk / EyeAuras.AI runtime for AI chat, model profiles, key resolution, Semantic Kernel tools, retrieval, MCP/Codex, app-integrated assistants, and reusable chat UI.
  • Prefer AiEngine, IAiProfileRepository, AiKernelPlugin, AiChatViewModel, and AiChatComponent over direct provider clients when building on EyeAuras AI infrastructure.
  • Use direct provider NuGets only when the script intentionally owns provider versioning, authentication, request shape, and response parsing.
  • Prefer storing API keys in variables, config providers, user settings, or secret storage rather than hardcoding.
  • Prefer bounded image sizes and compressed formats for vision calls.
  • Prefer separating capture logic from AI-client request logic.

Avoid

  • Avoid defaulting to direct provider SDKs when the built-in AI SDK can satisfy the workflow.
  • Avoid bypassing EyeAuras profiles, key resolution, tool approval, and session state for app-integrated AI.
  • Avoid confusing external provider SDKs with EyeAuras.AISdk / EyeAuras.AI runtime services.
  • Avoid hardcoding real API keys in scripts or docs.
  • Avoid sending large raw images when compressed bytes are sufficient.
  • Avoid blocking UI/render loops while waiting for network calls.
  • Avoid assuming provider SDK versions and model names stay current forever.

Research Anchors

  • EyeAuras.AISdk
  • EyeAuras.AI
  • AiEngine
  • IAiProfileRepository
  • IAiChatSession
  • AiModelProfile
  • AiKernelPlugin
  • AiChatViewModel
  • AiChatComponent
  • Microsoft.SemanticKernel
  • KernelFunctionAttribute
  • OpenAIClient
  • OpenAI.Chat
  • ChatClient
  • ChatMessage
  • ChatMessageContentPart
  • BinaryData
  • ToJpegData
  • IImageSearchTrigger
  • WindowImageProcessedEventArgs

Search Synonyms

  • EyeAuras AISdk
  • built-in AI SDK
  • built-in AI runtime
  • external AI client
  • direct OpenAI NuGet
  • chat client
  • vision model
  • GPT OCR
  • image to text
  • screenshot OCR
  • direct provider SDK escape hatch
  • AI profile
  • AI plugin

Assemblies and direct provider clients

At source revision c8b2304, EyeAuras.AI targets net8.0 and EyeAuras.AI.Desktop targets Windows; both set IsPackable=false. EyeAuras.AISdk references both projects. Do not assume every project name is downloadable through #r nuget; match actual assemblies to your installed host and exported project.

The historical vision example separately pins OpenAI 2.1.0 and Emgu.CV 4.9.0.5494, JPEG format and image ownership. Its package signatures were inspected; it does not claim a successful provider request. Its historical model is not a recommendation for a new project.