AI ML: Script AI Clients
AI-first map for choosing the built-in EyeAuras AI SDK/runtime first and using direct provider SDKs only when intentionally bypassing it.
Script AI Clients Discovery Map
Reference map for script mini-apps that need AI chat, vision, tool-calling, or
agent workflows. Start with the built-in EyeAuras.AI runtime and add EyeAuras.AI.Desktop for desktop integration. Direct provider clients are an explicit alternative when you need to own the request pipeline.
User Intents
- Use the built-in EyeAuras AI SDK/runtime from a script or mini-app.
- Call a chat/vision/OCR model through configured EyeAuras AI profiles.
- Send a captured image to an AI model.
- Register AI-callable tools through the existing Semantic Kernel-based infrastructure.
- Decide when a script should intentionally bypass the built-in AI SDK with a direct provider NuGet.
- Keep API keys and model configuration outside hardcoded source when possible.
Concept Model
EyeAuras.AIis the non-desktop runtime project;EyeAuras.AI.Desktopadds Windows integration.EyeAuras.AISdkreferences both as a compatibility aggregate. Project names do not establish published NuGet availability.- Built-in EyeAuras AI runtime is the app subsystem for chat sessions, profiles, retrieval, Semantic Kernel plugins, tool approval, MCP/Codex surfaces, and reusable chat UI.
- Use the built-in runtime as the default path for app-integrated AI. It already owns profile selection, key resolution, tool policy, session state, retrieval, and UI/event plumbing.
- External AI client NuGets are ordinary script dependencies used directly by a mini-app when the script deliberately bypasses the built-in runtime.
- External AI client assemblies are not default host/script references. Add
them explicitly with
#r "nuget:..."or project references when a script wants to call a provider SDK directly. - Direct clients are an escape hatch for provider-specific features, experiments, or standalone calls that should not participate in EyeAuras AI profiles, plugins, retrieval, or chat UI.
- Captured screen/window images usually come from CV/image APIs before being attached to an AI session or converted to provider-specific image input.
Invoke a built-in tool locally
This top-level EyeAuras script assumes compatible EyeAuras.AI and Semantic Kernel references have already been prepared. The host supplies Log and cancellationToken:
using EyeAuras.AI;
using EyeAuras.AI.PluginHost;
using Microsoft.SemanticKernel;
var host = new AiPluginHost();
host.AddOrUpdatePlugin("hello", new AiHelloWorldPlugin());
try
{
var result = await host.InvokeToolAsync(
"hello", "echo_with_time",
new KernelArguments { ["request"] = "Hello" },
cancellationToken: cancellationToken);
Log.Info(result.Succeeded ? $"Response: {result.Value}" : result.Error);
}
finally
{
host.RemovePlugin("hello");
}
The log receives the echoed text and host time. No provider key is needed; this is not a model conversation and does not apply session approval policy. For a real assistant, follow the session creation operation, including AddAiCoreServices, configuration, engine injection and engine-owned disposal. Editor factories/configurators come from the editor context, not arbitrary script globals.
Common Flow: Built-in AI SDK
- Prepare compatible host/project references to
EyeAuras.AIand the desktop layer when required; verify the actual assemblies and dependencies. - Resolve AI services through the script/project host surface, usually
AiEngine,IAiProfileRepository, session configurators, or UI factories. - Create an
IAiSession/IAiChatSessionthroughAiEngine. - Use configured
AiModelProfileentries instead of hardcoded provider clients and keys. - Attach
AiKernelPlugintools, retrieval plugins, or context snapshots when the workflow needs them. - Send turns through the session and render events with
AiChatViewModel/AiChatComponentwhen UI is needed. - Dispose session handles and subscriptions with script lifetime.
Common Flow: Direct Provider Escape Hatch
- Confirm the built-in AI SDK/runtime does not cover the required provider feature or experiment.
- Add the provider client explicitly with
#r "nuget:..."or a project reference. - Keep provider API keys in variables, config providers, user settings, or secret storage.
- Capture or receive inputs from EyeAuras APIs, then convert them to the provider-specific request format.
- Dispose provider clients, subscriptions, and buffers with script lifetime.
Prefer
- Prefer built-in
EyeAuras.AISdk/EyeAuras.AIruntime for AI chat, model profiles, key resolution, Semantic Kernel tools, retrieval, MCP/Codex, app-integrated assistants, and reusable chat UI. - Prefer
AiEngine,IAiProfileRepository,AiKernelPlugin,AiChatViewModel, andAiChatComponentover direct provider clients when building on EyeAuras AI infrastructure. - Use direct provider NuGets only when the script intentionally owns provider versioning, authentication, request shape, and response parsing.
- Prefer storing API keys in variables, config providers, user settings, or secret storage rather than hardcoding.
- Prefer bounded image sizes and compressed formats for vision calls.
- Prefer separating capture logic from AI-client request logic.
Avoid
- Avoid defaulting to direct provider SDKs when the built-in AI SDK can satisfy the workflow.
- Avoid bypassing EyeAuras profiles, key resolution, tool approval, and session state for app-integrated AI.
- Avoid confusing external provider SDKs with
EyeAuras.AISdk/EyeAuras.AIruntime services. - Avoid hardcoding real API keys in scripts or docs.
- Avoid sending large raw images when compressed bytes are sufficient.
- Avoid blocking UI/render loops while waiting for network calls.
- Avoid assuming provider SDK versions and model names stay current forever.
Research Anchors
EyeAuras.AISdkEyeAuras.AIAiEngineIAiProfileRepositoryIAiChatSessionAiModelProfileAiKernelPluginAiChatViewModelAiChatComponentMicrosoft.SemanticKernelKernelFunctionAttributeOpenAIClientOpenAI.ChatChatClientChatMessageChatMessageContentPartBinaryDataToJpegDataIImageSearchTriggerWindowImageProcessedEventArgs
Search Synonyms
- EyeAuras AISdk
- built-in AI SDK
- built-in AI runtime
- external AI client
- direct OpenAI NuGet
- chat client
- vision model
- GPT OCR
- image to text
- screenshot OCR
- direct provider SDK escape hatch
- AI profile
- AI plugin
Related Maps
- ml/ai.md
- ml/ai-runtime.md
- ml/ai-kernel-plugins.md
- computer-vision/images.md
- scripting/references-and-resources.md
Assemblies and direct provider clients
At source revision c8b2304, EyeAuras.AI targets net8.0 and EyeAuras.AI.Desktop targets Windows; both set IsPackable=false. EyeAuras.AISdk references both projects. Do not assume every project name is downloadable through #r nuget; match actual assemblies to your installed host and exported project.
The historical vision example separately pins OpenAI 2.1.0 and Emgu.CV 4.9.0.5494, JPEG format and image ownership. Its package signatures were inspected; it does not claim a successful provider request. Its historical model is not a recommendation for a new project.