AI OSD: Selection
AI-first map of OSD-based selection services and selected window data.
OSD Selection Discovery Map
Reference map for interactive on-screen selection: window, region, point, or color picking.
Drawing persistent annotations belongs to osd/screen-overlay.md.
User Intents
- Let the user pick a target window.
- Pick a screen region.
- Pick a screen point.
- Pick a color under cursor.
- Use selected window PID for CV, memory, or input workflows.
Concept Model
IOsdSelectionServiceis the user-facing picker.- Selection is interactive and temporary.
PickWindow()returnsIWindowHandle.- Region/point/color picking can be scoped to a target window.
- Region selection can provide screen/window coordinate transforms.
API Details
IOsdSelectionServicePickWindow()PickRegion(IWindowHandle targetWindow = default)PickPoint(IWindowHandle targetWindow = default)PickColor(IWindowHandle targetWindow = default)
OsdRegionSelectionResult- selected region, transform matrices, and window under cursor.OsdRegionTransformationMatrices-ScreenToWindowandWindowToScreen.OsdPointSelectionResult-Point,Matrices,WindowUnderCursor.OsdColorSelectionResult-Color,Point,Matrices,WindowUnderCursor.
Pick a window and a region
Run this top-level EyeAuras C# Script. An ordinary helper receives IOsdSelectionService explicitly:
using EyeAuras.Shared.Osd;
var selection = GetService<IOsdSelectionService>();
var window = await selection.PickWindow();
var result = await selection.PickRegion(window);
Log.Info($"Window: {window.Title}; region: {result.Region}");
Choose the window with the left mouse button, then select a rectangle. The log shows the chosen region. The service activates the supplied window before the second selection. WindowUnderCursor is sampled during selection and is separate from the supplied target window.
Right-click or Escape cancels selection and cancels the awaited task. These methods take no CancellationToken. A concurrent request on the same service throws InvalidOperationException. The selection window hides after completion or cancellation.
Pick a screen point and color
Run this as a separate C# Script:
using System;
using EyeAuras.Shared.Osd;
var selection = GetService<IOsdSelectionService>();
try
{
var point = await selection.PickPoint();
Log.Info($"Point: {point.Point}");
var underCursor = point.WindowUnderCursor;
Log.Info(underCursor == null || !underCursor.IsWindow
? "Window under cursor unavailable"
: $"Window under cursor: {underCursor.Title}, PID: {underCursor.ProcessId}");
var color = await selection.PickColor();
var c = color.Color;
Log.Info($"RGB({c.R}, {c.G}, {c.B}), HEX #{c.R:X2}{c.G:X2}{c.B:X2}; point: {color.Point}");
}
catch (OperationCanceledException)
{
Log.Info("Selection cancelled by user");
}
Without targetWindow, the matrices are identity and coordinates are screen coordinates. Color selection samples a desktop snapshot captured at the start of that selection; it is not a continuous CV capture.
All three result structures expose WindowUnderCursor. It is borrowed, may be null or stale, and is not the supplied target window. Supplying a target changes the coordinate origin to that window’s DwmFrameBounds, not its client area. These matrices do not include OCR capture/ROI transforms.
Coordinate contract and version
In EyeAuras 20fd3d424d886f6c99b979536a148774d289d110, matrix names disagree with the directions described in XML. The implementation builds ScreenToWindow with a positive translation by the target frame origin and WindowToScreen as its inverse. Returned Region and Point use WindowToScreen, subtracting the origin. For frame origin (100, 200), screen point (130, 240) becomes (30, 40); do not subtract the origin again. This does not promise client coordinates or DPI correction. Check implementation when changing versions.
Selection contracts · Selection implementation
Prefer
- Prefer
IOsdSelectionServicefor user-driven selection. - Prefer
IWindowHandlefromPickWindow()over raw handles. - Prefer windows-subsystems/window-handles.md for handle conversion/matching.
- Prefer osd/screen-overlay.md for drawing persistent overlays.
Avoid
- Avoid custom cursor polling or screenshots for picker behavior.
- Avoid using OSD selection as a persistent overlay renderer.
- Avoid assuming selected windows stay alive.
Research Anchors
IOsdSelectionService,PickWindow,PickRegion,PickPoint,PickColor,OsdRegionSelectionResult,WindowUnderCursor,OsdRegionTransformationMatrices.
Search Synonyms
- select window
- pick window
- OSD picker
- region picker
- point picker
- color picker
- screen selection
- window under cursor