AI Windows: Input, Hooks, Hotkeys

AI-first map of input sending, hooks, interception, and hotkey services.

Input Hooks / Hotkeys / Send Input Discovery Map

Reference map for keyboard/mouse hooks, hotkeys, script keybinds, send-input APIs, input simulators, smoothers, suppression, and native detours.

User Intents

  • Bind script methods to hotkeys.
  • Detect hotkey state as aura trigger.
  • Send keyboard/mouse/text input.
  • Select input simulator or smoother backend.
  • Track raw keyboard/mouse events.
  • Suppress keybind input when needed.

Concept Model

  • [Keybind] is script method metadata.
  • HotkeyIsActiveTrigger is the aura hotkey condition.
  • ISendInputScriptingApi is the script-facing input output surface.
  • IKeyboardEventsSource exposes host keyboard/mouse event streams.
  • IInputSimulatorProvider selects send-input backend.
  • IUserInputSmootherProvider selects mouse movement smoother.
  • Low-level hook/detour APIs are infrastructure, not the first script API.

API Details

  • KeybindAttribute - script hotkey binding metadata.
  • KeybindAttribute.Hotkey - string gesture such as F5 or Ctrl+Shift+P.
  • KeybindAttribute.SuppressKey - when true, marks the original key event as handled/suppressed for both key-down and key-up handling.
  • KeybindAttribute.IgnoreModifiers - when true, required modifiers must be present but extra modifiers are allowed.
  • KeybindAttribute.ActivationType - selects KeyDown or KeyUp invocation.
  • KeybindActivationType - key-down/key-up enum used by [Keybind].
  • HotkeyGesture - code-friendly hotkey value. Prefer constructors/enums when the hotkey is known in code.
  • IHotkeyConverter - string-to-HotkeyGesture conversion. Mostly useful for persisted user settings, editor input, or config text.
  • IHotkeyTracker - runtime hotkey tracking. IsEnabled defaults to true; set it only when intentionally disabling/re-enabling a tracker.
  • HotkeyIsActiveTrigger - aura trigger for hotkey state.
  • ISendInputScriptingApi - script send-input API.
  • IInputSimulatorProvider, IInputSimulatorEx - simulator discovery/backend.
  • IUserInputSmootherProvider, IUserInputSmoother - mouse smoothing.
  • Built-in smoother ids include BenLandLinearFast, BenLandRandomizedFast, BenLandRandomizedNormal, Kalon, and MlHumanMouseV1, MlHumanMouseFastV1.
  • MlHumanMouseV1 - ML Human Move smoother. Uses a local ML.NET FastTree model pack to predict compact movement parameters, generates a bounded Bezier/easing path, and applies the default 2x ML timing multiplier.
  • MlHumanMouseFastV1 - ML Human Move Fast smoother. Uses the same model-backed route planner with a 4x ML timing multiplier.
  • IKeyboardEventsSource - keyboard/mouse event streams.
  • HookEngine - advanced native function detour engine.

Assembly / Package Hints

  • IInputSimulatorProvider lives in the Roxy shared compile surface.
  • IInputSimulatorEx, ISendInputScriptingApi, simulator ids, and send-input contracts are shared/metadata APIs; Roxy runtime modules provide the actual simulator implementations.
  • Standalone embedded apps should request the Roxy capability with UseRoxy() and wait for readiness before resolving simulator-backed services.

Prefer

  • Prefer [Keybind] for simple script keyboard shortcuts.
  • Prefer HotkeyIsActiveTrigger for user-authored aura hotkey conditions.
  • Prefer new HotkeyGesture(...) in code when the key/modifiers are known.
  • Prefer IHotkeyConverter when reading a hotkey from persisted text or user input.
  • Prefer ISendInputScriptingApi for script input output.
  • Prefer provider-discovered simulator/smoother ids.
  • Prefer osd/selection.md for selecting a target window first.

Avoid

  • Avoid raw global hook subscriptions for simple hotkeys.
  • Avoid parsing hard-coded hotkey strings through IHotkeyConverter when a HotkeyGesture constructor would be clearer.
  • Avoid setting IHotkeyTracker.IsEnabled = true in examples unless the example first disabled it; true is already the default.
  • Avoid assuming simulator ids are installed on every machine.
  • Avoid using native detours for normal keyboard/mouse hooks.

Common Flows

  • Runtime hotkey with Rx:

    • create IHotkeyTracker through IFactory<IHotkeyTracker>.
    • set Hotkey with new HotkeyGesture(...) for code-defined hotkeys, or IHotkeyConverter for persisted/user-entered strings.
    • set HotkeyMode, SuppressKey, IgnoreModifiers, or HandleApplicationKeys only when the defaults are not right.
    • subscribe to WhenAnyValue(x => x.IsActive) and filter for true.
    • dispose the tracker and subscription with the owner lifetime; in top-level scripts, use ExecutionAnchors for per-run resources.
  • Script method [Keybind]:

    • decorate instance methods on the script object.
    • methods can be public or non-public.
    • multiple [Keybind] attributes can be placed on one method.
    • the runtime parses the string with IHotkeyConverter.
    • handler invocation is scheduled on the background scheduler.
    • method parameters are resolved from the script DI container before invoke.
    • if the method returns Task, runtime waits for task completion.
    • exceptions are written to aura event log as failed keybind events.
    • registration is active only while the script execution tracker is in the running state.

Send input without creating an aura component

In a C# Action, inject the service and send one key to the active application:

csharp
ISendInputScriptingApi SendInput { get; init; }

SendInput.KeyPress(Key.F1);
Log.Info("F1 request sent");

Roxy registers this API; the default simulator is Windows Input. The log confirms call completion, not that the application processed the key. TargetWindow does not guarantee background input. See SendInput operations for timing, coordinates and cancellation.

For key reactions use the complete Keybind example. A dynamic tracker and its subscription belong to the caller. Native event callbacks and background keybinds do not authorize direct UI mutation; use the UI dispatcher when needed. Ordinary classes receive these dependencies explicitly.

Research Anchors

  • KeybindAttribute, KeybindActivationType, HotkeyTrackerRuntimeVisitor, HotkeyGesture, IHotkeyConverter, IHotkeyTracker, HotkeyIsActiveTrigger, ISendInputScriptingApi, IInputSimulatorProvider, IUserInputSmootherProvider, IUserInputSmoother, MlHumanMouseV1, MlHumanMouseFastV1, IKeyboardEventsSource, HookEngine.

Search Synonyms

  • hotkey
  • keybind
  • keyboard hook
  • mouse hook
  • send input
  • input simulator
  • input smoother
  • ML Human Move smoother
  • MlHumanMouseV1
  • MlHumanMouseFastV1
  • target window input
  • suppress key
  • ignore modifiers
  • key down
  • key up
  • keybind activation type
  • method parameter injection
  • native detour

Tracker, provider and detour boundaries

IHotkeyTracker Hold follows key-down/up; Click toggles on key-up, so filtering IsActive == true in Click mode does not invoke on every press. With HandleApplicationKeys false, Click events are excluded while EyeAuras itself is active. This tracker behavior is separate from KeybindAttribute.ActivationType.

InputSimulatorProvider.ResolveById returns a registered object; Resolve(idOrName) returns an observable and may create a proxy before a real backend is registered. A known name does not establish IsAvailable. The smoother provider likewise resolves registered IDs; a catalogue entry does not install a driver.

HookEngine is a separate current-process native detour mechanism. It retains the detour delegate, returns an original-call delegate and requires explicit hook enabling. Dispose disables hooks and releases its allocator. This is not keyboard hotkey registration or a guarantee of safe patching while other threads execute the affected function.