web-infra-dev/midscene

Traditional UI automation and AI tools that parse the DOM or accessibility tree fail on dynamic or non-semantic elements (icon-only buttons, canvas, native apps, cross-origin iframes) and break with UI refactors. Midscene solves this by operating purely on screenshots, so it works on any platform and any visible element, and can assert visual states (colors, layout) rather than just DOM existence.

Developers can build end-to-end UI tests, automated workflows, and AI-agent-driven automation for web, Android, iOS, HarmonyOS, desktop apps, and custom interfaces using a unified JavaScript SDK or YAML. Use cases include form validation, mobile app interactions (e.g., ordering coffee, liking tweets), data extraction, and even robotic arm control for in-vehicle testing. The ceiling is limited only by what a human can see on screen, with integration into Playwright/Vitest and AI agents via Midscene Skills.