OCR-Based Element Identification

OCR-based element identification enables test automation on screens where elements are not accessible through standard widget or XPath-based strategies.

In some mobile applications, UI components are rendered in ways that make them invisible to conventional automation techniques. For example, screens built with custom rendering, image-based components, or applications where development gaps result in elements that cannot be reliably targeted.

With this capability, QApilot uses OCR during the recording phase to detect and display visible text elements on screen. When a user selects an element, QApilot stores its text content and bounding box coordinates. During execution, OCR runs again on the live device screen to locate the same text, recalculates the current bounding coordinates, and performs the interaction.

It's now possible to automate previously untestable screens without requiring changes to the underlying application.