Penn University researchers have developed an AI-based assistive technology for blind users, using vibrational and audio cues to give visually impaired individuals a sense of real-time spatial awareness to identify and locate objects in real time.
The visual aid smartphone app, NaviSense, was unveiled at the SIGACCESS ASSETS ’25 conference in Denver and received the Best Audience Choice Poster Award.
NaviSense has been shown to precisely guide a user’s hand directly toward the desired object using customized haptic feedback.
The researchers say current visual aid smartphone app technologies are advancing but still fall short of providing intuitive object recognition and real-time guidance, improving object retrieval accuracy and speed.
Their solution, NaviSense, integrates large-language models (LLM) and vision-language models (VLM) to recognize objects without preloaded data and guide users toward using vibration and audio cues.
How NaviSense Works to Assist the Visually Impaired
The tool responds to spoken prompts and searches surroundings to pinpoint the requested item functioning as a visual impairment object retrieval aid that expands user independence. If it is uncertain, it asks follow-up questions to enhance usability.
How NaviSense works becomes especially clear through its dynamic approach, “Using VLMs and LLMs, NaviSense can recognize objects in its environment in real time based on voice commands, without needing to preload models of objects,” said team lead Vijaykrishnan Narayanan, who described the development as “a major milestone for this technology.”
The team conducted interviews with visually impaired individuals before development.
“These interviews gave us a good sense of the actual challenges visually impaired people face,” explained the lead student investigator, Ajay Narayanan Sridhar.
One standout capability is hand-tracking the app’s monitors and phone movement to guide users’ hands directly to an object, functioning as a visual aid app with hand guidance unlike standard tools on the market.
Sridhar called it the tool’s most significant feature, noting, “There was really no off-the-shelf solution that actively guided users’ hands to objects, but this feature was continually requested in our survey.”
Testing, User Feedback and What Comes Next
Twelve participants compared the app to two commercial alternatives, often considered among the best visual aid apps, and NaviSense performed better on speed and accuracy. One tester remarked, “I like the fact that it is giving you cues to the location of where the object is, whether it is left or right, up or down, and then bullseye, boom, you got it.”
NaviSense relies on both LLM object recognition and VLM object recognition, enabling more flexible results than pre-trained databases alone.
Alongside recognition, the assistive technology for blind system provides vibrational feedback navigation, offering real-time orientation and distance cues that guide users toward objects naturally, a function that supports independence and reduces search frustration.
Researchers describe the tool as a conversational object identifier, capable of asking clarifying questions, interpreting responses, and adjusting guidance like an interactive assistant rather than a static tool.
The current version still requires refinement, including reductions in battery use and improved AI efficiency, but the framework is approaching commercialization. End-to-end functionality suggests potential as a non-visual object location guidance system that could ultimately define how future accessibility apps are built.
“This technology is quite close to commercial release, and we’re working to make it even more accessible,” Narayanan said.
The assistive technology for blind research team includes members across Penn State’s electrical engineering and computer science departments, alongside collaborators from the University of Southern California and independent research contributors. The project received support from the US National Science Foundation.
At its core, NaviSense is another AI-powered accessibility tool that respects privacy, improves autonomy, and incorporates feedback from the visually impaired community, turning research into real-world capability.
Inside Telecom provides you with an extensive list of content covering all aspects of the tech industry. Keep an eye on our Intelligent Tech sections to stay informed and up-to-date with our daily articles.