Shubham Jakhete

Shubham Jakhete

I study AI systems that interpret and act on digital interfaces for blind and low-vision people, and how they can catch interpretations that look valid but are wrong.

I want to learn which errors a user needs to hear about, and how to confirm important actions without interrupting every step. MS in Computer Science from Cal State Fullerton.

What a screen reader says about icon buttons

Without labelsbutton
With ContextLabeler

Category confirmed by the ontology before speaking

An illustration of the speech pattern, not a recording. The confirmed category comes first, and the model's guess is hedged. Select an icon to see its example.

Research

2026Independent research
Mentored by Richard Ladner, University of Washington

ContextLabeler

An NVDA screen reader add-on that labels unlabeled UI elements for blind users. An LLM (Claude Haiku) proposes each label, and SPARQL ASK queries check it against an RDF/Turtle ontology before NVDA speaks it. Speech gives the confirmed category first so users can judge how far to trust the label.

90.9% category accuracy with zero ontology-invalid outputs. With the ontology gate removed, 83.6% of outputs failed.

Method and results
  • 0.82 s median latency, 1.48 s at p95.
  • A fingerprinted label store kept labels 98.4% consistent across 550 calls.
  • Found a soundness and completeness trade-off in ontology gating through a VS Code tree-item failure case.
  • App-specific ontologies extend the vocabulary from 17 to 22 categories with no code change.

Python, NVDA, Claude API, RDF/Turtle, SPARQL, RDFLib Code for ContextLabeler

2026MS capstone (CPSC 597), CSUF
Advised by Dr. Duy H. Ho

CrossAssist

An on-device iOS navigation aid for low-vision pedestrians. It runs three YOLO models through Core ML, two of them custom-trained to classify pedestrian signals and find crosswalks. Vision OCR reads walk-signal countdowns, and the app speaks "cross" only after a number is confirmed.

In a pilot with three low-vision participants, all three read the walk-signal state correctly within 10 m.

When does the app say "cross"?

Camera frame
OCR reads
App says
An illustration of the decision rule, not app output. A walk symbol alone is not enough; the app waits until the countdown number is confirmed.
Method and results
  • Depth Anything V2 estimates distance, with a pinhole camera model between depth frames.
  • One participant struggled with the countdown. The cause was small display text, not OCR error.
  • Swift 6 concurrency safe, with no third-party dependencies.

Swift 6, SwiftUI, Core ML, YOLO, Vision, ARKit Code for CrossAssist

2026Independent prototype
Builds on Spellburst (Angert et al., UIST 2023)

Plan First

A human-in-the-loop checkpoint for AI creative coding. Before any generated p5.js code runs, the artist sees the model's reading of an ambiguous request such as "make it more dramatic" as editable Change and Preserve lists, and nothing changes until they approve it.

In 60 automated test cases with no human participants, naming a property in Preserve cut unwanted changes from 25 to 6.

Approve the plan before any code runs

"make it more dramatic"

Change
  • Darker background
  • Faster motion
Preserve
    An illustration, not the prototype itself. Untick Color palette to see the effect the tests measured: when a property is not named in Preserve, the model often changes it. Shapes are marked "not checked" because the system only reports properties it can measure, which are motion and dominant hue.
    Method and results
    • After each change, the system compares motion and dominant hue without another model call, then marks each preserved item as kept, changed anyway, or not checked.
    • The effect was strongest for color and "make it calmer" requests, and weak when a request asked to change and keep motion at once.

    TypeScript, p5.js, LLM evaluation Code for Plan First

    2025Course project (CPSC 583, Knowledge Representation and Reasoning), team of three

    Explainable Used Car Recommendation System

    An explainable recommender over 5,181 vehicles merged from Kaggle listings, NHTSA crash ratings, and NHTSA complaint data. Prolog expert rules, an RDF/OWL knowledge graph queried with SPARQL, propositional filtering, and a probabilistic reliability estimate combine so each result names the rule categories it meets.

    Data limits are stated openly: about 36% of vehicles had safety ratings and 22% had complaint records.

    Every result says why it was chosen

    • Family car
    • Under $20,000
    • Strong safety

    Example match: a midsize sedan

    • Within budget (Prolog expert rule)
    • Seats five, family body type (RDF/OWL graph, queried with SPARQL)
    • 5-star NHTSA crash rating (safety data)
    • Reliability estimate above threshold (probabilistic model)
    An illustrative result, not real output. Each recommendation names the rule categories it meets, so the user can see the reasoning.

    Prolog, RDF/OWL, SPARQL, FastAPI Code for the used car recommender

    Manuscripts

    • 2026ContextLabeler: Ontology-Constrained AI Labeling for Blind Users in NVDAShubham Jakhete. Demo paper.
    • 2026An AT-Native Voice Agent for Independent Computer Use by Blind UsersShubham Jakhete. Research proposal.
    • 2026CrossAssist: The Digital Shield for Independent Low-Vision MobilityShubham Jakhete. MS project report (CPSC 597), California State University, Fullerton. Supervisor: Dr. Duy H. Ho.

    Industry experience

    Sep 2026 to present

    One Community Inc.

    Volunteer Software Engineer

    Contributing to the Highest Good Network (React, Node.js, MongoDB), the app the organization uses for time tracking and weekly reporting. Reviewed and documented 25+ pull requests in a codebase shared by 95 dev teams, and fixed bugs in its Mastodon post scheduler.

    Aug 2025 to Dec 2025

    Beaver Ecoworks

    Software Development Engineer Intern

    Replaced an ingestion path that dropped data with a Python OPC UA client that polls 14 PLC tags every minute and writes straight to Supabase. Built a SCADA monitoring dashboard for an industrial effluent treatment skid as a static Next.js export backed by locked-down PostgreSQL functions.

    Jun 2023 to Jul 2024, maintained since

    Beaver Ecoworks

    Full Stack Engineer

    Built a monitoring dashboard for a wastewater treatment plant covering 17 sensor channels, with PHP REST APIs and SQL rollups over MySQL, and moved deploys from manual FTP to GitHub Actions.

    Production code for these roles is not public.

    Other projects

    SmartSky

    Flight booking platform with a prototype LLM layer that turns natural-language travel queries into structured search parameters.

    React, Node.js, MongoDB

    Blockchain Disk Space Rental

    dApp for listing and renting disk space. A Solidity factory contract deploys one rental contract per listing. Team of four.

    Solidity, Next.js, Web3.js, Truffle

    Education

    May 2026

    MS in Computer Science, California State University, Fullerton. GPA 3.89. Coursework in Advanced Neural Networks, Advanced Algorithms, and Knowledge Representation and Reasoning.

    May 2021

    MSc in Computer Science, Savitribai Phule Pune University

    Dec 2018

    BSc in Computer Science, Savitribai Phule Pune University