Shoelace is a React Native app I built for the Gemini Live Agent Challenge on Devpost. My original idea was to build a book-to-film converter by splicing clips from Google's Veo models, but that turned out to be prohibitively expensive, as my bank account soon confirmed. I still think it is a strong idea for the future, and I could imagine a startup doing well with it once the cost of video generation falls.
Shoelace guides people with disabilities through simple tasks such as tying shoes, brushing teeth, or making food. I was inspired by my younger brother, who has Down syndrome, and by how few attempts I had seen to use modern AI models for accessibility. Users select a task and record themselves completing it; Gemini then instructs them on the next step. I built the backend in FastAPI and used Gemini 2.0 Live as the main model. My focus was on letting the model react to changes in the user's state as quickly as possible. For that reason, I switched the backend to stream raw PCM data to the frontend.
The biggest challenge during development was the volume of testing. Much of the process was simply tying my shoes and tweaking the prompt. The project did not win the challenge, but I was happy with the progress and with what I had built. If I were to do it again, I might try some form of reinforcement learning to improve the outputs—I had little time during the challenge, and I assumed RL would have hurt my chances given that it was a Gemini-focused competition.