KnowWhere

Overview

KnowWhere is a native mobile app that turns your surroundings into a live, spoken guide. It's for people exploring a new place: open the app and it tells you about the most popular spots around you, out loud.

We built it in 24 hours at HackSussex 2026, and it won! Neither of us had used Google Cloud APIs or Generative AI APIs before. We chose the challenge deliberately, to combine real-world location data with generative AI.

The app

Tap Scan Area to find popular places within a radius of your location, then tap any place in the list to hear its AI-written audio guide, with a now-playing bar to stop it.

My role

KnowWhere was built by a two-person team. Most of my time went into frontend part of the mobile app. The backend was more of a shared effort. My teammate wrote most of it, but we made the key decisions together, such as architectural designs and optimisations.

How it works

When the app opens, it runs one pipeline:

1. Detect the user's location.
2. Scan nearby points of interest with Google Places.
3. Rank them and pick the five most popular within a radius.
4. Generate a short description of each with Gemini.
5. Speak each description aloud with ElevenLabs.

The frontend is React Native with Expo, whose sensors provide the location that sets the search radius and is sent to the API. The backend is a Django API that receives the coordinates, does the radius and ranking calculations, and integrates ElevenLabs, Google Places and Gemini.

The hard part: latency

The pipeline runs in sequence, so every delay adds up before the user hears anything. Generating the descriptions with Gemini was the slowest step, so that is where we focused.

We spent significant time optimising the Gemini prompt, cutting token usage to speed up generation.

Caching a moving target

Caching in Expo was harder than expected. Latitude and longitude change constantly as you move, so a cache keyed on exact coordinates rarely gets a hit.

The planned fix was to cache points of interest on the backend, so nearby requests can reuse them. Rounding coordinates to a grid would give requests from roughly the same spot the same cache key.

If I built it again

• Cache on the backend from the start, keyed on rounded coordinates, rather than fighting constantly changing locations on the client.
• Stream the narration, so speech starts as soon as the first description is ready instead of waiting for the whole pipeline.
• Treat latency as a design constraint from the get go, choosing smaller, faster models before spending time tuning prompts.
• Make the experience more interactive: tappable map pins with expanded summaries, and automatic scanning as you move.
• Add personalisation, letting users choose from a range of ElevenLabs voices to narrate their tour and pick what they care about, such as history, food or nature, so the descriptions are tailored to them.
• Spend more time polishing the UI: tidy up the layout, make the map and descriptions easier to scan, and add loading states.

Stack