An in-house meeting notes bot, and why the calendar was the hard part

A commercial real estate developer wanted the kind of AI notetaker most people have seen join a call, built in-house: recordings, transcripts, and notes in the firm's own database, on the same stack as its other internal tools.

The first version took one session. Making a bot reliably join the right meetings on its own took a good deal longer, and that part is the more useful story.

Version one: record, transcribe, summarize

A Next.js app on Supabase and Vercel with three ways to get a meeting in:

• Record in the browser, from the microphone, optionally mixed with a Teams or Zoom tab's audio, with pause, resume, and a live level meter.
• Upload any audio or video file up to 500MB, sent straight from the browser to a private storage bucket.
• Import a Teams or Zoom transcript file, with speaker names carried through and no transcription needed.

Deepgram transcribes with speaker separation. Claude then writes structured notes through a typed schema: overview, key points, decisions, action items with owners and due dates, open questions, topic chapters, and a draft recap email. OpenAI's Whisper is the fallback transcriber.

The meeting page is where people spend their time. Audio plays alongside a click-to-seek transcript. Speakers are renamed once and every line updates. Action items can be ticked off, and re-running the notes keeps the ticked ones. An Ask tab answers questions about the meeting, and there is sharing and Markdown export. Full-text search runs across every meeting a person can see, ranked title first, then overview, then transcript.

Sign-up is limited to company email domains and confirmed on the server, so no confirmation link has to survive the corporate email scanner, which opens every link in an inbound message before the person does. Every table has row-level security, recordings are private, and playback uses signed URLs that expire in an hour.

Version two: a bot that joins the call

Recording in a browser only works if someone remembers to start it. The next ask was a bot that joins Teams, Zoom, or Meet on its own and produces the same notes.

A bot that sits in a call for an hour holding a live audio stream and a headless browser is the opposite of serverless, so the work split in two. The bot runs on a small always-on server using Vexa, an open-source meeting bot under the Apache-2.0 license. Everything downstream stays in the existing app, and both paths feed one notes pipeline.

A few choices from that build:

• Protect the box first. The server had no swap, and a meeting bot can spike past 2GB of memory. We added swap before installing anything, so a spike could not take down the other service on that machine, and limited it to one bot at a time.
• Polling is primary, webhooks are a nudge. The bot's webhook delivery is unreliable upstream, so a timer on the server reconciles every 30 seconds and an open meeting page nudges the same check. Every path is idempotent, so running it twice changes nothing.
• Webhooks are signed. Valid signatures pass. Forged signatures and replayed old timestamps are refused.
• A fallback that needs no IT. If a tenant policy ever blocks anonymous guests, the transcript import path still works.

The calendar was the hard part

Joining a meeting from a pasted link worked quickly. Joining the right meetings automatically did not.

Outlook puts a Teams link on everything. Lunches, in-person meetings, and focus blocks all carry one, so the bot kept joining events nobody would open, then retrying. A calendar gate now stands those down before they fire: cancelled, marked free, in-person, or already attended. No-shows are labeled "No meeting" instead of failures, and retries dropped to three in 20 minutes, because in practice a retry never rescued a real meeting.

The bot was following the wrong calendar. When a scheduled meeting was missed, the trace showed the calendar feed the bot read was not the owner's. It belonged to someone else, while the app's rules were checking the owner's feed. The two disagreed, so the gate kept standing down real meetings and the bot kept joining someone else's lunches. The app now pushes the owner's feed to the bot server directly, and only trusts its own rules when the two calendars agree.

Calendar-dispatched bots were not recording. The bot server's own scheduler sent bots with recording switched off by default. Bots sent by hand recorded; bots sent from the calendar did not. So we took scheduling away from it. The app's gate now sends every bot itself, from two minutes before the start to ten minutes after, with recording on.

Our own cleanup call was damaging finished meetings. A call we had used to stand the calendar down after each meeting was flipping finished runs back into the planned set, which then made the bot server's calendar cleanup fail on every pass. We deleted that call and repaired the three affected records.

Then we put the owner in charge. An Upcoming page lists the next two weeks of meetings and what the bot will do for each ("Bot joins at 3:58 PM", "Skipped automatically. This is an in-person meeting", "You switched this off"). Every meeting has a switch that outranks every automatic rule, and a Send now button appears on a meeting already in progress.

What changed

The bot joins real meetings in production, and its transcripts land in the same notes pipeline as uploads. One meeting the bot attended while calendar recording was still broken was recovered from its live transcript lines: 387 lines after removing duplicates, rebuilt into notes with 13 key points and 11 action items.

On cost, Deepgram transcription was estimated at roughly 26 cents per meeting hour at the time of the build.

What other operators can take from this

• Transcription is a solved problem. The edge cases live in scheduling, permissions, and email security.
• Make every automatic rule visible and overridable by the person it affects.
• Trace the failure instead of retrying it. Retries never rescued a real meeting. Reading the actual records is what found the wrong calendar and the recording default.
• Keep a manual import path. It is the fallback that never needs IT.

If your team's meeting notes live in someone's head or someone's inbox, book a Systems Audit.