Recap

Last week, I built a basic video analysis engine. My first idea was to use Soundraw's API to generate music that fit my video. That failed for multiple reasons, mainly, chopping up pre-made music just ruins the song.

This week:

Can I build a drum pattern generator that actually follows the rhythm of any video, and still sounds musical?

Monday – Tuesday: Structure

Solidifying my Cursor workflow

• Tell Gemini what I want
• Gemini gives me an architecture/plan
• Paste that plan to Claude, ask for step-by-step Cursor prompts
• Paste those into Cursor, get code
• Paste any errors back to Claude, fix, repeat

My new plan:

1. Video Analysis → 2. Drum Pattern Plan → 3. Drum Audio Generation

The goal:

• Use last week's video analysis engine
• Instead of making a Soundraw prompt, actually generate a drum beat
• Feed the plan into a drum machine that will play it with actual audio samples

Drum Pattern Planning:

This step is the most important.

At first, my rule was just:

Find the best BPM based on video cuts, and generate a basic 4/4 beat.

That worked, but was boring and didn't really fit the video's energy.

With Claude, I started adding smarter rules:

• Analyse video for micro/meso/macro rhythm
• Account for motion intensity, not just cuts
• Assign genres and complexity levels
• Generate more "drummer-like" patterns

Now, the code is split into:

• rhythm_mapper.py: Converts video features into musical parameters (BPM, complexity, sections)
• pattern_generator.py: Uses those parameters to create drum hits (timing, instrument, velocity), using genre-specific and musical rules

Turning Plan Into Audio

I used the same AI workflow to finish the "drum machine" code:

• Load the drum plan and drum samples
• Create a silent audio track
• Place each drum sample at the right time and velocity as planned
• Export the final drum audio

First Results:

How does it sound?

• Pretty bad!
• Not musical yet, but for me right now, that's fine.
• The point is to see if it's even possible to make a drum beat that fits the video and doesn't sound completely random.

Wednesday: Borrowing From Research

Found some papers on music-to-video and video-to-music generation. Tried taking their approach to video analysis and improving my pipeline.

Big changes:

• Got rid of the old rhythm_mapper and pattern_generator

• New approach - two separate phases that:

  Phase 1:

  • Divide the video into sections
  • Assign BPM per section
  • Build musical measures and grids
  • Snap video events to the grid

• Phase 2:

  • Use a library of drum grooves as templates
  • Align hits and velocities to visual events
  • Add fills, intros, outros, and human timing

Results?

Better than yesterday.

Thursday: The MIDI Route

Today was a struggle, my brain wasn't working.

New idea:

• Build a database of MIDI drum files
• Pick the best fit for the video based on analysis
• Adjust the MIDI to match the video's rhythm

Didn't get far enough to test it, but the idea is there.

Friday: Back to the beginning

Tried the MIDI approach, but it didn't beat the previous generator I built on Wednesday.

Results:

Went back to improving what was did work (Wednesday code).

Tweaked the code to make the output sound more like a real drummer:

• Still hearing imperfections, some hits feel off-beat (probably in the swing/humanization layer)
• But it's improving, and I can tell I'm getting closer

End results for this week:

What's Next

• Fix the micro-level timing so the groove feels more natural
• Generate four or more beat variations per video to give more options
• Add more and better samples, organized by genre/style
• Try using drum samples split from real recordings, not just standalone hits
• Keep iterating until I get something that actually feels musical

Final Thoughts

It's been a tough week, but managed to make some progress.

The hardest part is telling the program what makes a drum beat sound "right" and how to fit the unpredictable rhythms of real videos.

Still, I'm making progress. If you've tried anything like this or have ideas, let me know!

Follow along as I either succeed or fail - probably the latter.