"How do I capture my game footage so I can design over it in REAPER?" A source of frustration to many, and a question someone asks in the REAPER community at least once a day. It's actually very simple... mostly... on Windows... Let's do it!
Step 0: Initial Conditions
I presume you have REAPER 5, 64 bit for Windows installed on a Windows 7 or 10 machine, not some WINE contraption or hacked onto XP or engraved in a stone tablet or whatnot.
Step 1: Install ShareX, FFmpeg, and VLC 64.
You're probably thinking, "That's 3 steps, I've already got VLC, and you're doing them out of order!" Do you have the 64 bit build? You might think you do, but you probably don't, because VideoLAN is cute and buries the installer for 64 bit VLC. It's here, but make sure you click the little drop down and select the 64 bit version. Look at the picture.
REAPER presumes VLC is in it's default install path in Program Files, so put it there. Yes, it's ugly, yeah, it could be better, but it's fine. Next, install ShareX. It's much less ornery about download links. Note, this is optional if you have a capture program you like, such as FRAPS (rubbish), Camtasia (expensive rubbish), or are ripping frames straight from the buffer with ffmpeg (love it). Also, ShareX will pull down a recent FFmpeg if you don't have it already, saving you a bit of clicking.
To install FFmpeg manually, grab a Windows build and copy FFmpeg.dll into your REAPER Plugins path, usually... just scroll down a bit.
Step 2: Confirm REAPER can see VLC
Screenshot above is of my setup. Note the path to VLC and the FFmpeg DLL in /Plugins. If VLC is missing, you'll see an error instead. This confounds people who have 32 bit VLC, because the path it's searching has a VLC in it, just not the one REAPER wants. Have I mentioned it's not brilliant?
Step 3: Configure ShareX (or just grab some footage however)
I like to skip transcoding my video, so that rules out FRAPS, and Shadow Play is a little less flexible than I like, so I use ShareX. You can scan the ShareX documentation, but I recommend installing the video and audio recorder devices and testing some window capture. It won't work for everything, but it's pretty good, and if you're working in Unity, there are some tricks to... that's another post.
Step 4: Open Your Video in REAPER
And there we are. VLC is installed. FFmpeg is installed. ShareX has rendered us some MP4's, let's try it out. Drop your video into a track in REAPER and open the video window (Ctrl+Shift+V). Hit play and there you are. Probably. Let's meet up in Step 5 and I'll explain.
Step 5: Uh, That Didn't Work?
Troubleshooting can be a pain, since there are a few moving parts. Let's start simple. Does your video playback in VLC? No? Capture side problems. Make sure ShareX (or whatever) is working. Does the video drop into REAPER or complain? Does it import but you get just black? That's a codec problem. Just a green frame? You didn't install VLC properly. Is the video slooooooooow or stuttery, or hesitates on start/stop? Codec issues again. I've tested a bunch of settings, here's what I like. Oh, you're... you're fine if I throw some FFmpeg commands at you, right? Great!
ffmpeg -i INPUTFILE -c:v libx264 -r 30 -preset ultrafast -tune zerolatency -crf 28 -pix_fmt yuv420p -c:a aac -strict -2 -ac 2 -b:a 192k OUTPUT.mp4
Run that on your input file, like... say, a misbehaving MOV, or a FRAPS AVI (though, there's some color stuff...), and it'll produce an MP4 that will run great in REAPER... 64... on Windows... with VLC... notice a trend? Audio is in AAC muxed with the MP4, but if you want WAV add "-c:a pcm_s24le OUTPUT.wav" to the end and you'll get a WAV alongside.
Oh, and if you drop that output MP4 into default REAPER w/o VLC installed, it'll crash REAPER. Hard. So save first?
Step 6: Share Your Process!
In my office, people use a mix of capture techniques, from Camtasia to FRAPS to pointing their phone at the screen. Standardizing everyone on one system makes it easier to share video, solve problems, and capture footage in a hurry. It also encourages people to capture more, so your bugs will have beautiful gifs from ShareX, your promos will have endless B-Roll, and your boss will have that 30 second clip for their GDC talk next year. It's a win win... on Windows 10... with VLC 64...
Now for some technical whatsits for the video nerds. I've ffprobed a bunch of videos that did or didn't work, and it seems REAPER's preference is for baseline h264 MP4's. ProRes and MJPEG both work with VLC, but crash REAPER native. You can turn off the video and the audio in a variety of ways, and it's better to do it in the video settings than muting the track, because that mutes the video... When it comes to getting video out of REAPER... well, that's a whole other thing and a half.