Searching through forums to figure out how to open another instance of VLC on a Mac brings up some pretty mixed results. So here's the method that actually works. I'm sure this applies equally well to any other application as well. Open the terminal and run the following command:
open -n /Applications/VLC.app
The -n option is what tells Mac to open another instance even if one already exists.
NOTE: Although you have two VLC icons in the dock now, dragging files to the icons will not work correctly. The Mac will link the icons to the same instance of VLC. You'll need to drag files to the player control panel instead or just open them through the file menu.