This sounds very much like some of your AVI files have video encoded with MPEG-2 or VC1 (or possibly some other unsupported codec) and you don't have the MPEG-2 or VC1 codec installed (only h.264/MPEG-4 content can be played by default). I had exactly the same issue when I first installed RaspBMC and tried to play DVD content (it plays like an audio file, in the background, as you describe). The RaspBMC FAQ says:
Q: Can Raspbmc play back MPEG2 or VC1?
A: Yes! Raspbmc offers hardware decoding of these codecs, provided the
codec pack has been purchased from the Raspberry Pi foundation website
here
You can find out the particular codec you need by opening the "Media Information" window in VLC then clicking the "Codec Details" tab (the screenshot below was taken on a Mac, but I imagine the Linux version will be similar). The video stream is usually Stream 0. If its MPEG-2 or VC1 then you should be able to purchase a codec and play it just fine.
Note that (at time of writing) the Raspberry Pi does not support playback of files encoded with the DIVX3 codec, which appear in VLC as type "MPEG-4 Video v3 (DIV3)". This Raspberry Pi forum topic has a reply from dom (near the bottom) which confirms this:
That is not MPEG 4 compliant and not supported. DIVX3 is actually a
codec reverse engineered from an old Microsoft codec, than was a
non-standard variant of MPEG 4. The hardware codec can't support it.
In order to get around this, you should be able to re-encode the file to one of the supported formats. The answer to this question includes a script that claims to convert from DIVX3 to H.264 (which then should be playable on the Pi without purchasing a codec). It looks like it uses avconv to do the legwork. I've not tried it, so use at your own risk.
The How-To Geek article How to Add MPEG-2 and VC-1 Video Codec Support to Your Raspberry Pi contains an excellent tutorial on what to do from here, but in a nutshell, once you've purchased the relevant codec, the license file will get emailed to you. You can then install the codec by following the instructions in the How-To Geek article above or How Do I Add Codecs? section of the RaspBMC/FAQ.
Tools
>Codec Information
it should tell you what CODECs are being used for the file, it does sound like the AVI in question uses a video CODEC RaspBMC does not have installed