I don't make a habit of unplugging the pi in the sense of eschewing shutting them down properly except when I've lost networking on a headless pi, in which case I am usually too lazy to plug in a keyboard, etc.
Generally I always check to make sure the green ACT light is not on at that point; for recent models (or firmware?) this will be off when the SD card isn't being accessed. Which is what you want to make sure of. Unless you are writing to it constantly, this should be simple enough; as long as there is a decent amount of headroom in free RAM (say 50-100+ MB, depending on context), then things that are prone to being re-used frequently but aren't actually loaded by a process at any given moment will be in cached in free memory and reloaded from there by the OS, not the real physical medium. This is how all contemporary general purpose operating systems operate.
If the pi is making use of the SD card, here's what the risk is minimally: the filesystem on the card is out of sync with the in memory state. Normally, this is probably not a big problem; for starters, the filesystem journalling used by default on most pis might be a defense against it, as is the fsck which should be applied automatically at boot if the filesystem was not cleanly unmounted. I'll explain why I say "might" and not "is" shortly, because in this context I think it often might not.
As far as I know, I've never ended up with filesystem corruption or data loss when pulling the plug, which I may have done a hundred or more times over the years. However, again, I don't do it habitually. If you do it several times a day you may eventually run into whatever the statistical risk level is, and there is a potentially nasty catch involved.
Here's the PROBLEM:
Recently it occurred to me that there is an issue with SD cards which OS/filesystem mechanisms may be powerless against, and which may explain why some people seem to have persistent problems with fs corruption, particularly those who get into yanking the cord regardless of system state -- e.g., recently someone here claimed to be running compute modules where corruption left the systems unbootable in ~1/40 such power cuts.
In abstract terms, without taking the nature of SD cards into account, that should not happen even if the system is busy, because what is most likely to end up corrupted is non-critical stuff that's being written, not any system software that is effectively read-only and only gets altered during updates.
It could happen if bits are being juggled around, and so the filesystem meta information which stores where various bits are is corrupted. Again, however, journalling and fsck should be capable of dealing with this, which does require the kernel to boot, but the kernel on the pi is on a separate boot partition which might as well be unmounted during use (except for when updating) because none of it is used after the system has booted. Meaning the information about the partition should be, effectively, incorruptible, even when it is left mounted.
But...
SD cards are a black box to the operating system. There is no way out of that. While there are specific drivers for SD card controllers that are part of the computer's hardware (on the pi, this is part of the SoC) there is no such thing as a driver for different, specific makes and models of card.
And yet, they all have microcontrollers inside them that may operate in very different ways.1. This is what makes the card a black box; it interacts with the operating system via a standardized SD protocol, and that is the last point of control the OS has.
One of the things SD cards and other solid state media do which is different from traditional spinning disks is use opaque virtual addressing; they do not store information physically in the sequences the operating system perceives. This is mostly a good thing, or else we might really need different drivers for different brands of cards, etc., and it permits cards to (opaquely) implement wear leveling, which significantly prolongs their lifespan.
Another thing they rely on is relatively large "erase blocks"; when data in a block needs to be changed, the entire block is erased and re-written. Filesystems also do this kind of thing as a matter of course, but note that is at a system software level, and problems resulting from this are exactly the kinds of problems journalling and fsck deal with.
The crux of the more insurmountable problem is that the scale filesystems do that on is usually much smaller than the scale SD cards do it on. If it weren't, you'd end up wasting a good deal of storage space, because a filesystem block can only contain data from one file. If the block is 2 MB and there are only a few kB of data, the rest is wasted. So filesystem blocks tend to range from 1/2 KiB to 4 KiB.
It is fairly obvious SD cards don't do that, and in fact, couldn't, because the controller in an SD card has no awareness of boundaries such as "file", "filesystem", or even "device partition". It just deals with the chunks of data the OS wants, via an opaque layer from inside a black box where anything could be happening on a physical level.
One of the reasons it is obvious (aside from the premise cards "couldn't do that in the first place") is those blocks are frequently quite large, yet the cards seem to do a decent job of utilizing all space. An erase block may be several megabytes in size. What's more, the details are proprietary. While there may be mechanisms by which the OS can request the erase block size from the card, the card does not have to provide this, it can lie about it, and it would become absurd if the OS tried to leverage it.
It follows from this that since:
The SD card controller has no concept of what data "rightfully" belongs where in the sense of coherent filesystems and parititions, and,
The SD card is a black box into which the OS cannot truly see,
Then what is in a given 1 MB erase block, where a card contains several partitions using 4 KiB or smaller filesystem blocks, is bound to be arbitrary once the card has been used enough (and possibly even if it hasn't been used much at all). This is probably true even if you try to force the OS to use larger/corresponding block sizes; they will end up misaligned.
So:
If the power is cut while the SD card is in the middle of altering a block, that could result in the loss of a fairly large volume of arbitrary data that could be anything from the card. It could be information marked "read-only". It could be information from a partition which isn't even mounted, and, obviously enough, from the mostly unused boot partition.
If that happens and the card doesn't have some kind of safeguard system of its own (which some may have, but I am sure most do not), then you could be looking at a FUBAR situation. Journalling will not protect against random MB size corruption which does not even recognize partition boundaries. Or fsck.
Or, since the card's hardware is generally proprietary, could anything else except software produced by the card's manufacturer -- presuming that's plausible. I've never heard of such. This would make cards more complicated, more expensive, and more of a hassle to use. Which is not the goal.
Put another way, SD cards are not intended to be used reliably this way. They are cheap and very handy but that is the result of a trade-off in the protocol: In general, there is no guarantee of data integrity for anything on the card if you arbitrarily kill the power to it.
What can go wrong with just unplugging the Pi? Should I start simply unplugging? Note: In this case I'm not too worried about data loss.
It will not physically harm the pi, no, just beware "data loss" could extend to "useless card" which needs to be completely reformatted. However, I'd say the chance of that if you are doing it with the green ACT light off are very low.
- This may be significant when considering why some specific makes/models of card clearly fair worse than others for some people. Unfortunately, while two cards which are labelled identically may be identical in terms of stated features (size, speed, etc.), manufacturers are not bound to making them truly 100% identical in terms of constituent parts.
