There's no clear and simple answer to that question since like any single drug or group of drugs a lot depends on dose and duration as well as the individual's biochemistry. After all, even a simple OTC drugs such as aspirin can be unsafe if you take too much of it, or for too long, or are allergic.
Of course "safer" can mean many different things, and there are different classes of both so there may be some crossover. But if we went by claims alone SARMs would likely be the easy winner. They're designed to target specific receptors and leave the rest of the body's systems unaffected. One reason why you don't need to take nearly as much (mg for mg) for them to be effective. They're certainly safer to administer for the most part.
On the other hand, steroids tend to be a more broad spectrum drug that can effect your entire biochemistry, and almost always require an additional drug protocol to help fix that imbalance (PCT). This is possible with SARMs too (in larger doses), but the impact is typically not nearly as severe. If you have a negative reaction to SARMs they're generally out of your body within a matter of hours. Conversely, many steroids can take months to totally clear your system entirely.
Steroid do have the added benefit of having been around for many generations, so we know better how to administer them, treat the side-effects, and how they will effect the body over a life-time. SARMs however, are a new class of drug and at this point still under research. That means nobody really knows what the long-term effects of those drugs will be. Given that, and barring any and all of the legal ramifications and social stigma, the clear "safer" winner would be steroids.