I guess it depends on what you consider "the surrounding area". Like the Harbor is walkable from there, and is a fine place, and so are places like Federal Hill and Fells Point. But there's certainly areas directly around the stadium that I would want no part of walking around at night and barely even in the daylight. Anything right outside the Casino or in similar areas.
There's a reason why a lot of football stadiums aren't built right inside a City area per-say. Size and traffic (which M&T is on the smaller side and the traffic sucks balls) are one, and safety is the other, because most cities, regardless of size, are going to have poverty, homeless people, and crime in the surrounded areas.
That being said... I've never once felt "unsafe" going to a game during daylight areas (I'm not the kind to feel unsafe really at night either). The tailgating is generally great. The A-C lots are wide open in a friendly area surrounded by fans, however I will say the G-H lots are kind of sketchy for tailgating, given they're like off to the side and right underneath an overpass. But I don't find them unsafe.
The sheer volume of people around makes it difficult to be too worried about crime. I'm quite certain that in some of the smaller lots (and even in the big one's) people just walk around and steal stuff routinely while people are inside during the game, but I've seen that as prevalent at road stadiums also.
