In cases of violent animal cruelty or severe neglect, the abuse is clear and unquestionable. In other scenarios, however, the situation may be less clear.
"Many times what appears to be neglect seen from over the neighbor's fence is simply misinterpreted. If you believe the dog is not being fed, or left for hours without water, or has no way to get out of the rain, sun, or wind, look more closely during different times of the day to be sure.
"Sometimes the food and water bowls are kept inside an animal's shelter to keep them away from bugs or to prevent them from freezing. Maybe the owner feeds the dog everyday before he goes to work at 3 am when you're asleep, and then takes the bowl back inside. Maybe the pet's shelter is a cleverly disguised dog door into the shed or garage. Perhaps the dog just prefers to be outside no matter what the weather, so you never see him go inside." - American Humane
Identifying neglect can be harder than it seems, so it is critical that you look at the entire situation.
But is it Criminal?
There may be situations where you feel an animal is being neglected because they are not being treated the way you yourself would treat a pet. For example, if a dog is left tethered in the backyard at all times. Depending on where you live, that may not be against the law. As long as the pet owner provides adequate food, water and shelter, they may not actually be violating any laws - which means you have to take a different approach if you want to get involved.
Try making friends with the neighbor and gently suggesting different approaches to how their animals are treated.
Lastly, if you're not sure - report it. The Animal Control or Humane Law Enforcement officers are trained to evaluate the situation.