just like things can wear out / dry rot, being used or not over time, yah the rubber seals, and fluids can suffer ageing issues as well. Even though you keep them clean, pollutants in the air, Sox Nox can eat at them over time too. Some super fine grit / silt manages to roll up in there past the wiper / seal, and now it's churning around.
If you ever notice your brake fluid reservoir, over time that can turn darker as the fluid picks up crap out of the air, oxidizes, gets superheated during stops etc etc, that's the fluid breaking down over time, discoloring it, from the should be pretty much clear when brand new. The oil in your shocks is constantly agitated from riding and being forced thru tiny ports over and over again. While we can see our brake fluid and know oh yah its getting worn, there really isn't a way to see the forks w/o opening them up, which is not really a standard thing to do.
Depending on where you live and condition of the roads, can really take a toll on them as well
aaron