It's a risk if you don't swap the filter regularly as recommended by the manufacturer, and everything downstream of the filter (tubing etc) is at elevated risk for biofilm buildup when the filter is stripping out chlorine/chloramine.
It's an even bigger risk, generally, with pitcher/reservoir type systems (e.g. Brita or Pur) that require manual fills, given that they get a lot of environmental exposure. If you use a Brita pitcher for a while and then leave the filter media and pitcher somewhere at room temp for a couple weeks, you can often see evidence of fungal/microbial growth popping out of the bottom of the filter cartridge. It's pretty gross. I've also had a relative end up in the hospital with a pretty severe amoebic infection, which investigators traced back to her Brita filter cartridge she had neglected to change for some time.
Part of the reason I purposefully bypassed the filter on my fridge. The tap water is fine and doesn't need filtering but an old water filter can cause all sorts of nastiness.
It's an even bigger risk, generally, with pitcher/reservoir type systems (e.g. Brita or Pur) that require manual fills, given that they get a lot of environmental exposure. If you use a Brita pitcher for a while and then leave the filter media and pitcher somewhere at room temp for a couple weeks, you can often see evidence of fungal/microbial growth popping out of the bottom of the filter cartridge. It's pretty gross. I've also had a relative end up in the hospital with a pretty severe amoebic infection, which investigators traced back to her Brita filter cartridge she had neglected to change for some time.