

Even if you have a password for your ssh key, malware on your system can just wait until you enter the password.
My point is that SSH access is very powerful, and effectively means that the security of the SSH server is reduced to the security of the SSH client. If your SSH client is pwned, so is your server. If you have 10 devices each with ssh access to each other, then if any one device is pwned, all devices are pwned as well.
This is not the case for systems designed for file sharing only. For example with syncthing, if one device gets pwned, all it can do is send files to the other devices.
Depending on the number of devices you have, your threat model, it can be helpful to set up a security hierarchy. So you only need to worry about securing the devices at the top of the hierarchy, and can play loose and careless with the devices lower down. That way it’s less likely to lose everything due to one mistake