If you're the one declining the invite, the onus is on you to clarify which it is, not for the initiator to guess your intentions. In my life, I usually decline with "That doesn't work for me, but how about ${ALTERNATIVE_PLAN}?" It signals that the invitation is welcome, but doesn't work for practical reasons, so I'll return the kindness by proposing something else and leaving the other person in a clear and actionable position.
As the parent comment said, clarity is kindness. If you're leaving people guessing, then you're being very unkind.
