Musical intervals are among the smallest structures of musical organization. Symphonies are made from it, melodies are built from it, all chords are constructed from it.
The problem with thinking about intervals as distances is that it implies measuring with a ruler. You might say that the chromatic scale is the musical ruler, and that a major third is ‘defined’ as an interval of 4 semitones, and that a perfect fifth is an interval of 7 semitones. Fundamentally, this removes you from the musical experience. Not to mention that it’s imprecise.
Intervals seem to be really flexible things that are precise and concrete relations at their core. Everyone who sings in choir and plays instrumental chamber music experiences this. They rely on their ear to achieve these specific resonances. The body doesn’t need to calculate to achieve this level of precision, it knows what resonates perfectly.
The useful idea is to think of intervals as resonances. Resonances come together to form the larger musical structures that we know to be true. When you perform an interval, you are completing an action to achieve a specific goal. If you base your theory of music on something as fixed and rigid as the chromatic scale, then you can never transcend sound relations that will require your calculation, and music for you will always remain intellectual. If you want to connect more with others musically, then connect with your body and find a theory of music that resonates in the literal sense of the word.