Only you can answer that.
A lot of folks have gone into debt over that "hobby" and then turned it into a career.
I never got used to heights. Still don't like ladders. My first 100 parachute jumps, it never changed. Neither did the next...and I still get the same feeling near the edge of a building or cliff.
An aircraft, though...the floor is only three feet beneath you.
If you don't want to fly, however, don't. Nobody is forcing you.
Why go into debt? Because you want to do it. Flying isn't a necessity, but it does ring true for a lot of people who feel they've got to do it. Most never regret it. One or two flights isn't enough to tell.