Some people just don't fit the normal mode.
My brother had a 60-70% hearing loss (at age 4 from GE measles), was very small for his age, had few friends and did (very) poorly in school. But he drug up an old 2.5hp Evinrude off the bottom of lake Erie and rebuilt it to working order in elementary school. He was a motorhead and loved to take things apart and rebuild them. He became a master carpenter, plumber, pipefitter, welder, electrician and there is simply nothing he cannot build or repair. He has an aircraft hanger full of rebuilt trucks, dump truck, front end loader/backhoe, Hi-lo, a row of old outboards, a couple small hydroplanes, three big farm tractors, giant stump grinder, backup V8s to go in his trucks, a whole machine shop, and all of it was old used junk when he got it. In the last three years, he has built a single seat aircraft (and added flaps to the design so it could do shorter takeoffs and landings), and a fishing trawler style 30' wood boat. All from scratch, and all by himself (he doesn't mind if you watch, but you can't help). Seeing the prices of boat upholstery/canvas, he purchased an ancient industrial sewing machine, rebuilt it into a fine handmade wood bench, and is teaching himself how to sew all his new canvas and cushions for his boat.
Early on, everyone thought he was slow. He wasn't slow at all, but he was only interested in his own drummer, not what anyone else thought he should be interested in. He also ended up taller than anyone else in the family, about 5' 11". He now has about an 80% hearing loss, and if he doesn't like the conversation, he takes his hearing aid out and ignores you.

Interesting story about your brother Jess. Growing up in AK I got to know many people like him. A lot of people that don't fit in would move there and thrive. They can be aloof and cantankerous but once you get to know them they are just solid people that aren't worried about what others think about them.