I have been interesting in Physics as a hobby for a very long time. Back in the mid-90's, I was reading about Quantum Mechanics and Relativity on a fairly regular basis. I had to really dig to find books that were deeper than a purely "popular" treatment, but not so deep that an engineer could actually make sense of them. Some math, not all math.
Having been trained as an engineer is "almost enough" to handle "some" of the math, but I ended up with an awfully long road ahead. I eventually got a bit discouraged, but never quite gave up. (Emulation came along and I got a bit distracted for a number of years.)
In 2004 Roger Penrose wrote a giant book on Math and Physics, call "The Road to Reality: A Complete Guide to the Universe". It is a highly mathematical, modern treatment, but it builds on itself such that, theoretically, someone with some math background can actually understand it all. (At least, they'll be able to self-direct to other resources as needed.) There are 16 chapters of Math (the first 1/3 or so), followed by as many on Physics. For comparison, my formal math training ended at chapter 7. I now find myself in chapter 15, nearly finished with the math section. It is definitely not for everyone, but it's exactly what I needed.
Finally, I have found that discussing the details of the book can help a lot with the understanding. I recently created a Yahoo Group called RTRFANS, for just this purpose. With this book, and with internet resources that weren't available in the mid-90's, I've now got a shot at understanding truly modern physics.