I formerly worked for a company that makes breathable waders. You should be okay with polartec fleece under your wader. I highly suggest you invest in some midweight underwear, something that wicks (Capilene from Patagonia, REI's midweight wicking underwear).
People get cold in active sports because they are sweaty and if it soaks their clothes on their skin the wind chill or weather will make them lose their core temperature. When in waders, the cold water outside your wader around your legs and midsection will refrigerate the sweat that is accumulating inside. So a wicking underwear like Capilene will sponge the sweat from your skin and move it to the breathable part of your wader fabric. You can wear your polartec over the underwear, it will not impede the wicking.
Also to note, do you have a neoprene footed wader or a sock footed wader?
Neoprene will be much warmer but stinkier. Again invest in a good wicking sock for the same reasons above.
Lastly, guys, keep your breathable wader clean. Salt crystals from your sweat will render your breathable wader useless. It basically clogs the breathable membrane. SO WASH your waders regularly!
Take care, have fun... grrlfeeeesh

PS---Most breathable waders are designed to layer product underneath. Usually a polarfleece and a long underwear were fitted onto the wader in the design during sizing.