I spent the last few years doing lots of insulation upgrades to my home, which was built in 1960 and had empty wall cavities and 4" of attic insulation. The first winter and summer were horrible and even dealing with very uncomfortable temperatures inside the house resulted in expensive heating/cooling bills (I have natural gas heat). So in 2009/2010 it was possible to get energy credits on your taxes, so I stacked up. I ripped all the walls open, insulated past current code requirements, then cleaned out the attic and double rolled the thing past code requirements. Spent a few thousand dollars and quite a bit of my time on all of it. But now, my same furnace burns half as much gas in similar weather (my gas bill gives me graphs with historical usage and last year's average temps). In the summer the experience has been similar, my electric bill was slashed significantly. I sealed up around all the outside wall receptacles with those foam gaskets, re-sealed my window sashes to cut drafts, installed seals around my attic stairs, sealed and insulated all the ductwork in the attic, etc. It was a ton of "hidden" work that doesn't make my house any prettier or more marketable but I can now leave it set at 72° all winter and have bills that are a fraction of what other people pay to stay "colder" in their homes. That comfort is what I was after and it was all worth it.
Good luck. I would see if you can get someone to come in with a thermal camera and find all your leaks. You may have just one big one, who knows? When it comes to air leaks, even small ones make a big difference.
