Johnny, the nice thing about your issues is that you have the history of the dog.
The first thing I'm going to tell you is to not to feel sorry or bad about his past. Obviously there was nothing you could do about it and focusing on it now will not help him at all.
You need to give the dog what he's been missing all his life, stability and a routine. The first aspect of that is exercise. Obviously walks are key and should be done at least once a day, but you need to get that guy out in the water! Now, I know doing that everyday is going to be impossible, but you should do it as much as you can. It's in his DNA to get out there and once he starts to wake that part of himself up again, you're going to notice a dog that is going to be more relaxed.
On the walks I suggest getting him a backpack. What you're doing there is giving him a job to do. Now the walks have meaning to him. It's now more than just going outside to sniff around and find a place to potty. It's now about getting that important cargo (a couple of water bottles, or his treats) to it's destination.
What that will accomplish for you, is getting him tired, both physically and mentally. If he is concentrating on carrying is cargo, then when he gets home, he'll be too tired to worry about much else.
After the walk is when you want to feed him. Here we want to use nature to our advantage. Think about it, if you're tired and full, what do you want to do? REST!
I talked about the feeding ritual a little bit before, but you need to make that time a calm and relaxing time for him. Make sure that the bowl isn't put down until he is calm and after 10 minuets the bowl should be pick up and put away.
The walks and feeding need to be done on a consistent schedule. By doing this, you're taking the guess work out of it and eventually your guy is going to understand when things are going to happen.
The other big issue here is learning when to give affection. The key here is to remember that affection will reinforce the specific behavior that your dog is exhibiting. For example, if he is anxious and you pet him, you are rewarding him for being anxious. Always keep that in mind when you're interacting with him. You have to remember that not only does petting equal affection but food does also.