The symptoms do sound like the servo is being emptied and can't replenish vacuum quickly enough, however check also for flexible vacuum pipe collapsing, same effect.

Have the brake fluid replaced if it hasn't been done for a long time, are all calipers free ie not sticking.

Have you repeatedly pumped the brake with the engine running to simulate hill decent? If that produces no such hardening of the brake pedal i wonder if you might have very poor rear brake effort, resulting in the front brakes doing all the work and fading out when very hot.

Are any of the brakes getting ridiculously hot after heavy use, maybe coupled with the rears being too cool.