I want to thank Klopzi, Fuel, and $mokee for replying to my call for input.
I wish Fuel would have expounded a little on his reason for folding.
In the moment of play I had the exact same thought process $mokee had. So exact it's scary. I decided to push to force out the other two players. The original player had pocket 5's for the flopped set. I hit my K on the river for a bigger set. Still didn't manage to win though because the chip leader called preflop gambling it up with Ad7d. His flopped nut flush was enough to win the hand after he called my all in re-raise.
Second guessing myself afterwards I was convinced I made the wrong play here. With four of us seeing that flop and since I wasn't re-raised a set or straight draw is very possible. That flop should have been scary even if their wasn't the three diamonds. Since I had the second nut flush draw I figured it was worth the play at the time. I'm still a little undecided on what the right course of action should have been.
I guess that's why this game hasn't been solved yet.