not only that he has two dreams after all , a worthy death like Roger, Oden and WB , so if Luffy kills him he will get that,so Luffy will not fulfill that for him, the other dream is being Joyboy and Luffy is Joyboy after all, so indirectly he takes his another dream too.
to be honest if he dies , it will be only by taking his own life since his dream were crushed.