The promises to Abram concerning Ishmael were different than the promises made to Abraham concerning Isaac.
Genesis 17:7 “I will establish my covenant between me and you and your seed after you throughout their generations for an everlasting covenant, to be a God to you and to your seed after you.”
Yahweh never promised to be the God of Ishmael. Isaac was the child of promise for Abraham.