Hello there, and MerryXmas.
Hmm-- well I have three mangas that are very similar to what you look, I hope it is one of them.

The first one, here the MC is considered to be an Assassin and decides to leave his friends.
Title: My Status as an Assassin Obviously Exceeds the Brave

Second.. here the MC is left to die by his friend, and finally leaves to know about the world and decides to return.
Title: Arifureta Shokugyou de Sekai Saikyou

And finally, here the MC already knows magic, is very powerful but the one summoned are only three people.. counting him.
He decides to leave his friends and look for a way to return both of them back to their world.
Title: Isekai mahou wa okureteru

From all the three, I find the last one the better. But everyone to their like.

If it's not the thing you are looking for, maybe you can tell me more details.