I tried out the Samsung Note 5 for a while and liked the slim feel, but I decided to go back to the Note 4 because it lacked the following features that are found on the Note 4.
There is no built in IR on the Note 5 so any remote app you download won’t help you control devices or TV’s. With the Note 4–I enjoyed using the remote app that looks so much like the real Samsung remote for my Samsung TV.
Ability to flash AOSP roms like Cyanogenmod, Slim Roms etc.
The Note 5 Exynos processor source code isn’t yet available freely so that makes it a pain for Cyanogenmod to start development on a custom pure android Rom (I don’t like Touchwhiz for various reasons). The default messaging app in Touchwhiz really sucks. Touchwhiz has come a long way, but I would go with a stable custom AOSP rom any day–over touchwhiz.
External SD Card and Battery
You can’t remove battery yourself–plus you can’t insert your own SD Card. The S6/S6 Edge+ and Note 5 don’t give you the options. I just can’t live without a removable battery and SD expansion slot.
If you are thinking about flashing Android M Cyanogenmod 13 on your Note 4–you might want to hold off. The current Unofficial release that is on other sites and XDA is not a daily driver. Too many bugs and issues for you to use. SlimRoms is what I would recommend, because they always have nearly stable roms.
The Note 5 does have the right look and overall feel. I loved how the S-pen pops out when I pushed it in. The S-Pen and Phone itself both feel premium with metallic and glass combination.