I started using Svorak A5 (https://svorak.ahall.se/) the summer before I started college in 2016 (with the standard bindings on the numerical keys as well, so it's redundant). Since I have started using more alphabets I have modified the layout quite a lot over the years. I now have Norwegian, Icelandic and Sami characters as well. I currently bind up to four characters per key.
It's really true that the extra layers doesn't disturb at all. Some characters I don't even know where they are, but most I use daily. There are also some that are placed strategically, so that I can find them when i need them (ŋ is Ctrl+Shift+AltGr+N, the ◌̌ dead key is Ctrl+Shift+AltGr+^ and ° is Ctrl+Shift+AltGr+0).
I use a similar keyboard layout, it took me a couple of weeks to get back to being able to competently type when I first started using it and now I wouldn't go back. It's not really any different to getting used to the fact that when you hold shift the 1 key inserts and exclamation mark instead, you probably don't consciously hold the shift key and then look around your keyboard trying to work out which key you need to hit.
If I did not switch to chording with plover, I would try this out. I prefer chording now, but colemak was my layout of choice when I still typed a lot. Layers are more confusing for me than chords, and I find chords + phonetics to be more flexible in most cases.