Draft: fix/rapid-cdn-upgradability
Currently rapid-cdn is unupgradable. (It never was, operators were lucky).
This time adding error-page-manager revealed it fully.
As it changes what is published to shared instances, master is uninstalling all shared instances and them installs them with new parameters.
As production clusters are asynchronous and work in parallel frontend nodes are picking up list of updated shared instances, resulting with 404 to given shared. This can lead to 100% of unavailable frontends for the cluster upgrade taking many hours, which supposed to be invisible to users.
Note: For the upgrade from 1.0.469 to 1.0.506 no hacks were possible, as new error-page-manager needs own shared instances to be instantiated, but frontend nodes were disabled to avoid 100% frontend 404, which resulted with chicken and egg problem. Even if it would move forward, then enabling new error-page-manager would result with updated shared instance connection parameters and the full uninstall full install would happen again.
Tasks:
-
prove with test non-upgradability of the rapid-cdn -
find more cases -
fix everything -- no hacks allowed -
maybe other parts of slapos ecosystem needs improvements
-
-
retest on clusters with own reference slapos master