Openstack Magnum kube_master: Went to status ERROR due to "Message: Exceeded maximum number of retries

3/5/2021

I tried to deploy a kubernetes cluster with one master node and one worker node in openstack magnum. But it returned an error:

ResourceInError: resources.kube_masters.resources0.resources.kube-master: Went to status ERROR due to "Message: Exceeded maximum number of retries. Exceeded max scheduling attempts 3 for instance c477d0d3-6176-4cec-b5aa-3a2fb7c77435. Last exception: Argument must be bytes or unicode, got 'NoneType', Code: 500"

Additional info:\ Image used: Fedora-CoreOS 32\ Flavor: 4VCPUs, 4GB RAM, 25GB disk\ docker storage driver: devicemapper\ docker volume size: 10GB

Nova logs:

2021-03-05 02:58:29.180 22 ERROR nova.scheduler.utils req-aa6a7c8b-3d9a-4c8d-9d56-dfcb104ae828 53dd56f7fefc4bd38e2e4b1e8dde2b51 a67073edfee14079a6dda119969895c9 - default default Error from last host: kolla-ceph-compute17 (node kolla-ceph-compute17): 'Traceback (most recent call last):\n', ' File "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/compute/manager.py", line 2437, in _build_and_run_instance\n
block_device_info=block_device_info)\n', ' File "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/virt/libvirt/driver.py", line 3550, in spawn\n mdevs=mdevs)\n', ' File "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/virt/libvirt/driver.py", line 6158, in _get_guest_xml\n xml = conf.to_xml()\n', ' File "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/virt/libvirt/config.py", line 79, in to_xml\n root = self.format_dom()\n', ' File "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/virt/libvirt/config.py", line 2726, in format_dom\n self._format_devices(root)\n', ' File "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/virt/libvirt/config.py", line 2680, in _format_devices\n
devices.append(dev.format_dom())\n', ' File "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/virt/libvirt/config.py", line 1037, in format_dom\n auth.set("username", self.auth_username)\n', ' File "src/lxml/etree.pyx", line 815, in lxml.etree._Element.set\n', ' File "src/lxml/apihelpers.pxi", line 593, in lxml.etree._setAttributeValue\n', ' File "src/lxml/apihelpers.pxi", line 1525, in lxml.etree._utf8\n', "TypeError: Argument must be bytes or unicode, got 'NoneType'\n", '\nDuring handling of the above exception, another exception occurred:\n\n', 'Traceback (most recent call last):\n', ' File "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/compute/manager.py", line 2161, in _do_build_and_run_instance\n filter_properties, request_spec)\n', ' File "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/compute/manager.py", line 2537, in _build_and_run_instance\n
instance_uuid=instance.uuid, reason=six.text_type(e))\n', "nova.exception.RescheduledException: Build of instance 8fd65139-c43e-40bd-836e-0ec57ea78960 was re-scheduled: Argument must be bytes or unicode, got 'NoneType'\n"
2021-03-05 02:59:08.970 20 ERROR nova.scheduler.utils req-aa6a7c8b-3d9a-4c8d-9d56-dfcb104ae828 53dd56f7fefc4bd38e2e4b1e8dde2b51 a67073edfee14079a6dda119969895c9 - default default Error from last host: kolla-ceph-compute18 (node kolla-ceph-compute18): 'Traceback (most recent call last):\n', ' File "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/compute/manager.py", line 2437, in _build_and_run_instance\n
block_device_info=block_device_info)\n', ' File "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/virt/libvirt/driver.py", line 3550, in spawn\n mdevs=mdevs)\n', ' File "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/virt/libvirt/driver.py", line 6158, in _get_guest_xml\n xml = conf.to_xml()\n', ' File "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/virt/libvirt/config.py", line 79, in to_xml\n root = self.format_dom()\n', ' File "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/virt/libvirt/config.py", line 2726, in format_dom\n self._format_devices(root)\n', ' File "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/virt/libvirt/config.py", line 2680, in _format_devices\n
devices.append(dev.format_dom())\n', ' File "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/virt/libvirt/config.py", line 1037, in format_dom\n auth.set("username", self.auth_username)\n', ' File "src/lxml/etree.pyx", line 815, in lxml.etree._Element.set\n', ' File "src/lxml/apihelpers.pxi", line 593, in lxml.etree._setAttributeValue\n', ' File "src/lxml/apihelpers.pxi", line 1525, in lxml.etree._utf8\n', "TypeError: Argument must be bytes or unicode, got 'NoneType'\n", '\nDuring handling of the above exception, another exception occurred:\n\n', 'Traceback (most recent call last):\n', ' File "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/compute/manager.py", line 2161, in _do_build_and_run_instance\n filter_properties, request_spec)\n', ' File "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/compute/manager.py", line 2537, in _build_and_run_instance\n
instance_uuid=instance.uuid, reason=six.text_type(e))\n', "nova.exception.RescheduledException: Build of instance 8fd65139-c43e-40bd-836e-0ec57ea78960 was re-scheduled: Argument must be bytes or unicode, got 'NoneType'\n"
root@kolla-infra2:~# cat /var/log/kolla/magnum/log | egrep -i "(2021-03-05).error"

Magnum logs:

2021-03-05 02:18:26.681 19 ERROR magnum.drivers.heat.k8s_fedora_template_def req-bc1f4cad-1636-42b4-ab80-f7e7c3459c5d - - - - - Failed to load default keystone auth policy: FileNotFoundError: Errno 2 No such file or directory: '/etc/magnum/keystone_auth_default_policy.json' 2021-03-05 02:20:25.721 32 ERROR magnum.drivers.heat.k8s_fedora_template_def req-4e8ac06c-debd-4980-9cbf-7a48684e75e1 - - - - - Failed to load default keystone auth policy: FileNotFoundError: Errno 2 No such file or directory: '/etc/magnum/keystone_auth_default_policy.json' 2021-03-05 02:45:17.069 27 ERROR magnum.drivers.heat.k8s_fedora_template_def req-205dd35a-5546-4919-875a-cfd15feeeadf - - - - - Failed to load default keystone auth policy: FileNotFoundError: Errno 2 No such file or directory: '/etc/magnum/keystone_auth_default_policy.json' 2021-03-05 03:00:00.418 6 ERROR magnum.drivers.heat.driver req-17a7777b-0e70-4b8f-9199-026cd3f3d2ae - - - - - Nodegroup error, stack status: CREATE_FAILED, stack_id: 1671e517-8227-47c3-9c7a-4642ececde33, reason: Resource CREATE failed: ResourceInError: resources.kube_masters.resources0.resources.kube-master: Went to status ERROR due to "Message: Exceeded maximum number of retries. Exceeded max scheduling attempts 3 for instance c477d0d3-6176-4cec-b5aa-3a2fb7c77435. Last exception: Argument must be bytes or unicode, got 'NoneType', Code: 500" 2021-03-05 03:00:00.621 6 ERROR magnum.drivers.heat.driver req-17a7777b-0e70-4b8f-9199-026cd3f3d2ae - - - - - Nodegroup error, stack status: CREATE_FAILED, stack_id: 1671e517-8227-47c3-9c7a-4642ececde33, reason: Resource CREATE failed: ResourceInError: resources.kube_masters.resources0.resources.kube-master: Went to status ERROR due to "Message: Exceeded maximum number of retries. Exceeded max scheduling attempts 3 for instance c477d0d3-6176-4cec-b5aa-3a2fb7c77435. Last exception: Argument must be bytes or unicode, got 'NoneType', Code: 500"

Heat logs:

2021-03-05 02:59:49.270 21 ERROR heat.engine.resource Traceback (most recent call last): 2021-03-05 02:59:49.270 21 ERROR heat.engine.resource File "/var/lib/kolla/venv/lib/python3.6/site-packages/heat/engine/resource.py", line 920, in _action_recorder 2021-03-05 02:59:49.270 21 ERROR heat.engine.resource yield 2021-03-05 02:59:49.270 21 ERROR heat.engine.resource File "/var/lib/kolla/venv/lib/python3.6/site-packages/heat/engine/resource.py", line 1033, in _do_action 2021-03-05 02:59:49.270 21 ERROR heat.engine.resource yield self.action_handler_task(action, args=handler_args) 2021-03-05 02:59:49.270 21 ERROR heat.engine.resource File "/var/lib/kolla/venv/lib/python3.6/site-packages/heat/engine/scheduler.py", line 346, in wrapper 2021-03-05 02:59:49.270 21 ERROR heat.engine.resource step = next(subtask) 2021-03-05 02:59:49.270 21 ERROR heat.engine.resource File "/var/lib/kolla/venv/lib/python3.6/site-packages/heat/engine/resource.py", line 982, in action_handler_task 2021-03-05 02:59:49.270 21 ERROR heat.engine.resource done = check(handler_data) 2021-03-05 02:59:49.270 21 ERROR heat.engine.resource File "/var/lib/kolla/venv/lib/python3.6/site-packages/heat/engine/resources/stack_resource.py", line 409, in check_create_complete 2021-03-05 02:59:49.270 21 ERROR heat.engine.resource return self._check_status_complete(self.CREATE) 2021-03-05 02:59:49.270 21 ERROR heat.engine.resource File "/var/lib/kolla/venv/lib/python3.6/site-packages/heat/engine/resources/stack_resource.py", line 463, in _check_status_complete 2021-03-05 02:59:49.270 21 ERROR heat.engine.resource action=action) 2021-03-05 02:59:49.270 21 ERROR heat.engine.resource heat.common.exception.ResourceFailure: ResourceInError: resources0.resources.kube-master: Went to status ERROR due to "Message: Exceeded maximum number of retries. Exceeded max scheduling attempts 3 for instance c477d0d3-6176-4cec-b5aa-3a2fb7c77435. Last exception: Argument must be bytes or unicode, got 'NoneType', Code: 500" 2021-03-05 02:59:49.270 21 ERROR heat.engine.resource 2021-03-05 02:59:49.291 21 INFO heat.engine.stack req-9b34fa4a-da80-45aa-9596-6880e4a5d1ce - admin - default default Stack CREATE FAILED (k8s-test-cluster-nyrxxu2pqvnj-kube_masters-5w5xewm2u4c3): Resource CREATE failed: ResourceInError: resources0.resources.kube-master: Went to status ERROR due to "Message: Exceeded maximum number of retries. Exceeded max scheduling attempts 3 for instance c477d0d3-6176-4cec-b5aa-3a2fb7c77435. Last exception: Argument must be bytes or unicode, got 'NoneType', Code: 500"

Thank you.

-- Michael Reuben Alagon
kubernetes
openstack

0 Answers