Bug 1729811

Summary: [scale] updatevmdynamic broken if too many users logged in - psql ERROR: value too long for type character varying(255)
Product: Red Hat Enterprise Virtualization Manager Reporter: Germano Veit Michel <gveitmic>
Component: ovirt-engineAssignee: Steven Rosenberg <srosenbe>
Status: CLOSED ERRATA QA Contact: Petr Matyáš <pmatyas>
Severity: high Docs Contact:
Priority: high    
Version: 4.3.4CC: ahadas, amashah, bwalton, lleistne, michal.skrivanek, pelauter, rdlugyhe, srosenbe
Target Milestone: ovirt-4.4.1Keywords: ZStream
Target Release: ---   
Hardware: x86_64   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Previously, the guest_cur_user_name of the vm_dynamic database table was limited to 255 characters, not enough for more than approximately 100 user names. As a result, when too many users logged in, updating the table failed with an error. The current release fixes this issue by changing the field type from VARCHAR(255) to TEXT.
Story Points: ---
Clone Of:
: 1734671 (view as bug list) Environment:
Last Closed: 2020-08-04 13:19:49 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: Virt RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1734671    

Description Germano Veit Michel 2019-07-15 01:19:14 UTC
Description of problem:

If a VM has several users logged in, the engine cannot update the vm_dynamic table via the UpdateVmDynamic procedure, leaving everything out of sync, which can be dangerous.

The problem is guest_cur_user_name is limited to 255 chars. If there are enough users logged in to exceed this, the SQL procedure fails.

CREATE OR REPLACE FUNCTION UpdateVmDynamic (
    v_app_list TEXT,
    v_guest_cur_user_name VARCHAR(255),
    ...

2019-07-15 11:14:18,926+10 ERROR [org.ovirt.engine.core.dal.dbbroker.BatchProcedureExecutionConnectionCallback] (EE-ManagedThreadFactory-engineScheduled-Thread-33) [] Can't execute batch: Batch entry 0 select * from public.updatevmdynamic(CAST ('kernel-3.10.0-957.12.2.el7,cloud-init-18.2-1.el7.centos.2,ovirt-guest-agent-common-1.0.16-1.el7,qemu-guest-agent-2.12.0' AS text),CAST ('' AS varchar),CAST (NULL AS varchar),CAST (NULL AS varchar),CAST (NULL AS uuid),CAST ('3.10.0-957.12.2.el7.x86_64' AS varchar),CAST (NULL AS uuid),CAST ('b89ccd0c-7e74-415f-a7c4-f8f3abced3fb'::uuid AS uuid),CAST (1 AS int4),CAST ('f3e6d39a-a404-4a1e-8729-9d37563f8205'::uuid AS uuid),CAST ('engine.kvm' AS varchar),CAST ('192.168.150.253 fe80::5054:ff:fe19:c1ff' AS varchar),CAST ('engine.kvm' AS varchar),CAST (NULL AS timestamptz),CAST (NULL AS timestamptz),CAST (0 AS int8),CAST ('2019-07-05 13:24:42.316+10' AS timestamptz),CAST ('TRUE' AS bool),CAST (0 AS int4),CAST (0 AS int4),CAST (0 AS int4),CAST ('' AS varchar),CAST (NULL AS int4),CAST (0 AS int4),CAST (5 AS int4),CAST (NULL AS varchar),CAST (-1848952666 AS int4),CAST (NULL AS numeric),CAST (NULL AS varchar),CAST ('FALSE' AS bool),CAST ('FALSE' AS bool),CAST (NULL AS varchar),CAST (NULL AS varchar),CAST (NULL AS varchar),CAST (NULL AS varchar),CAST (-1 AS int4),CAST (2 AS int4),CAST (5901 AS int4),CAST (5902 AS int4),CAST ('192.168.150.1' AS varchar),CAST (5900 AS int4),CAST ('192.168.150.1' AS varchar),CAST (0 AS int4),CAST (0 AS int4),CAST (600 AS int4),CAST ('AEST' AS varchar),CAST (1 AS int4),CAST ('' AS varchar),CAST ('CentOS Linux' AS varchar),CAST ('3.10.0-957.12.2.el7.x86_64' AS varchar),CAST ('Linux' AS varchar),CAST ('7' AS varchar),CAST ('[]' AS text)) as result was aborted: ERROR: value too long for type character varying(255)
  Where: SQL statement "UPDATE vm_dynamic
    SET app_list = v_app_list,
        guest_cur_user_name = v_guest_cur_user_name,
        console_cur_user_name = v_console_cur_user_name,
        runtime_name = v_runtime_name,
        console_user_id = v_console_user_id,
        guest_os = v_guest_os,
        migrating_to_vds = v_migrating_to_vds,
        run_on_vds = v_run_on_vds,
        status = v_status,
        vm_host = v_vm_host,
        vm_ip = v_vm_ip,
        vm_fqdn = v_vm_fqdn,
        last_start_time = v_last_start_time,
        boot_time = v_boot_time,
        downtime = v_downtime,
        last_stop_time = v_last_stop_time,
        acpi_enable = v_acpi_enable,
        session = v_session,
        boot_sequence = v_boot_sequence,
        utc_diff = v_utc_diff,
        client_ip = v_client_ip,
        guest_requested_memory = v_guest_requested_memory,
        exit_status = v_exit_status,

Version-Release number of selected component (if applicable):
ovirt-engine-4.3.4.3-1.el7.noarch

How reproducible:
Always

Steps to Reproduce:
1. Have a Guest with agent running
2. Create 100 users in the VM, set password
$ for i in `seq 0 100` ; do useradd germano$i; done;
$ for i in `seq 0 100` ; do echo 'redhat' | passwd --stdin germano$i; done;
3. Login 100 users at the same time
$ for i in `seq 0 100` ; do screen -d -m sshpass -predhat ssh germano$i.150.159; done;

Comment 1 Germano Veit Michel 2019-07-15 02:01:50 UTC
This is the one with the reproduction steps above:

2019-07-15 11:58:42,296+10 ERROR [org.ovirt.engine.core.dal.dbbroker.BatchProcedureExecutionConnectionCallback] (EE-ManagedThreadFactory-engineScheduled-Thread-74) [] Can't execute batch: Batch entry 0 select * from public.updatevmdynamic(CAST ('kernel-3.10.0-957.el7,ovirt-guest-agent-common-1.0.16-1.el7,qemu-guest-agent-2.12.0' AS text),CAST ('germano1, germano5, germano13, germano3, germano2, germano14, germano6, germano7, germano9, germano0, germano11, germano12, germano8, germano4, germano18, germano16, germano21, germano15, germano27, germano20, germano24, germano26, germano28, germano22, germano25, germano30, germano31, germano29, germano34, germano33, germano36, germano32, germano35, germano41, germano43, germano40, germano44, germano47, germano42, germano45, germano46, germano48, germano50, germano51, germano52, germano49, germano56, germano54, germano57, germano59, germano58, germano60, germano62, germano67, germano64, germano63, germano70, germano66, germano69, germano65, germano68, germano75, germano79, germano76, germano72, germano78, germano73, germano74, germano81, germano83, germano77, germano85, germano82, germano92, germano91, germano86, germano95, germano89, germano84, germano93, germano88, germano87, germano98, germano94, germano99, germano96, germano100' AS varchar),CAST (NULL AS varchar),CAST ('CentOS7' AS varchar),CAST (NULL AS uuid),CAST ('3.10.0-957.el7.x86_64' AS varchar),CAST (NULL AS uuid),CAST ('3034ef0c-9a41-44f9-8c1b-0520f83dd04d'::uuid AS uuid),CAST (1 AS int4),CAST ('372ec622-0878-429c-9eaf-bdb1b304b5e7'::uuid AS uuid),CAST ('localhost.localdomain' AS varchar),CAST ('192.168.150.159 fe80::23d6:15db:409c:2b58' AS varchar),CAST (NULL AS varchar),CAST ('2019-07-15 11:42:46.594+10' AS timestamptz),CAST ('2019-07-15 11:42:46.594+10' AS timestamptz),CAST (0 AS int8),CAST ('2019-07-15 11:26:35.776+10' AS timestamptz),CAST ('TRUE' AS bool),CAST (0 AS int4),CAST (0 AS int4),CAST (0 AS int4),CAST ('' AS varchar),CAST (NULL AS int4),CAST (0 AS int4),CAST (5 AS int4),CAST (NULL AS varchar),CAST (-2016404065 AS int4),CAST (NULL AS numeric),CAST (NULL AS varchar),CAST ('FALSE' AS bool),CAST ('FALSE' AS bool),CAST ('Skylake-Client,+spec-ctrl,+ssbd,+md-clear' AS varchar),CAST ('pc-i440fx-rhel7.6.0' AS varchar),CAST (NULL AS varchar),CAST (NULL AS varchar),CAST (-1 AS int4),CAST (1 AS int4),CAST (5901 AS int4),CAST (5902 AS int4),CAST ('192.168.150.2' AS varchar),CAST (5900 AS int4),CAST ('192.168.150.2' AS varchar),CAST (0 AS int4),CAST (0 AS int4),CAST (600 AS int4),CAST ('AEST' AS varchar),CAST (1 AS int4),CAST ('' AS varchar),CAST ('CentOS Linux' AS varchar),CAST ('3.10.0-957.el7.x86_64' AS varchar),CAST ('Linux' AS varchar),CAST ('7' AS varchar),CAST ('[]' AS text)) as result was aborted: ERROR: value too long for type character varying(255)
  Where: SQL statement "UPDATE vm_dynamic
    SET app_list = v_app_list,
        guest_cur_user_name = v_guest_cur_user_name,
        console_cur_user_name = v_console_cur_user_name,
        runtime_name = v_runtime_name,
        console_user_id = v_console_user_id,
        guest_os = v_guest_os,

Comment 3 Martin Perina 2019-07-15 09:33:30 UTC
Ryan, could you please take a look?

Comment 4 Ryan Barry 2019-07-15 11:17:32 UTC
This will require some careful testing, but it's worth checking whether we can change this to plain TEXT, or at least increase the size of the varchar field

Comment 5 Michal Skrivanek 2019-07-16 04:20:10 UTC
Yeah.. did that for couple other fields already

Comment 9 Daniel Gur 2019-08-28 13:11:55 UTC
sync2jira

Comment 10 Daniel Gur 2019-08-28 13:16:08 UTC
sync2jira

Comment 12 RHV bug bot 2019-12-13 13:13:30 UTC
WARN: Bug status (ON_QA) wasn't changed but the folowing should be fixed:

[Found non-acked flags: '{}', ]

For more info please contact: rhv-devops: Bug status (ON_QA) wasn't changed but the folowing should be fixed:

[Found non-acked flags: '{}', ]

For more info please contact: rhv-devops

Comment 13 RHV bug bot 2019-12-20 17:43:29 UTC
WARN: Bug status (ON_QA) wasn't changed but the folowing should be fixed:

[Found non-acked flags: '{}', ]

For more info please contact: rhv-devops: Bug status (ON_QA) wasn't changed but the folowing should be fixed:

[Found non-acked flags: '{}', ]

For more info please contact: rhv-devops

Comment 14 RHV bug bot 2020-01-08 14:48:11 UTC
WARN: Bug status (ON_QA) wasn't changed but the folowing should be fixed:

[Found non-acked flags: '{}', ]

For more info please contact: rhv-devops: Bug status (ON_QA) wasn't changed but the folowing should be fixed:

[Found non-acked flags: '{}', ]

For more info please contact: rhv-devops

Comment 15 RHV bug bot 2020-01-08 15:14:07 UTC
WARN: Bug status (ON_QA) wasn't changed but the folowing should be fixed:

[Found non-acked flags: '{}', ]

For more info please contact: rhv-devops: Bug status (ON_QA) wasn't changed but the folowing should be fixed:

[Found non-acked flags: '{}', ]

For more info please contact: rhv-devops

Comment 16 Petr Matyáš 2020-01-08 15:38:40 UTC
Verified on ovirt-engine-4.4.0-0.13.master.el7.noarch

Comment 17 RHV bug bot 2020-01-24 19:49:59 UTC
WARN: Bug status (VERIFIED) wasn't changed but the folowing should be fixed:

[Found non-acked flags: '{}', ]

For more info please contact: rhv-devops: Bug status (VERIFIED) wasn't changed but the folowing should be fixed:

[Found non-acked flags: '{}', ]

For more info please contact: rhv-devops

Comment 22 errata-xmlrpc 2020-08-04 13:19:49 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Important: RHV Manager (ovirt-engine) 4.4 security, bug fix, and enhancement update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2020:3247