Bug 1734671 - [scale] updatevmdynamic broken if too many users logged in - psql ERROR: value too long for type character varying(255)
Summary: [scale] updatevmdynamic broken if too many users logged in - psql ERROR: valu...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Enterprise Virtualization Manager
Classification: Red Hat
Component: ovirt-engine
Version: 4.3.6
Hardware: x86_64
OS: Linux
high
high
Target Milestone: ovirt-4.3.6
: 4.3.6
Assignee: Steven Rosenberg
QA Contact: Petr Matyáš
URL:
Whiteboard:
Depends On: 1729811
Blocks:
TreeView+ depends on / blocked
 
Reported: 2019-07-31 07:43 UTC by Steven Rosenberg
Modified: 2019-10-23 07:33 UTC (History)
8 users (show)

Fixed In Version: ovirt-engine-4.3.6.1
Doc Type: Bug Fix
Doc Text:
The Database field guest_cur_user_name in the vm_dynamic Database Table was limited to a size of 255 characters, which was not enough to accept a large number of users (for example, more than 100) to log in. In this release, there is no character limit for the guest_cur_user_name field.
Clone Of: 1729811
Environment:
Last Closed: 2019-10-10 15:36:58 UTC
oVirt Team: Virt
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Red Hat Knowledge Base (Solution) 4282351 0 Troubleshoot None Engine logs filled with ERROR: value too long for type character varying(255) 2019-08-01 07:26:02 UTC
Red Hat Product Errata RHEA-2019:3010 0 None None None 2019-10-10 15:37:11 UTC
oVirt gerrit 102297 0 ovirt-engine-4.3 MERGED engine: Change Guest User field to Type TEXT 2020-08-04 14:21:39 UTC

Description Steven Rosenberg 2019-07-31 07:43:33 UTC
+++ This bug was initially created as a clone of Bug #1729811 +++

Description of problem:

If a VM has several users logged in, the engine cannot update the vm_dynamic table via the UpdateVmDynamic procedure, leaving everything out of sync, which can be dangerous.

The problem is guest_cur_user_name is limited to 255 chars. If there are enough users logged in to exceed this, the SQL procedure fails.

CREATE OR REPLACE FUNCTION UpdateVmDynamic (
    v_app_list TEXT,
    v_guest_cur_user_name VARCHAR(255),
    ...

2019-07-15 11:14:18,926+10 ERROR [org.ovirt.engine.core.dal.dbbroker.BatchProcedureExecutionConnectionCallback] (EE-ManagedThreadFactory-engineScheduled-Thread-33) [] Can't execute batch: Batch entry 0 select * from public.updatevmdynamic(CAST ('kernel-3.10.0-957.12.2.el7,cloud-init-18.2-1.el7.centos.2,ovirt-guest-agent-common-1.0.16-1.el7,qemu-guest-agent-2.12.0' AS text),CAST ('' AS varchar),CAST (NULL AS varchar),CAST (NULL AS varchar),CAST (NULL AS uuid),CAST ('3.10.0-957.12.2.el7.x86_64' AS varchar),CAST (NULL AS uuid),CAST ('b89ccd0c-7e74-415f-a7c4-f8f3abced3fb'::uuid AS uuid),CAST (1 AS int4),CAST ('f3e6d39a-a404-4a1e-8729-9d37563f8205'::uuid AS uuid),CAST ('engine.kvm' AS varchar),CAST ('192.168.150.253 fe80::5054:ff:fe19:c1ff' AS varchar),CAST ('engine.kvm' AS varchar),CAST (NULL AS timestamptz),CAST (NULL AS timestamptz),CAST (0 AS int8),CAST ('2019-07-05 13:24:42.316+10' AS timestamptz),CAST ('TRUE' AS bool),CAST (0 AS int4),CAST (0 AS int4),CAST (0 AS int4),CAST ('' AS varchar),CAST (NULL AS int4),CAST (0 AS int4),CAST (5 AS int4),CAST (NULL AS varchar),CAST (-1848952666 AS int4),CAST (NULL AS numeric),CAST (NULL AS varchar),CAST ('FALSE' AS bool),CAST ('FALSE' AS bool),CAST (NULL AS varchar),CAST (NULL AS varchar),CAST (NULL AS varchar),CAST (NULL AS varchar),CAST (-1 AS int4),CAST (2 AS int4),CAST (5901 AS int4),CAST (5902 AS int4),CAST ('192.168.150.1' AS varchar),CAST (5900 AS int4),CAST ('192.168.150.1' AS varchar),CAST (0 AS int4),CAST (0 AS int4),CAST (600 AS int4),CAST ('AEST' AS varchar),CAST (1 AS int4),CAST ('' AS varchar),CAST ('CentOS Linux' AS varchar),CAST ('3.10.0-957.12.2.el7.x86_64' AS varchar),CAST ('Linux' AS varchar),CAST ('7' AS varchar),CAST ('[]' AS text)) as result was aborted: ERROR: value too long for type character varying(255)
  Where: SQL statement "UPDATE vm_dynamic
    SET app_list = v_app_list,
        guest_cur_user_name = v_guest_cur_user_name,
        console_cur_user_name = v_console_cur_user_name,
        runtime_name = v_runtime_name,
        console_user_id = v_console_user_id,
        guest_os = v_guest_os,
        migrating_to_vds = v_migrating_to_vds,
        run_on_vds = v_run_on_vds,
        status = v_status,
        vm_host = v_vm_host,
        vm_ip = v_vm_ip,
        vm_fqdn = v_vm_fqdn,
        last_start_time = v_last_start_time,
        boot_time = v_boot_time,
        downtime = v_downtime,
        last_stop_time = v_last_stop_time,
        acpi_enable = v_acpi_enable,
        session = v_session,
        boot_sequence = v_boot_sequence,
        utc_diff = v_utc_diff,
        client_ip = v_client_ip,
        guest_requested_memory = v_guest_requested_memory,
        exit_status = v_exit_status,

Version-Release number of selected component (if applicable):
ovirt-engine-4.3.4.3-1.el7.noarch

How reproducible:
Always

Steps to Reproduce:
1. Have a Guest with agent running
2. Create 100 users in the VM, set password
$ for i in `seq 0 100` ; do useradd germano$i; done;
$ for i in `seq 0 100` ; do echo 'redhat' | passwd --stdin germano$i; done;
3. Login 100 users at the same time
$ for i in `seq 0 100` ; do screen -d -m sshpass -predhat ssh germano$i.150.159; done;

--- Additional comment from Germano Veit Michel on 2019-07-15 02:01:50 UTC ---

This is the one with the reproduction steps above:

2019-07-15 11:58:42,296+10 ERROR [org.ovirt.engine.core.dal.dbbroker.BatchProcedureExecutionConnectionCallback] (EE-ManagedThreadFactory-engineScheduled-Thread-74) [] Can't execute batch: Batch entry 0 select * from public.updatevmdynamic(CAST ('kernel-3.10.0-957.el7,ovirt-guest-agent-common-1.0.16-1.el7,qemu-guest-agent-2.12.0' AS text),CAST ('germano1, germano5, germano13, germano3, germano2, germano14, germano6, germano7, germano9, germano0, germano11, germano12, germano8, germano4, germano18, germano16, germano21, germano15, germano27, germano20, germano24, germano26, germano28, germano22, germano25, germano30, germano31, germano29, germano34, germano33, germano36, germano32, germano35, germano41, germano43, germano40, germano44, germano47, germano42, germano45, germano46, germano48, germano50, germano51, germano52, germano49, germano56, germano54, germano57, germano59, germano58, germano60, germano62, germano67, germano64, germano63, germano70, germano66, germano69, germano65, germano68, germano75, germano79, germano76, germano72, germano78, germano73, germano74, germano81, germano83, germano77, germano85, germano82, germano92, germano91, germano86, germano95, germano89, germano84, germano93, germano88, germano87, germano98, germano94, germano99, germano96, germano100' AS varchar),CAST (NULL AS varchar),CAST ('CentOS7' AS varchar),CAST (NULL AS uuid),CAST ('3.10.0-957.el7.x86_64' AS varchar),CAST (NULL AS uuid),CAST ('3034ef0c-9a41-44f9-8c1b-0520f83dd04d'::uuid AS uuid),CAST (1 AS int4),CAST ('372ec622-0878-429c-9eaf-bdb1b304b5e7'::uuid AS uuid),CAST ('localhost.localdomain' AS varchar),CAST ('192.168.150.159 fe80::23d6:15db:409c:2b58' AS varchar),CAST (NULL AS varchar),CAST ('2019-07-15 11:42:46.594+10' AS timestamptz),CAST ('2019-07-15 11:42:46.594+10' AS timestamptz),CAST (0 AS int8),CAST ('2019-07-15 11:26:35.776+10' AS timestamptz),CAST ('TRUE' AS bool),CAST (0 AS int4),CAST (0 AS int4),CAST (0 AS int4),CAST ('' AS varchar),CAST (NULL AS int4),CAST (0 AS int4),CAST (5 AS int4),CAST (NULL AS varchar),CAST (-2016404065 AS int4),CAST (NULL AS numeric),CAST (NULL AS varchar),CAST ('FALSE' AS bool),CAST ('FALSE' AS bool),CAST ('Skylake-Client,+spec-ctrl,+ssbd,+md-clear' AS varchar),CAST ('pc-i440fx-rhel7.6.0' AS varchar),CAST (NULL AS varchar),CAST (NULL AS varchar),CAST (-1 AS int4),CAST (1 AS int4),CAST (5901 AS int4),CAST (5902 AS int4),CAST ('192.168.150.2' AS varchar),CAST (5900 AS int4),CAST ('192.168.150.2' AS varchar),CAST (0 AS int4),CAST (0 AS int4),CAST (600 AS int4),CAST ('AEST' AS varchar),CAST (1 AS int4),CAST ('' AS varchar),CAST ('CentOS Linux' AS varchar),CAST ('3.10.0-957.el7.x86_64' AS varchar),CAST ('Linux' AS varchar),CAST ('7' AS varchar),CAST ('[]' AS text)) as result was aborted: ERROR: value too long for type character varying(255)
  Where: SQL statement "UPDATE vm_dynamic
    SET app_list = v_app_list,
        guest_cur_user_name = v_guest_cur_user_name,
        console_cur_user_name = v_console_cur_user_name,
        runtime_name = v_runtime_name,
        console_user_id = v_console_user_id,
        guest_os = v_guest_os,

--- Additional comment from Germano Veit Michel on 2019-07-15 03:51:23 UTC ---

Looks like the customer VMs are SSH proxies, waiting for confirmation of the use case.

--- Additional comment from Martin Perina on 2019-07-15 09:33:30 UTC ---

Ryan, could you please take a look?

--- Additional comment from Ryan Barry on 2019-07-15 11:17:32 UTC ---

This will require some careful testing, but it's worth checking whether we can change this to plain TEXT, or at least increase the size of the varchar field

--- Additional comment from Michal Skrivanek on 2019-07-16 04:20:10 UTC ---

Yeah.. did that for couple other fields already

--- Additional comment from Michal Skrivanek on 2019-07-16 06:45:30 UTC ---

zstream?

--- Additional comment from Ryan Barry on 2019-07-16 10:02:32 UTC ---

Clone once there's a patch

Comment 4 Petr Matyáš 2019-08-26 10:39:08 UTC
Verified on ovirt-engine-4.3.6.3-0.1.el7.noarch

Comment 5 Daniel Gur 2019-08-28 13:11:28 UTC
sync2jira

Comment 6 Daniel Gur 2019-08-28 13:15:40 UTC
sync2jira

Comment 7 RHV bug bot 2019-09-25 08:46:31 UTC
INFO: Bug status (VERIFIED) wasn't changed but the folowing should be fixed:

[Tag 'ovirt-engine-4.3.5.6' doesn't contain patch 'https://gerrit.ovirt.org/102297']
gitweb: https://gerrit.ovirt.org/gitweb?p=ovirt-engine.git;a=shortlog;h=refs/tags/ovirt-engine-4.3.5.6

For more info please contact: rhv-devops

Comment 10 errata-xmlrpc 2019-10-10 15:36:58 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHEA-2019:3010


Note You need to log in before you can comment on or make changes to this bug.