Bug 1696808

In local copy of rhel-7-server-rpms repository.  Copy the now Pasckages/[0-9a-z]/*.rpm to Packages/ directory.  This mimics what reposync left laying around when Red Hat recently added this subdirectory.

  # cd rhel-7-server-rpms/

  # ls -1 Packages/*.rpm
  ls: cannot access Packages/*.rpm: No such file or directory

  # cp -arp Packages/?/*.rpm Packages/.


So now with duplicates rpms (1 copy in Packages/blah.rpm and one in Packages/b/blah.rpm)
Note RC=0 indicates fail here,as grep has match. Seems to be ~33% fail rate for me (so may have try multiple times)

  # createrepo_c -d --update --keep-all-metadata --local-sqlite -s sha256 --skip-stat rhel-7-server-rpms/
  # bunzip2 -k -c $(ls -1 rhel-7-server-rpms/repodata/*primary.sqlite.bz2) > /tmp/file.db
  # sqlite3 /tmp/file.db '.dump' > /tmp/file.dump
  # grep -axv '.*' /tmp/file.dump > /dev/null
  # echo $?
  0

And repoview originally reports fails too
  # repoview rhel-7-server-rpms/
  Examining repository...done
  Opening primary database...done
  Opening changelogs database...done
  Parsing comps.xml...done
  Examining state db...done
  Collecting letters...done
  Writing package ElectricFence.html
  ...
  Writing package compat-libical1.html
  Traceback (most recent call last):
    File "/bin/repoview", line 940, in <module>
      main()
    File "/bin/repoview", line 937, in main
      Repoview(opts)
    File "/bin/repoview", line 191, in __init__
      packages = self.do_packages(repo_data, group_data, pkgnames)
    File "/bin/repoview", line 544, in do_packages
      pkg_data = self.get_package_data(pkgname)
    File "/bin/repoview", line 442, in get_package_data
      rows = pcursor.fetchall()
  @�'ite3.OperationalError: Could not decode to UTF-8 column 'location_href' with text '�



Here is createrepo_c RPM version I am using
  # rpm -q createrepo_c
  createrepo_c-0.10.0-18.el7.x86_64

Even with Fedora's createrepo_c-0.12.2 this issue appears to be present.
In run below, can see invalid characters in 2nd and 4th to last fields...


Duplicate the Packages/blah.rpm & Packages/b/blah.rpm manually.
Then run createrepo_c 
NOTE: Have to have multiple workers, unable replicate with --workers=1
  # rpm -q createrepo_c
  createrepo_c-0.12.2-1.fc29.x86_64

  # cd rhel-7-server-rpms
  # cp -ap Packages/?/*.rpm Packages/.

  # createrepo_c -d --update --keep-all-metadata --local-sqlite -s sha256 --skip-stat .
  08:06:16: Version: 0.12.2 (Features: DeltaRPM LegacyWeakdeps )
  08:06:16: Signal handler setup
  08:06:16: Thread pool ready
  08:06:16: Dir to scan: ./.repodata
  08:06:16: Dir to scan: ./Packages
  08:06:16: Dir to scan: ./repodata
  <snip>
  08:06:26: Memory cleanup
  08:06:26: All done


Now do some manual checks for non-ascii, which we find
  # bunzip2 -k -c $(ls -1 repodata/*primary.sqlite.bz2) > /tmp/file.db
  # sqlite3 /tmp/file.db '.dump' > /tmp/file.dump
  # grep -naxv '.*' /tmp/file.dump > /tmp/OUTPUT
  # echo $?
  0 <------------ rc=0 means have non ASCII here


Here are the three packages found, and fields with issue
  # awk -F, '{print $1,$3," ... ", $(NF-4), $(NF-2)}' /tmp/OUTPUT 
  107:INSERT INTO packages VALUES(102 'PackageKit-glib'  ...  491028 'P\��'
  5137:INSERT INTO packages VALUES(5132 'libkexiv2-devel'  ...  replace(' $\n��' char(10))
  6953:INSERT INTO packages VALUES(6948 'openldap'  ...  1037392 '(��'


Now do strace, to get the garbage chars
  # strace -ttTvf -s 4096 -o /tmp/strace repoview .
  <snip>
  sqlite3.OperationalError: Could not decode to UTF-8 column 'location_href' with text ' $
  ��'
  # echo $?
  1

  # grep 'Could not decode' /tmp/strace 
  12393 08:15:24.869960 write(2, "Could not decode to UTF-8 column 'location_href' with text ' $\n\220\317\177'", 67) = 67 <0.000004>


With the chars from strace, use printf to set var to them
Then grep /tmp/OUTPUT to show matches one of lines we found above (libkexiv2-devel in this case).
  # var=$(printf "%s\n" $'\n\220\317\177')
  # grep -a $var /tmp/OUTPUT  | awk -F, '{print $1,$3," ... ", $(NF-4), $(NF-2)}'
  5137:INSERT INTO packages VALUES(5132 'libkexiv2-devel'  ...  replace(' $\n��' char(10))

I created PR for it here: https://github.com/rpm-software-management/createrepo_c/pull/151

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2019:3986