Bug 1719647

(In reply to Richard W.M. Jones from comment #0)
>   /usr/share/doc/ruby-hivex/api/created.rid

Note this is not a gzip'ed js file, although it changes across architectures.

(In reply to Pino Toscano from comment #1)
> (In reply to Richard W.M. Jones from comment #0)
> >   /usr/share/doc/ruby-hivex/api/created.rid
> 
> Note this is not a gzip'ed js file, although it changes across architectures.

Yes this one is likely a separate bug.  The file contains timestamps
which are all the same for source files, but then there's another
timestamp which is different although it's not clear what it refers to:

$ cat ./i686/usr/share/doc/ruby-hivex/api/created.rid
Sat, 25 May 2019 04:07:14 +0000
./README.rdoc	Mon, 03 Dec 2012 16:05:45 +0000
./lib/hivex.rb	Mon, 03 Dec 2012 16:05:45 +0000
./ext/hivex/_hivex.c	Mon, 26 Feb 2018 10:50:23 +0000
./ext/hivex/extconf.h	Sat, 25 May 2019 04:07:14 +0000

$ cat x86_64//usr/share/doc/ruby-hivex/api/created.rid 
Sat, 25 May 2019 04:07:09 +0000
./README.rdoc	Mon, 03 Dec 2012 16:05:45 +0000
./lib/hivex.rb	Mon, 03 Dec 2012 16:05:45 +0000
./ext/hivex/_hivex.c	Mon, 26 Feb 2018 10:50:23 +0000
./ext/hivex/extconf.h	Sat, 25 May 2019 04:07:09 +0000

See also: https://bugs.ruby-lang.org/issues/13627

We could probably just rm this file.

And this is probably workaround for the .gz:

https://github.com/ruby/rdoc/pull/569

Hm, so we have RDoc 6.0.1 while both these issues were resolved/workarounded in 6.0.2, but you still would need to use the ENV variable. Not sure what we can do about this issue in RHEL.

But the env variable usage feels as a material for Fedora ...

(In reply to Vít Ondruch from comment #4)
> Hm, so we have RDoc 6.0.1 while both these issues were resolved/workarounded
> in 6.0.2, but you still would need to use the ENV variable. Not sure what we
> can do about this issue in RHEL.

It looks like both the patches use the SOURCE_DATE_EPOCH approach of reproducible builds.
It is easy enough to use it in a spec file -- for example Jirka Denemar and me came out with
  export SOURCE_DATE_EPOCH=$(stat --printf='%Y' %{_specdir}/%{name}.spec)
for libvirt.

> But the env variable usage feels as a material for Fedora ...

Not sure, what do you mean with that?

Shall we assign this to hivex to add that line as suggested by Pino,
and duplicate it for several products:
  Fedora / hivex
  Fedora / libguestfs
  RHEL 8 / libguestfs
?

Hit "Save" too early...

Vít: if you backport the two patches (or rebase rdoc), and reassign this bug to rdoc, I can clone this for hivex in RHEL & RHEL AV, and fix it once the fixed rdoc is available.
What do you think?

(In reply to Pino Toscano from comment #5)
> (In reply to Vít Ondruch from comment #4)
> > Hm, so we have RDoc 6.0.1 while both these issues were resolved/workarounded
> > in 6.0.2, but you still would need to use the ENV variable. Not sure what we
> > can do about this issue in RHEL.
> 
> It looks like both the patches use the SOURCE_DATE_EPOCH approach of
> reproducible builds.
> It is easy enough to use it in a spec file -- for example Jirka Denemar and
> me came out with
>   export SOURCE_DATE_EPOCH=$(stat --printf='%Y' %{_specdir}/%{name}.spec)
> for libvirt.

But this is not supported by:

~~~
Version-Release number of selected component (if applicable):

rubygem-rdoc-6.0.1-103.module+el8+2671+ebcc7ee0.noarch
~~~
 
> > But the env variable usage feels as a material for Fedora ...
> 
> Not sure, what do you mean with that?

Not sure how the noarch packages are processed on Fedora, but I assume that without the env variable, we have to face such issues in Fedora at some circumstances. And it would be probably useful to set the env varaible by default. But this should be developed and tested in Fedora first.

(In reply to Vít Ondruch from comment #8)
> (In reply to Pino Toscano from comment #5)
> > (In reply to Vít Ondruch from comment #4)
> > > Hm, so we have RDoc 6.0.1 while both these issues were resolved/workarounded
> > > in 6.0.2, but you still would need to use the ENV variable. Not sure what we
> > > can do about this issue in RHEL.
> > 
> > It looks like both the patches use the SOURCE_DATE_EPOCH approach of
> > reproducible builds.
> > It is easy enough to use it in a spec file -- for example Jirka Denemar and
> > me came out with
> >   export SOURCE_DATE_EPOCH=$(stat --printf='%Y' %{_specdir}/%{name}.spec)
> > for libvirt.
> 
> But this is not supported by:
> 
> ~~~
> Version-Release number of selected component (if applicable):
> 
> rubygem-rdoc-6.0.1-103.module+el8+2671+ebcc7ee0.noarch
> ~~~

Yes, I know, I meant that it will be easy to use SOURCE_DATE_EPOCH once the patches/fixes are available in rdoc.

> > > But the env variable usage feels as a material for Fedora ...
> > 
> > Not sure, what do you mean with that?
> 
> Not sure how the noarch packages are processed on Fedora, but I assume that
> without the env variable, we have to face such issues in Fedora at some
> circumstances. And it would be probably useful to set the env varaible by
> default. But this should be developed and tested in Fedora first.

I don't see these problems in the Fedora packaging, so I assume it is not needed there.

When the patches are applied, the reproducer is as easy as:

~~~
$ cd into/ruby/spec/directory

$ rdoc test_systemtap.rb # or any other .rb file
... snip ...

$ md5sum doc/created.rid doc/js/*.js.gz 
5cc8afe1bdfeca5abd9c53c7be0aff02  doc/created.rid
6675f6a42a0685d5feb02c3802c4311e  doc/js/navigation.js.gz
cb45ba52509d3c1486f93594c3577481  doc/js/searcher.js.gz
e12315987bf7a8525658cac12b41165c  doc/js/search_index.js.gz

$ rdoc test_systemtap.rb # or any other .rb file
... snip ...

$ md5sum doc/created.rid doc/js/*.js.gz 
86f57716613a1a7bb519692c672435b2  doc/created.rid
6675f6a42a0685d5feb02c3802c4311e  doc/js/navigation.js.gz
cb45ba52509d3c1486f93594c3577481  doc/js/searcher.js.gz
6830b12fa4a33b34d0954243eff4b4db  doc/js/search_index.js.gz

$ # Checksums are modified due to timestamps

$ SOURCE_DATE_EPOCH=$(stat --printf='%Y' ruby.spec) rdoc test_systemtap.rb
... snip ...

$ md5sum doc/created.rid doc/js/*.js.gz 
b64541e4046243f4272aee9749ac4fc4  doc/created.rid
6675f6a42a0685d5feb02c3802c4311e  doc/js/navigation.js.gz
cb45ba52509d3c1486f93594c3577481  doc/js/searcher.js.gz
03852b912f354db9e92d4bc79479558e  doc/js/search_index.js.gz

$ SOURCE_DATE_EPOCH=$(stat --printf='%Y' ruby.spec) rdoc test_systemtap.rb
... snip ...

$ md5sum doc/created.rid doc/js/*.js.gz 
b64541e4046243f4272aee9749ac4fc4  doc/created.rid
6675f6a42a0685d5feb02c3802c4311e  doc/js/navigation.js.gz
cb45ba52509d3c1486f93594c3577481  doc/js/searcher.js.gz
03852b912f354db9e92d4bc79479558e  doc/js/search_index.js.gz

$ # Checksums are stable, because the predefined timestamp is used.
~~~

I thinking to include this PR along the upstream patches:

https://src.fedoraproject.org/rpms/ruby/pull-request/47

This sets the SOURCE_DATE_EPOCH for every rubygem- package. It won't help this particular case, but others might hopefully benefit.

Note: this will be fixed new version redhat-rpm-config.
We do not need to change ruby.
https://src.fedoraproject.org/rpms/ruby/pull-request/47#comment-26983
https://src.fedoraproject.org/rpms/redhat-rpm-config/pull-request/57

(In reply to Jun Aruga from comment #12)
> Note: this will be fixed new version redhat-rpm-config.
> We do not need to change ruby.

This is RHEL ticket not Fedora

* So unless somebody is going to request fix for redhat-rpm-config in RHEL, the SOURCE_DATE_EPOCH is not going to be set by rpmbuild
* The Ruby 2.5 does not have the rubygem-rdoc up2date, so the patches needs to be applied.

Just summarizing this issue.

If rubygem-rdoc < v6.0.2, need below patch.
https://github.com/ruby/rdoc/pull/569

rubygem-rdoc versions in rpms/ruby
* stream-ruby-2.6-rhel-8.1.0 branch: 6.1.0
* stream-ruby-2.5-rhel-8.1.0 branch: 6.0.1

(In reply to Jun Aruga from comment #14)
> If rubygem-rdoc < v6.0.2, need below patch.
> https://github.com/ruby/rdoc/pull/569

This ^^ is incorrect. Let me fix this for you:

If rubygem-rdoc < v6.0.2, need below patches.
https://github.com/ruby/rdoc/pull/570
https://github.com/ruby/rdoc/pull/569

I'm a bit cnfused about where we are with this bug.  We can certainly patch hivex to
fix this (just for hivex of course); if so pleae reassign the bug to hivex.  But it
looks as if to fix this "properly" we'll need to patch redhat-rpm-config.  Contrary
to claims here, it's perfectly possible for us to do this, we will need to file a bug
against redhat-rpm-config.  I assume (correctly?) that if we fix redhat-rpm-config
then no change will be needed to hivex.

> if we fix redhat-rpm-config then no change will be needed to hivex.

I think it depends what you want to fix.

Right now the logic to create rdoc document in hivex is in the process of "make".

https://src.osci.redhat.com/rpms/hivex/blob/stream-rhel-rhel-8.1.0/f/hivex.spec#_191

```
+ make V=1 INSTALLDIRS=vendor -j4
...
rake rdoc
Parsing sources...
 25% [ 1/ 4]  README.rdoc
 50% [ 2/ 4]  ext/hivex/_hivex.c
 75% [ 3/ 4]  ext/hivex/extconf.h
100% [ 4/ 4]  lib/hivex.rb
BUILDSTDERR: Generating Darkfish format into /builddir/build/BUILD/hivex-1.3.15/ruby/doc/site/api...
  Files:       4
  Classes:     2 (2 undocumented)
  Modules:     1 (1 undocumented)
  Constants:   0 (0 undocumented)
  Attributes:  0 (0 undocumented)
  Methods:    30 (0 undocumented)
  Total:      33 (3 undocumented)
   90.91% documented
  Elapsed: 0.2s
```

Below Rakefile.in is used for the "rake rdoc".
https://github.com/libguestfs/hivex/blob/v1.3.15/ruby/Rakefile.in#L79-L83

```
RDOC_FILES = FileList[
    "@srcdir@/README.rdoc",
    "@srcdir@/lib/**/*.rb",
    "@srcdir@/ext/**/*.[ch]"
]
```

We tried to set a value of SOURCE_DATE_EPOCH in %gem_install macro in rpms/ruby ruby.spec that is used to install rubygem-* RPM package. But in case of ruby-hivex, it is not used. I suppose that is the reason SOURCE_DATE_EPOCH is not set for hte ruby-hivex's rdoc document gz file's meta data (?).

If we can fix redhat-rpm-config like Fedora rawhide to set SOURCE_DATE_EPOCH value, you do not have to add below kind of hivex.spec before "make". Otherwise you need to add the logic.

```
export SOURCE_DATE_EPOCH=$(stat --printf='%Y' %{SOURCE0})
```

Let me debug a little bit more. I want to see what happens when set "export SOURCE_DATE_EPOCH" in hivex.spec.

(In reply to Richard W.M. Jones from comment #21)
> But it looks as if to fix this "properly" we'll need to patch redhat-rpm-config. 
> Contrary to claims here, it's perfectly possible for us to do this, we will need to
> file a bug against redhat-rpm-config. 

That change is not without issues, since there is still ongoing discussion in the Fedora ticket:

https://src.fedoraproject.org/rpms/redhat-rpm-config/pull-request/57

> I assume (correctly?) that if we fix
> redhat-rpm-config then no change will be needed to hivex.

Right.

> ```
> export SOURCE_DATE_EPOCH=$(stat --printf='%Y' %{SOURCE0})
> ```
>
> Let me debug a little bit more. I want to see what happens when set "export SOURCE_DATE_EPOCH" in hivex.spec.

* searcher.js.gz : OK as we know.
* search_index.js.gz : OK as SOURCE_DATE_EPOCH in hivex.spec is set correctly.
* navigation.js.gz : NOT OK, the reason is ruby copies the file navigation.js in the template directory to installed directory. But in the ruby.spec, we are patching navigation.js file with system date. That is a bug.

* created.rid : NOT OK.

Because 1st line of the file is updated as a value of SOURCE_DATE_EPOCH.
But it seems that actually extconf.h is generated with system date in the make process of hivex.
That causes the difference between each files.

```
$ diff ./new_hivex_set_source_date/result.x86_64/usr/share/doc/ruby-hivex/api/created.rid ./new_hivex_set_source_date/result.i686/usr/share/doc/ruby-hivex/api/created.rid
5c5
< ./ext/hivex/extconf.h	Wed, 10 Jul 2019 15:11:30 +0200
---
> ./ext/hivex/extconf.h	Wed, 10 Jul 2019 15:18:14 +0200

$ cat ./new_hivex_set_source_date/result.x86_64/usr/share/doc/ruby-hivex/api/created.rid
Fri, 04 May 2018 21:38:37 -0000 <= This line is ok.
./README.rdoc	Mon, 03 Dec 2012 17:05:45 +0100
./lib/hivex.rb	Mon, 03 Dec 2012 17:05:45 +0100
./ext/hivex/_hivex.c	Mon, 26 Feb 2018 11:50:23 +0100
./ext/hivex/extconf.h	Wed, 10 Jul 2019 15:11:30 +0200
```

For created.rid, you can patch like this.

```
$ git diff
diff --git a/ruby/Rakefile.in b/ruby/Rakefile.in
index cd99507..83125bb 100644
--- a/ruby/Rakefile.in
+++ b/ruby/Rakefile.in
@@ -76,11 +76,14 @@ Rake::TestTask.new(:test) do |t|
 end
 task :test => :build
 
+CREATED_FILES = [
+    "@srcdir@/ext/hivex/extconf.h"
+]
 RDOC_FILES = FileList[
     "@srcdir@/README.rdoc",
     "@srcdir@/lib/**/*.rb",
     "@srcdir@/ext/**/*.[ch]"
-]
+] - CREATED_FILES
 
 Rake::RDocTask.new do |rd|
     rd.main = "@srcdir@/README.rdoc"
```

> * navigation.js.gz : NOT OK, the reason is ruby copies the file navigation.js in the template directory to installed directory. But in the ruby.spec, we are patching navigation.js file with system date. That is a bug.

I fixed navigation.js.gz in rpms/ruby.

```
$ find . -name "navigation.js.gz" | xargs md5sum
8e4ae8f305b2a1cd808057bc1db70253  ./result.x86_64/usr/share/doc/ruby-hivex/api/js/navigation.js.gz
8e4ae8f305b2a1cd808057bc1db70253  ./result.i686/usr/share/doc/ruby-hivex/api/js/navigation.js.gz
```

So, summarizing, Steps hivex can do on is below.

1. Wait rpms/ruby 2.5 new release applying patches.
2. hivex can set export SOURCE_DATE_EPOCH in hivex.spec or wait redhat-rpm-config fixing it.
3. hivex can apply above patch for created.rid to fix the issue, or waive it as false positive.

> 1. Wait rpms/ruby 2.5 new release applying patches.

It was fixed at ruby-2.5.5-105.module+el8.1.0+3656+f80bfa1d in modules/ruby 2.5

QA team for rpms/ruby, I show you how to test rpms/ruby for this ticket.

This ticket's issue is below files in ruby-hivex binary RPM are different between x86_64 and i686 RPM file.

> https://bugzilla.redhat.com/show_bug.cgi?id=1719647#c0
> https://rpmdiff.engineering.redhat.com/run/407212/18/

/usr/share/doc/ruby-hivex/api/js/searcher.js.gz
/usr/share/doc/ruby-hivex/api/js/search_index.js.gz
/usr/share/doc/ruby-hivex/api/js/navigation.js.gz
/usr/share/doc/ruby-hivex/api/created.rid

searcher.js.gz, navigation.js.gz, search_index.js.gz and created.rid are created at the build time of ruby-hivex.
The differences are 

searcher.js.gz and navigation.js.gz are created as below steps.

1. Copy ./usr/share/gems/gems/rdoc-6.0.1/lib/rdoc/generator/template/json_index/js/searcher.js
in rubygem-rdoc-*.noarch RPM to /usr/share/doc/ruby-hivex/api/js/searcher.js
2. searcher.js.gz is created from /usr/share/doc/ruby-hivex/api/js/searcher.js using the /usr/share/doc/ruby-hivex/api/js/searcher.js 's modified date as a searcher.js.gz file's metadata. So, when /usr/share/doc/ruby-hivex/api/js/searcher.js's modified date is not fixed value, the created gz file's content is different.

This ticket's patch is to make below copy preserving the file's modified date like "cp -p".
As a result, the created gz file has fixed content between x86_64 and i686 (architectures).

> 1. Copy ./usr/share/gems/gems/rdoc-6.0.1/lib/rdoc/generator/template/json_index/js/searcher.js
in rubygem-rdoc-*.noarch RPM to /usr/share/doc/ruby-hivex/api/js/searcher.js

navigation.js.gz is same as well as searcher.js.gz.
I also preserved the modified date for ./usr/share/gems/gems/rdoc-6.0.1/lib/rdoc/generator/template/json_index/js/navigation.js . Last time navigation.js was updated as system date at the build time of rpms/ruby. But this modification was not necessary. I made mistake.
Because rubygem-rdoc RPM is noarch package. The common rubygem-rdoc is used by each arch's ruby-hivex. It does not matter that a file in rubygem-rdoc RPM is updated as system date or not. Sorry for that. I tested this ticket with wrong way. Though it's not harmful.


search_index.js.gz are created as below steps.
1. ruby-hivex's search_index.js is newly created at build time of ruby-hivex.
2. The search_index.js's modified date is updated as a value of a environment variable SOURCE_DATE_EPOCH if SOURCE_DATE_EPOCH exists.
3. search_index.js.gz is created from the search_index.js using the search_index.js's modified date as the gz file's metadata.

created.rid is created at build time of ruby-hivex.
1st line (date string) in created.rid is updated as string of SOURCE_DATE_EPOCH if SOURCE_DATE_EPOCH exists. Otherwise it is updated as system date.


So, QA team. There are 2 tests to verify this ticket for rpms/ruby.

1. Check the result of build.log. If below tests are passed, it's okay.

http://download.eng.bos.redhat.com/brewroot/vol/rhel-8/packages/ruby/2.5.5/105.module+el8.1.0+3656+f80bfa1d/data/logs/x86_64/build.log
> [11694/17422] TestRDocGeneratorJsonIndex#test_generate = 0.00 s
> [11695/17422] TestRDocGeneratorJsonIndex#test_generate_search_index_with_reproducible_builds = 0.00 s


2. Check rubygem-rdoc RPM. Check below files are not updated as system date (July 12th 2019), that is okay, but preserved as the original file's modified date.

https://brewweb.engineering.redhat.com/brew/buildinfo?buildID=929456
$ curl -O http://download.eng.bos.redhat.com/brewroot/vol/rhel-8/packages/ruby/2.5.5/105.module+el8.1.0+3656+f80bfa1d/noarch/rubygem-rdoc-6.0.1-105.module+el8.1.0+3656+f80bfa1d.noarch.rpm

Extract the RPM file.

```
$ find . -name searcher.js | xargs ls -l
-rw-r--r-- 1 jaruga jaruga 6633 Sep  5  2016 ./usr/share/gems/gems/rdoc-6.0.1/lib/rdoc/generator/template/json_index/js/searcher.js
$ find . -name navigation.js | xargs ls -l
-rw-r--r-- 1 jaruga jaruga 3652 Nov 27  2012 ./usr/share/gems/gems/rdoc-6.0.1/lib/rdoc/generator/template/json_index/js/navigation.js
```

For rpms/ruby-hivex, I have already wrote how to modify.

https://bugzilla.redhat.com/show_bug.cgi?id=1719647#c26

> So, summarizing, steps hivex can do on is below.
> 
> 1. Wait rpms/ruby 2.5 new release applying patches.
> 2. hivex can set export SOURCE_DATE_EPOCH in hivex.spec or wait redhat-rpm-config fixing it.
> 3. hivex can apply above patch for created.rid to fix the issue, or waive it as false positive.

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2019:3384