Bug 2173531

Summary: ruby: FTBFS due to test suite failures
Product: Red Hat Enterprise Linux 9 Reporter: Florian Weimer <fweimer>
Component: rubyAssignee: Jun Aruga <jaruga>
Status: CLOSED NEXTRELEASE QA Contact: Lukáš Zachar <lzachar>
Severity: unspecified Docs Contact:
Priority: unspecified    
Version: 9.3CC: jaruga, jprokop
Target Milestone: rcKeywords: Patch, Triaged
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: No Doc Update
Doc Text:
If this bug requires documentation, please select an appropriate Doc Type value.
 Story Points: ---
Clone Of: Environment:
Last Closed: 2023-07-25 13:34:16 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Attachments:
Description Flags
build.log.gz
none
0001-Fix-build-failure-in-test-suite-2173531.patch none

Description Florian Weimer 2023-02-27 08:36:06 UTC 
Created attachment 1946677 [details]
build.log.gz

The test suite has the following failures:

  1) Failure:
TestGemSourceGit#test_checkout_submodules [/builddir/build/BUILD/ruby-3.0.4/test/rubygems/test_gem_source_git.rb:72]:
fatal: transport 'file' not allowed
fatal: clone of '/builddir/build/BUILD/ruby-3.0.4/tmp/test_rubygems_20230225-1166610-5ujnie/git/b' into submodule path '/builddir/build/BUILD/ruby-3.0.4/tmp/test_rubygems_20230225-1166610-5ujnie/git/a/b' failed

Probably needs:

commit 86096a91d55f72620e0f8ca8634da5fa342dc35b
Author: Hiroshi SHIBATA <hsbt>
Date:   Wed Oct 19 20:39:07 2022 +0900

    [rubygems/rubygems] Use [] instead of double-quotes

commit dae843f6b7502f921a7e66f39e3714a39d860181
Author: Hiroshi SHIBATA <hsbt>
Date:   Wed Oct 19 19:40:00 2022 +0900

    Bypass git submodule add/update with git config protocol.file.allow=always option.
    
    Co-authored-by: Nobuyoshi Nakada <nobu>

  2) Failure:
TestTimeTZ#test_asia_singapore [/builddir/build/BUILD/ruby-3.0.4/test/ruby/test_time_tz.rb:143]:
TZ=Asia/Singapore Time.local(1981, 12, 31, 23, 59, 59).
<"1981-12-31 23:59:59 +0730"> expected but was
<"1982-01-01 00:29:59 +0800">.
  3) Failure:
TestTimeTZ#test_gen_Asia_Singapore_22 [/builddir/build/BUILD/ruby-3.0.4/test/ruby/test_time_tz.rb:380]:
TZ=Asia/Singapore Time.utc(1981, 12, 31, 16, 29, 59).localtime.
<"1981-12-31 23:59:59 +0730"> expected but was
<"1982-01-01 00:29:59 +0800">.
  4) Failure:
TestTimeTZ#test_gen_Asia_Singapore_45 [/builddir/build/BUILD/ruby-3.0.4/test/ruby/test_time_tz.rb:398]:
TZ=Asia/Singapore Time.local(1981, 12, 31, 23, 59, 59).
<"1981-12-31 23:59:59 +0730"> expected but was
<"1982-01-01 00:29:59 +0800">.

Probably fixed by:

commit 58cc3c9f387dcf8f820b43e043b540fa06248da3
Author: Nobuyoshi Nakada <nobu>
Date:   Wed Dec 7 13:08:40 2022 +0900

    [Bug #19187] Fix for tzdata-2022g
Comment 1 Florian Weimer 2023-02-27 08:37:02 UTC 
Failing build: https://kojihub.stream.centos.org/koji/taskinfo?taskID=2020221
Comment 2 Florian Weimer 2023-02-27 09:55:06 UTC 
Created attachment 1946685 [details]
0001-Fix-build-failure-in-test-suite-2173531.patch

Scratch build: https://kojihub.stream.rdu2.redhat.com/koji/taskinfo?taskID=2022174
Comment 3 Vít Ondruch 2023-02-27 10:59:00 UTC 
I believe these are the relevant fixes in Fedora:

https://src.fedoraproject.org/rpms/ruby/c/8d62f79680354624c93943a7ba613707097f2c67
https://src.fedoraproject.org/rpms/ruby/c/a4040936aef930df86fdb88a85295eb8bfbd2c9e
Comment 4 Jun Aruga 2023-02-27 11:02:46 UTC 
> The test suite has the following failures:

Thanks for the report, the investigation and the patch file! As Vit reported, we can backport the fixes to RHEL.

Here are the commits on Ruby rawhide branch.

For the 1st test failure:
https://src.fedoraproject.org/rpms/ruby/c/79d75fdcddac78d2c73000a6ae7569eabb05f7bc?branch=rawhide

For the 2nd test failure:
https://src.fedoraproject.org/rpms/ruby/c/f8ef5964d0f872f70fa1f5b997de49bec02755d8?branch=rawhide
Comment 5 Florian Weimer 2023-02-27 11:52:54 UTC 
Sorry, I forgot to mention that there's another failure after fixing the first batch:

1)
Time.local timezone changes correctly adjusts the timezone change to 'CEST' on 'Europe/Amsterdam' FAILED
Expected [0, 0, 0, 16, 5, 1940, 4, 137, true, "WEST"] == [0, 40, 1, 16, 5, 1940, 4, 137, true, "CEST"]
to be truthy but was false
/builddir/build/BUILD/ruby-3.0.4/spec/ruby/core/time/shared/local.rb:13:in `block (5 levels) in <top (required)>'
/builddir/build/BUILD/ruby-3.0.4/spec/ruby/core/time/shared/local.rb:12:in `block (4 levels) in <top (required)>'
/builddir/build/BUILD/ruby-3.0.4/spec/ruby/core/time/local_spec.rb:5:in `<top (required)>'

2)
Time.mktime timezone changes correctly adjusts the timezone change to 'CEST' on 'Europe/Amsterdam' FAILED
Expected [0, 0, 0, 16, 5, 1940, 4, 137, true, "WEST"] == [0, 40, 1, 16, 5, 1940, 4, 137, true, "CEST"]
to be truthy but was false
/builddir/build/BUILD/ruby-3.0.4/spec/ruby/core/time/shared/local.rb:13:in `block (5 levels) in <top (required)>'
/builddir/build/BUILD/ruby-3.0.4/spec/ruby/core/time/shared/local.rb:12:in `block (4 levels) in <top (required)>'
/builddir/build/BUILD/ruby-3.0.4/spec/ruby/core/time/mktime_spec.rb:5:in `<top (required)>'

3)
Time.new timezone changes correctly adjusts the timezone change to 'CEST' on 'Europe/Amsterdam' FAILED
Expected [0, 0, 0, 16, 5, 1940, 4, 137, true, "WEST"] == [0, 40, 1, 16, 5, 1940, 4, 137, true, "CEST"]
to be truthy but was false
/builddir/build/BUILD/ruby-3.0.4/spec/ruby/core/time/shared/local.rb:13:in `block (5 levels) in <top (required)>'
/builddir/build/BUILD/ruby-3.0.4/spec/ruby/core/time/shared/local.rb:12:in `block (4 levels) in <top (required)>'
/builddir/build/BUILD/ruby-3.0.4/spec/ruby/core/time/new_spec.rb:11:in `<top (required)>'

It needs:

commit b7577b4d9e0fd92522fc30e10fe712e245adee8c
Author: Nobuyoshi Nakada <nobu>
Date:   Wed Aug 17 00:45:27 2022 +0900

    The tzdata 2022c removed Amsterdam Mean Time

This one has seen quite a bit follow-up activity upstream, I haven't tried to backport the changes that the lead to this testing being renabled upstream.
Comment 6 Jarek Prokop 2023-02-27 12:14:08 UTC 
Multiple updates of tzdata broke Ruby builds in Fedora, this is not the first (maybe not even last :/) time this happened.

We can assume more shipped Ruby versions across RHEL versions are unbuildable the same way if they include those test cases regarding tzdata.
Comment 7 Jun Aruga 2023-02-27 13:13:13 UTC 
> Sorry, I forgot to mention that there's another failure after fixing the first batch:

On the test failures, I think the patch on the Fedora rawhide is below.
https://src.fedoraproject.org/rpms/ruby/c/ad7f78b5c865f3cbf8721d7eecf950a508a6bc5d?branch=rawhide
Comment 8 Florian Weimer 2023-06-14 08:11:29 UTC 
MR: https://gitlab.com/redhat/centos-stream/rpms/ruby/-/merge_requests/17

The pipeline fails because there is no rhel-9.2.0 branch. Should we just create the rhel-9.2.0 branch and merge this, to unlock the tree for further updates?

I saw an unrelated sporadic build failure on s390-074.build.eng.bos.redhat.com:

  1) Error:
TestBundledCA#test_accessing_new_index:
Errno::ECONNREFUSED: Failed to open TCP connection to index.rubygems.org:443 (Connection refused - connect(2) for "index.rubygems.org" port 443)
    /builddir/build/BUILD/ruby-3.0.4/lib/net/http.rb:987:in `initialize'
    /builddir/build/BUILD/ruby-3.0.4/lib/net/http.rb:987:in `open'
    /builddir/build/BUILD/ruby-3.0.4/lib/net/http.rb:987:in `block in connect'
    /builddir/build/BUILD/ruby-3.0.4/lib/timeout.rb:97:in `block in timeout'
    /builddir/build/BUILD/ruby-3.0.4/lib/timeout.rb:107:in `timeout'
    /builddir/build/BUILD/ruby-3.0.4/lib/net/http.rb:985:in `connect'
    /builddir/build/BUILD/ruby-3.0.4/lib/net/http.rb:970:in `do_start'
    /builddir/build/BUILD/ruby-3.0.4/lib/net/http.rb:959:in `start'
    /builddir/build/BUILD/ruby-3.0.4/lib/net/http.rb:1512:in `request'
    /builddir/build/BUILD/ruby-3.0.4/lib/net/http.rb:1270:in `get'
    /builddir/build/BUILD/ruby-3.0.4/test/rubygems/test_bundled_ca.rb:34:in `assert_https'
    /builddir/build/BUILD/ruby-3.0.4/test/rubygems/test_bundled_ca.rb:58:in `test_accessing_new_index'
  2) Error:
TestBundledCA#test_accessing_rubygems:
Errno::ECONNREFUSED: Failed to open TCP connection to rubygems.org:443 (Connection refused - connect(2) for "rubygems.org" port 443)
    /builddir/build/BUILD/ruby-3.0.4/lib/net/http.rb:987:in `initialize'
    /builddir/build/BUILD/ruby-3.0.4/lib/net/http.rb:987:in `open'
    /builddir/build/BUILD/ruby-3.0.4/lib/net/http.rb:987:in `block in connect'
    /builddir/build/BUILD/ruby-3.0.4/lib/timeout.rb:97:in `block in timeout'
    /builddir/build/BUILD/ruby-3.0.4/lib/timeout.rb:107:in `timeout'
    /builddir/build/BUILD/ruby-3.0.4/lib/net/http.rb:985:in `connect'
    /builddir/build/BUILD/ruby-3.0.4/lib/net/http.rb:970:in `do_start'
    /builddir/build/BUILD/ruby-3.0.4/lib/net/http.rb:959:in `start'
    /builddir/build/BUILD/ruby-3.0.4/lib/net/http.rb:1512:in `request'
    /builddir/build/BUILD/ruby-3.0.4/lib/net/http.rb:1270:in `get'
    /builddir/build/BUILD/ruby-3.0.4/test/rubygems/test_bundled_ca.rb:34:in `assert_https'
    /builddir/build/BUILD/ruby-3.0.4/test/rubygems/test_bundled_ca.rb:46:in `test_accessing_rubygems'
  3) Error:
TestBundledCA#test_accessing_staging:
Errno::ECONNREFUSED: Failed to open TCP connection to staging.rubygems.org:443 (Connection refused - connect(2) for "staging.rubygems.org" port 443)
    /builddir/build/BUILD/ruby-3.0.4/lib/net/http.rb:987:in `initialize'
    /builddir/build/BUILD/ruby-3.0.4/lib/net/http.rb:987:in `open'
    /builddir/build/BUILD/ruby-3.0.4/lib/net/http.rb:987:in `block in connect'
    /builddir/build/BUILD/ruby-3.0.4/lib/timeout.rb:97:in `block in timeout'
    /builddir/build/BUILD/ruby-3.0.4/lib/timeout.rb:107:in `timeout'
    /builddir/build/BUILD/ruby-3.0.4/lib/net/http.rb:985:in `connect'
    /builddir/build/BUILD/ruby-3.0.4/lib/net/http.rb:970:in `do_start'
    /builddir/build/BUILD/ruby-3.0.4/lib/net/http.rb:959:in `start'
    /builddir/build/BUILD/ruby-3.0.4/lib/net/http.rb:1512:in `request'
    /builddir/build/BUILD/ruby-3.0.4/lib/net/http.rb:1270:in `get'
    /builddir/build/BUILD/ruby-3.0.4/test/rubygems/test_bundled_ca.rb:34:in `assert_https'
    /builddir/build/BUILD/ruby-3.0.4/test/rubygems/test_bundled_ca.rb:54:in `test_accessing_staging'
  4) Error:
TestBundledCA#test_accessing_www_rubygems:
Errno::ECONNREFUSED: Failed to open TCP connection to www.rubygems.org:443 (Connection refused - connect(2) for "www.rubygems.org" port 443)
    /builddir/build/BUILD/ruby-3.0.4/lib/net/http.rb:987:in `initialize'
    /builddir/build/BUILD/ruby-3.0.4/lib/net/http.rb:987:in `open'
    /builddir/build/BUILD/ruby-3.0.4/lib/net/http.rb:987:in `block in connect'
    /builddir/build/BUILD/ruby-3.0.4/lib/timeout.rb:97:in `block in timeout'
    /builddir/build/BUILD/ruby-3.0.4/lib/timeout.rb:107:in `timeout'
    /builddir/build/BUILD/ruby-3.0.4/lib/net/http.rb:985:in `connect'
    /builddir/build/BUILD/ruby-3.0.4/lib/net/http.rb:970:in `do_start'
    /builddir/build/BUILD/ruby-3.0.4/lib/net/http.rb:959:in `start'
    /builddir/build/BUILD/ruby-3.0.4/lib/net/http.rb:1512:in `request'
    /builddir/build/BUILD/ruby-3.0.4/lib/net/http.rb:1270:in `get'
    /builddir/build/BUILD/ruby-3.0.4/test/rubygems/test_bundled_ca.rb:34:in `assert_https'
    /builddir/build/BUILD/ruby-3.0.4/test/rubygems/test_bundled_ca.rb:50:in `test_accessing_www_rubygems'

Looking at the test, I suspect this happens because the builder has accidentally working DNS, but no connection to the Internet. But I haven't been able to hit the builder again to confirm this, using additional instrumentation.
Comment 10 Vít Ondruch 2023-06-14 09:24:26 UTC 
@Florian we will likely address this sooner or later by rebase of Ruby. Therefore I am reluctant to accept the MR and branch and what not.

However, it seems it might block you for some reason. Do you mind to elaborate? And also should you convince us, I'd really prefer if the MR was composed from Fedora cherry-picks referenced above then from custom patches (I am mainly concerned about the ruby-rh2173531-4.patch, which just disables the test case instead of applying the fix. But I have not take a closer look).
Comment 11 Florian Weimer 2023-06-14 09:40:05 UTC 
(In reply to Vít Ondruch from comment #10)
> @Florian we will likely address this sooner or later by rebase of Ruby.
> Therefore I am reluctant to accept the MR and branch and what not.
> 
> However, it seems it might block you for some reason. Do you mind to
> elaborate? And also should you convince us, I'd really prefer if the MR was
> composed from Fedora cherry-picks referenced above then from custom patches
> (I am mainly concerned about the ruby-rh2173531-4.patch, which just disables
> the test case instead of applying the fix. But I have not take a closer
> look).

It does not block me anymore, we have forked ruby for the ISA SIG rebuilds, so the damage (if you want to call it that) is already done.

Note that the CentOS Zuul tester has a different file system and hits a time_t range issue. I have not fixed that in the backports.
Comment 12 Vít Ondruch 2023-06-14 10:22:01 UTC 
(In reply to Florian Weimer from comment #11)
> It does not block me anymore

Perfect

> we have forked ruby for the ISA SIG rebuilds,
> so the damage (if you want to call it that) is already done.

Sorry for that.

> Note that the CentOS Zuul tester has a different file system and hits a
> time_t range issue. I have not fixed that in the backports.

Interesting. I assume you are referring to this issue:

~~~
DEBUG: 1)
DEBUG: File.utime allows Time instances in the far future to set mtime and atime (but some filesystems limit it up to 2446-05-10 or 2038-01-19) FAILED
DEBUG: Expected [559444, 2446, 2038].include? 2486
DEBUG: to be truthy but was false
DEBUG: /builddir/build/BUILD/ruby-3.0.4/spec/ruby/core/file/utime_spec.rb:84:in `block (4 levels) in <top (required)>'
DEBUG: /builddir/build/BUILD/ruby-3.0.4/spec/ruby/core/file/utime_spec.rb:3:in `<top (required)>'
DEBUG: Finished in 79.017842 seconds
DEBUG: 3769 files, 30807 examples, 179300 expectations, 1 failure, 0 errors, 0 tagged
~~~

Not sure I have seen this previously. We will need to investigate. Thank you for pointing this out.
Comment 13 Florian Weimer 2023-06-14 10:27:13 UTC 
(In reply to Vít Ondruch from comment #12)
> Interesting. I assume you are referring to this issue:
> 
> ~~~
> DEBUG: 1)
> DEBUG: File.utime allows Time instances in the far future to set mtime and
> atime (but some filesystems limit it up to 2446-05-10 or 2038-01-19) FAILED
> DEBUG: Expected [559444, 2446, 2038].include? 2486
> DEBUG: to be truthy but was false
> DEBUG:
> /builddir/build/BUILD/ruby-3.0.4/spec/ruby/core/file/utime_spec.rb:84:in
> `block (4 levels) in <top (required)>'
> DEBUG:
> /builddir/build/BUILD/ruby-3.0.4/spec/ruby/core/file/utime_spec.rb:3:in
> `<top (required)>'
> DEBUG: Finished in 79.017842 seconds
> DEBUG: 3769 files, 30807 examples, 179300 expectations, 1 failure, 0 errors,
> 0 tagged
> ~~~

Yes, this is the issue I meant.  I can reproduce it with kernel-5.14.0-284.11.1.el9_2.x86_64 on an XFS file system (created fresh during the installation of RHEL 9 pretty much using defaults, I believe), in case that helps.
Comment 14 Jun Aruga 2023-06-22 14:18:19 UTC 
I take this task to fix the FTBFS on c9s and RHEL 9 based on the PR, <https://gitlab.com/redhat/centos-stream/rpms/ruby/-/merge_requests/17>.
Comment 15 Jun Aruga 2023-06-27 18:46:24 UTC 
I am working in progress in the following pull-request.

https://gitlab.com/redhat/centos-stream/rpms/ruby/-/merge_requests/18
Comment 18 Jun Aruga 2023-06-28 14:15:29 UTC 
(In reply to Vít Ondruch from comment #12)
> (In reply to Florian Weimer from comment #11)
> > It does not block me anymore
> 
> Perfect
> 
> > we have forked ruby for the ISA SIG rebuilds,
> > so the damage (if you want to call it that) is already done.
> 
> Sorry for that.
> 
> > Note that the CentOS Zuul tester has a different file system and hits a
> > time_t range issue. I have not fixed that in the backports.
> 
> Interesting. I assume you are referring to this issue:
> 
> ~~~
> DEBUG: 1)
> DEBUG: File.utime allows Time instances in the far future to set mtime and
> atime (but some filesystems limit it up to 2446-05-10 or 2038-01-19) FAILED
> DEBUG: Expected [559444, 2446, 2038].include? 2486
> DEBUG: to be truthy but was false
> DEBUG:
> /builddir/build/BUILD/ruby-3.0.4/spec/ruby/core/file/utime_spec.rb:84:in
> `block (4 levels) in <top (required)>'
> DEBUG:
> /builddir/build/BUILD/ruby-3.0.4/spec/ruby/core/file/utime_spec.rb:3:in
> `<top (required)>'
> DEBUG: Finished in 79.017842 seconds
> DEBUG: 3769 files, 30807 examples, 179300 expectations, 1 failure, 0 errors,
> 0 tagged
> ~~~
> 
> Not sure I have seen this previously. We will need to investigate. Thank you
> for pointing this out.


Maybe I found the upstream patch to pass the test.
https://github.com/ruby/ruby/commit/8d1109c03bacc952b6218af2e4ae9b74c9855273
Comment 19 Jun Aruga 2023-06-29 16:46:19 UTC 
> I am working in progress in the following pull-request.
>
> https://gitlab.com/redhat/centos-stream/rpms/ruby/-/merge_requests/18

The pull-request was merged to the c9s branch.

> https://bugzilla.redhat.com/show_bug.cgi?id=2173531#c13

I opened the ticket below to ask to use the same environment between CentOS Koji and CentOS Zuul CI to avoid the Zuul CI specific test failures.

Aligning CentOS Zuul CI server environments with CentOS Koji servers.
https://issues.redhat.com/browse/CS-1652
Comment 22 Jun Aruga 2023-07-25 13:34:16 UTC 
I would close this ticket as status CLOSED, CURRENTRELEASE.

For the record the reason is because I heard from Lukas Zachar (@lzacar) that Python maintainers are doing this way for this kind of issue to fix FTBFS issues.
Comment 23 Jun Aruga 2023-08-02 10:39:25 UTC 
> I would close this ticket as status CLOSED, CURRENTRELEASE.

I closed as status: CLOSED NEXTRELEASE, as the updated RHEL 9.3 is the next release of the RHEL.

In the context of the CentOS 9 Stream (c9s), it is CURRENTRELEASE.
In the context of the RHEL 9.3, it is  NEXTRELEASE.