Bug 2008773

This is OpenSSL 3.0.0 breakage.  See upstream patches

https://git.postgresql.org/gitweb/?p=postgresql.git;a=commitdiff;h=8e7199453bf9fe142f3f4a5e17010320c24867e7

and its two immediate predecessors

The only thing that will not be handled by upstream patches is the use of the legacy provider, that I expect we cannot enable in Fedora koji.

Since the legacy provider is problematic I proposed a patch that disables testing of obsolete functions. 
https://gitlab.com/fjanus/postgresql/-/blob/c9s/postgresql-pgcrypto-openssl3-tests.patch

During my investigation, I found an issue with rijndael test suite.
AFAIK aes should be supported by openssl3 but it behaves weirdly.
I am not sure where the problem is, but maybe I've already seen a similar issue in the case of mistaken initialization of ciphers.
@Tom do you have any idea what could it cause?

-- decrypt
 select decrypt(encrypt('foo', '0123456', 'aes'), '0123456', 'aes');
- decrypt 
----------
- foo
+                       decrypt                        
+------------------------------------------------------
+ P\273\005\004\353 \330\253<\024\211\007*\272\320\255
 (1 row)

 -- iv
@@ -106,9 +106,9 @@

 select decrypt_iv(decode('2c24cb7da91d6d5699801268b0f5adad', 'hex'),
                '0123456', 'abcd', 'aes');
- decrypt_iv 
-------------
- foo
+                   decrypt_iv                   
+------------------------------------------------
+ ,$\313}\251\035mV\231\200\022h\260\365\255\255
 (1 row)

 -- long message
@@ -119,8 +119,8 @@
 (1 row)

 select decrypt(encrypt('Lets try a longer message.', '0123456789', 'aes'), '0123456789', 'aes');
-          decrypt           
-----------------------------
- Lets try a longer message.
+                      decrypt                       
+----------------------------------------------------
+ Lets try a longe>\331<\245I0\025?,;\236\307xPV\255
 (1 row)

(In reply to Filip Januš from comment #3)
> @Tom do you have any idea what could it cause?

Sorry, no idea --- I know little about this code.

I will note that in the upstream buildfarm, this code is passing its tests fine
on a Fedora rawhide machine as of v13 branch tip.
Are you sure you copied all three of the relevant commits?

Thanks,
that's weird, my builds[1] are failing on Fedora 36 and centos stream due to rijndael test suite.

[1] https://kojipkgs.fedoraproject.org//work/tasks/9951/76569951/build.log

Could something about the legacy-provider situation have changed very recently?
Our buildfarm member:

https://buildfarm.postgresql.org/cgi-bin/show_log.pl?nm=caiman&dt=2021-09-30%2021%3A10%3A19

currently reports it's running

uname -r = 5.14.0-61.fc36.x86_64
uname -v = #1 SMP Mon Aug 30 22:37:03 UTC 2021

so that does look to be rawhide, but it looks like it might not've been updated
for a couple of weeks.

Is it possible the buildfarm has still openssl 1.x? the openssl 3.0 got to rawhide only in September

(In reply to Honza Horak from comment #7)
> Is it possible the buildfarm has still openssl 1.x? the openssl 3.0 got to
> rawhide only in September

No, that animal is definitely using openssl 3.0.0 since mid-September,
because it broke in the back branches until we back-patched the
compatibility patches I referred to.  Compare

https://buildfarm.postgresql.org/cgi-bin/show_log.pl?nm=caiman&dt=2021-09-23%2018%3A28%3A14

which I believe is showing the same symptoms you mention, and the next build

https://buildfarm.postgresql.org/cgi-bin/show_log.pl?nm=caiman&dt=2021-09-27%2006%3A15%3A48

Thanks,
this patch fixed my issue

https://git.postgresql.org/gitweb/?p=postgresql.git;a=commitdiff;h=135d8687ad

Fixed in postgresql-13.4-3.fc36