Bug 442047
Summary: | no good way to pull in multilib runtime that isn't explicit dependencies | ||
---|---|---|---|
Product: | [Fedora] Fedora | Reporter: | Bill Nottingham <notting> |
Component: | distribution | Assignee: | Václav Pavlín <vpavlin> |
Status: | NEW --- | QA Contact: | |
Severity: | low | Docs Contact: | |
Priority: | low | ||
Version: | rawhide | CC: | adam, asliotentik, belegdol, ceski, jacquesdong, jzeleny, redhat, rransom.8774, r_stevenz, scottt.tw, sgallagh, tlestach, yaneti |
Target Milestone: | --- | Keywords: | FutureFeature |
Target Release: | --- | ||
Hardware: | All | ||
OS: | Linux | ||
Whiteboard: | |||
Fixed In Version: | Doc Type: | Enhancement | |
Doc Text: | Story Points: | --- | |
Clone Of: | Environment: | ||
Last Closed: | Type: | --- | |
Regression: | --- | Mount Type: | --- |
Documentation: | --- | CRM: | |
Verified Versions: | Category: | --- | |
oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
Cloudforms Team: | --- | Target Upstream Version: | |
Embargoed: | |||
Bug Depends On: | |||
Bug Blocks: | 663269 |
Description
Bill Nottingham
2008-04-11 13:47:44 UTC
Adding Panu, I think we need to make it important for us to add arch-specific deps. We can do it now but we have to do it manually. I'm not sure how arch-specific deps help you here. It's the sort of thing where if you have pam_krb5 installed, you need all arches, but no app is going to require pam_krb5. Then you'll need someway of tagging these in the rpm. Either way - adding panu is a good idea. :) Changing version to '9' as part of upcoming Fedora 9 GA. More information and reason for this action is here: http://fedoraproject.org/wiki/BugZappers/HouseKeeping This message is a reminder that Fedora 9 is nearing its end of life. Approximately 30 (thirty) days from now Fedora will stop maintaining and issuing updates for Fedora 9. It is Fedora's policy to close all bug reports from releases that are no longer maintained. At that time this bug will be closed as WONTFIX if it remains open with a Fedora 'version' of '9'. Package Maintainer: If you wish for this bug to remain open because you plan to fix it in a currently maintained version, simply change the 'version' to a later Fedora version prior to Fedora 9's end of life. Bug Reporter: Thank you for reporting this issue and we are sorry that we may not be able to fix it before Fedora 9 is end of life. If you would still like to see this bug fixed and are able to reproduce it against a later version of Fedora please change the 'version' of this bug to the applicable version. If you are unable to change the version, please add a comment here and someone will do it for you. Although we aim to fix as many bugs as possible during every release's lifetime, sometimes those efforts are overtaken by events. Often a more recent Fedora release includes newer upstream software that fixes bugs or makes them obsolete. The process we are following is described here: http://fedoraproject.org/wiki/BugZappers/HouseKeeping *** Bug 460188 has been marked as a duplicate of this bug. *** I just got bitten by this issue: wine which is currently in updates-testing pulled nss-mdns.i386. This caused yum to fail (I'm on x86_64 system). Can get preetty serious, it seems. Yes, wine-core-1.1.24-2.fc11.1.i586 pulls in nss-mdns-0.10-7.fc11.i586 as a dependency. On a x86_64 (mulitlib) install, if the x86_64 nss-mdns is not installed, name resolution fails because of the change to nsswitch.conf. "Install the nss-mdns x86_64 package" is going to become a faq. Hm, that almost sounds like a bug in the wine package. Sorry, Bill - what's the bug? The fact that wine-core has a dependency on nss-mdns, or the fact that not installing the x86_64 nss-mdns package will result in broken resolving behaviour after the script from the nss-mdns.i586 changes the nsswitch.conf to, "hosts: files mdns4_minimal [NOTFOUND=return] dns" without the 64 bit versions of libnss_mdns*.so.* having been installed. Sorry, for the noise - nss-mdns(x86-32) has been added as a 'hard' dependency to wine-core. See #492700. (And someone has raised the fact under that bug that installing nss-mdns.i586 without the x86_64 package will break name resolution on x86_64). Yeah, it's by no means bug in wine. Wine needs nss-mdns to work correctly, so it requires it. It's a 'bug' in that introducing the update shouldn't break the system. However, I'm not sure how to solve it. One way to solve it would be to add support to glibc to do something like /etc/nsswitch.d/... instead of packages doing sed on /etc/nsswitch.conf. Then of course it would have to be per-arch. Or, add the RPM dependency handling that allows this: (in package nss-mdns.x86_64) Requires: nss-mdns.i586 if installed glibc.i586 (in package nss-mdns.i586) Requires: nss-mdns.x86_64 if installed glibc.x86_64 Both of these solutions are potentially messy and complicated. Maybe the way forward is to eliminate nss modules in favor of a daemon process running that provides lookups, like what SSSD is starting to do? http://fedoraproject.org/wiki/Features/SSSD (The daemon has the plugins, not the C library.) *** Bug 508481 has been marked as a duplicate of this bug. *** glibc not finding /lib64/libnss_mdns4_minimal.so.2 and treating the result as NOTFOUND instead of UNAVAIL sounds like a bug to me *** Bug 512351 has been marked as a duplicate of this bug. *** *** Bug 509357 has been marked as a duplicate of this bug. *** I think I agree with comment 16 - glibc needs to handle the failure better. (In reply to comment #13) > It's a 'bug' in that introducing the update shouldn't break the system. > However, I'm not sure how to solve it. Actually after the system breaks in this way, yum stops working. This is very serious for a lot of users as they can't "unbreak" it. I have seen this break on two systems now. I just installed a brand new Fedora 11 x86_64 system with default "software development" package set, got all updates, the installed Chromium via the spot yum repository: http://www.fergytech.com/2009/07/a-chromium-rpm-on-fedora-11/ After these steps (and nothing else), numerous programs in the system (yum, wget, ...) couldn't resolve hostnames. I don't know if it was updates or installing Chromium that did it, but the point is that numerous unrelated core parts of the system broke. On one of the systems Firefox also broke, on both Chromium also broke -- this makes it hard to find a solution to the problem too. https://admin.fedoraproject.org/updates/glibc-2.10.1-3 may be of interest here. Question: Is this bug solved yet? I experience dns timeouts and unresolvable host on a clean Fedora 13 also. I wonder if it's related? Eddie: Is this on wifi? Check dmesg, there's a kernel bug in .35/.36 that causes those sorts of errors on wifi with a certain combination of wireless chipset and AP. "service NetworkManager restart" as root may cure it for 30 seconds or so until it buckles under load again. There's a patch going upstream for this. I don't have either the error message or bug report reference handy unfortunately. Hello Luke, It's not on wireless. I installed a fresh Fedora 13 here on a pc with a wired lan connection to serve my lan dns, dhcp and other intranet services. It is a replacement for an older Fedora Core 4 server. That older server never caused me any problems with dns resolving hosts on the internet but it seems that now the newer Bind version has come in place of the old one I have problems all the time. It could be my Zyxel P-2602R-D1A modem/router's firewall that is the culprit but that is what I am trying to find out. Hence my question. I hope this is explanation makes it a bit more clear? Regards, Eddie. Oh, and BTW, I don't use the NetworkManager on that new server. I disabled NetworkManager and configured eth0 with static addresses and use service "network" to enable networking at boot time. I guess the first thing to do is try plugging directly into the Internet, bypassing the router. This bug is on a completely different topic though, you should file a separate bug report if you are still having problems. I'd like to chime in here and mention that this bug is becoming much more highly-visible as people begin to use SSSD instead of nscd for caching identity lookups. A classic example is bug 663269, where acroread fails on x86_64 systems with SSSD unless sssd-client.i686 is installed. nsswitch modules (and PAM modules) really need to be able to specify "SSSD requires sssd-client.x86_64. It also requires sssd-client.i686 if glibc.i686 is installed" I recently upgraded some (not all) desktops from F14 to F15 and I did several updates of F15. On the F14 desktops, I cannot use wget without getting the error message "failed: Name or service not known." (but yum is working fine), while on the F15 desktops, that problem seems to have been solved. on F14: $ wget http://download.fedoraproject.org/pub/fedora/linux/releases/15/Live/i686/Fedora-15-i686-Live-Desktop.iso --2011-07-13 09:27:07-- http://download.fedoraproject.org/pub/fedora/linux/releases/15/Live/i686/Fedora-15-i686-Live-Desktop.iso Resolving download.fedoraproject.org... failed: Name or service not known. wget: unable to resolve host address “download.fedoraproject.org” $ cvlc http://vipicecast.yacast.net/europe1 VLC media player 1.1.10 The Luggage (revision exported) Blocked: call to unsetenv("DBUS_ACTIVATION_ADDRESS") Blocked: call to unsetenv("DBUS_ACTIVATION_BUS_TYPE") [0xca2c50] dummy interface: using the dummy interface module... [0x7f0ad80014c0] main access error: cannot resolve vipicecast.yacast.net port 80 : Name or service not known [0x7f0ad80014c0] access_http access error: cannot connect to vipicecast.yacast.net:80 [0x7f0ad80014c0] main access error: cannot resolve vipicecast.yacast.net port 80 : Name or service not known [0x7f0ad80014c0] access_mms access error: cannot connect to vipicecast.yacast.net:80 [0x7f0ae0000c90] main input error: open of `http://vipicecast.yacast.net/europe1' failed: (null) [0x7f0ae0000c90] main input error: Your input can't be opened [0x7f0ae0000c90] main input error: VLC is unable to open the MRL 'http://vipicecast.yacast.net/europe1'. Check the log for details. ^C[0xc76d60] signals interface error: Caught Interrupt signal, exiting... on F15: [09:28][beauduin@rigoletto ~]$ wget http://download.fedoraproject.org/pub/fedora/linux/releases/15/Live/i686/Fedora-15-i686-Live-Desktop.iso --2011-07-13 09:28:52-- http://download.fedoraproject.org/pub/fedora/linux/releases/15/Live/i686/Fedora-15-i686-Live-Desktop.iso Resolving download.fedoraproject.org... 80.239.156.215, 85.236.55.6, 209.132.181.16, ... Connecting to download.fedoraproject.org|80.239.156.215|:80... connected. HTTP request sent, awaiting response... 302 FOUND Location: http://mirrors.ircam.fr/pub/fedora/linux/releases/15/Live/i686/Fedora-15-i686-Live-Desktop.iso [following] --2011-07-13 09:28:57-- http://mirrors.ircam.fr/pub/fedora/linux/releases/15/Live/i686/Fedora-15-i686-Live-Desktop.iso Resolving mirrors.ircam.fr... 129.102.1.37, 2001:660:3004:4003::37:80 Connecting to mirrors.ircam.fr|129.102.1.37|:80... connected. HTTP request sent, awaiting response... 200 OK Length: 592445440 (565M) [application/octet-stream] Saving to: “Fedora-15-i686-Live-Desktop.iso” 0% [ ] 4,404,523 1.70M/s ^C $ cvlc http://vipicecast.yacast.net/europe1 VLC media player 1.1.10 The Luggage (revision exported) Blocked: call to unsetenv("DBUS_ACTIVATION_ADDRESS") Blocked: call to unsetenv("DBUS_ACTIVATION_BUS_TYPE") [0x8fd1e0] dummy interface: using the dummy interface module... [0x7f56300014c0] access_http access: Raw-audio server found, mp3 demuxer selected [0x7f562c000ac0] pulse audio output error: cannot connect to server: Connection refused ALSA lib pulse.c:229:(pulse_connect) PulseAudio: Unable to connect: Connection refused [0x7f562c000ac0] oss audio output error: cannot open audio device (/dev/dsp) ALSA lib pulse.c:229:(pulse_connect) PulseAudio: Unable to connect: Connection refused [0x8d8550] signals interface error: Caught Interrupt signal, exiting... Is this now solved with F15? Not sure comment #28 is really related. The original issue hasn't undergone any change. (In reply to comment #29) > Is this now solved with F15? No. A friend of mine had this problem with fedora 15, one day ago. He installed Ubuntu now and the problem is gone. This bug has been discussed on packaging tools team IRC meeting today. The outcome of the discussion: Since NSS/sssd is not a dependency of the glibc(32) library, it should never rely on it existing on the system, configuration options created by any other package (sssd(64) in this case) notwithstanding. It was suggested glibc checks for the presence of NSS and in case it is missing it fallbacks on whatever it would do without NSS. It was also mentioned this is the case in more recent glibc and suggested this bug is closed in that case. I'm having difficulty parsing this response. First of all, NSS (name-service switch) is *part* of glibc. It is included in the 'glibc' package and is a necessary component of any Linux system (it's how the system looks up usernames, DNS entries and a host of other things). The core of the problem is that NSS runs in-process by necessity. All C applications are linked against glibc directly and therefore run NSS modules (automatically loaded when needed) from the user process. It's not a system daemon of any kind. This means that in order for a user to look up an entry while running a 64-bit application, there must be a 64-bit plugin available for NSS. On multilib platforms (e,g. i686 runtime on x86_64 kernel), NSS also must have a 32-bit plugin available for glibc so that 32-bit applications can access name-service data. This is a problem for ALL name-service providers, including SSSD, winbind and nss-pam-ldapd. If the corresponding 32-bit version of a plugin is not installed, 32-bit applications (such as any third-party apps with no 64-bit available) will not be able to access name-service data and will generally either fail or behave erratically. This problem is NOT limited to NSS either. The same issue occurs with PAM, if an i686 application's PAM stack relies on e.g. pam_winbind.so but only the samba-winbind-clients.x86_64 package is installed, authentication will fail. We do not want to force SSSD and winbind to have an explicit requires on both the 32-bit and 64-bit client libraries, because that would force the presence of the 32-bit runtime on any system wishing to perform centralized user management. This is far too heavy-handed and is why we would prefer an RPM/yum solution to ensure that the appropriate NSS and PAM plugins are pulled in for systems with both runtimes available. First of all sorry the issue was presented in a simplified version on the meeting. I see now it's more complex than blaming misconfiguration. (In reply to comment #33) > This problem is NOT limited to NSS either. The same issue occurs with PAM, > if an i686 application's PAM stack relies on e.g. pam_winbind.so but only > the samba-winbind-clients.x86_64 package is installed, authentication will > fail. > I agree this is a valid issue, e.g. admin knows what package needs to be installed on the particular machine, installs them and then finds some 32 bit apps can not be authenticated. Before we discuss this on a future meeting again, a quick idea: wouldn't it be enough if every 64-bit client library depended on the 32-bit counterpart? Maybe we could introduce some per-package 'multilib-policy: all'. (In reply to comment #34) > First of all sorry the issue was presented in a simplified version on the > meeting. I see now it's more complex than blaming misconfiguration. > > (In reply to comment #33) > > This problem is NOT limited to NSS either. The same issue occurs with PAM, > > if an i686 application's PAM stack relies on e.g. pam_winbind.so but only > > the samba-winbind-clients.x86_64 package is installed, authentication will > > fail. > > > > I agree this is a valid issue, e.g. admin knows what package needs to be > installed on the particular machine, installs them and then finds some 32 > bit apps can not be authenticated. > > Before we discuss this on a future meeting again, a quick idea: wouldn't it > be enough if every 64-bit client library depended on the 32-bit counterpart? > Maybe we could introduce some per-package 'multilib-policy: all'. "We do not want to force SSSD and winbind to have an explicit requires on both the 32-bit and 64-bit client libraries, because that would force the presence of the 32-bit runtime on any system wishing to perform centralized user management." This solution isn't acceptable because it forces a minimum of 14MB of additional data (the installed size of the glibc.i686 package) onto the x86_64 minimal install (including LiveCDs). https://www.genjosholiday.com/3-cara-mudah-merawat-ban-bus-pariwisata/ https://www.genjosholiday.com/4-alasan-mengapa-membutuhkan-jasa-inspeksi-independen-sebelum-beli-mobil-bekas/ https://www.genjosholiday.com/4-keuntungan-gaul-sama-komunitas-mobil/ https://www.genjosholiday.com/4-tips-memilih-oli-yang-tepat-untuk-mobil-anda/ https://www.genjosholiday.com/5-faktor-penyebab-mesin-mobil-panas/ https://www.genjosholiday.com/5-keuntungan-beli-mobil-bekas-tangan-pertama/ https://www.genjosholiday.com/5-langkah-sederhana-membuat-mesin-mobil-awet-sepanjang-masa/ https://www.genjosholiday.com/5-langkah-yang-harus-dilalui-jika-anda-memilih-mobil-bekas-perorangan/ https://www.genjosholiday.com/5-perawatan-mobil-penting-yang-sering-diabaikan/ https://www.genjosholiday.com/6-indikasi-problem-pada-rem-mobil-bekas/ https://www.genjosholiday.com/6-keuntungan-membeli-mobil-bekas-yang-perlu-anda-ketahui/ https://www.genjosholiday.com/6-tips-nyaman-naik-bus-pariwisata-saat-traveling/ https://www.genjosholiday.com/6-tips-untuk-memastikan-mobil-anda-sudah-siap-untuk-mudik/ https://www.genjosholiday.com/7-lampu-indikator-pada-mobil-yang-penting-anda-ketahui/ https://www.genjosholiday.com/pantai-nampu/ https://www.genjosholiday.com/berniat-membuka-po-bus-pahami-dulu-persyaratannya/ https://www.genjosholiday.com/big-bus/ https://www.genjosholiday.com/bikin-rugi-cermati-ciri-ciri-aki-mobil-rekondisi-berikut-ini/ https://www.genjosholiday.com/blog/ https://www.genjosholiday.com/bus-medium/ https://www.genjosholiday.com/candi-barong-yogyakarta/ https://www.genjosholiday.com/candi-ijo/ https://www.genjosholiday.com/candi-mendut/ https://www.genjosholiday.com/candi-ratu-boko/ https://www.genjosholiday.com/canting-mas-puncak-dipowono/ https://www.genjosholiday.com/cara-membedakan-jetbus-hdd-mhd-dan-shd/ https://www.genjosholiday.com/cara-modifikasi-mobil-bekas-jadi-tampak-baru/ https://www.genjosholiday.com/harga-tiket-masuk-tempat-wisata-jogja/ https://www.genjosholiday.com/dapat-berakibat-fatal-ini-dia-kesalahan-berkendara-yang-masih-banyak-dilakukan-orang/ https://www.genjosholiday.com/dapat-berakibat-fatal-ini-dia-kesalahan-mengemudi-bus-pariwisata-yang-masih-banyak-dilakukan/ https://www.genjosholiday.com/disclaimer/ https://www.genjosholiday.com/fakta-fakta-mobil-bekas-modifikasi/ https://www.genjosholiday.com/fakta-fakta-tentang-busi-yang-perlu-anda-ketahui/ https://www.genjosholiday.com/faktor-harga-sewa-bus-pariwisata-terbaru/ https://www.genjosholiday.com/fitur-toyota-hiace-luxury-terbaru/ https://www.genjosholiday.com/air-terjun-sri-gethuk/ https://www.genjosholiday.com/ https://www.genjosholiday.com/harga-toyota-hiace-luxury/ https://www.genjosholiday.com/hutan-pinus-mangunan/ https://www.genjosholiday.com/hutan-pinus-pengger/ https://www.genjosholiday.com/ini-dia-5-keuntungan-membeli-mobil-bekas-perusahaan/ https://www.genjosholiday.com/ini-dia-5-penyebab-lain-mobil-susah-distarter-selain-karena-aki/ https://www.genjosholiday.com/ini-dia-untung-ruginya-jika-membeli-mobil-bekas-secara-kredit/ https://www.genjosholiday.com/ini-dia-7-cara-agar-mobil-anda-lebih-menghemat-bbm/ https://www.genjosholiday.com/karoseri-bus-populer/ https://www.genjosholiday.com/istilah-jual-beli-mobil-yang-perlu-dipahami-sebelum-ke-dealer-atau-pameran/ https://www.genjosholiday.com/jogja-bay/ https://www.genjosholiday.com/kalibiru/ https://www.genjosholiday.com/kapan-sebaiknya-ganti-oli-mobil/ https://www.genjosholiday.com/kapan-waktu-terbaik-untuk-ganti-ban-bus-pariwisata/ https://www.genjosholiday.com/kapan-waktu-terbaik-untuk-ganti-ban-mobil/ https://www.genjosholiday.com/kebiasaan-kebiasaan-yang-membuat-mesin-bus-pariwisata-cepat-rusak/ https://www.genjosholiday.com/kebiasaan-kebiasaan-yang-membuat-mesin-mobil-cepat-rusak/ https://www.genjosholiday.com/kebijakan-privasi/ https://www.genjosholiday.com/kebun-buah-mangunan/ https://www.genjosholiday.com/taman-sari/ https://www.genjosholiday.com/keraton-yogyakarta/ https://www.genjosholiday.com/keuntungan-dan-kerugian-membeli-mobil-bekas-secara-tunai/ https://www.genjosholiday.com/kontak/ https://www.genjosholiday.com/lakukan-hal-ini-pada-kendaraan-guna-cegah-virus-corona/ https://www.genjosholiday.com/langkah-sederhana-merawat-cat-bus-pariwisata/ https://www.genjosholiday.com/langkah-sederhana-merawat-cat-mobil/ https://www.genjosholiday.com/lava-tour-merapi/ https://www.genjosholiday.com/lima-komponen-mesin-mobil-dan-jenis-perbaikannya-yang-mahal/ https://www.genjosholiday.com/tebing-breksi/ https://www.genjosholiday.com/pantai-seruni/ https://www.genjosholiday.com/menjaga-performa-bus-pariwisata-dengan-perawatan-yang-benar/ https://www.genjosholiday.com/merawat-kaca-bus-pariwisata/ https://www.genjosholiday.com/mobil-sedan-kurang-diminati-di-indonesia-mungkin-inilah-sebabnya/ https://www.genjosholiday.com/museum-gunung-merapi/ https://www.genjosholiday.com/museum-ullen-sentalu/ https://www.genjosholiday.com/nasib-mobil-bekas-banjir-bagaimana-ini-beberapa-faktanya/ https://www.genjosholiday.com/paket-wisata/ https://www.genjosholiday.com/pantai-anora/ https://www.genjosholiday.com/pantai-ayah-lokasi-spot-foto-dan-harga-tiket-masuk/ https://www.genjosholiday.com/pantai-baros/ https://www.genjosholiday.com/pantai-blebak/ https://www.genjosholiday.com/pantai-bopong-lokasi-spot-foto-dan-harga-tiket-masuk/ https://www.genjosholiday.com/pantai-bugel-lokasi-spot-foto-dan-harga-tiket-masuk/ https://www.genjosholiday.com/pantai-cahaya-kendal/ https://www.genjosholiday.com/pantai-cemoro-sewu-lokasi-spot-foto-dan-harga-tiket-masuk-genjos-holiday/ https://www.genjosholiday.com/pantai-depok/ https://www.genjosholiday.com/pantai-drini/ https://www.genjosholiday.com/pantai-genjik/ https://www.genjosholiday.com/pantai-goa-cemara-lokasi-spot-foto-dan-harga-tiket-masuk/ https://www.genjosholiday.com/pantai-greweng/ https://www.genjosholiday.com/pantai-jetis/ https://www.genjosholiday.com/pantai-jodo/ https://www.genjosholiday.com/pantai-jogan/ https://www.genjosholiday.com/pantai-kalipat/ https://www.genjosholiday.com/pantai-karang/ https://www.genjosholiday.com/pantai-karang-hawu/ https://www.genjosholiday.com/pantai-karang-2/ https://www.genjosholiday.com/pantai-klotok/ https://www.genjosholiday.com/pantai-kukup/ https://www.genjosholiday.com/pantai-kuwaru/ https://www.genjosholiday.com/pantai-lampon/ https://www.genjosholiday.com/pantai-minajaya/ https://www.genjosholiday.com/pantai-moro-kendal/ https://www.genjosholiday.com/pantai-muara-kencan/ https://www.genjosholiday.com/pantai-ngebum/ https://www.genjosholiday.com/pantai-ngobaran/ https://www.genjosholiday.com/pantai-ngrenehan-lokasi-rute-fasilitas-harga-tiket-masuk/ https://www.genjosholiday.com/pantai-parangendog-lokasi-spot-foto-dan-harga-tiket-masuk/ https://www.genjosholiday.com/pantai-parangkusumo/ https://www.genjosholiday.com/pantai-parangtritis/ https://www.genjosholiday.com/pantai-pasir-puncu/ https://www.genjosholiday.com/pantai-pecaron/ https://www.genjosholiday.com/pantai-porok/ https://www.genjosholiday.com/pantai-sadeng/ https://www.genjosholiday.com/pantai-siung/ https://www.genjosholiday.com/pantai-slili/ https://www.genjosholiday.com/pantai-suwuk/ https://www.genjosholiday.com/pantai-widarapayung/ https://www.genjosholiday.com/pantai-wonokerto/ https://www.genjosholiday.com/penyebab-dan-solusi-kenapa-mesin-cepat-panas-saat-ac-mobil-hidup/ https://www.genjosholiday.com/puncak-becici/ https://www.genjosholiday.com/puncak-kosakora/ https://www.genjosholiday.com/puncak-kuda-sembrani/ https://www.genjosholiday.com/puncak-suroloyo/ https://www.genjosholiday.com/puncak-widosari/ https://www.genjosholiday.com/review-toyota-hiace-commuter/ https://www.genjosholiday.com/goa-pindul/ https://www.genjosholiday.com/sewa-bus-pariwisata/ambon/ https://www.genjosholiday.com/sewa-bus-pariwisata/bali/ https://www.genjosholiday.com/sewa-bus-pariwisata/banda-aceh/ https://www.genjosholiday.com/sewa-bus-pariwisata/bandar-lampung/ https://www.genjosholiday.com/sewa-bus-pariwisata/bandung/ https://www.genjosholiday.com/sewa-bus-pariwisata/banjarmasin/ https://www.genjosholiday.com/sewa-bus-pariwisata/banjarnegara/ https://www.genjosholiday.com/sewa-bus-pariwisata/banyuwangi/ https://www.genjosholiday.com/sewa-bus-pariwisata/bengkulu/ https://www.genjosholiday.com/sewa-bus-pariwisata/cilacap/ https://www.genjosholiday.com/sewa-bus-pariwisata/gorontalo/ https://www.genjosholiday.com/sewa-bus-pariwisata/jakarta/ https://www.genjosholiday.com/sewa-bus-pariwisata/jambi/ https://www.genjosholiday.com/sewa-bus-pariwisata/jogja/ https://www.genjosholiday.com/sewa-bus-pariwisata/kebumen/ https://www.genjosholiday.com/sewa-bus-pariwisata/lombok/ https://www.genjosholiday.com/sewa-bus-pariwisata/malang/ https://www.genjosholiday.com/sewa-bus-pariwisata/manado/ https://www.genjosholiday.com/sewa-bus-pariwisata/padang/ https://www.genjosholiday.com/sewa-bus-pariwisata/palangkaraya/ https://www.genjosholiday.com/sewa-bus-pariwisata/palembang/ https://www.genjosholiday.com/sewa-bus-pariwisata/pangkal-pinang/ https://www.genjosholiday.com/sewa-bus-pariwisata/ https://www.genjosholiday.com/sewa-bus-pariwisata/semarang/ https://www.genjosholiday.com/sewa-bus-pariwisata/pekanbaru/ https://www.genjosholiday.com/sewa-bus-pariwisata/pontianak/ https://www.genjosholiday.com/sewa-bus-pariwisata/samarinda/ https://www.genjosholiday.com/sewa-bus-pariwisata/sidoarjo/ https://www.genjosholiday.com/sewa-bus-pariwisata/solo/ |