Attachment 294950 Details for Bug 432877 – IRC chat log

IRC chat log

flah.txt (text/plain), 20.53 KB, created by Jonathan Earl Brassow on 2008-02-14 21:45:30 UTC

(hide)

Description:

Filename:

MIME Type:

Creator: Jonathan Earl Brassow

Created: 2008-02-14 21:45:30 UTC

Size: 20.53 KB

patch

obsolete

>Feb 14 13:46:05 <visegrips>     riley_dt, ping
>Feb 14 13:51:20 -->     anders (~akarlsso@vpn-6-33.fab.redhat.com) has joined #cluster
>Feb 14 13:54:47 -->     JuanDRay (~jrfuller@vpn-14-45.rdu.redhat.com) has joined #cluster
>Feb 14 13:59:49 <riley_dt>      visegrips pong
>Feb 14 14:00:04 <visegrips>     riley_dt, I'm having some trouble with chkpoints
>Feb 14 14:00:13 <riley_dt>      could you be more speicifc :)
>Feb 14 14:00:21 <visegrips>     riley_dt, of course, one sec
>Feb 14 14:00:27 <riley_dt>      want to tlak on phone or irc ?
>Feb 14 14:00:50 <visegrips>     riley_dt, I can start up all machines just fine (1st one sends checkpoints to others),
>Feb 14 14:01:25 <visegrips>     riley_dt, but if I kill a machine and bring it back, the others say they have sent it the checkpoint, but there are always '0' sections...
>Feb 14 14:01:37 <visegrips>     riley_dt, I'm not sure what I should be looking for.
>Feb 14 14:02:00 <visegrips>     riley_dt, perhaps I am not closing the checkpoint properly... so the second round conflicts with the first...
>Feb 14 14:02:10 <visegrips>     riley_dt, or perhaps something else?
>Feb 14 14:02:21 <riley_dt>      checkpoints are like posix
>Feb 14 14:02:29 <visegrips>     riley_dt, ?
>Feb 14 14:02:38 <visegrips>     files?
>Feb 14 14:02:42 <riley_dt>      when you close one and open another eveyrone referencing the old checkpoing keeps it open
>Feb 14 14:02:43 <visegrips>     unlink and what not
>Feb 14 14:02:46 <riley_dt>      yeah
>Feb 14 14:02:50 <riley_dt>      are you unlinking ?
>Feb 14 14:02:57 <visegrips>     y
>Feb 14 14:03:07 <riley_dt>      well it is possible there is a bug in unlink
>Feb 14 14:03:36 <riley_dt>      so you unlink the checkpoint and open a new one ?
>Feb 14 14:03:50 <visegrips>     riley_dt, sort of... let me explain:
>Feb 14 14:03:51 <riley_dt>      show me code pls
>Feb 14 14:04:00 <visegrips>     cluster/cmirror/src/cluster.c
>Feb 14 14:04:19 <visegrips>     1) export_checkpoint
>Feb 14 14:04:28 <visegrips>     1.1) open/create checkpoint
>Feb 14 14:04:35 <visegrips>     1.2) fill 3 sections
>Feb 14 14:04:40 <visegrips>     1.3) close
>Feb 14 14:04:52 <visegrips>     1.4) notify entering node that checkpoint is ready
>Feb 14 14:05:04 <visegrips>     2) import_checkpoint (on joining node)
>Feb 14 14:05:11 <visegrips>     2.1) open
>Feb 14 14:05:17 <visegrips>     2.2) unlink
>Feb 14 14:05:21 <visegrips>     2.3) read
>Feb 14 14:05:26 <visegrips>     2.4) close
>Feb 14 14:05:39 <visegrips>     * final close should eliminate the checkpoint
>Feb 14 14:06:06 <visegrips>     when a machine dies and comes back, the above process is repeated...
>Feb 14 14:06:19 <visegrips>     1.* succeeds (as best I can tell)
>Feb 14 14:06:43 <visegrips>     2.3 fails... saying "import_checkpoint: 0 checkpoint sections found"
>Feb 14 14:07:08 <riley_dt>      which node does the unlink
>Feb 14 14:07:15 <visegrips>     riley_dt, so I'm not sure if it's because I am missing a close somewhere, or what
>Feb 14 14:07:26 <visegrips>     riley_dt, 2.2 --- the entering node
>Feb 14 14:07:34 <riley_dt>      ok entering node is which ip
>Feb 14 14:07:49 <visegrips>     bp-xen-01
>Feb 14 14:07:59 <visegrips>     (.lab.msp.redhat.com)
>Feb 14 14:08:05 <riley_dt>      is that the lowest ip in the system
>Feb 14 14:08:09 <visegrips>     y
>Feb 14 14:08:35 <riley_dt>      does it work with killin of other nodes
>Feb 14 14:08:38 <visegrips>     n
>Feb 14 14:08:42 <visegrips>     same thing
>Feb 14 14:09:20 <visegrips>     riley_dt, the other nodes correctly send the info:
>Feb 14 14:09:26 -->     thorsten (~tscherf@vpn-4-100.str.redhat.com) has joined #cluster
>Feb 14 14:09:37 <visegrips>     Feb 14 08:52:57 bp-xen-03 clogd[1893]: [BhE28KAP] Checkpoint prepared for 1
>Feb 14 14:09:37 <visegrips>     Feb 14 08:52:57 bp-xen-03 clogd[1893]: Sending checkpointed data to 1
>Feb 14 14:09:37 <visegrips>     Feb 14 08:52:57 bp-xen-03 clogd[1893]: [BhE28KAP] Checkpoint ready, notification sent to 1
>Feb 14 14:10:06 <riley_dt>      so the 2's above are on the joining node
>Feb 14 14:10:10 <visegrips>     riley_dt, and node 1 gets the notification and is able to open, but there are no sections
>Feb 14 14:10:42 <visegrips>     riley_dt, right  :)
>Feb 14 14:10:56 <riley_dt>      and the 1's above are done by which node ?
>Feb 14 14:11:26 <visegrips>     riley_dt, all of the others... but the one that succeeds in this case is nodeid == 3, bp-xen-03
>Feb 14 14:11:36 <visegrips>     riley_dt, the others would say:
>Feb 14 14:11:52 <visegrips>     Feb 14 08:52:57 bp-xen-02 clogd[1894]: [BhE28KAP] Checkpoint prepared for 1
>Feb 14 14:11:52 <visegrips>     Feb 14 08:52:57 bp-xen-02 clogd[1894]: Sending checkpointed data to 1
>Feb 14 14:11:52 <visegrips>     Feb 14 08:52:57 bp-xen-02 clogd[1894]: export_checkpoint: sync checkpoint section already exists
>Feb 14 14:11:52 <visegrips>     Feb 14 08:52:57 bp-xen-02 clogd[1894]: -- closing checkpoint
>Feb 14 14:12:15 <--     anders has quit (Ping timeout: 630 seconds)
>Feb 14 14:12:27 <riley_dt>      how does it know when to do the checkpoint sync
>Feb 14 14:12:31 <riley_dt>      using cpg service ?
>Feb 14 14:13:06 <visegrips>     riley_dt, y.  when a config callback is recieved, they know that node is going to need a checkpoint (unless it is the first to join)
>Feb 14 14:13:20 -->     anders (~akarlsso@vpn-6-1.fab.redhat.com) has joined #cluster
>Feb 14 14:14:39 -->     deepthot (~dwysocha@vpn-14-77.rdu.redhat.com) has joined #cluster
>Feb 14 14:15:08 ---     bmarson_wfh is now known as bmarson_wfh-away
>Feb 14 14:15:18 <riley_dt>      how do you notify of the checkpoint has been updated
>Feb 14 14:17:28 <riley_dt>      i dont see "notification sent" in the src tree on jb-xen-01
>Feb 14 14:18:05 <--     jmh has quit (Read error: Connection reset by peer)
>Feb 14 14:18:26 -->     jmh (~jmh@vpn-248-26.boston.redhat.com) has joined #cluster
>Feb 14 14:18:48 <visegrips>     riley_dt, sorry, that's because I added that debug print to the end of export_checkpoint (just before the return)
>Feb 14 14:19:29 <visegrips>     riley_dt, you also won't find the "closing checkpoint" message... i added that too
>Feb 14 14:19:56 <riley_dt>      cluster_send
>Feb 14 14:19:59 <riley_dt>      is this a cman interface ?
>Feb 14 14:20:43 <riley_dt>      i see you use cpg mcast
>Feb 14 14:20:46 <visegrips>     riley_dt, no... it's at the top of the file, but it is only used to send notice to the in-coming node -
>Feb 14 14:20:49 <riley_dt>      well
>Feb 14 14:21:04 <riley_dt>      the node definately gets the notification huh?
>Feb 14 14:21:04 ---     jrummy is now known as jrummy_afk
>Feb 14 14:21:19 <visegrips>     - yes, because it tries to read teh checkpoint
>Feb 14 14:22:51 <visegrips>     riley_dt, I could put an open (without the create) after the close on the node that unlinks... that would tell us if it is reusing the old checkpoint somehow...
>Feb 14 14:23:18 ---     xixi is now known as xixi_afk
>Feb 14 14:23:19 <visegrips>     riley_dt, I also am unaware of how to open exclusively (or only if the file is also created)
>Feb 14 14:23:28 <--     thorsten has quit (Remote host closed the connection)
>Feb 14 14:23:48 <riley_dt>      there is no exlusive open
>Feb 14 14:24:07 <visegrips>     that's fine, I guess... that's how it is operating now
>Feb 14 14:24:53 ---     toure is now known as toure_gone
>Feb 14 14:25:44 <riley_dt>      IterationNext doesn't test for SA_AIS_ERR_RETRY
>Feb 14 14:25:53 <riley_dt>      when counting len
>Feb 14 14:26:36 <riley_dt>      what is rv on the break on line 490
>Feb 14 14:27:18 ---     jrummy_afk is now known as jrummy
>Feb 14 14:28:01 <riley_dt>      is the system under heavy io load ?
>Feb 14 14:28:09 <riley_dt>      i mean network communication
>Feb 14 14:28:18 <riley_dt>      soryr i mean ipc calls into the system :)
>Feb 14 14:28:21 <visegrips>     riley_dt, it isn't under any load
>Feb 14 14:28:57 <visegrips>     490?
>Feb 14 14:29:04 <visegrips>     mine must be pretty different from yours
>Feb 14 14:29:11 <riley_dt>      i am looking on jb-xen-01
>Feb 14 14:29:15 <riley_dt>      cluster.c
>Feb 14 14:29:35 <riley_dt>              while (1) {
>Feb 14 14:29:35 <riley_dt>                      rv = saCkptSectionIterationNext(itr, &desc);
>Feb 14 14:29:35 <riley_dt>                      if (rv == SA_AIS_OK)
>Feb 14 14:29:35 <riley_dt>                              len++;
>Feb 14 14:29:35 <riley_dt>                      else
>Feb 14 14:29:36 <riley_dt>                              break;
>Feb 14 14:29:38 <riley_dt>              }
>Feb 14 14:29:40 <riley_dt>
>Feb 14 14:30:19 <riley_dt>      could be a bug in checkpoint service
>Feb 14 14:30:33 <riley_dt>      or oculd be a usage bug
>Feb 14 14:31:03 <visegrips>     riley_dt, ok, I just added:
>Feb 14 14:31:13 <visegrips>             while (1) {
>Feb 14 14:31:13 <visegrips>                     rv = saCkptSectionIterationNext(itr, &desc);
>Feb 14 14:31:13 <visegrips>                     if (rv == SA_AIS_OK)
>Feb 14 14:31:13 <visegrips>                             len++;
>Feb 14 14:31:13 <visegrips>                     else if (rv != SA_AIS_ERR_RETRY)
>Feb 14 14:31:13 <visegrips>                             break;
>Feb 14 14:31:13 <visegrips>             }
>Feb 14 14:31:42 <riley_dt>      replace the 0 in iterationinitialize with SA_TIME_END
>Feb 14 14:32:37 <visegrips>     init_retry:
>Feb 14 14:32:37 <visegrips>             rv = saCkptSectionIterationInitialize(h, SA_CKPT_SECTIONS_ANY, SA_TIME_END, &itr);
>Feb 14 14:32:37 <visegrips>             if (rv == SA_AIS_ERR_TRY_AGAIN) {
>Feb 14 14:32:58 <riley_dt>      ws that already there ?
>Feb 14 14:33:01 <visegrips>     n
>Feb 14 14:33:07 <visegrips>     just showing you I did it
>Feb 14 14:33:37 <visegrips>     riley_dt, ok for me to reboot and try again?
>Feb 14 14:34:02 <visegrips>     riley_dt, (I also added an open check after the final unlink/close... just to test)
>Feb 14 14:34:03 <riley_dt>      did you fi the IterationNext chekcing to print out rv
>Feb 14 14:34:30 <riley_dt>      did you print out rv after IterationNext
>Feb 14 14:34:37 <visegrips>     riley_dt, no, but I can do that now too
>Feb 14 14:34:45 <riley_dt>      around like 490
>Feb 14 14:34:52 <riley_dt>      rv isn't tested for SA_AIS_ERR_RETRY
>Feb 14 14:35:03 <riley_dt>      so if SA_AIS_ERR_RETRY is returned iterationnext wont find the next entry
>Feb 14 14:35:14 <riley_dt>      it will just break with len == 0
>Feb 14 14:37:06 <visegrips>     riley_dt, what's the difference between SA_AIS_ERR_TRY_AGAIN and SA_AIS_ERR_RETRY?
>Feb 14 14:37:53 <riley_dt>      sorry there is no _RETRY
>Feb 14 14:37:56 <riley_dt>      there is only _TRY_AGIAN
>Feb 14 14:38:13 <riley_dt>      where you do the iteration around line 490 can you add a call to checkpointstatusget
>Feb 14 14:38:18 <riley_dt>      to determine the number of sections
>Feb 14 14:38:22 <riley_dt>      and print out the checkpoint descriptor
>Feb 14 14:38:52 <riley_dt>      or if i can edit on jb-xen-01 i can do it there if you like
>Feb 14 14:38:59 <robk>  long ping
>Feb 14 14:39:04 <robk>  lon ping
>Feb 14 14:39:15 <riley_dt>      long has a new nickname!!
>Feb 14 14:39:19 <robk>  heh
>Feb 14 14:39:31 <robk>  (better than lon pig)
>Feb 14 14:39:47 <riley_dt>      yeah
>Feb 14 14:40:25 <riley_dt>      3.5.7 is the spec section which describesit
>Feb 14 14:40:47 <visegrips>     riley_dt, give me a sec... and I'll upload the new files and you can change the code
>Feb 14 14:40:55 <riley_dt>      ok
>Feb 14 14:41:03 <visegrips>     riley_dt, line 490 is in export_checkpoint....
>Feb 14 14:41:35 <riley_dt>      not on jb-xen-01
>Feb 14 14:41:38 <riley_dt>      its import_checkpoint
>Feb 14 14:41:43 <riley_dt>      i guess the code on jb-xen-01 is old ?
>Feb 14 14:41:56 -->     dash (~dash@vpn-68-17.bne.redhat.com) has joined #cluster
>Feb 14 14:42:06 <visegrips>     riley_dt, shouldn't be, I'm using rsync to update
>Feb 14 14:42:23 *       visegrips checks
>Feb 14 14:42:55 <riley_dt>      what is the path to the files you are using
>Feb 14 14:42:57 <--     sbradley has quit (Quit: leaving)
>Feb 14 14:43:53 <visegrips>     cluster-rhel5/cmirror/src/cluster.c
>Feb 14 14:43:59 <visegrips>     riley_dt, should be up-to-date
>Feb 14 14:44:09 <visegrips>     they have the changes I just made
>Feb 14 14:44:33 <riley_dt>      /root/cluster-rhel5/cmirror/src
>Feb 14 14:44:37 <riley_dt>      that path ?
>Feb 14 14:44:46 <riley_dt>      has no changes on jb-xen-01
>Feb 14 14:45:03 <visegrips>     riley_dt, hahahahahahaq
>Feb 14 14:45:33 <visegrips>     riley_dt, I'm using ***bp-xen-01***   I was misreading your jb...
>Feb 14 14:45:42 <riley_dt>      whats bp
>Feb 14 14:45:51 -->     sbradley (~sbradley@mvmware.rdu.redhat.com) has joined #cluster
>Feb 14 14:46:12 <visegrips>     riley_dt, the name of the machine
>Feb 14 14:46:22 <riley_dt>      is it hosted on bid-08 ?
>Feb 14 14:46:27 <visegrips>     riley_dt, yup
>Feb 14 14:46:33 <visegrips>     riley_dt, sorry, no
>Feb 14 14:46:45 <visegrips>     riley_dt, bp-01 -> bp-xen-0[1234]
>Feb 14 14:47:38 <visegrips>     riley_dt, I haven't been using the jb-xen-* machines, because I thought you were using them to track down the chkpoint init issue (the restart of clogd)
>Feb 14 14:47:50 <riley_dt>      yes
>Feb 14 14:47:55 <riley_dt>      i am at night i thoguth u were using during day
>Feb 14 14:48:17 <visegrips>     :)
>Feb 14 14:48:47 <visegrips>     no, I have plenty of clusters/machines with bp-0[1234] -> bp-xen-0[1 - 16]
>Feb 14 14:49:06 <--     anders has quit (Ping timeout: 630 seconds)
>Feb 14 14:50:19 <riley_dt>      will LOG_PRINT come out in the log
>Feb 14 14:50:25 -->     thorsten (~tscherf@vpn-4-54.str.redhat.com) has joined #cluster
>Feb 14 14:50:33 <visegrips>     y
>Feb 14 14:52:11 ---     bmarson_wfh-away is now known as bmarson_wfh
>Feb 14 14:52:21 <--     bmarson_wfh has quit (Quit: Leaving)
>Feb 14 14:53:45 <riley_dt>      one problem i see
>Feb 14 14:54:00 <riley_dt>      line 513 checkpoint handle "h"
>Feb 14 14:54:17 <riley_dt>      nevemrind
>Feb 14 14:55:45 <visegrips>     riley_dt, let me know when you get the code changed, and I'll compile/test
>Feb 14 14:58:15 <riley_dt>      yup its set
>Feb 14 14:58:18 <riley_dt>      give it a roll
>Feb 14 14:58:22 <visegrips>     riley_dt, all machines?
>Feb 14 14:58:24 <riley_dt>      afk for 2 mins
>Feb 14 14:58:27 <riley_dt>      its on bp-xen-01
>Feb 14 14:58:31 <riley_dt>      cluster.c
>Feb 14 14:58:41 <visegrips>     riley_dt, ok, I'll transfer to 2 3 4 too
>Feb 14 14:59:00 <--     cverhoef has quit (Ping timeout: 630 seconds)
>Feb 14 15:00:53 <lon>   robk: ponmg
>Feb 14 15:01:54 <riley_dt>      back
>Feb 14 15:02:41 <visegrips>     riley_dt, wow, lots worse now... won't even work the first time
>Feb 14 15:03:03 <visegrips>     riley_dt, node1 joins first (doesn't need checkpoint)... others are constantly complaining
>Feb 14 15:05:08 <riley_dt>      hrm
>Feb 14 15:05:32 <visegrips>     riley_dt, crap
>Feb 14 15:05:44 <visegrips>             while (1) {
>Feb 14 15:05:44 <visegrips>                     rv = saCkptSectionIterationNext(itr, &desc);
>Feb 14 15:05:44 <visegrips>                     if (rv != SA_AIS_ERR_TRY_AGAIN) {
>Feb 14 15:05:44 <visegrips>                             LOG_ERROR("saCkptSectionIterationNext failure: %d", rv);
>Feb 14 15:05:44 <visegrips>                             break;
>Feb 14 15:05:44 <visegrips>                     }
>Feb 14 15:05:44 <visegrips>                     if (rv == SA_AIS_OK)
>Feb 14 15:05:44 <visegrips>                             len++;
>Feb 14 15:05:44 <visegrips>             }
>Feb 14 15:05:50 <visegrips>     that's not going to work :)
>Feb 14 15:05:57 <visegrips>     will redo
>Feb 14 15:06:01 <riley_dt>      that should work
>Feb 14 15:06:20 <riley_dt>      i looked at it a good while it seemed ok to me
>Feb 14 15:06:35 <riley_dt>      rv incresaes on each interation of sa-ais-ok
>Feb 14 15:06:46 <riley_dt>      on error that isn't try again an error is logged and break out of while occurs
>Feb 14 15:06:53 <riley_dt>      otherwise it repeats the loop
>Feb 14 15:08:34 <riley_dt>      your right it wont work
>Feb 14 15:08:51 <riley_dt>      if (rv == SA_AIS_OK) { len++} else
>Feb 14 15:08:59 <riley_dt>      if (rv !+ SA_AIS_ERR_TRY_AGAIN) {log error}
>Feb 14 15:09:20 <--     silberb has quit (Quit: Leaving)
>Feb 14 15:09:25 -->     silberb (~bsilberm@sebastian-int.corp.redhat.com) has joined #cluster
>Feb 14 15:09:32 <riley_dt>      Feb 14 10:09:26 bp-xen-02 clogd[1831]: printing [Before unlink] sections [3] result [1]
>Feb 14 15:09:33 <riley_dt>      Feb 14 10:09:26 bp-xen-02 clogd[1831]: printing [After unlink] sections [3] result [1]
>Feb 14 15:09:42 <riley_dt>      well that is a really good sign that the checkpoint sections exist
>Feb 14 15:09:54 <riley_dt>      and something isn't working with the iteration code in the executive or in the logic of the iteration
>Feb 14 15:10:42 ---     bmr is now known as bmr_afk
>Feb 14 15:10:53 <riley_dt>      there are 3 sectoins, correct ?
>Feb 14 15:11:27 -->     chrissie_ (~christin@vpn-6-3.fab.redhat.com) has joined #cluster
>Feb 14 15:11:28 <visegrips>     y
>Feb 14 15:12:23 <--     ldimaggi_ has quit (Quit: Leaving)
>Feb 14 15:12:50 ---     sly is now known as sly_mtg
>Feb 14 15:14:26 <--     JuanDRay has quit (Quit: Leaving)
>Feb 14 15:14:47 <--     chrissie has quit (Read error: No route to host)
>Feb 14 15:16:30 <--     mnewma1 has quit (Remote host closed the connection)
>Feb 14 15:16:39 refried riley_dt rmccabe rmunilla robk rohara_home
>Feb 14 15:16:46 <visegrips>     riley_dt, ok.... here we go.
>Feb 14 15:17:07 <--     jhunt has quit (Remote host closed the connection)
>Feb 14 15:17:25 <riley_dt>      cool
>Feb 14 15:17:33 <riley_dt>      well i have faith this will be an easy problem to solve
>Feb 14 15:17:46 <visegrips>     node3 replies fine, node1 gets this:
>Feb 14 15:17:49 <visegrips>     Feb 14 10:24:33 bp-xen-01 clogd[1776]: printing [Before unlink] sections [0] result [1]
>Feb 14 15:17:49 <visegrips>     Feb 14 10:24:33 bp-xen-01 clogd[1776]: printing [After unlink] sections [0] result [1]
>Feb 14 15:17:49 <visegrips>     Feb 14 10:24:33 bp-xen-01 clogd[1776]: saCkptSectionIterationNext failure: 27
>Feb 14 15:17:49 <visegrips>     Feb 14 10:24:33 bp-xen-01 clogd[1776]: import_checkpoint: 0 checkpoint sections found
>Feb 14 15:19:11 <riley_dt>      bp-xen-01 was restarted ?
>Feb 14 15:19:26 <visegrips>     y
>Feb 14 15:19:35 <visegrips>     killed and restarted
>Feb 14 15:20:54 <riley_dt>      well that implies there is some problem with syncronizatoin
>Feb 14 15:22:36 <riley_dt>      just turned into a hard problem to solve
>Feb 14 15:24:29 <riley_dt>      at this point i would recommend filing a bugzilla with a log of this conversation
>Feb 14 15:24:45 <riley_dt>      i really need to address this blocker issue with rhel5
>Feb 14 15:25:12 <riley_dt>      in order to determine what is going on i will have to instrument the checkpoint code on the bp nodes
>Feb 14 15:25:28 -->     mmcallis (~mmcallis@dhcp-64-222.bne.redhat.com) has joined #cluster
>Feb 14 15:26:06 <--     dash has quit (Quit: Ah sleep)
>Feb 14 15:26:10 <riley_dt>      was this happening before ?
>Feb 14 15:26:13 <riley_dt>      or did some code change
>Feb 14 15:26:50 <--     kapow has quit (Quit: Leaving)
>Feb 14 15:28:12 <visegrips>     riley_dt, it's always been happening
>Feb 14 15:28:33 <riley_dt>      can you try just one more thing for me
>Feb 14 15:28:50 <visegrips>     riley_dt, sure
>Feb 14 15:28:58 <riley_dt>      is it ok to sleep in cluster.c code ?
>Feb 14 15:29:11 <visegrips>     riley_dt, Bug 430296 already concerns this issue
>Feb 14 15:29:25 <visegrips>     riley_dt, not really... depends on how long
>Feb 14 15:29:29 <visegrips>     and where
>Feb 14 15:30:22 <riley_dt>      visegrips need a seperate bug for openais on the issue
>Feb 14 15:30:30 <riley_dt>      that other bug is cmirror
>Feb 14 15:30:37 <riley_dt>      two seperate bugs :)
>Feb 14 15:30:55 <visegrips>     riley_dt, I can give you another bug if you want...
>Feb 14 15:31:00 <riley_dt>      yeah
>Feb 14 15:31:11 <riley_dt>      just for the purposes of logging this debug session
>Feb 14 15:31:35 ---     bmarson_gone is now known as bmarson
>Feb 14 15:32:21 <riley_dt>      ok cluste.rc on bp-xen-01 is ready to go
>Feb 14 15:32:22 <riley_dt>      afk 2 mins
>Feb 14 15:32:52 <riley_dt>      actually jon I wanted a seperate bugzilla
>Feb 14 15:32:57 <riley_dt>      not that bugzilla assigned to me.. :)
>Feb 14 15:33:00 <riley_dt>      i will just open one up
>Feb 14 15:33:30 <visegrips>     riley_dt, crap.  didn't mean to do that
>Feb 14 15:33:38 <visegrips>     riley_dt, I'm already opening one
>Feb 14 15:33:47 <visegrips>     Bug 432877
>Feb 14 15:34:45 <riley_dt>      ok bio 2 mins afk :)
>>      afk 2 mins
>Feb 14 15:32:52 <riley_dt>      actually jon I wanted a seperate bugzilla
>Feb 14 15:32:57 <riley_dt>      not that bugzilla assigned to me.. :)
>Feb 14 15:33:00 <riley_dt>      i will just open one up
>Feb 14 15:33:30 <visegrips>     riley_dt, crap.  didn't mean to do that
>Feb 14 15:33:38 <visegrips>     riley_dt, I'm already opening one
>Feb 14 15:33:47 <visegrips>     Bug 432877
>Feb 14 15:34:45 <riley_dt>      ok bio 2 mins afk :)

Actions: View

Attachments on bug 432877: 294950