Bug 1082327

I think case-based scheme should replace current one, if that is not feasible we are better off with post-analysis and pre-cleanup hooks that would execute scripts.

In case hooks get implemented, retrace should provide config options +/- follows:
HookUser = blah
HookGroup = blah
HookPostAnalysis = /some/dir
HookPreCleanup = /some/other/dir

Then it would execute each file in respective directories as HookUser:HookGroup, in alphabetical order.

Scripts would receive 2 arguments: task id and task directory

(yes, I know task id can be currently obtained from the path, but this does look like a good thing to depend on)

Created attachment 880444 [details]
Simple bash script to create an index of retrace tasks by caseno and finished_time

Fixed 'find' in script - otherwise we get a lot of these, I think from the 32-bit vmcores.
find: `/cores/retrace/tasks/610785335-kernel': Permission denied

(In reply to Mateusz Guzik from comment #1)
> I think case-based scheme should replace current one, if that is not
> feasible we are better off with post-analysis and pre-cleanup hooks that
> would execute scripts.
> 
Not sure what you mean - are you saying replace the taskid with something like '/cores/retrace/tasks/<caseno>/..."?

If so I'm not sure about it since it's a bigger change.  Right now 'caseno' is optional too.

> In case hooks get implemented, retrace should provide config options +/-
> follows:
> HookUser = blah
> HookGroup = blah
> HookPostAnalysis = /some/dir
> HookPreCleanup = /some/other/dir
> 
> Then it would execute each file in respective directories as
> HookUser:HookGroup, in alphabetical order.

I think the hooks should be a different bz - feel free to file it or I will do it soon.  I agree we need these if there is not already something in retrace.  Need to think about it to make sure existing retrace functionality is not destabilized by some post-analysis hook.

(In reply to Dave Wysochanski from comment #4)
> (In reply to Mateusz Guzik from comment #1)
> > I think case-based scheme should replace current one, if that is not
> > feasible we are better off with post-analysis and pre-cleanup hooks that
> > would execute scripts.
> > 
> Not sure what you mean - are you saying replace the taskid with something
> like '/cores/retrace/tasks/<caseno>/..."?
> 
> If so I'm not sure about it since it's a bigger change.  Right now 'caseno'
> is optional too.
> 

Well, not requiring case number when submitting a vmcore reduces usefulness of this feature in the first place. So case number should become mandatory (and maybe we can work something out to make it easier to submit it).

With case-related numbers ids current id scheme become redundant, thus it makes sense to get rid of it.

However, if that cannot be done for now, it will be easier for everyone to have custom scripts doing the job.

On the other hand we would need hooks even if current id scheme goes away, so I agree separate bz is needed - https://bugzilla.redhat.com/show_bug.cgi?id=1082376

That said, this bz should be: replace current scheme with case number based one, if that is not going to work we should close it.

(In reply to Mateusz Guzik from comment #5)
> (In reply to Dave Wysochanski from comment #4)
> > (In reply to Mateusz Guzik from comment #1)
> > > I think case-based scheme should replace current one, if that is not
> > > feasible we are better off with post-analysis and pre-cleanup hooks that
> > > would execute scripts.
> > > 
> > Not sure what you mean - are you saying replace the taskid with something
> > like '/cores/retrace/tasks/<caseno>/..."?
> > 
> > If so I'm not sure about it since it's a bigger change.  Right now 'caseno'
> > is optional too.
> > 
> 
> Well, not requiring case number when submitting a vmcore reduces usefulness
> of this feature in the first place. So case number should become mandatory
> (and maybe we can work something out to make it easier to submit it).
> 
True but assuming people working on cases are submitting the cores there should be incentives to submit the case # with the core.  That said, if we can make it easier we should do it.  I already filed one bz in this regard and it's been addressed (https://bugzilla.redhat.com/show_bug.cgi?id=999643)

Maybe someone needs to work out a portal app to take a vmcore name and submit from the case, then the case # would be there.  After all, the customer just gives us the tarball name of the vmcore they uploaded and that is how it is identified through case updates.  It then makes sense to have some way directly from the case to submit to retrace-server and we should have most vmcores with caseno file.



> With case-related numbers ids current id scheme become redundant, thus it
> makes sense to get rid of it.
> 
Not really.  Why would you think everyone using retrace-server for vmcores would have a case#?  Remember people can use it outside of case work.  That is just our primary use case.  However, I think we're agreeing on the general idea that task#'s are just numbers, and they don't add value to a user of retrace - they kinda get in the way a bit but I understand why we have them.

On a slightly related topic, it may be nice to be able to load crash just based on the vmcore tarball, or perhaps we need tools to look it up (kinda what this bz is).  I do agree the 'taskid' is just something people need to remember which is really unrelated to the important information which identifies a vmcore (case#, vmcore tarball name, machine name timestamp of the crash, customer name, bugzilla if associated, etc).  So it may be we need to think about a better interface into retrace rather than using the taskid, which is really what is the overarching reason for this bz I think.


> However, if that cannot be done for now, it will be easier for everyone to
> have custom scripts doing the job.
> 
Ok so you'd have this index outside of retrace-server package then?
I didn't like the idea of custom scripts the more I thought about it but yeah it can be done.  We'd need to have cleanup script too which is not too hard.


> On the other hand we would need hooks even if current id scheme goes away,
> so I agree separate bz is needed -
> https://bugzilla.redhat.com/show_bug.cgi?id=1082376
> 
> That said, this bz should be: replace current scheme with case number based
> one, if that is not going to work we should close it.

I don't think I agree with the above but I'll have to take a crack at a patch and/or see what Michal thinks.  Maybe you're right and we should just close the bz and add scripts in some other package, which would go on top of existing retrace-server taskid based design to make the user interface a bit more "workflow friendly"

(In reply to Mateusz Guzik from comment #5)
> That said, this bz should be: replace current scheme with case number based
> one, if that is not going to work we should close it.

One thing we always need is some disambiguation for cores with the same characteristics (same case#, submitted at same time, tarball same name, etc).  If we ditch taskid, we'd have to come up with some scheme for uniqueness problem (such as time of submission to retrace-server if it can be guaranteed unique).  So I think the existing 'taskid' type approach is fine for uniqueness problem and the base of retrace-server.

(In reply to Dave Wysochanski from comment #6)
> (In reply to Mateusz Guzik from comment #5)
> > (In reply to Dave Wysochanski from comment #4)
> > > (In reply to Mateusz Guzik from comment #1)
> > > > I think case-based scheme should replace current one, if that is not
> > > > feasible we are better off with post-analysis and pre-cleanup hooks that
> > > > would execute scripts.
> > > > 
> > > Not sure what you mean - are you saying replace the taskid with something
> > > like '/cores/retrace/tasks/<caseno>/..."?
> > > 
> > > If so I'm not sure about it since it's a bigger change.  Right now 'caseno'
> > > is optional too.
> > > 
> > 
> > Well, not requiring case number when submitting a vmcore reduces usefulness
> > of this feature in the first place. So case number should become mandatory
> > (and maybe we can work something out to make it easier to submit it).
> > 
> True but assuming people working on cases are submitting the cores there
> should be incentives to submit the case # with the core.  That said, if we
> can make it easier we should do it.  I already filed one bz in this regard
> and it's been addressed (https://bugzilla.redhat.com/show_bug.cgi?id=999643)
> 
> Maybe someone needs to work out a portal app to take a vmcore name and
> submit from the case, then the case # would be there.  After all, the
> customer just gives us the tarball name of the vmcore they uploaded and that
> is how it is identified through case updates.  It then makes sense to have
> some way directly from the case to submit to retrace-server and we should
> have most vmcores with caseno file.

This sounds like a good idea.  The case can then maintain it's own list of vmcores submitted (and possibly include details from the 'sys' command so we can distinguish different cores).

> > Well, not requiring case number when submitting a vmcore reduces usefulness
> > of this feature in the first place. So case number should become mandatory
> > (and maybe we can work something out to make it easier to submit it).
> > 
> True but assuming people working on cases are submitting the cores there
> should be incentives to submit the case # with the core.  That said, if we
> can make it easier we should do it.  I already filed one bz in this regard
> and it's been addressed (https://bugzilla.redhat.com/show_bug.cgi?id=999643)
> 
> Maybe someone needs to work out a portal app to take a vmcore name and
> submit from the case, then the case # would be there.  After all, the
> customer just gives us the tarball name of the vmcore they uploaded and that
> is how it is identified through case updates.  It then makes sense to have
> some way directly from the case to submit to retrace-server and we should
> have most vmcores with caseno file.
> 

That sounds reasonable for the short term.

I think the real problem is that customers upload files to an anonymous server. Instead, an account should be created along with the case in sfdc and you would get credentials in return. Files uploaded with such credentials end up in case-specific directory thus we get stuff automagically classified and can have it acted upon in similar manner.
 
> > With case-related numbers ids current id scheme become redundant, thus it
> > makes sense to get rid of it.
> > 
> Not really.  Why would you think everyone using retrace-server for vmcores
> would have a case#?  Remember people can use it outside of case work.  That
> is just our primary use case.  However, I think we're agreeing on the
> general idea that task#'s are just numbers, and they don't add value to a
> user of retrace - they kinda get in the way a bit but I understand why we
> have them.
> 

I don't really see why would you use retrace for something else than case work, but such usage only strenghtens my argument. :)

As was noted, the problem is that we just have a directory with semi-random numbers known as task id. Nobody knows what the core is based on directory content unless it happens to have caseno files.

That said, some outside-of-sfdc usage with no additional information would only contribute to the mess. Instead all cores should be "resolvable" to a case number, and if there is none we have to know who owns the core - so it needs some user-specific id, username or something else. I don't see why this could not be included in task id.

> On a slightly related topic, it may be nice to be able to load crash just
> based on the vmcore tarball, or perhaps we need tools to look it up (kinda
> what this bz is).  I do agree the 'taskid' is just something people need to
> remember which is really unrelated to the important information which
> identifies a vmcore (case#, vmcore tarball name, machine name timestamp of
> the crash, customer name, bugzilla if associated, etc).  So it may be we
> need to think about a better interface into retrace rather than using the
> taskid, which is really what is the overarching reason for this bz I think.
>

Just core name will lead to ambiguity (read: problems) in the long run, case number is needed. 

For instance if per-case dirs are created we can suddenly get files with the same name from different cases (and the one you want is not submitted yet, or maybe just got removed, while the other one is ready), you don't want to open a different core by accident, do you? :->

... and if you allow people to lookup the core based only on the name this will happen in the future.

But if we will have links/dirs with cases (and we will) this loses most of its value.

> 
> > However, if that cannot be done for now, it will be easier for everyone to
> > have custom scripts doing the job.
> > 
> Ok so you'd have this index outside of retrace-server package then?
> I didn't like the idea of custom scripts the more I thought about it but
> yeah it can be done.  We'd need to have cleanup script too which is not too
> hard.
> 

Not sure what you don't like. With custom scripts we have more flexibility to do various stuff.

> 
> > On the other hand we would need hooks even if current id scheme goes away,
> > so I agree separate bz is needed -
> > https://bugzilla.redhat.com/show_bug.cgi?id=1082376
> > 
> > That said, this bz should be: replace current scheme with case number based
> > one, if that is not going to work we should close it.
> 
> I don't think I agree with the above but I'll have to take a crack at a
> patch and/or see what Michal thinks.  Maybe you're right and we should just
> close the bz and add scripts in some other package, which would go on top of
> existing retrace-server taskid based design to make the user interface a bit
> more "workflow friendly"

Again, custom scripts = more flexibility. On the other hand replacing current scheme with case-based is cleaner and would have to happen in retrace.

Leaving the scheme as it is and implementing this feature in retrace gives us the worst of two worlds. :)

(In reply to Dave Wysochanski from comment #7)
> (In reply to Mateusz Guzik from comment #5)
> > That said, this bz should be: replace current scheme with case number based
> > one, if that is not going to work we should close it.
> 
> One thing we always need is some disambiguation for cores with the same
> characteristics (same case#, submitted at same time, tarball same name,
> etc).  If we ditch taskid, we'd have to come up with some scheme for
> uniqueness problem (such as time of submission to retrace-server if it can
> be guaranteed unique).  So I think the existing 'taskid' type approach is
> fine for uniqueness problem and the base of retrace-server.

Uniqueness is not a problem. We can have: caseno/coreno, caseno-coreno or some other variant where coreno is a counter.

(In reply to Mateusz Guzik from comment #10)
> (In reply to Dave Wysochanski from comment #7)
> > (In reply to Mateusz Guzik from comment #5)
> > > That said, this bz should be: replace current scheme with case number based
> > > one, if that is not going to work we should close it.
> > 
> > One thing we always need is some disambiguation for cores with the same
> > characteristics (same case#, submitted at same time, tarball same name,
> > etc).  If we ditch taskid, we'd have to come up with some scheme for
> > uniqueness problem (such as time of submission to retrace-server if it can
> > be guaranteed unique).  So I think the existing 'taskid' type approach is
> > fine for uniqueness problem and the base of retrace-server.
> 
> Uniqueness is not a problem. We can have: caseno/coreno, caseno-coreno or
> some other variant where coreno is a counter.

Yeah that's a good idea.  But again you're assuming we mandate the case#.  I'm not sure about this since not everyone that uses retrace has 'case numbers'.  For us it would be great though.

I am also concerned about instability should we go the route of moving away from the taskid as the basis.  Recently we've addressed a lot of problems but we've still got multiple issues in production.  It's one thing to add an index, but another thing to change the design.  We need Michal to weigh in on it, someone to create a patch, and probably some help with testing.  It doesn't seem like it would be too bad but we won't know until someone does a patch.

I already noted that in #9:

"That said, some outside-of-sfdc usage with no additional information would only contribute to the mess. Instead all cores should be "resolvable" to a case number, and if there is none we have to know who owns the core - so it needs some user-specific id, username or something else. I don't see why this could not be included in task id."

Replace caseno with ident, where ident is either a case number or user login.

This message is a reminder that EPEL 6 is nearing its end of life. Fedora will stop maintaining and issuing updates for EPEL 6 on 2020-11-30. It is our policy to close all bug reports from releases that are no longer maintained. At that time this bug will be closed as EOL if it remains open with a 'version' of 'el6'.

Package Maintainer: If you wish for this bug to remain open because you plan to fix it in a currently maintained version, simply change the 'version' to a later EPEL version.

Thank you for reporting this issue and we are sorry that we were not able to fix it before EPEL 6 is end of life. If you would still like to see this bug fixed and are able to reproduce it against a later version of Fedora, you are encouraged  change the 'version' to a later Fedora version prior this bug is closed as described in the policy above.

EPEL el6 changed to end-of-life (EOL) status on 2020-11-30. EPEL el6 is
no longer maintained, which means that it will not receive any further
security or bug fix updates. As a result we are closing this bug.

If you can reproduce this bug against a currently maintained version of
EPEL please feel free to reopen this bug against that version. If you
are unable to reopen this bug, please file a new report against the
current release. If you experience problems, please add a comment to this
bug.

Thank you for reporting this bug and we are sorry it could not be fixed.