r96 - 10 Aug 2007 - 15:05:56 - TedLeungYou are here: OSAF >  Projects Web  >  CommunityHome > QuantitativeCommunityInvolvementMetrics

Quantitative Community Involvement Metrics

This area is for tracking community involvement by several different quantitative metrics. This is what we want to track:

Mailing List Posting Metrics

Number of messages posted to OSAF lists by community members in a week, and the number of unique posters. Each are broken down into a total, volunteers (non-OSAF staff), and OSAF staff.

Date Messages Unique Posters PyLucene
Week ending total vols OSAF total vols OSAF

Failed to include URL http://kumu.osafoundation.org/~bear/metrics/countMessages.txt

This shows how many people are subscribed to the following OSAF mailing lists on the given day.

Date Announce Design subscr Dev subscr General Process PyLucene

Failed to include URL http://kumu.osafoundation.org/~bear/metrics/countListSubscriptions.txt

Wiki accesses to meeting minutes and status for week ending

This table shows the weekly Wiki hits, excluding hits from OSAF's Howard Street office.
  • Total wiki hits
  • Staff meeting minutes ("Staff")
  • Management meeting minutes ("Mgmt")
  • Comprehensive status reports ("Status")
  • Newsletters ("Newsletters")
  • Design working group meetings ("Design")
  • Apps working group meetings ("Apps")
  • Repository working group meetings ("Repository")
  • Community working group meetings ("Comm")
  • The number of accounts on the given day
  • The number of RSS requests in a week

Week ending wiki hits Staff Mgmt Status Newsletters Design Apps Repos Comm Wiki accounts RSS gets

Failed to include URL http://kumu.osafoundation.org/~bear/metrics/countWikiPageHits.txt

Web hits in the week ending

Weekly Web hits, excluding hits from OSAF's Howard Street offices (from http://stats.osafoundation.org):

Week ending total web hits page hits downloads

Failed to include URL http://kumu.osafoundation.org/~bear/metrics/countWebHits.txt

Note: the third column shows all downloads, including 206s (partial GETs), 304s (no modification), and HEADs (excluding accesses from Howard Street).

Note that for the 9 May and 16 May 2004 lines, we believe that roughly 20,000 hits are due to a change in the build system. The builds started being run much more frequently, and the Tinderbox page refreshes itself automatically. On 17 May, Markie modified the Tinderbox status page so that it does not pull the OSAF logo nor the css file from hula, so we thought the number of hits should've dropped significantly. We don't yet understand why they didn't.

On around July 1, we moved the Web server, and it took a while to get back to normal. Hence the missing data. Also, the log parsing tool needed to be rewritten because of a change in practice.

Downloads

This table shows the number of possibly successful Chandler downloads in a given week (excluding downloads from the Howard Street office):

  Release: Milestones older releases 0.4 release 0.5 release 0.6 release
Date Total Total Win Linux Mac Total Win Linux Mac Total Win Linux Mac Total Win Linux Mac Total Win Linux Mac

Failed to include URL http://kumu.osafoundation.org/~bear/metrics/countDownloads.txt

This table shows only the downloads that had a chance of retrieving the entire package (i.e. were GETs with a status code of 200).

Bugzilla

This table shows the number of bugs filed in a week, both by OSAF employees and others in the community ("vols").

  Bugs Opened Bugs Closed Patches BZ accounts % regressed
Week ending total vols OSAF total vols OSAF total vols OSAF (cumulative)  

Failed to include URL http://kumu.osafoundation.org/~bear/metrics/countBugzilla.txt

CVS in week ending

This shows how much CVS activity there was in a week. We don't yet have a way to count who is OSAF and who is not..
  files lines
Week ending # added # removed # changed added removed net change comments

Failed to include URL http://kumu.osafoundation.org/~bear/metrics/countChanges.txt

IRC Office Hours

This table shows the number of people ("talkers") or present but not participating ("lurkers") in an IRC Office Hour. "OH lines" gives the number of lines in the transcript of the hour, and the topic gives what the Office Hour was about (and a link to the transcript).

Date vol talkers OSAF talkers vol lurkers OSAF lurkers total vols total OSAF OH lines topic

Failed to include URL http://kumu.osafoundation.org/~bear/metrics/countIRCs.txt

Future Metrics

Here are some metrics that we want to track but don't yet:
  • Number of patches per week submitted by the community
  • code reviews done by community
  • (Not a community metric per se, but useful) Bug regressions, i.e. how many bugs introduced when fixing other bugs
  • Lines of code in Chandler
  • number of hits to the RSS feed of the wiki
  • StarterProjects projects posted / active / participants / completed
  • attendance at public events
  • Media exposure: number of references / citations / articles / blogs / etc. referring to Chandler and OSAF from the community and general press, e.g.:
    • news links (number of articles from news.google.com mentioning "open source applications foundation")
    • web links (Google of "link:www.osafoundation.org")
    • web references (Google of "osaf chandler -site:osafoundation.org" for the "last 3 months")
    • blog mentions1 (Google of "open source applications foundation blog")
    • blog mentions2 (Google of "osafoundation.org ext:xml" and ext:rdf and ext:rss)
    • blog mentions3 (Feedster search of osafoundation.org)

Date News Links references Blog1 Blog2 Blog3
18 Mar 2004 1 795 3,570 7480 1121 353

If you have an idea for a good metric that we aren't tracking, please send email to ducky at osafoundation.org. ("Number of active volunteers" is something we'd like to track, but it isn't clear how we would count it.)

-- DuckySherwood - 13 Jan 2004


Comments

This looks like a great idea. I like what the community group is doing here to try to track some quantitative metrics of success. I hope the design group and the other groups will end up following the lead of the community group.

After you have some raw data for different metrics, it seems like the next step would be to try to figure out how to use these metrics to keep track of how successful OSAF's community outreach effort is. Towards that end, it might be good to start thinking about what the metrics might look like for a successful effort. In my opinion, it seems like the main goals of the community effort are (1) to help get Chandler built as quickly as possible, (2) to help make sure Chandler is a quality product, and (3) to help speed the adoption of Chandler. I think that "community involvement" should not be a goal in an of itself, unless it helps achieve one of the 3 main goals.

So for goal #1, one of the metrics of success might be "number of lines of code contributed by volunteers". Or, more accurately, the metric should be "number of lines of contributed code which end up included in 1.0", and that would be a positive metric. And actually, there might also be a metric for "number of lines of contributed code which end up not included in 1.0", and that would be a negative metric. The long term goal would be to try to increase the values of positive metrics and decrease the values of negative metrics.

  • BrianDouglasSkinner - 26 Jan 2004


Heikki explained how to get the bugs in a week:
    1. Go to Bugzilla query page: http://bugzilla.osafoundation.org/query.cgi
    2. Select product (Chandler)
    3. Clear Status (none selected)
    4. In the Email and numbering section, uncheck bug owner, check reporter, select "doesn't match regexp" from the drop down list and write "osafoundation.org" (without quotes) in the edit box
    5. In Bug Changes section select [bug creation] field, and write the dates from-to below


I don't think it is as straight forward as Brian suggested (positive/negative metrics). If an extension will not ship with 1.0, but is usable and downloadable from somewhere with 1.0, I think it is a positive thing (think of all the Mozilla extensions).

Also, I could easily imagine someone getting involved with Chandler on something that will not make it into 1.0 but would make it into 2.0. In the long run that is clearly positive, but short term distracting (if we did not support it in the short term, the long term gain would probably never happen).

Still, having said that, OSAF's number one priority is shipping 1.0 in good time.

-- HeikkiToivonen - 28 Jan 2004


Just FYI -- you might want to look at this TWiki ChartPlugin?.

  • BrianDouglasSkinner - 26 Jan 2004


Release download numbers as of xx/xx/xxxx:
Release Windows Linux Mac OS X Total
0.3 Release xxx xxx xxx xxx

We might also want to consider debug versions and source tarballs.

-- HeikkiToivonen - 02 Mar 2004

(DONE Done. -- DuckySherwood)


To get a count of patches in bugzilla try this:

  select profiles.login_name AS email, count(a.bug_id) AS bugcount FROM profiles, attachments a 
  WHERE a.submitter_id=profiles.userid and a.ispatch = 1 GROUP BY email ORDER BY bugcount;
  

-- MikeT - 11 Feb 2004


I think Brian is on the right track, but I would redefine the main goals; I think goal #1 in particular is too narrow. I would rewrite it as something like "Making Chandler a long-term succesful offering" and then define success in terms of quality, adoption, functionality, and the other things OSAF discussed in its vision documents. Chandler 1.0 is only the beginning. We know that 1.0 won't be aversion aimed at institutions. We also know that 1.0 might be a version for early individual adopters but not necessarily the mass consumer market. So defining community success in terms of 1.0 seems too narrow, since the success of the Chandler project itself won't be evaluated this way. Mitchell


I just realized Bonsai can also provide some metrics since the tree was last opened (currently that is a LONG period for us):

-- HeikkiToivonen - 31 Mar 2004


PageInfo
PageType ProjectPage?
MaintainedBy DuckySherwood
PageStatus Proposal -- available for review?
Trash.CommentsWelcome2 Feel free to contribute comments?, either by adding to the Comments Welcome section of this page, or by posting to the dev list, or by sending mail directly to the person listed as maintaining the page.
Edit | WYSIWYG | Attach | Printable | Raw View | Backlinks: Web, All Webs | History: r96 < r95 < r94 < r93 < r92 | More topic actions
 
Open Source Applications Foundation
Except where otherwise noted, this site and its content are licensed by OSAF under an Creative Commons License, Attribution Only 3.0.
See list of page contributors for attributions.