17
$\begingroup$

We're currently two-thirds of the way through the beta period. As I understand it, the site will get shut down at the end of that period if it's not considered successful. This blog post says:

We’ll set strict criteria (number of new questions per day, number of registered users, percentage of answered questions, number of people who vote, etc.) to define a site that we consider to be successful.

and this one says

We’re looking at lots and lots of metrics, but the most important ones are people and questions.

People: Do we have a lot of people visiting the site? Are a lot of people signing up? How many people are answering questions? How many page views does the site generate?

Questions: Are questions getting answered? Are they answered well? Are they answered quickly? Are a lot of answers accepted, indicating that the person who asked them was satisfied? Are a lot of answers upvoted, indicating that some third party thought they were quality answers?

Does anyone know the cutoff for success? Anyone (e.g. moderators) know what the stack exchange overlords think of the site?

The proposal site lists a few metrics along with the numbers they consider "healthy" and "worrying." Currently we're listed as "worrying" for number of questions and only "okay" for number of users and visits per day. You can also look at how we compare with other beta sites: we're currently way behind on number of questions but doing comparatively well in some other areas.

My guess would be that we should try to avoid being "worrying" in any category at the end of the beta period in 30 days. Any ideas of how to boost numbers?

$\endgroup$
5
  • 2
    $\begingroup$ It might (or might not) be useful to see what will happen to the beta sites which are below the threshold on some metrics. Home Improvement, 88 days in the public beta, has 598 questions (on pace to get 611 (<750) questions after 90 days), 170 (<500) visits per day. GIS, 87 days in the public beta, has 650 questions (on pace to get 672 (<750) questions after 90 days), 70 users with ≥200 rep points (on pace to have 72 (<75) users after 90 days). $\endgroup$ Oct 18, 2010 at 1:48
  • 2
    $\begingroup$ Our community leaders (in terms of reputation) were very active in terms of providing answers. However, The number of questions asked by some of them is far below the number of answers they generously provided. I hope our leaders (especially the well known experts) continue their generosity by asking more good questions since our weakest metric is the number of questions. No one wants our collective efforts and contributions to go in vain. $\endgroup$ Oct 18, 2010 at 5:40
  • 3
    $\begingroup$ OK, OK, y'all have shamed me enough. :) I will try to post more questions. In fact I had some more questions in the early stages of the beta period but unfortunately forgot them... $\endgroup$ Oct 21, 2010 at 1:20
  • 3
    $\begingroup$ Highly relevant discussion for another SX site: meta.stackexchange.com/questions/68079/… (summary: if the questions and answers are good, the site will not be deleted, but the beta period might be extended) $\endgroup$ Oct 22, 2010 at 11:05
  • 3
    $\begingroup$ Also this blog post, specially why we need users with higher reputation (over 3000) after the public beta period. $\endgroup$
    – Kaveh
    Oct 22, 2010 at 17:06

10 Answers 10

13
$\begingroup$

This is a comment about the number of questions.

Personally I now feel less motivated to ask questions. In my experience good questions take more effort to generate than good answers, yet garner half the reputation per upvote. I find it relatively easy to generate a question, but require nontrivial effort to turn this into a good question, including filtering out the questions of lesser quality.

Further, the questions that I believe are likely to further research (as opposed to being interesting but less productive) tend to be quite specific. Such questions receive little attention overall, and even less when someone posts a precise answer that is then quickly accepted. The number of questions per user is definitely quite low, and I detect a tendency toward shorter questions that are less likely to be research-level (this is just a perception and may be incorrect). The site metrics essentially encourage such questions, which has implications for the long term character of the site.

I don't strongly care about my personal site reputation, but "reputation points" form the main mechanism used by the site to direct activity in certain ways. The mechanics and metrics of the site currently direct us toward posting simpler questions that generate multiple quick, piecemeal answers. I find it irritating to see my quick off-the-cuff responses sometimes garner many reputation points while posts that require substantial insight (and effort) are ignored by nearly everyone.

I have said this before, so I will now shut up about it: MO awards 10 points per question upvote, but we seem to be stuck with the StackExchange model where questions are expected to be fairly straightforward. Oh, well.

$\endgroup$
7
  • 2
    $\begingroup$ I agree with you: asking good questions here is very difficult. But regarding the reputation points: since asking good questions is difficult, we should be very generous in rewarding good questions by upvoting them! $\endgroup$ Oct 21, 2010 at 13:46
  • 2
    $\begingroup$ By the way, we have currently 1860 users and as many as 97 active users with 200+ reputation. But then questions like cstheory.stackexchange.com/questions/1346/… have got only 27 upvotes. 27! That is, we have at least 70 active users who didn't think that this question is "useful and clear"? This is a truly amazing question that really demonstrated the strengths of this site, what else do you need?-) $\endgroup$ Oct 21, 2010 at 13:53
  • $\begingroup$ @Jukka: there is a gold badge cstheory.stackexchange.com/badges/28/electorate which is awarded for voting on 600 questions as long as at least 25% of one's votes are for questions. I suppose it is a subtle nudge in a specific direction (only vote for really good answers, vote for all good questions), but since we only recently hit 600 questions this is going to be quite rare! $\endgroup$ Oct 21, 2010 at 16:26
  • $\begingroup$ @Jukka: agreed about David's question, it's the one I nominated for outreach. The standards applied to questions appear to be really high here -- when even quite fuzzy questions are crossposted to MO they receive more upvotes there than here, and perhaps even more supportive comments. $\endgroup$ Oct 21, 2010 at 16:29
  • 1
    $\begingroup$ @András Salamon: I think part of the reason for questions receiving more votes in MO is a result of having much more active users. $\endgroup$
    – Kaveh
    Oct 21, 2010 at 18:49
  • 2
    $\begingroup$ I agree that the reputation system has a tendency to encourage popular questions/answers rather than well-thought-out questions/answers. The reputation system is silly, but I guess that we have to live with it because it is difficult to come up with a better system. Let’s hope that we know better than letting the reputation system ruin the whole website. $\endgroup$ Oct 21, 2010 at 21:29
  • $\begingroup$ Garnering lots of reputation for simpler answers is something that's happened to me on MO, too. I think you just have to keep in mind that there are a lot of things more important than reputation. $\endgroup$ Oct 22, 2010 at 1:24
10
$\begingroup$

There are three specific things I had in mind, in increasing order of commitment and difficulty, that I think would get us on a firm footing.

  1. Do the article for SIGACT News/EATCS news that I mentioned.
  2. SIGACT News curently has an opening for an open problems column/editor. We should suggest that in each issue we have a 'featured on cstheory' column with a few key questions/answers, and maybe some open questions.
  3. (and this is the most significant) see if we can get SIGACT to take over this site to institutionalize it.

Let me explain the last point. There's an ubuntu SE site that was created around the same time as this one. It's already been promoted to a full site, because Canonical (the folks that make ubuntu) took it over and are using it as a replacement of sorts for their forums. I contacted the SE admins about this, and they are more than willing to do the same thing (go into partnership with another organization) for us, if we can work out the logistics.

I was going to either email Lance (SIGACT chair) or see if he'd be at FOCS and bring up the idea. The logistics of doing this is not entirely clear, but it sounds like the main added value is branding, and that SE would still maintain the software base etc. I have a contact at SE to liaise with on this front if we wanted to go forward with this.

While I don't think SIGACT will miraculously make things better in terms of participation, and I think we should continue doing what Kaveh and Ian are proposing (although I'm not so sure about Reddit/Slashdot), I think that getting the imprimatur of an organization means that

  • we could pick our own domain name (under sigact) maybe
  • we wouldn't need to constantly worry whether SE would pull the plug (I think)
  • We get the kind of centralized "conference room" that Lance has also been talking about

Now I'll be honest. I have enough work with my regular job that I can't focus a lot of time on all these things. But there are enough people here on meta that I think together we can achieve a fair amount of this, especially the tasks that Kaveh suggests. I can volunteer to talk with SIGACT about the first few points above, with the idea of eventually broaching the third.

p.s as an aside, I believe that in the early days of MO, the participants did do a lot of explicit solicitations to get people involved in the site. I think if each of us could get 3-4 new people to start contributing, we'd do well.

Update from FOCS: I have informal blessing to do all three of the things I mentioned above, btw. I'll start talking with SE.

$\endgroup$
4
  • 7
    $\begingroup$ I like the idea of having SIGACT backing. It could give us a lot of immunity from any decisions or policies made by SE that we don't like. $\endgroup$
    – Ian
    Oct 18, 2010 at 2:28
  • $\begingroup$ Excellent news! :) $\endgroup$
    – Kaveh
    Oct 24, 2010 at 22:31
  • $\begingroup$ I like 1 and 2 but feel 3 might be too much. Given that the ASL, the programming languages community, and several other bodies are "theoretical computer science" but do not fall within the current ambit of SIGACT, "ownership" by SIGACT might reinforce these divisions? $\endgroup$ Oct 25, 2010 at 9:46
  • $\begingroup$ I'm happy to consider other communities, but it gets trickier to coordinate. the SIGACT is run by ACM, which is larger than just theory. $\endgroup$ Oct 25, 2010 at 9:50
7
$\begingroup$

good question. it seems to me that to get more questions we need to get more active involvement (one of the other metrics on which aren't doing that well is the number of active users - people with rep above 200). For that, more publicity in the larger research community might help. One of the more recent discussions involved a proposal to write an article for the SIGACT news and EATCS news (maybe jointly) advertising the site: as an example, Mathoverflow had such an article recently in the Notices of the AMS.

I'm also going ahead with publicity material for FOCS, and I'm hoping that someone who goes to LICS/ICALP might be able to do something similar there as well.

I've also been browsing the user lists, and while this is very subjective, it seems to me that a large fraction of the users signed on are not professional researchers (defined as someone in a Ph.D program in TCS, or a researcher in the area). While I welcome amateurs to this site, and think it's healthy to have this element of outreach, I suspect that amateurs are less likely to be supplying questions.

On the flip side, even an active user like myself has only asked 12 questions or so, which is pretty sad. Now we can't force questions (otherwise they won't be good ones), but if each of our 80 odd users with rep above 200 asked 10 questions each, we'd be doing a lot better than we are right now on that front.

$\endgroup$
6
$\begingroup$

On the positive side, I think the quality of questions and answers are really high. We have some involvement from some famous names in theory (for example Scott Aaronson, Eric Allender, Boaz Barak, Andrej Bauer, Paul Beame, David Eppstein, Jeff Erickson, Joe Fitzsimons, Lance Fortnow, Sariel Har-Peled, Russell Impagliazzo, Gil Kalai, Shiva Kintali, Pascal Koiran, Dana Moshkovitz, Noam Nisan, Mihai Pătraşcu, Luca Trevisan, Emanuele Viola, Noam Zeilberger, ...).

On the negative side, there are many others that are not using the site. (I find it a little bit ironic that mathematicians are faster than cs theorist in adopting new technologies.) Ideally we should have every active theory prof and every tcs graduate student in good universities actively use the site, post questions and answers. We should look more carefully on why this is not the case and address the issues that make people uninterested in using the site. We are also not getting reasonable involvement from researchers in areas outside algorithms, complexity, and computational geometry.

Theory B is very weak on the site, not enough crpto/privacy, combinatorics, and algorithmic game theory and many other very active research areas in our scope.

  • more involvement from community (profs, grad students)

  • more questions and answers

  • more active areas

I think we need a number of active core users who are experts in each area to keep those area active on the site.

Here are some suggestions:

  1. There are number of reasons that I have heard from people who I have talked with and tried to get interested in using the site. It might help if we collect those reasons and try to address them if possible.

  2. Moderators can contact some of the top experts that have used the site, and ask their opinion and for suggestions on making the site more interesting and useful.

  3. List a number of reasons why every theory prof/grad student should use the site: "Why I should check cstheory? Because ..."

  4. Contact a number of researchers in areas that are not active on the site, and tell them that their area is not active on the site and ask them to help us in getting more questions and answers in those areas.

  5. We can also hold special weeks where every user is encouraged to ask a question (even very basic ones from people unfamiliar with those areas, you have attended a talk in some area and you have a very basic question you haven't asked) in a specific area, if we get enough question in those areas, they may generate enough interest to make those areas active. We can also directly invite some of the experts in those areas for the week assigned to each area.

(Most of these are already suggested by others in other questions.)

$\endgroup$
7
  • 1
    $\begingroup$ Many people have probably heard of this site by way of blogs: Lance Fortnow, Scott Aaronson, Noam Nisan, and of course Suresh have all mentioned the site on their blogs (possibly some others I've forgotten). Not surprisingly, the areas those researchers cover are among the better represented ones. If we can get bloggers in poorly-represented areas like crypto to promote the site we might have better coverage in those areas. $\endgroup$
    – Ian
    Oct 17, 2010 at 6:44
  • 1
    $\begingroup$ Jonathan Katz at Maryland might be one person to contact. $\endgroup$ Oct 17, 2010 at 8:27
  • $\begingroup$ Hmm, among this list of "big names", Suresh should also be mentioned :) $\endgroup$
    – arnab
    Oct 17, 2010 at 15:45
  • $\begingroup$ @arnab: You are right, I didn't include our moderators. :) $\endgroup$
    – Kaveh
    Oct 17, 2010 at 17:06
  • 1
    $\begingroup$ "(I find it a little bit ironic that mathematicians are faster than cs theorist in adopting new technologies.)" Really, you're surprised? Computer scientists are hard-core luddites! $\endgroup$
    – Jeffε
    Oct 19, 2010 at 15:43
  • $\begingroup$ I wouldn't say that mathematicians are faster, but that there are a lot more of them. This site only attracts a small fraction of computer science researchers, but mathoverflow.com attracts all mathematicians. $\endgroup$ Oct 21, 2010 at 16:56
  • $\begingroup$ JeffE, "hard core luddites" huh? I gotta remember that one. kaveh-- would be interested in that [1] list of reasons why people dont want to use the site. is that what you are referring to? (its phrased ambiguously) also like [4] and [5], figuring out ways of encouraging questions, any new thinking or progress on that? $\endgroup$
    – vzn
    Sep 13, 2012 at 3:14
4
$\begingroup$

This link may include relevant pieces of information:

http://area51.stackexchange.com/proposals/8766/theoretical-computer-science.

It gives statistics on what is going on, and what they expect from us.

In particular, we are excellent in:

  • Ratio of questions answered;
  • Ratio of answers per question.

Yet we are OK (on the edge of being "worrying") in:

  • # of questions;
  • # of total & avid users;
  • # of visits per day.

I recommend the following best practices (though I make this CW, so as others can edit it appropriately):

Asking more questions

We have (as of now) about 100 users with 200+ reputation (which I assume are active), and about 650 questions. We are 70 days into beta, with 20 days remaining. To reach the 1500 questions limit, every active user should ask 8-9 questions within the next 20 days. Seems impossible, though we should try as much as we can.

NOT asking too narrow questions

While trying to ask more questions, we must consider the ratio of users who can possibly answer it. If we ask questions that are too narrow, we damage our "ratio of questions answered". In addition, let enough time before accepting an answer. See if more users can answer the question (this strengthens the "ratio of answers per question").

Try to reach more vivid users

Let's see the statistics (as of now):

  • 150 users with 200+ rep (on pace for 137 users at 90 days)
  • 10 users with 2,000+ rep (on pace for 12 users at 90 days)
  • 5 users with 3,000+ rep (on pace for 2 users at 90 days)

So, we need more users with 200+ rep, and more users with 3000+ rep. Here, I focus on the latter. As of now, we have 5 users with rep in (2500, 3000). We need 3 of these users to try to earn (at most) 500 reps within the next 20 days. I insist that they all try their best!

$\endgroup$
1
  • 7
    $\begingroup$ I think we will reach the targets for numbers of active users quite easily, now that people seem to be voting more actively. In the race to meet arbitrary milestones (see the links I posted in comments above) we should not sacrifice the superb quality of the site built up so far. I would rather see an extended beta period than a site with many half-thought-through questions. $\endgroup$ Oct 26, 2010 at 11:36
3
$\begingroup$

Not sure how other people feel about this, but it might worth trying to promote the site on places like Slashdot, Reddit, Hacker News, Digg, etc. I imagine some people might disagree since most of the people it would attract would be amateurs rather than researchers. However, some of them will be interested and knowledgeable, and sites like those can reach far more people than we ever could on academic blogs and at conferences. The only links to here I can find on any of those sites are on Reddit to "Algorithms from the Book" and "What's new in purely functional data structures since Okasaki?", and those are the two highest voted questions on the site.

I can help with this promotion, but I don't usually read any of these sites, and someone who is more familiar with the culture of, say, Slashdot, could probably write a far more effective pitch than I could. For sites that rely on users to upvote submissions, it would also help if people mentioned their submission here in meta, so that others can then go vote it up.

$\endgroup$
2
  • 2
    $\begingroup$ I do not think that Slashdot, Digg, etc. are best places to find valuable contributors. There is some overlap with net culture (e.g. readers of xkcd), but the signal-to-noise ratio is very low. $\endgroup$
    – Jakob
    Oct 18, 2010 at 8:20
  • $\begingroup$ Since this idea has drawn several votes of skepticism (which I actually mostly agree with) and no support, I won't try to do any promotion on any of these sites. $\endgroup$
    – Ian
    Oct 19, 2010 at 21:11
3
$\begingroup$

This discussion reminds me on the early discussions in Wikipedia community in the early 2000s when the site was relatively unknown. When we had more press and new contributors, we first were very happy, but then there came the trolls and similar social phenomena. Today the Wikipedia-community is kind of harsh, there are hidden power structures etc. I do not know whether you can avoid it, but I think that slow but constant growth is much more healthy for a community. So please do not try to quickly meet some arbitrary metrics, but try to spread the word target-oriented. If decisions in social systems (like this forum) are based on metrics only, instead of arguments, it is too late anyway, and you should better leave the place.

$\endgroup$
2
$\begingroup$

Currently, it doesn't look bad at all. According to the metric overview people get 'excellent' answers to their questions. The site will also be at least close to reach the recommended number of users with a certain reputation.

The number of visits/day is also decent. If you have a look at the visits/day number shown on the main page, the number is even better (2,983 vs. 1,326). So one of them seems wrong (or they, confusingly, measure different things). Can anybody clarify this?

Only the number of questions is a bit on the low side but I really don't think it is worth worrying about it too much.

$\endgroup$
5
  • $\begingroup$ the discrepancy comes from the fact that our site uses a different mechanism to the main site (which uses google analytics). $\endgroup$ Oct 24, 2010 at 17:37
  • $\begingroup$ We need more question in more areas and more active users (+200) independent of what SE thinks/says. We also need more people voting since we need more users with +3000 reputation for managing the site. $\endgroup$
    – Kaveh
    Oct 24, 2010 at 22:40
  • $\begingroup$ Currently we have only two: Suresh and Ryan (who are already moderators), we need more high-reputation users to help them in managing the site. I think David Eppstein will make it, and hopefully Sadeq Dousti, Noam Nisan, Joshua Grochow will also get over 3000 in the next three weeks. $\endgroup$
    – Kaveh
    Oct 24, 2010 at 22:41
  • $\begingroup$ But the original post was exactly about what SE thinks. Of course it cannot hurt to have more of everything. The most important thing however is that people get good answers to the questions they have (what else should be the point of a Q&A site) and that appears to happen. Then more people may actually be tempted to stick around and ask questions. Also, as you point out, some people will pass the 3000 rep mark in the next weeks. While I think that e.g. Suresh's FOCS initiative is great, I believe that the worrying tone, of some people, is understandable but unnecessary. Stay positive. $\endgroup$
    – Matthias
    Oct 25, 2010 at 8:30
  • $\begingroup$ A couple things have changed since I posted this question. First, our numbers have improved a bit, and second, SE has indicated (blog.stackoverflow.com/2010/10/when-will-my-site-graduate) that they're going to be much more lenient with beta sites than I had thought when I read the earlier blog posts mentioning "strict criteria" for being considered successful. I no longer think we have anything to worry about. $\endgroup$
    – Ian
    Oct 26, 2010 at 4:29
2
$\begingroup$

Thank you for bringing up this issue here.

Honestly speaking, I am not interested in the numbers very much.

People can focus on the numbers if they want. However, in my opinion, the numbers are of secondary importance. We should just be a good community, and any promotion should aim at building a better community, not at increasing the number of users/questions/answers/etc.

The website is fun and useful, but not absolutely necessary. If Stack Overflow continues to run the site after the 90-day beta period, it is fine. If it shuts it down, that’s the company’s choice, and it is not a big deal. After all, I may not be very enthusiastic about this website.

Added: Just to clarify, I do not mind seeing more users/questions/answers/etc. if it is a result of building a better community. I am just against considering the thresholds as our aim. If the company were stupid enough to shut down a website just because of low numbers, I would be happy to leave the website run by that company. Fortunately, the company seems to know better than that.

$\endgroup$
4
  • 7
    $\begingroup$ I'm a little surprised (and disappointed). Especially given the time you've spent on this site, I don't understand why you'd be fine with it disappearing. While we can't force people to use the site if it doesn't work for them, my interest in looking at metrics and other measures is merely to determine whether we're doing the best we can to create a viable place for theoreticians to exchange ideas, and if we can do more. If we do the best we can, and it still fails, then I can live with that. But I wouldn't want to feel that we failed because of lack of effort. $\endgroup$ Oct 17, 2010 at 19:46
  • $\begingroup$ Dear @Suresh: Sorry for disappointing you, and just to make sure, I appreciate your hard work put into this website. However, targeting the metric seems dangerous to me. I am all for making the website better (although I have done little about it), but I am really unsure if the numbers reflect the quality of the website correctly. (more) $\endgroup$ Oct 17, 2010 at 20:07
  • $\begingroup$ (cont’d) Moreover, the threshold values look arbitrary to me, and who knows whether the company will not change them in 30 days? (You can see that the company lost part of my trust by changing the domain-name policy.) I do not feel like working hard to meet their own seemingly-arbitrary criteria. If the company shuts down the site, I am sure that it will make me sad, but that is not the end of the world. $\endgroup$ Oct 17, 2010 at 20:07
  • $\begingroup$ Right - you should not forget that there are many other institutions with similar goals (Lambda the Ultimate, Complexity Zoo, personal blogs, Wikipedia etc., and last but least the whole academia with university courses, conferences, journals etc.). I am sure that cstheory nicely fills some gap here but it is not the only place. By the way the best way to grow is to collaborate with existing institutions as those just mentioned, and growth is not a sign of quality as such. $\endgroup$
    – Jakob
    Oct 18, 2010 at 8:39
1
$\begingroup$

I agree with Tsuyoshi Ito's response (except for his very last point). Any metrics that the Stack Exchange folks might have for general-interest sites are not going to be very applicable for something like cstheory, which has an inherently small base. While it makes sense to advertise the site within the theory community and try to encourage participation, I doubt that cstheory will ever be comfortably above all of the metric thresholds and I don't think it's worth worrying about.

I would be surprised if the Stack Exchange admin decided to pull the plug on cstheory, but so what if they do? The site infrastructure is not exactly rocket science. I'm sure someone could scrape together a little grant money and pay a few students to rebuild it from scratch. I think you've already proven that the site is a worthwhile asset to the community and it would be worth the effort to keep it going. The main downside I see is that the already accumulated questions, answers, and comments could be lost. It might be worth talking to the Stack Exchange management and seeing if you can get a dump of the cstheory database entries in the event that they do decide to shut it down.

$\endgroup$
4
  • 3
    $\begingroup$ I think that's the problem. retaining all the questions and answers accumulated thus far. Hence my suggestion about getting SIGACT involved, which would allow us to eliminate any focus on metrics. $\endgroup$ Oct 18, 2010 at 2:26
  • 2
    $\begingroup$ There are already at least 2 open source SE clones-- OSQA (osqa.net) and shapado (gitorious.org/shapado), and MO seems to have contemplated moving to OSQA. I don't think either of these platforms is very mature, but it would be an option. Nevertheless, I don't think we should be cavalier about the possibility of having to migrate the site. $\endgroup$
    – Ian
    Oct 18, 2010 at 2:54
  • 3
    $\begingroup$ i spent some time exploring OSQA as an option while we appeared to be stuck in commit hell in area51. it's definitely a viable option, and the OSQA folks are quite happy to import data from SE if we can get it out. But moving things around is hard, especially when people are slowly getting used to this site $\endgroup$ Oct 18, 2010 at 4:13
  • $\begingroup$ OSQA looks elegant: Python built on top of Django, clean modular structure, should be easy to extend. Now we just need an SE export function -- I suspect the time tags required to reconstruct the journal are the trickiest to extract without access to the database itself. (I'm not proposing we do this, but it is always nice to know one is not locked into a proprietary platform.) $\endgroup$ Oct 26, 2010 at 11:51

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .