4
$\begingroup$

Could we use the data dump that stack exchange publishes for theoretical computer science and create an ontology out of it? The data is heavily structured and the things I can think of what we could do with it.

  • Create robots that aid in question answering
  • Machine learn what kind of questions are asks and who asks them
  • Learn more about theoretical computer science sociologically

The things that help us are as follows

  • Everything is ordered and tagged by humans
  • It will be constantly changed and updated

The problems with it are as follows

  • Knowledge engineering these types of questions is hard by definition (some questions have no answer or can't be answered!)

The strategy I thought of would be to go for the "easy" questions first (questions tagged as off topic) and work from there.

$\endgroup$
3
  • 1
    $\begingroup$ Well of course the answer is YES ! Go for it if you want to. I think you have access to the data dump. $\endgroup$
    – Suresh Venkat
    Jun 14, 2012 at 2:45
  • 2
    $\begingroup$ you may also want to check MO's data dump. Anton regularly posts the links to them in a meta.MO post. $\endgroup$
    – Kaveh
    Jun 14, 2012 at 4:57
  • $\begingroup$ I'm going to try to do a classification and then make a cstheory blog post about it. $\endgroup$
    – Joshua Herman
    Jun 26, 2012 at 16:35

2 Answers 2

Reset to default
5
$\begingroup$

Some questions that you could ask with the data dumps:

  • Activity levels: typically, when does a question get its first answer and when does an answer get accepted ?
  • If a question is to be closed, how soon this does happen ? Is this via moderator, or via 5 close votes ?
  • If you define "neighborhoods" for users via commenting on the same question, or the same tagged questions, are there local communities that have formed ?
  • how many "polymaths" are there who have upvoted answers on questions in many different topics ? Conversely, how many people are specialists ?
  • If you have information on user visits, how often does the average user visit the site (this could be inferred indirectly from postings of course).

I could go on for a while here :). Also, for each of the above questions, there may be global answers, or significantly different answers when grouped by tag or tag collection

$\endgroup$
3
  • $\begingroup$ Nice questions. For the first two (and perhaps the fourth), I guess that the Data Explorer website is sufficient. But for neighborhood relations of users and more advanced things, Data Explorer is probably too limited. $\endgroup$
    – Tsuyoshi Ito
    Jun 15, 2012 at 0:48
  • $\begingroup$ At some point. Stackexchange was offering money for research related to the SE sites. I wonder if they are still doing this. $\endgroup$
    – Suresh Venkat
    Jun 15, 2012 at 0:59
  • $\begingroup$ @TsuyoshiIto I downloaded the data dump. I am going to filter it for cstheory. If I got money that would be nice for hosting but I don't think I require anything yet. $\endgroup$
    – Joshua Herman
    Jun 16, 2012 at 11:15
1
$\begingroup$

I am not sure what you are suggesting here. If you want to build a system like that, then I agree with Suresh: go for it, data is available. If you want someone to build it, then my guess is that things like this will not happen unless you already have something to start with or you are extremely lucky.

Note that most closed questions have no answers and zero or negative score, and such questions are subject to automatic deletion. This means that closed questions in the data dump do not probably reflect typical closed questions, unless you take the union of multiple data dumps.

$\endgroup$
5
  • $\begingroup$ I'm just asking for advice here really on how to approach the problem. $\endgroup$
    – Joshua Herman
    Jun 14, 2012 at 15:06
  • $\begingroup$ @Joshua Herman: I gave you advice on how to approach the problem, even though you did not ask for it in OP, didn’t I? $\endgroup$
    – Tsuyoshi Ito
    Jun 14, 2012 at 15:21
  • $\begingroup$ Sorry I must have misread your comment. $\endgroup$
    – Joshua Herman
    Jun 14, 2012 at 16:21
  • $\begingroup$ I think I will do a classification of off topic questions and on topic questions. $\endgroup$
    – Joshua Herman
    Jun 19, 2012 at 0:53
  • $\begingroup$ @Joshua Herman: Good luck! $\endgroup$
    – Tsuyoshi Ito
    Jun 19, 2012 at 1:59

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .