Website data

Discussion in 'Ask the Admin' started by shmmeee, Oct 24, 2017.

  1. shmmeee

    shmmeee Well-Known Member

    Joined:
    Jul 11, 2011
    Messages:
    13,318
    Likes Received:
    9,370
    Location:
    Coventry, United Kingdom, United Kingdom
    Nick:

    I was thinking about playing around with some data from the site, like post sentiment or like graphs. Just to see if I can and because I thought the results would be fun (who likes who, who is the happiest and grumpiest poster).

    Do you have access to that, or if not would there be an issue with me writing a scraper to get it, or would that hurt your hosting costs/run up against anti-bot measures?

    No promises on output, I might get bored and wander off halfway through. But if you’re OK with it I’ll have a crack.
     
    Otis likes this.
  2. Nick

    Nick Administrator

    Joined:
    Feb 25, 2008
    Messages:
    74,268
    Likes Received:
    23,602
    Location:
    Coventry
    I can try :)

    What sort of data do you need? Send me a PM and happy to try and find things.

    Depending on the scraper it might get blocked by security if it looks dodgy and hammers the site :)
     
  3. shmmeee

    shmmeee Well-Known Member

    Joined:
    Jul 11, 2011
    Messages:
    13,318
    Likes Received:
    9,370
    Location:
    Coventry, United Kingdom, United Kingdom
    Bump.

    Thought I’d have a go at working out a league table of likes per post just for fun. I was gonna just scrape the data but the members page only seems to show the top twenty or so for likes and posts. Is there a full member list somewhere that has how many posts and how many likes each user has?
     
  4. Nick

    Nick Administrator

    Joined:
    Feb 25, 2008
    Messages:
    74,268
    Likes Received:
    23,602
    Location:
    Coventry
    What format do you want? If it's just username, number of posts and number of likes I could probably sort something.
     
  5. shmmeee

    shmmeee Well-Known Member

    Joined:
    Jul 11, 2011
    Messages:
    13,318
    Likes Received:
    9,370
    Location:
    Coventry, United Kingdom, United Kingdom
    Yeah a csv would be perfect.

    As that was so easy, is there any way to view back past 2016 for who gave you a like? Then I could do who likes who as well.
     
  6. Nick

    Nick Administrator

    Joined:
    Feb 25, 2008
    Messages:
    74,268
    Likes Received:
    23,602
    Location:
    Coventry
    Dont think that would be as simple as Id need to query every post to see who it was who liked it and then tally it all up.

    Who works with CSV any more? It's all JSON nowadays ;)
     
  7. shmmeee

    shmmeee Well-Known Member

    Joined:
    Jul 11, 2011
    Messages:
    13,318
    Likes Received:
    9,370
    Location:
    Coventry, United Kingdom, United Kingdom
    You can give it as json if you like. As long as it’s not XML, I’m working with that at work as we speak and I am not a fan.
     
  8. Nick

    Nick Administrator

    Joined:
    Feb 25, 2008
    Messages:
    74,268
    Likes Received:
    23,602
    Location:
    Coventry
    Will have a look when I get chance, need an hour or so to get the query right.
     

Share This Page