Back to all discussions

  •   almost 6 years ago

Judging Criteria

Hello. I really enjoyed looking at the winner projects of the hackathon. They are all very good. However, I have a question regarding the judging criteria. Since my bot is hooked on a Facebook page and I am an admin of the page, I can see the inbox of the page. Up until the winners were announced, I have not seen a judge interacting with my chatbot. I just want to confirm how the judge interact with the chatbot so that I can improve in future hackathons. Thanks!

  • 8 comments

  •   •   almost 6 years ago

    From the rules section:
    "C. Submission Review:Judges are not required to test the Application and may choose to judge based solely on the text description, images and video provided in the Submission."

    For the records: I did not see any interaction either on my bot... ;-)

  •   •   almost 6 years ago

    Thank you for the response!

  •   •   almost 6 years ago

    EDIT (8/16): I actually do see that there were logins to my slack setup and that my bot was used.

  •   •   almost 6 years ago

    I'd be curious to see the scoring, at least for my own submission. The lack of any interaction or feedback makes me wonder if it could have been missed. I'm also wondering why it says there are around 40 additional submissions, when it was at something like 150 when the hackathon closed, and yet now it shows 180 something.

  •   •   almost 6 years ago

    If the winner does not even have any interaction with the judge, that makes me doubt the judging process again.... It seems as though if the pitch is good, you will have more chance to win. The technical aspect is unfortunately ignored....

  •   •   almost 6 years ago

    Matt, there were actually qualified 179 submissions and the number was 182 before the judging period started. 3 additional entries were entertained as the copyright related issues were sorted out and AWS did provide insights on that in the slack channel. However, since the results have been announced, I see that number at 198. Maybe a software bug?

    I do agree with what has been discussed on this topic so far. There was a discussion on slack channel during the final week of judging when many contests observed that the bots have not received any traffic. Even though there is a judging criteria mentioned on the home page of this challenge as well as the panel of six judges, based on the limited facts that we have as contestants, it is now evident that the only criteria that seems to have been applied alludes to this fineprint:

    "C. Submission Review:Judges are not required to test the Application and may choose to judge based solely on the text description, images and video provided in the Submission."

    Just look at the number of submissions. To be able to look at each entry based on this sole criteria, it would take approx. 5 minutes to watch video, another 5 minutes to read the description and another 5 minutes to grasp the value add the application brings. This means 180 times 15 minutes, that is almost 45 hours of effort from each judge. Then add notes taking and comparing. That is a lot of time commitment. I am sure they have a busy schedule just like us, maybe busier.

    So unless AWS provides some transparency in terms of what they actually did and how the solutions were compared, there would only be speculations and theories as to what actually happened.

    Twitch session to discuss winning chatbots, that might have given some insights, was cancelled at the last moment for some inexplicable reason and that might create further doubts.

    I hope AWS realizes the effort and time that has been put by all the contestants into their projects and at least out of a goodwill to reciprocate the same and create a vibrant community around Lex, provides feedback as to what the contestants could have done better in their projects and what was it that was unique about some of the projects that might have made the judges scratch their head.

  •   •   almost 6 years ago

    We also did not see any traffic to our bot, so assumed the judging would be based on the other materials provided.

    Developing our bot was a great experience and allowed us to explore various AWS technologies, however, it'd be great if we could also get some feedback. Of course, I do understand if it's not possible.

  •   •   almost 6 years ago

    Could you kindly share the feedback for the ( Edubot - The Smart Catalyst to pursue your dreams ) - AWS Chat Bot Challenge for us to improve the product.

Comments are closed.