thebitterroost

Idea for improving judging speed.

Recommended Posts

Hi everyone. I'm a newb here, but hopefully that won't detract too much from an idea as to how to potentially improve the speed of track judgment/posting without sacrificing quality. I searched for a thread that was about this topic, but only saw it addressed as a sort of sub-topic in this thread. Hopefully not beating a dead horse here.

Anyway, I have an idea that may be useful for both the community and especially the judges, who seem to have a rather large workload to deal with. Considering how much they type out for each review, and how much time it takes for me to even compose a post like this one, it's not a task I envy. Given that the purpose of the judging system is primarily to set a minimum bar of quality, I feel like a large part of this task could be distributed to a larger part of the community. My idea is a two-stage approach, with the initial stage being something along these lines:

  • Create a user category of "sub-judges" (or whatever title would work) comprised of volunteers whose basic judging credentials are based on having some number (1-4?) of already accepted remixes.
  • Create a standard judging form that resembles the checklist from this forum to reduce the level of time commitment from sub-judges (with space for freeform commentary at the end). The aggregate of the forms can show a succinct basis of the reasons for the ultimate binary result of yes or no

The results of these sub-judgments could be used as either a simple reference for the actual judges, or could be used to automatically reject remixes where the ratio of "yes" votes falls below a certain percentage (say, 20-30% or so?) with a minimum of number of sub-judgment submissions (I dunno, 10? 20?). Hopefully the end result of a system like this would be to reduce the workload of the judges while keeping things fair and transparent.

I know this isn't a perfect solution by any means, and would create some extra work for the site designers/admin to create this little system, but hopefully not too much.

Thoughts? :|

 

Share this post


Link to post
Share on other sites

This is already in place via the Workshop evaluation system with the criteria you linked, not with % system, but qualitative feedback. That said, we're behind on THAT as well, with about 30 tracks to review there. Just need to keep on trucking. :-)

Share this post


Link to post
Share on other sites

Yeah the workshop is supposed to fill that role.  The recruiting process for workshop mods is even the same as those for judges so they are effectively another layer. Not every artist knows or uses the workshop though, so a lot of subs don't go through that.  One thing to consider is, burnout/real life happens to the guys in charge of that too so it can get clogged as well.  It all boils down to: it's a hard, volunteer, pay-less and sometimes thankless job, and having a large judge panel doesn't exactly help either, only having motivated people does, but motivation is hard to mantain for years and years (I have so much respect for Larry for that).

Not an easy problem to solve, if we want to mantain quality.

 

Edited by Sir_NutS

Share this post


Link to post
Share on other sites

Thanks for the clarifications, guys!
I read the "What is evaluation and when do I use it?" thread, but didn't get the notion that it had any effect on the actual judging process, time-wise or qualitative. I was also thrown off a bit by the "When you're ready to submit a remix to OC ReMix, but you're not sure if the remix fits the site's submission standards" qualifier. That made me think I was better off skipping the workshop arena, since I was confident in the quality of my work. Not trying to sound cocky or anything :oops:, it's just that I've recorded/mixed bands on a semi-professional basis for several years at this point and feel comfortable with that take. I guess I interpreted the purpose of the workshop to be more about soliciting help/feedback from the community at large.

Anyway, with that in mind, I suppose I'll hit up the workshop with the track I recently submitted. I definitely appreciate the level of work you guys put into that process, and to reiterate, I don't envy you for that. I hope you all can find a way to lighten that load, and apologies in advance for adding to it! :P

Share this post


Link to post
Share on other sites

The Workshop doesn't have a direct effect in the judging process in the sense that, we don't go look at the workshop thread and feedback for every single mix (I do it for some though).  But it does serve as an indirect help in the sense that every track that doesn't meet the bar and reaches the queue adds more time and work to the panel and thus slows the queue down.  When a track is approved by a workshop moderator, which again, go through the same interviews, tests etc. as judges, it has a very high chance of making it through the panel and makes evaluation quicker and easier.  It also saves time and headaches for the artists, as submitting a song, waiting X months only to get it rejected is not ideal, most artists would've moved on from that track already into new things, or even evolved beyond what they showed on that track and probably wouldn't be that keen on going back to it.  The workshop feedback is supposed to be more immediate to help you fix issues in a much shorter timeframe, but again, that's not always working as intended because the workshop staff can get burnout or life gets in the way and that queue gets clogged too.

Anyways, it doesn't hurt to show off your track in the workshop or even join the workshop discord channel and ask for a listen there, possibly get some feedback and have a better chance at making it past the bar, especially if you're new to creating music or don't have a posted remix yet.  Even though I should have a good idea of where the bar is, I always try to get a friend to listen to it and tell me if it's good enough or check for issues.

Share this post


Link to post
Share on other sites

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.

Guest
Reply to this topic...

×   Pasted as rich text.   Paste as plain text instead

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.