Wiki: Difference between revisions of "Judges Panel"

(→‎Are the judges biased?: added list of bias factors avoided)
m
(24 intermediate revisions by the same user not shown)
Line 1: Line 1:
:''For information on the evaluation criteria used at OC ReMix, please see [[Submission Standards and Instructions]].''
+
:''For information on the evaluation criteria used at OC ReMix, please see '''[[Submission Standards and Instructions]]'''.''
[[image:Judges_panel.jpg|250px|right|Big Giant Circles, Jillian Aversa, zircon & Liontamer hard at work, as djpretzel looks on]]
+
[[image:Judges_panel.jpg|250px|right|Big Giant Circles, Jillian Aversa, zircon & Liontamer listen in as djpretzel ultimately casts the tiebreaking vote in favor of GSlicer's Legend of Zelda: Majora's Mask OC ReMix, "Inverting the Tower Temple"]]
 
== Who are they and what do they do? ==
 
== Who are they and what do they do? ==
  
Line 6: Line 6:
 
:''For panelists throughout OCR's history, please see [[Talk:Judges Panel|List of current & former judges]].''
 
:''For panelists throughout OCR's history, please see [[Talk:Judges Panel|List of current & former judges]].''
  
OverClocked ReMix began in late 1999 with djpretzel evaluating all submissions personally. As the site grew, the job of evaluating submissions became too large for one person. In early 2002, after a community poll of ReMixers chose the most acceptable solution, djpretzel established the judges panel to decide (with his oversight) which submissions best fulfill the vision for OCR. The judges were initially selected and invited by djpretzel, but since then have been added or removed based on panel consensus, with djpretzel's input.
+
OverClocked ReMix began in late 1999 with founder djpretzel evaluating all submissions personally. As the site grew, the job of evaluating submissions became too large for one person. In early 2002, after a community poll of ReMixers chose the most acceptable solution, djpretzel established the judges panel to decide (with his oversight) which submissions best fulfill the vision for OCR. The judges were initially chosen and invited by djpretzel, but since then have been added or removed based on panel consensus, with djpretzel's input.
  
Before a submission is sent to the judges panel for evaluation, it goes through an initial review by djpretzel or a designated member of the judges panel. As a result of this review, submissions are either [[Direct Rejection Letter|rejected outright]], forwarded to the panel for consideration, or accepted without further evaluation.
+
Before a submission is sent to the judges panel for evaluation, it goes through an initial review by djpretzel or a designated member of the judges panel. As a result of this review, submissions are either [[Direct Rejection Letter|rejected outright]], [[Panel Evaluation Letter|forwarded to the panel for consideration]], or [[Direct Acceptance Letter|accepted without further evaluation]].
  
Decisions on submissions forwarded to the panel are reached by a majority of a panel quorum; a vote will be closed after three negative votes or four positive votes. Any decision can be further extended to a majority vote of five at the request of a judge. In addition to his power to veto any submission regardless of the panel's decision, djpretzel may also serve as a tiebreaker during particularly divisive decisions.
+
Decisions on submissions forwarded to the panel are reached by a majority of a panel quorum; a unanimous vote will be closed after three negative votes or four positive votes. A split decision will be closed after a difference of three votes (e.g. 5 YES's vs. 2 NO's), or extended to a majority vote at the request of a judge. In addition to his power to veto any submission regardless of the panel's decision, djpretzel may also serve as an expediting vote, a tiebreaker during particularly divisive decisions, as well as a guide on where the overall acceptance bar should be.
  
 
=== General requirements for consideration ===
 
=== General requirements for consideration ===
  
Newcomers often ask how they can become a judge; the judges panel does not have an open selection process. When a vacancy within the panel is created, it is typically not announced until after there is a consensus among the group as to who would make the best replacement candidates. When the panel decides a new judge will be necessary, candidates must demonstrate the following qualifications prior to or during an evaluation:
+
Community members often ask how they can become a judge; the judges panel does not have an open interview process. When a panel vacancy opens, it is typically not announced until after a group consensus regarding candidates. When the panel decides a new judge will be necessary, candidates must demonstrate the following qualifications prior to or during an evaluation:
  
 
*Musical knowledge (formal music theory knowledge and instrument/performance experience are not requirements);
 
*Musical knowledge (formal music theory knowledge and instrument/performance experience are not requirements);
*Extensive commitment to the community;
+
*A strong grasp of current [[Submission Standards and Instructions|OCR standards and guidelines]] as expressed in evaluation of ReMixes on the site, through submissions, and/or during the candidate evaluation process;
*A strong grasp of current OCR standards and guidelines as expressed in evaluation of ReMixes on the site, through submissions, and/or during the candidate evaluation process;
+
*Ability to clearly and constructively advise artists as to how they can refine and improve their material;
*Ability to clearly and constructively advise submitters as to how they can refine and improve their material;
 
 
*Strong communication skills;
 
*Strong communication skills;
 
*Openness to all musical styles/genres;
 
*Openness to all musical styles/genres;
 +
*Extensive commitment to the community;
 
*Ability to work proactively with the variety of artists and personalities in the community; and
 
*Ability to work proactively with the variety of artists and personalities in the community; and
*Time, availability and enthusiasm for the workload involved.
+
*Time, availability, and enthusiasm for the workload involved.
  
Many people who want to be judges or are considered as judges fall short in at least one of those areas. Because the selection process is not open, candidates who feel they may be qualified to become a judge are discouraged from lobbying judges to obtain a position on the panel.
+
Many people who want to be judges or are considered as judges fall short in at least one of those areas. Because the process of choosing judges is not open, candidates who feel they may be qualified to become a judge are discouraged from lobbying judges to obtain a position on the panel.
  
 
== What percentage of submissions are accepted? ==
 
== What percentage of submissions are accepted? ==
Line 31: Line 31:
 
The total acceptance rate of all music sent to OCR's Submissions Inbox ranges between 10-15%. The acceptance rate of music forwarded to the judges panel ranges between 15-20%.
 
The total acceptance rate of all music sent to OCR's Submissions Inbox ranges between 10-15%. The acceptance rate of music forwarded to the judges panel ranges between 15-20%.
  
After initial review, about 45% of all submissions are [[Form Rejection Letter|rejected with no further evaluation]], 50% are forwarded to the judges panel for further evaluation, and 5% are accepted with no further evaluation (also known as a 'direct post').
+
After initial review, about 45% of all submissions are [[Direct Rejection Letter|rejected with no further evaluation]], 50% are [[Panel Evaluation Letter|forwarded to the judges panel for further evaluation]], and 5% are [[Direct Acceptance Letter|accepted with no further evaluation]] (also known as a 'direct post').
  
A submission that is directly posted vs. posted after panel evaluation is neither better nor worse; often times submissions are sent to the panel out of unfamiliarity with the source material, a desire to provide feedback, or because a precedent is being set in some way.
+
A submission that is [[Direct Acceptance Letter|directly posted]] vs. [[Panel Acceptance Letter|posted after panel evaluation]] is neither better nor worse; oftentimes submissions are sent to the panel out of unfamiliarity with the source material, a desire to provide feedback, or because a precedent is being set in some way.
  
 
=== Why is the acceptance bar so high? ===
 
=== Why is the acceptance bar so high? ===
  
Since the panel's inception in early 2002, the bar for evaluating ReMixes has gradually but substantially risen. As the site's vision and standards became clearer and the number of submissions increased, this change was necessary to maintain a consistent posting schedule and to give each posted ReMix the time and attention it deserves.
+
Since the panel's inception in early 2002, the bar for evaluating ReMixes has gradually but substantially risen. As the site's vision and standards became clearer and the number of submissions increased, this change was necessary to maintain a consistent posting schedule and to give each posted ReMix the time and attention it deserves. Even with the high bar that's been set, OC ReMix is a hobbyist community, so the bar purposefully remains below a professional bar, especially on production.
  
 
=== Why are there so many rejections in a row? ===
 
=== Why are there so many rejections in a row? ===
  
Unlike rejections, the judges' decisions on each accepted ReMix are not made public (i.e. moved into the [http://www.ocremix.org/forums/forumdisplay.php?f=15 Judges Decisions] forum) until the ReMix is posted onto OCR's front page. As a result, there may be instances where the first 1-3 pages of decisions seem to all be rejections without a single acceptance.
+
Unlike rejections, the judges' decisions on each accepted ReMix are not made public (i.e. moved into the [https://ocremix.org/community/forum/11/ Judges Decisions] forum) until the ReMix is posted onto OCR's front page. As a result, there may be instances where the first 1-3 pages of decisions seem to all be rejections without a single acceptance.
 +
 
 +
=== My submission was rejected. What if I strongly disagree or want more clarification? ===
 +
 
 +
:''To appeal a decision, message a '''[[Staff#Judges Panel|judge]]''', email us at '''<span style="color:orange">[mailto:admin@ocremix.org admin@ocremix.org]</span>''', or use our '''[https://ocremix.org/community/contact/ Contact form]'''.''
 +
 
 +
OCR takes the integrity and consistency of the judging process seriously. Whenever an artist strongly disagrees with a straightforward rejection of their submission (e.g. 3 NOs, 1 YES/4 NOs), or feels the feedback was unclear or impractical, the judges are willing to revisit it, and have — [https://ocremix.org/remix/OCR01448 on] [https://ocremix.org/remix/OCR03679 rare] [https://ocremix.org/remix/OCR03760 occasion] — reversed their original decision.
 +
 
 +
The judges are expected to frame their votes, particularly rejections, in the context of the arrangement and production criteria in the '''[[Submission Standards and Instructions|Submission Standards]]''', as well as provide constructive criticism. We ask artists to respect the judges and the judging process when challenging a decision.
  
 
== How long does it take for a submission to be judged? ==
 
== How long does it take for a submission to be judged? ==
  
:''To view how far initial reviews have reached into the Submissions Inbox, as well as which submissions are currently being evaluated, recently rejected, or accepted and awaiting postage to OCR's main page, please see [http://www.ocremix.org/forums/showthread.php?t=2819 Currently in the Judging Process].''
+
:''To receive more timely feedback for your arrangement '''before''' submitting it, please post in the [https://ocremix.org/community/forum/12/ Post Your Game ReMixes] section of the Workshop forums.''
 +
:''To view how far initial reviews have reached into the Submissions Inbox, as well as which submissions are currently being evaluated, recently rejected, or accepted and awaiting postage to OCR's main page, please see [https://ocremix.org/community/topic/2765/ Currently in the Judging Process].''
  
Submissions currently take 7 months, at most, to receive initial review. Once the initial review is complete, if the submission is neither rejected nor accepted without further evaluation, it is forwarded to the judges panel. Due to the lengthy queue in the Judges Panel, panel evaluation currently takes 9 months, at most. The speed of the judging process depends on a variety of factors including judge availability, familiarity with source material, and disputed votes requiring the attention of a greater number of judges.
+
OC ReMix is run by an all-volunteer staff, including the judging process. Submissions currently take 8 months, at most, to receive initial review. Once the initial review is complete, if the submission is neither rejected nor accepted without further evaluation, it is forwarded to the judges panel. Due to the lengthy queue in the Judges Panel, panel evaluation currently takes 10 months, at most. The speed of the judging process depends on a variety of factors including judge availability, familiarity with source material, and disputed votes requiring the attention of a greater number of judges.
  
 
=== Fall-throughs ===
 
=== Fall-throughs ===
  
A priority "fall-through" status may be requested for a submission when its creator suspects it is lost in the queue (hence, "fell through the cracks") and contacts a judge after an unusual amount of time has passed on an initial review (one month or more). The judge will decide whether or not to forward the submission to the judging panel, typically for factors such as elapsed wait time and probability of acceptance.
+
A priority "fall-through" status may be requested for a submission when its creator suspects it is lost in the queue (hence, "fell through the cracks") and contacts a judge after an unusual amount of time has passed on an initial review (9 months or more). The judge will decide whether or not to forward the submission to the judging panel, typically for factors such as elapsed wait time and probability of acceptance.
 
 
Verifying each and every fall-through request as legitimate can be difficult, as it involves combing through the entire submission queue. For this reason, mixers who've requested fall-throughs for two submissions which were approved will not be able to request any more. Note, this does not include questions as to the general status of the mix or suggestions and feedback.
 
  
 
== Are the judges biased? ==
 
== Are the judges biased? ==
  
Though the evaluation process is partially subjective by design, the judges panel does its best to eliminate personal and collective bias when evaluating submissions. Individual judges will frequently be in positions where they either reject something they enjoy or accept something they do not personally care for. The fact that OCR employs a panel of anywhere between 7-12 judges also helps compensate for dissenting votes that may hinge on personal bias, improperly applying the standards and guidelines, being too lenient, or being too harsh.
+
Though the evaluation process is partially subjective by design, the judges panel does its best to eliminate personal and collective bias when evaluating submissions. Individual judges will frequently be in positions where they either reject something they enjoy or accept something they do not personally care for. The fact that OCR employs a panel of anywhere between 7-13 judges also helps compensate for dissenting votes that may hinge on personal bias, improperly applying the standards and guidelines, being too lenient, or being too harsh.
  
 
Factors that have no bearing on judges' decisions include:
 
Factors that have no bearing on judges' decisions include:
  
*The submission's genre;
+
*The submission's genre or enjoyability;
 +
*The submission's release on other platforms;
 +
*The submission's recency or oldness;
 
*The game or original music being arranged;
 
*The game or original music being arranged;
*An artist's tenure in the OCR community;
+
*The amount of time spent creating the submission;
*An artist's experience level;
+
*The cost of the samples, instruments, or equipment used;
*An artist's following or notoriety as a professional musician;
+
*An artist's tenure in the community or prior submissions outcomes;
*An artist's friendship or personal differences with the site staff or community;
+
*An artist's experience level or temperament;
*An artist's criticisms about OCR's policies;
+
*An artist's following, notoriety, or connections;
*An artist's temperament;
+
*An artist's praise or criticisms about OCR;
*Donating to the site; and
+
*An artist's personal differences with the staff or community;
*Being a member of the site staff or judges panel.
+
*An artist's promotional or financial support of OCR; and
 +
*Being a member of the staff or judges panel.
  
In other words, don't overthink or politicize the process. The [[Submission Standards and Instructions|Submission Standards]] on arrangement and production are ultimately what guide the judges' evaluations.
+
In other words, don't overthink or politicize the process. The '''[[Submission Standards and Instructions|Submission Standards]]''' on arrangement and production are ultimately what guide the judges' evaluations.
  
 
== Why not allow ReMixes to be approved by popular vote? ==
 
== Why not allow ReMixes to be approved by popular vote? ==
  
OverClocked ReMix fosters a sense of community built upon the common goal of highlighting high-quality video game music arrangements made by amateurs and professionals alike and making them easily available to those who appreciate this genre of music. A popular vote system does not work in OC ReMix's case because most fans are not interested with whether a submission fits OCR's standards and guidelines.  
+
OverClocked ReMix fosters a sense of community built upon the common goal of highlighting high-quality video game music arrangements made by amateurs and professionals alike and making them easily available to those who appreciate this subset of music. A popular vote system does not work in OC ReMix's case because most fans are not interested with whether a submission fits OCR's '''[[Submission Standards and Instructions|standards and guidelines]]'''.  
  
OCR's selection process and judges panel are established to spotlight excellent content by any video game music fan or mainstream music listener's standards. While the lack of popular participation in the submissions process may frustrate some, as a whole, OCR fans understand the necessity of the judges panel and trust that the decisions it makes are ultimately in the best interests of the site.
+
OCR's evaluation process and judges panel are established to spotlight excellent content by any video game music fan or mainstream music listener's standards. While the lack of popular participation in the submissions process may frustrate some, as a whole, OCR fans understand the necessity of the judges panel and trust that the decisions it makes are ultimately in the best interests of the site.
 
  
 
[[Category:Judging Process]]
 
[[Category:Judging Process]]
 
[[Category:Submitting]]
 
[[Category:Submitting]]

Revision as of 02:47, 29 May 2022

For information on the evaluation criteria used at OC ReMix, please see Submission Standards and Instructions.
Big Giant Circles, Jillian Aversa, zircon & Liontamer listen in as djpretzel ultimately casts the tiebreaking vote in favor of GSlicer's Legend of Zelda: Majora's Mask OC ReMix, "Inverting the Tower Temple"

Who are they and what do they do?

For the list of current panelists, please see Staff.
For panelists throughout OCR's history, please see List of current & former judges.

OverClocked ReMix began in late 1999 with founder djpretzel evaluating all submissions personally. As the site grew, the job of evaluating submissions became too large for one person. In early 2002, after a community poll of ReMixers chose the most acceptable solution, djpretzel established the judges panel to decide (with his oversight) which submissions best fulfill the vision for OCR. The judges were initially chosen and invited by djpretzel, but since then have been added or removed based on panel consensus, with djpretzel's input.

Before a submission is sent to the judges panel for evaluation, it goes through an initial review by djpretzel or a designated member of the judges panel. As a result of this review, submissions are either rejected outright, forwarded to the panel for consideration, or accepted without further evaluation.

Decisions on submissions forwarded to the panel are reached by a majority of a panel quorum; a unanimous vote will be closed after three negative votes or four positive votes. A split decision will be closed after a difference of three votes (e.g. 5 YES's vs. 2 NO's), or extended to a majority vote at the request of a judge. In addition to his power to veto any submission regardless of the panel's decision, djpretzel may also serve as an expediting vote, a tiebreaker during particularly divisive decisions, as well as a guide on where the overall acceptance bar should be.

General requirements for consideration

Community members often ask how they can become a judge; the judges panel does not have an open interview process. When a panel vacancy opens, it is typically not announced until after a group consensus regarding candidates. When the panel decides a new judge will be necessary, candidates must demonstrate the following qualifications prior to or during an evaluation:

  • Musical knowledge (formal music theory knowledge and instrument/performance experience are not requirements);
  • A strong grasp of current OCR standards and guidelines as expressed in evaluation of ReMixes on the site, through submissions, and/or during the candidate evaluation process;
  • Ability to clearly and constructively advise artists as to how they can refine and improve their material;
  • Strong communication skills;
  • Openness to all musical styles/genres;
  • Extensive commitment to the community;
  • Ability to work proactively with the variety of artists and personalities in the community; and
  • Time, availability, and enthusiasm for the workload involved.

Many people who want to be judges or are considered as judges fall short in at least one of those areas. Because the process of choosing judges is not open, candidates who feel they may be qualified to become a judge are discouraged from lobbying judges to obtain a position on the panel.

What percentage of submissions are accepted?

The total acceptance rate of all music sent to OCR's Submissions Inbox ranges between 10-15%. The acceptance rate of music forwarded to the judges panel ranges between 15-20%.

After initial review, about 45% of all submissions are rejected with no further evaluation, 50% are forwarded to the judges panel for further evaluation, and 5% are accepted with no further evaluation (also known as a 'direct post').

A submission that is directly posted vs. posted after panel evaluation is neither better nor worse; oftentimes submissions are sent to the panel out of unfamiliarity with the source material, a desire to provide feedback, or because a precedent is being set in some way.

Why is the acceptance bar so high?

Since the panel's inception in early 2002, the bar for evaluating ReMixes has gradually but substantially risen. As the site's vision and standards became clearer and the number of submissions increased, this change was necessary to maintain a consistent posting schedule and to give each posted ReMix the time and attention it deserves. Even with the high bar that's been set, OC ReMix is a hobbyist community, so the bar purposefully remains below a professional bar, especially on production.

Why are there so many rejections in a row?

Unlike rejections, the judges' decisions on each accepted ReMix are not made public (i.e. moved into the Judges Decisions forum) until the ReMix is posted onto OCR's front page. As a result, there may be instances where the first 1-3 pages of decisions seem to all be rejections without a single acceptance.

My submission was rejected. What if I strongly disagree or want more clarification?

To appeal a decision, message a judge, email us at admin@ocremix.org, or use our Contact form.

OCR takes the integrity and consistency of the judging process seriously. Whenever an artist strongly disagrees with a straightforward rejection of their submission (e.g. 3 NOs, 1 YES/4 NOs), or feels the feedback was unclear or impractical, the judges are willing to revisit it, and have — on rare occasion — reversed their original decision.

The judges are expected to frame their votes, particularly rejections, in the context of the arrangement and production criteria in the Submission Standards, as well as provide constructive criticism. We ask artists to respect the judges and the judging process when challenging a decision.

How long does it take for a submission to be judged?

To receive more timely feedback for your arrangement before submitting it, please post in the Post Your Game ReMixes section of the Workshop forums.
To view how far initial reviews have reached into the Submissions Inbox, as well as which submissions are currently being evaluated, recently rejected, or accepted and awaiting postage to OCR's main page, please see Currently in the Judging Process.

OC ReMix is run by an all-volunteer staff, including the judging process. Submissions currently take 8 months, at most, to receive initial review. Once the initial review is complete, if the submission is neither rejected nor accepted without further evaluation, it is forwarded to the judges panel. Due to the lengthy queue in the Judges Panel, panel evaluation currently takes 10 months, at most. The speed of the judging process depends on a variety of factors including judge availability, familiarity with source material, and disputed votes requiring the attention of a greater number of judges.

Fall-throughs

A priority "fall-through" status may be requested for a submission when its creator suspects it is lost in the queue (hence, "fell through the cracks") and contacts a judge after an unusual amount of time has passed on an initial review (9 months or more). The judge will decide whether or not to forward the submission to the judging panel, typically for factors such as elapsed wait time and probability of acceptance.

Are the judges biased?

Though the evaluation process is partially subjective by design, the judges panel does its best to eliminate personal and collective bias when evaluating submissions. Individual judges will frequently be in positions where they either reject something they enjoy or accept something they do not personally care for. The fact that OCR employs a panel of anywhere between 7-13 judges also helps compensate for dissenting votes that may hinge on personal bias, improperly applying the standards and guidelines, being too lenient, or being too harsh.

Factors that have no bearing on judges' decisions include:

  • The submission's genre or enjoyability;
  • The submission's release on other platforms;
  • The submission's recency or oldness;
  • The game or original music being arranged;
  • The amount of time spent creating the submission;
  • The cost of the samples, instruments, or equipment used;
  • An artist's tenure in the community or prior submissions outcomes;
  • An artist's experience level or temperament;
  • An artist's following, notoriety, or connections;
  • An artist's praise or criticisms about OCR;
  • An artist's personal differences with the staff or community;
  • An artist's promotional or financial support of OCR; and
  • Being a member of the staff or judges panel.

In other words, don't overthink or politicize the process. The Submission Standards on arrangement and production are ultimately what guide the judges' evaluations.

Why not allow ReMixes to be approved by popular vote?

OverClocked ReMix fosters a sense of community built upon the common goal of highlighting high-quality video game music arrangements made by amateurs and professionals alike and making them easily available to those who appreciate this subset of music. A popular vote system does not work in OC ReMix's case because most fans are not interested with whether a submission fits OCR's standards and guidelines.

OCR's evaluation process and judges panel are established to spotlight excellent content by any video game music fan or mainstream music listener's standards. While the lack of popular participation in the submissions process may frustrate some, as a whole, OCR fans understand the necessity of the judges panel and trust that the decisions it makes are ultimately in the best interests of the site.

Promotion

Latest Albums

Latest ReMixes