ASK A JUDGE: While we're busy NOT voting - your questions, we want 'em

[Liontamer]

As a means of supplementing my blog, VG Frequency, I'm throwing this thread out there for questions towards myself and the other judges. Any legit questions on our thoughts on arranging/judging/the community/music recommendations/site stuff, I'm willing to answer; maybe some others will chime in, but when I've got the free time, I'll try to answer as much as I can for the peoples.

I'll also answer relationship questions, because some of you guys need a shitload of help.

[BardicKnowledge]

Here's a couple judging questions for you:

What percentage of submissions are asked to resubmit after fixing a few things?

And as a followup: How many of those actually follow through, tweak what's been asked, and resub?

[Liontamer]

Here's a couple judging questions for you:

What percentage of submissions are asked to resubmit after fixing a few things?

That's an interesting one, in that we really don't catalog what rejections get a more positive response or seem really close to passing. I've got no idea of a solid percentage where we have a consensus that things are pretty good, even though it's not a YES yet. Conservatively, I'd say maybe 5-7% of rejections, you either get a mix of YESs and the rest are positive NOs, or all of the NOs react encouragingly towards the track.

And as a followup: How many of those actually follow through, tweak what's been asked, and resub?

Way fewer than I would like to see. In terms of resubs that pass, I'm fairly sure those make up less than 5% of all submissions. If anything, I'd love to know how many submitting artists actually read their decision threads. Out of 1800+ subs that I've voted on, I'm again just taking a stab, but I'm sure I've heard maybe 50 where IMO the artist only had to do pretty minor stuff, could have tweaked it, and it would have made it to the front page.

I think the best example I can think of is way back when I started, we got a mix from Kailem: Tetris Attack "Poochy Meets Froggy DX". Kind of an unwieldy title, but the arrangement was excellent.

Assuming you've now had a chance to read that decision, when we get situations like that, as much as some people like to claim we're production whores, we've been pretty lenient on stuff. (Zoltan Vegvari's Mega Man 3 "Intro Jazz" is the best example on that level.) But the mixing/balance was so terrible on "Poochy" that it got rejected. I really thought Kailem would work on it some more and send it back, but it never happened.

A good example of someone actually following through would be Navij11's Kirby & The Amazing Mirror "Kirby's Mystical Mirror", the resub of which has yet to go up. Personally, I have 0 idea whether Jacob was mad at the initial rejection or whatever, but he did a great job of running with the criticisms and revising the arrangement to give it more substance while not needlessly overhauling it. We've had plenty of close ones where the artist, instead of just tweaking a couple of things, majorly changed what was there, and ended up moving backwards.

Anyway, for me, I tend to say "resubmit" when a concept is promising but could use further development, even if I don't think the artist could easily get it to pass; there could certainly be a lot of issues. I'd rather see how much they can improve an already solid concept.

"refine/resubmit", I'll use for stuff that I feel doesn't need too much more work to pass. The real gold to me though, I'll use "borderline". Any time I use that in any context, that's a pretty close one, and I would hope the artist didn't choose to call it a day.

[Sengin]

Do you guys take a look at the author before listening? In other words, if you know the person submitting has had other remixes on the site before, does the name even subconciuosly affect your decision if you're on the fence? And the same for new remixers, but with the other way? If you're on the fence, is it more generally a "no" so that you can hopefully get a better quality mix in a little while as a resub?

If I were to submit a mix now, what would be the average time it takes to get judged? And if it got accepted, what's the average time it would take to make the front page? How many mixes that have been accepted are still waiting to make the front page?

Good idea for a topic, by the way.

[AMT]

Why does it take so long for accepted remixes to get posted? I don't mean this as a confrontational question, I know you guys have a lot going on in your own lives on top of OCR, but is there a particular reason it takes a while?

[Ramaniscence]

Why does it take so long for accepted remixes to get posted? I don't mean this as a confrontational question, I know you guys have a lot going on in your own lives on top of OCR, but is there a particular reason it takes a while?

Real-life and write-ups, I would assume. Those things don't write themselves, and if you're thinkin' "How hard could it be to write a couple paragraphes and post some songs a few times a week?" you got another thing coming.

This is also assuming the mixer/game/system/composer/company are all already in the database. Otherwise you ahve to go out and get all that info, find a game image, edit and upload that image, add everything to the dabatase, check and recheck to make sure it's working, and then post.

Edit: It's also been said before, and rightfully so, that the space between mixes is also sometimes on purpose. If they posted every mix as it got accepted you'd get days with 5-7 mixes, and weeks with no mixes at all. And from a mixer perspective you have more time to have your mix on the front page and get more exposure, rather than having it forced out in 2 updates.

[SoloGamer]

Can the judges DJP? [/obligatory]

[Murmeli Walan]

Can the judges DJP? [/obligatory]

I think there's more than enough evidence to support that they can.

[Liontamer]

Glad you like the topic.

Do you guys take a look at the author before listening? In other words, if you know the person submitting has had other remixes on the site before, does the name even subconciuosly affect your decision if you're on the fence?

I know Vigilante doesn't pay attention to names much, but it's specifically an effort to make himself more impartial. I'd be foolish to say that in the 6 years of the panel that not a single vote by a judge has gone one way or the other based on name. When you say "subconsciously" though, by definition, that's something we wouldn't pick up on.

At the same time though, I don't feel like I've seen it in my 3 1/2 years on board. Honestly, if I could think of one where I really felt someone was biased, I'd point it out to you right now and link it.

For me, I always look at the author. To me, it doesn't have any affect on my voting. At the end of the day, I pride myself on my objectivity. I've been in plenty of cases where I'm sure I've lost goodwill with people because I've NOed their material. Not that every group I'm gonna mention has had someone complain, but I've NOed past judges, future judges, current judges, old-school legends, music majors, and artists with professional music, film and game credits.

You've just gotta go with your honest take on everything, I think that's much easier at the end of the day. I dunno how others feel, and can't speak for them, but I feel great knowing that I've been around so long and never been close to compromising my standards. It's also good to have colleagues where, if we disagree, they want to change your convictions, instead of wanting you to vote against your convictions.

I've definitely felt conflicted before, certainly on name. I think NOing Mazedude and Mustin were some of the more difficult ones for me, in the sense that I'd hope they wouldn't get angry and not want to be involved in OCR, submitting, the community, etc. But I've never actually had it come down to actually going "well, I don't truly think it's a YES, but should I YES it?" I'm thankful I don't have those kind of moral dilemmas. When it comes to integrity vs. Hands Across America, I'd rather keep my integrity.

And the same for new remixers, but with the other way? If you're on the fence, is it more generally a "no" so that you can hopefully get a better quality mix in a little while as a resub?

Nah. If anything, NOing new/unposted artists that are close to passing should feel more difficult, in the sense that maybe they'll get frustrated and not stay involved in the community. But for me, it's not difficult. Much like dealing with established ReMixers, you just gotta vote even-handed and hope for the best in terms of the person's attitude.

If I were to submit a mix now, what would be the average time it takes to get judged? And if it got accepted, what's the average time it would take to make the front page? How many mixes that have been accepted are still waiting to make the front page?

To get judged? Right now, it would vary from 2 weeks to 3 months. Just like in the Judge FAQ, it depends on a myriad of reasons. As soon as we hit Christmas last year, we all basically went on hiatus, and we're back to where were at before in terms of judging wait time. Even now, I haven't had the time to get back into it as much. That being said, I think we can pick it back up. When we have momentum, we start going through a lot of submissions at once.

Wait time after acceptance? It's all on djp's schedule. It's a packed schedule. 2-3 months at least, AFAIK. We've got a pretty big posting queue of stuff ready and waiting to go up.

Number of mixes waiting? Currently 24 that definitely could be posted any time right now, around 12 that are part of unreleased album projects and about 12 that need to be tweaked a bit before they're ready to be posted.

Why does it take so long for accepted remixes to get posted? I don't mean this as a confrontational question, I know you guys have a lot going on in your own lives on top of OCR, but is there a particular reason it takes a while?

Life. There's always going to be an excuse/reason why. Nowadays, djp's got so many irons in the fire, with the planning of the Street Fighter II HD Remix soundtrack, conventions, contests, interviews and real life with a girlfriend and job. So djp's gotta fit the posting of mixes into all that. That's really the only reason. Like Rama mentioned, song writeups aren't as easy as it looks, and stuff needs to get the proper spotlight time on the front page. But yeah, life. I would really love a streamlined enough system where we had stuff from the inbox to the frontpage in 2-3 weeks.

[zircon]

Do you guys take a look at the author before listening? In other words, if you know the person submitting has had other remixes on the site before, does the name even subconciuosly affect your decision if you're on the fence? And the same for new remixers, but with the other way? If you're on the fence, is it more generally a "no" so that you can hopefully get a better quality mix in a little while as a resub?

Personally, half the time I don't even read who the remixer is. I often find out simply by reading the votes of the other judges. I don't think the person necessarily has an impact on my final vote, either, though how I write it might change. If I'm NOing a sub from an established remixer, I'll phrase it differently than if I'm NOing a sub from someone that hasn't gotten passed yet.

[Hausdog]

How hard is it to phrase exactly what the song does well/poorly? Sometimes, when I'm in the WIP forums, I'll really like something but not be able to deliver anything helpful beyond encouragement.

How often do you "yes" mixes you dislike and "no" mixes you like?

What are some of your pet peeves that submitters do?

[FR]

I'll give you an easy one: How do you become a judge?

[Tex]

Which are your actions when you don't find the ost from an evaluation?

[zircon]

Hopefully Larry won't mind me answering some of these

I'll give you an easy one: How do you become a judge?

We look for a number of qualities in potential new judges. There are no formal requirements, and we've brought people on who didn't have all these qualities, but the list should give you a general idea:

* Active and constructive involvement in the community. Specifically we look for people that regularly spend time in the Reviews and WIP forums, as well as ReMixing and #ocrwip.

* A personality that we think we can get along with. If you are a regular troublemaker on the forums, or if you have a huge ego, or you tend to fly off the handle constantly, that doesn't bode well for the group interaction. We've definitely had to let judges go in the past because of this.

* Good musicianship. If you get rejected all the time, how are you going to give useful feedback to other people? The exception to this is people with an excellent musical ear who are not themselves musicians, such as Larry and CHz.

* The ability to be articulate and give constructive criticism. Probably obvious, but you might meet all the above criteria but have poor English, for example, in which case it might be hard for you to give useful feedback to people. Alternatively, you might be a great musician but simply not be good at critiquing. This comes up more often than you'd think.

* A good sense for where our standards are at, and a general agreement with how we do things. If you like the site but think that we really ought to accepting covers and MIDI rips, you ain't gonna be a judge. Not to say that we don't allow disagreement within the panel, in fact, we regularly discuss issues of standards; especially on borderline votes. However, any new judge has to at least agree generally with where we are now.

This last point is maybe one of the biggest reasons we turn people down - they don't have a grasp of what we're looking for in submissions and either accept or reject things that should be obviously going in the other direction, or simply have poor reasoning for why a mix should or should not be posted.

--

As for the actual process, we're always looking out for people with the above qualities in the community, but believe me they are in pretty short supply. If there's someone really outstanding we may look to test just them and bring them on without doing a whole multiple-candidate evaluation - this is what we did with Harmony.

MOST of the time, however, we look for new judges when we feel like our ranks have either thinned to the point that we need replacements for people that have left, or when things just seem to be moving too slow and we need fresh blood. When this happens we begin a process that usually takes a couple months to complete. We compile a list of names of people who might be good candidates, then have a lengthy discussion and eliminate ones we don't think would be a good fit. We're then left with a smaller list (<12 people) who we'll interview, then test by giving them a number of remixes to judge as if they were already on the panel. After the test, we interview them again and talk about their votes.

While this is going on, we typically post the "test batches" of all the candidates as well as the interview logs, where possible. Finally, we review everything as a group and determine which people would be the strongest to add to the panel.

In short, be active in the community, give feedback in the WIP and Reviews forums, sharpen your skills as a ReMixer, and regularly read the Judges Decisions forum. And don't call us, we'll call you.

Which are your actions when you don't find the ost from an evaluation?

To my knowledge, this has only happened once, where we absolutely could not find the source anywhere. Luckily, the production in that case was such that even if it had a hot arrangement, the production issues alone were enough to NO it. I think if we were really unsure and could not find the OST at all, we would have to solicit the community and the ReMixer to help us find it. Though I think that if something is really that obscure, it might not even qualify as acceptable material (eg. random homebrew PC game from 1987.)

[Liontamer]

How hard is it to phrase exactly what the song does well/poorly? Sometimes, when I'm in the WIP forums, I'll really like something but not be able to deliver anything helpful beyond encouragement.

Kind of a copout, but it all depends on the track. If it's hard to articulate the issues, it's also hard to articulate WHY we can't articulate the issues.

Usually though, it's not too difficult to be able to say what's working and what's not. For me, a non-musician, I just stick to what I know (or purport to know). For example, with instruments, I can't tell the difference between various groups of brass, mallet percussion, woodwinds or synth techniques. Instead of trying to bullshit that, I just generalize those things when referring to the instruments.

I wish I could be able to look back and just point to XYZ method as what made me improve at music critique, but I don't know of one. After listening to all the ReMixes and having a good idea of the standards in 2002-2003, I just understood where the bar generally was and what kind of creativity and production quality was being looked for here.

The best thing I can say is to be detail-oriented. Have some decent headphones (I'll never judge on anything but headphones.) They don't have to be overly expensive, but some generic earbuds can't help you pick out details. GrayLightning recommended me the Sennheiser HD497s a few years ago, and while a little bit trebbly, they're nonetheless excellent. I think they were $60. But I judged for maybe 1 year and some change on some $30 ones and got by.

Writing down thoughts in a stream of consciousness always helps me, especially when I'm finding it difficult to articulate issues/problems. Even if what you wrote down isn't very organized, you'll have your thoughts out in front of you. Part of the benefits of having colleagues on the panel are that they can look at what you're thinking and possibly corroborate your POV.

How often do you "yes" mixes you dislike and "no" mixes you like?

Definitely couldn't give a percentage on that, but often enough. Many YESs, they're objectively good, but I wouldn't put them on my iPod vs. other songs out there. That's not a slight against those tracks, but not everything can be a personal favorite just because it's well made. Then again, I don't actively dislike those tracks. I don't think I've ever encountered tracks I've hated that I've YESed, but rather I'm personally indifferent to them, even though I'm impressed as a judge. That being said, I liked maybe 1/3rd of the OCR's back in the 700s. Nowadays, the proportion is higher because the consistency from mix to mix is higher.

I keep about 5% of the rejections I've encountered in a folder. I like 'em, I still listen to 'em. It's definitely a perk. There are a lot of decent NOs.

What are some of your pet peeves that submitters do?

Ha, that's funny. Good question. Nowadays, there are a lot fewer peeves, now that we've revised the submission standards. There used to be so many MIDI rips and files that would 404 because they were hosted on Putfile or whatever and expired in 7 days, before we could get to them. MIDI rips and uncreative covers have definitely gone down since the Standards were cleaned up a couple of months ago.

Maybe the only one is when submitters who have forum accounts go "I have no idea what my forum ID # is". It's not hard to get the ID #. I'm #1211. Doesn't really matter, since I handle those details for the database anyway. If I think of something else, offhand, I'll let you know.

[The wingless]

Can former judges answer stuff? Oh, pweeese?

[Liontamer]

Hopefully Larry won't mind me answering some of these

Yeah, definitely. This is for all the judges. Former judges too, whenever relevant.

To my knowledge, this has only happened once, where we absolutely could not find the source anywhere. Luckily, the production in that case was such that even if it had a hot arrangement, the production issues alone were enough to NO it. I think if we were really unsure and could not find the OST at all, we would have to solicit the community and the ReMixer to help us find it. Though I think that if something is really that obscure, it might not even qualify as acceptable material (eg. random homebrew PC game from 1987.)

Yeah, following up on that, I'd quit before we posted a track with no source tune to compare. At the end of the day, I need the proof that the arrangement is interpretive enough. The video game music fan community is resourceful though. Nowadays, there's a very low likelihood that we'd get stuck with something where we definitely couldn't find the source.

[Fishy]

plz voet yus on ma beatz plz?

Also how does the direct post system actually work, cause I remember hearing from someone that you guys have a possible DP list that I presume is visible to the judges, even if they don't get input. Is it like entirely Larry's decision, or does he pick a few and show them to Dave to get his thoughts? Do any of the judges get any input on that?

[Palpable]

And as a followup: How many of those actually follow through, tweak what's been asked, and resub?

Throwing in my own two cents here as a mixer rather than a judge: I didn't know about the judge decision forums until submitting my third mix here. The only thing I did was constantly check the front page for months. In the case of my first mix, I wasn't even aware it had gone up until months afterward because I had stopped checking! By the time I submitted my fourth mix (which was rejected) I had learned about the judge decisions forums, but I was essentially doing the same thing: checking every few days for a couple months. In that case, I got so tired of waiting that I lost interest in OCR completely.

I would love if we could e-mail mixers once their mix had completing judging, informing them of the decision, but there are at least a few problems with it I can think of. One is just remembering to do it, unless we can automate the process. Two is that sometimes the decision can be changed. In that case, when is the decision finalized such that we can tell the mixer what the decision was?

These are certainly some things I'd like us to think about. I think a lot of great potential resubmits fall by the wayside because the mixer isn't familiar enough with OCR or doesn't check back often enough. There was one rejected mix back in November that I absolutely loved that nevertheless had some major problems. I've thought about e-mailing the mixer just because I very badly want to see it completed, and I have no idea if he ever intends to go back to it, or is even aware it was rejected. More communication would be a very good thing, I think, if it's possible.

[Liontamer]

Also how does the direct post system actually work, cause I remember hearing from someone that you guys have a possible DP list that I presume is visible to the judges, even if they don't get input. Is it like entirely Larry's decision, or does he pick a few and show them to Dave to get his thoughts? Do any of the judges get any input on that?

I did have a list that the judges could view, but they don't really care too much about the DPs, or at least no one has told me, so I don't maintain it anymore. That being said, anything I flag for DP is on the judges FTP, so they can check out anything any time.

Yeah I suppose de facto, I'm now single-handedly responsible for DPs. So really when it says "Evaluated by: djpretzel", it should be saying "Evaluated by: Liontamer".

I've never had djp or the judges complain about what gets DPed. The judges, that could be an issue of them simply not listening to the DPs when they're posted, because maybe someone would have an issue, but I've never heard of one yet. Dave clearly hears everything, at least before doing the writeups, and I've never had him complain either.

I'd argue that my DP bar is higher than djp's, but I don't think he would take offense, because he's said before on VGF #50 that he feels the site's gotten better since the panel got more influence on song selection, because they have higher collective standards.

When I flag something for DP, I pretty much make sure, to the best that I can, that it's cut-and-dry YES quality.

[JJT]

How hard is it to phrase exactly what the song does well/poorly?

Depends on how well a judge knows the genre in question. I'm generally more comfortable critiquing jazz and solo piano arrangements than say, Tibetan throat singing. The less I know about a genre, the more general I have to be with my criticism.

How often do you "yes" mixes you dislike and "no" mixes you like?

I'd estimate that I like about 70% of the stuff I YES, and maybe 10% of the stuff I NO. There have been a few heart breakers for me, definitely.

What are some of your pet peeves that submitters do?

Probably my biggest pet peeve is when first time submitters write a fuggin' novel about the remix they threw together in FL in 6 hours. They write 4 paragraphs about playing the game for the first time at age 7. They mention the names of everyone who listened to earlier drafts. They thank God, and their parents. And after all of this they suck profusely. There's nothing that calls down the fire (as far as my votes are concerned) like a longass pretentious submission email.

Once you get a mix or two posted, feel free to write a ton, as Dave likes putting that stuff in the writeups. I do it all the time.

[Arek the Absolute]

Depends on how well a judge knows the genre in question. I'm generally more comfortable critiquing jazz and solo piano arrangements than say, Tibetan throat singing. The less I know about a genre, the more general I have to be with my criticism.

I'd estimate that I like about 70% of the stuff I YES, and maybe 10% of the stuff I NO. There have been a few heart breakers for me, definitely.

Probably my biggest pet peeve is when first time submitters write a fuggin' novel about the remix they threw together in FL in 6 hours. They write 4 paragraphs about playing the game for the first time at age 7. They mention the names of everyone who listened to earlier drafts. They thank God, and their parents. And after all of this they suck profusely. There's nothing that calls down the fire (as far as my votes are concerned) like a longass pretentious submission email.

Once you get a mix or two posted, feel free to write a ton, as Dave likes putting that stuff in the writeups. I do it all the time.

Since you mentioned "write ups", what is your favorite "write up" that you got for a mix that was submitted?

[Liontamer]

Throwing in my own two cents here as a mixer rather than a judge: I didn't know about the judge decision forums until submitting my third mix here. The only thing I did was constantly check the front page for months. In the case of my first mix, I wasn't even aware it had gone up until months afterward because I had stopped checking! By the time I submitted my fourth mix (which was rejected) I had learned about the judge decisions forums, but I was essentially doing the same thing: checking every few days for a couple months. In that case, I got so tired of waiting that I lost interest in OCR completely.

I would love if we could e-mail mixers once their mix had completing judging, informing them of the decision, but there are at least a few problems with it I can think of. One is just remembering to do it, unless we can automate the process. Two is that sometimes the decision can be changed. In that case, when is the decision finalized such that we can tell the mixer what the decision was?

These are certainly some things I'd like us to think about. I think a lot of great potential resubmits fall by the wayside because the mixer isn't familiar enough with OCR or doesn't check back often enough. There was one rejected mix back in November that I absolutely loved that nevertheless had some major problems. I've thought about e-mailing the mixer just because I very badly want to see it completed, and I have no idea if he ever intends to go back to it, or is even aware it was rejected. More communication would be a very good thing, I think, if it's possible.

I would love to do that, and I've definitely thought about that, but have so much on my plate and have been content enough with our current system that I haven't taken the time to move an idea like this forward.

Keeping mixers in the loop is one aspect where our system really has no plan other than "they should follow the site", which is definitely not conducive to getting new people to become community regulars. Admittedly, it has worked for us so far, and honestly, it would continue to as is, because enough people become regulars. But contacting people regarding acceptances and rejections would be great.

That would also be yet another thing I'd have to be personally responsible for, since I maintain the inbox. IF we developed a form letter for notifying artists about acceptances and rejections, I would love to start doing that. I feel that kind of contact/updating is something that would only benefit the site, even though we'd have our share of complainers who would resent bad news.

[Palpable]

IF we developed a form letter for notifying artists about acceptances and rejections, I would love to start doing that. I feel that kind of contact/updating is something that would only benefit the site, even though we'd have our share of complainers who would resent bad news.

Yeah, this is sort of what I was thinking. Maybe one form letter for NO, one for NO (resubmit), one for YES, with maybe some space for comments if we feel the need. (Of course, NO (resubmit) is an individual's decision, not a group decision, so maybe we'd have to be clear about what gets classified as NO (resubmit).) Getting people to resubmit is the most crucial part of this to me. A decent amount of stuff we get is close but doesn't make it.

[Kanthos]

Why not a form letter with a link to the Judges' Decision thread for that mix? That'd be less work than cutting-and-pasting comments into the e-mail.