Fumble OS

by zenquaker

I’m giving up on Chess.

It’s been a rough week in some ways. I say “in some ways” because while I dropped into some pretty damn dark moods earlier this week, I was able to see what was going on, understand the desires at play, and let go of them. Everything was good life after that.

A little background. We used to buy this data set at work. We’d had data quality problems with the data, but the vendor assured us they would do more quality control. We even put it in the contract. But we continued to have data quality concerns, and our conversations with the vendor only made us more nervous. Finally, after one public presentation by the vendor, my team leader directed me to do more search data quality checks on the data.

I found two real problems. The first dealt with the data constraints. Things such as “if ageunit is month, age can only be in the range 1-31.” There were several dozen of these in the data, and it seemed like a good place to start because it was easy to program. About 5% of the observations had one or more violations of the data constraints. This is not a big deal in and of itself. Any large data set is going to have errors, and the effort to get rid of them is inversely proportional to how many there are. Eventually you hit a point of diminishing returns and just deal with it. But since these data errors were so easy to find, it said to us that they weren’t doing any quality control, despite their assurances and the contract. They later confirmed this.

The other problem was in an important variable that we used a lot. In our data that variable has an objective scale. In their data that variable has a subjective scale, and that worried me. I analyzed that variable across the different data collection centers, using clustering analysis on the distribution of values. I found three distinct clusters of data collection centers with three different distributions of coding the data. Again, this is not necessarily a problem. There could be geographic or demographic factors that affect the data being collected. When we asked the vendor if they could provide any reason for the differences, they got very mad at us for even thinking of doing quality control checks on their data.

In the end, we stopped buying the data. My analysis combined with their response made it so that we just don’t trust it any more. Some people at work were really attached to that data, and they are still pissed off at me. They are still trying to get back at me, in some rather underhanded ways.

We stopped buying the data a few years ago. This week I was required to put summaries of the data from the vendor’s annual report into one of my reports. Additionally, I was told that upper management is very interested in pursuing further projects with the vendor. After sticking my neck out to get rid of this data, and after taking years worth of harassment over it, I was really upset. I was seriously thinking about quitting my job.

You’re sitting there wondering what the hell this has to do with Chess.

I came home from work that night trying to remain calm and on an even keel so that I could look at the situation rationally. I had my dinner and went to do my evening Chess tactics training. I choked, I got upset about it, and it sent me into a downward spiral that left me collapsed in bed and barely able to get up in the morning.

I realized that the reason I play Chess is because I need to win, to show that I am better than someone else, and thereby validate myself. But it’s never going to work. I am going to continue losing a significant chunk of my games, and my losses will be directly attributable to errors on my part that will make me feel like a moron. If I improve, I will just face tougher opponents and make more subtle mistakes. It’s the nature of Chess. It is such a deep game that there is always someone better than you, and the game is always won because the loser made a mistake.

Similarly, I realized that I needed to be right at work. I have long realized that my technical skills have long been important to my self validation. When I was a young freak, demonstrating my intellectual prowess was the only way I ever got any recognition. I saw the requirement to use the data as attack on my technical evaluation of it, and thereby an attack on me. But there is no me. All there is to attack is my desire to be smarter than everyone else. If I let go of that, there is nothing left to attack.

If management wants bad data in the report, management gets bad data in the report. I’m not going to let them ruin my life because they insist on it. And I’m not going to ruin it myself with some stupid need to be a better Chess player.

I’ve been feeling very relieved since I gave up on those desires. I’ve been getting more work done on my other projects, and I’ve been learning new and interesting things doing that work.

I will probably even have more time for this blog.