Turtles all the way down

At any one point in my career, I’ve always been writing something. But now that I more or less write for a living, I’ve had to get more serious about doing it well and making sure that what I write is correct. Proofing your own work is hard, and as a bonus, also boring. But, over the last few months I’ve found two techniques that seem to work reasonably well. Neither technique is foolproof and mistakes will still slip through, but I’ve used both to improve my writing.

Read it backwards: Normal people proof their writing by starting on page one and working linearly through the document. You need to do this because you want to make sure that the document flows logically from point to point. However, this logical flow can distract you from analyzing the mechanics of your writing, such as spelling and grammar. I’m notorious for making typos that slip past word processor spell-checking as well as my own.

The solution? Read it backwards. Start on the last sentence of the last page. Read each sentence in order, but read the sentences out of order (what I mean is read the last sentence first, then the second-to-last sentence next, and so on, but read each sentence itself in order).

This will disrupt the logical flow of the document and allow you to focus on spelling, grammar, and other low-level mechanical aspects of your writing. The result? I always find those hidden typos using this technique.

Another advantage to reading it backwards occurs when you’re proofing a lengthy document. Once your document is 30 pages or longer, it’s likely that when you read it in order, you’ll concentration and focus will be far higher on page one than on page 30. By reading it backwards, your focus on page 30 will be higher than it otherwise would be, thus increasing the likelihood that you’ll find mistakes on those later pages.

Give it a break: Once you finish a draft, let it rot. Don’t try to proof something you’ve just finished writing. Give it some time off to in order for your brain to regain some objectivity. Ideally, give your document a week between passes of proofreading, but in reality that is often too long. But try to let it rot for at least a day or two. You’ll read your own writing as if it were someone else’s, and you won’t be so close to it. Your concentration and focus will remain higher throughout the task because your writing will seem more fresh.

I spent a little too much time in the 90’s studying queuing theory. Why, you ask? After all, how exciting is it to be able to mathematically model the dynamics of…people lining up?

Queuing theory is more and less than that. It models the dynamics of anything that is waiting in line for just about any purpose. However the typical queuing models are usually either too simple to be accurate or too complex to be tractable.

For many years, queuing theory was used to model telecommunications system capacity. This worked reasonably well until the amount of data traffic in these systems began to grow. Packet data exhibits dynamics far different from circuit-switched telephony. After some noble, but ultimately futile attempts to “save” queuing theory, most of the academic community reluctantly admitted that other modeling techniques might prove more accurate.

Over the years I’ve found three basic queuing principles to be the most applicable and useful in practical settings. Below are these principles and (hopefully) intuitive explanations of why they are true. Of course, if you don’t believe me then you can look up proofs in various textbooks.

Little’s Result: If the arrival rate of customers into a queue is lambda (e.g., 5 customers per second), N denotes the average number of of customers in the queue, and T denotes the average amount of time that any given customer waits in the queue. The following relationship always holds:

N = lambda T

The reason for this can be explained with an example. Suppose a customer, Bob, arrives in the queue. On average there will be N customers ahead of Bob, and Bob will wait T seconds in the queue. During these T seconds, on average lambda T customers will arrive in the queue behind Bob. Thus when Bob is finally served, there is once again N customers in the queue.

Another means for understanding Little’s Result is to consider that if you double the arrival rate to the queue per unit time (lambda) or the waiting time in the queue (T) then you will also double the number of customers in the queue.

The remarkable thing about Little’s Result is that it universally applies to any queue, regardless of how customers arrive or how they are serviced. Thus, it applies equally to a steady stream of customers arriving to a queue that is served in a first-in-first-out basis, and to a highly-variable stream of customer arrivals to a priority queue.

The more variance in the service, the longer the average waiting time: Another relatively simple concept that is fairly intuitive. The more variability in the time it takes to serve customers, the more likely you will run into a pathological situation where a bunch of customers arrive behind a given customer who requires an abnormally long period of time to be served. You might feel like you run into this situation every time you get in line. In fact, the average waiting time in the queue grows linearly with the variance of the service time. Thus, it behooves us all to reduce the variability of service time. If that can’t happen then the next section describes one way of mitigating the damage.

When you have multiple servers, always maintain a single queue: Suppose that, at the airport, there are two agents (servers) checking in customers. The airline has two reasonable options for arranging queues. The first approach is to allow a dedicated queue for each agent. The second approach is to have a single, common queue in which all customers wait, and the agent calls the customer at the head of the queue when the agent finishes serving a previous customer.

The surprising result is that, all other things the same, the average waiting time for the two-queue system is twice that of the single queue system. Thus, simply arranging customers to line up in a single queue is twice as efficient as using two queues. As the number of servers increases, so does this efficiency. A single queue with four servers is four times as efficient as the same system with four queues, a single queue with eight servers is eight times as efficient as the same system with eight queues, and so on.

Luckily, most airports already use this practice, as do some motor vehicle departments and movie theaters. It is a simple way of getting a large number of people through the servers as quickly as possible. Why does it work? Again, it is all about avoiding the situation where a customer who requires a long service time ties up everyone behind him. When there are multiple servers and a single queue, the system approximates a fair first-come-first-served discipline.

Bonus: Is it queuing or queueing? Now that you know all you need to about queuing theory (and pretty much all I know), here’s a question I don’t have an answer for… The modern usage of the word seems to be “queuing,” but traditional texts tend to use “queueing.” I believe the latter is the British spelling and the former is the Americanized version. Anyone know for sure?

As you might expect, here is my outline for contracts. Enjoy.

Today is my last day as a 1L, unless you count the summer session, which I’m not going to. In many ways, my experience of law school has turned out very differently than I had expected. Off the top of my head a few of these differences include:

• Law school was much harder than I expected. I thought it would be easy to cruise by with B’s. It isn’t.
• I thought I’d make it through on raw brain power and not have to work too hard. Instead, my raw brain power got me very little. What few successes I’ve had so far have been through hard work.
• My fellow students are very bright, hard working and dedicated. I expected this, but the extent of their brightness, work ethic and dedication is more than I thought it would have been.
• There is way too much emphasis on grades. I figured that in law school, like any other type of school I’ve been to, grades would be a just a rough proxy of intelligence, ability and hard work. This is true, but hiring firms seem to put so much emphasis on grades that the law school do so as well with ridiculously severe curves.
• Having said that, my fellow students are pretty non-competitive bunch, which is a welcome relief.
• In law school the professors don’t teach, they profess. The students teach themselves the material.
• By far the best and most useful classes are legal writing and research. Grades in these classes are what hiring firms should look at.
• The time commitment for a part-time law student is enormous. It is easily a 30-40 hours a week on average. (Says the guy who spent 90 hours on his appellate brief.)
• For the most part, the law is reasonable, rational and fair. Also quite intellectual. The stereotype of lawyers being fast-talking scam artists is so wrong. Thankfully, those types are few and far between.
• The classroom and the textbooks do not prepare you for exams. However the study guides, hornbooks and online materials do.

Finals season is upon us.

Yesterday was the last day of class. I have the Property final in one week and the Contracts final one week after that. My outlines for both classes are essentially done though both need tweaking.

I spent today working on a practice exam for Property with limited success. It is taking me longer to answer the questions than I’d like and I’m having to plug holes in my outline. But that is to be expected, I suppose.

I’m finding myself less motivated by the whole “learning” aspect this time around and just wishing it was over with.

Part of that feeling, I’m sure, is to do with how challenging this semester was. It wasn’t too bright of me to take the patent bar in the middle of the second semester, in between the appellate brief and oral argument assignments. Not to mention the business trip during Spring Break which kept me from getting too much down time.

I’m not a big fan of classroom learning. It is effective to an extent. It is not effective when class time is used inefficiently.

Currently I get about 10-11 hours of classroom instruction per week, and I think it is too much. While face time with experts (i.e., professors) is a critical aspect of learning, other effective avenues are group and solo studying.

The particular quirks of law school lecture classes - no homework and one big final being the only grade - is probably less conducive to effective learning than others formats.

Right now, I’m thinking that 5-6 hours of classroom instruction would be best, while there should be more emphasis on doing and less on reading.

Today I took the patent agent registration exam, aka, the Patent Bar.

I passed, so now I can practice in front of the US Patent and Trademark Office, and represent clients just like an attorney. Except that I’m not an attorney. Yet. I’m just an agent.

The patent bar is hard. The pass rate is typically around 50%, whereas the pass rate for the Illinois Bar Exam is about 85%.

Various people have told me that you need to spend between 150 and 400 hours preparing for the exam. I spent about 80-90, but my 10+ years of working with patents on the client side of the fence may have given me a head start.

Like too many law-related exams, the patent bar is all about speed. You have six hours to answer 100 questions, and you’ll probably need all of it. The best way to study is to take a broad overview course, then to take as many practice exams as possible.

I took the PLI self-study course, then did exam drills and focused studying with Bullseye. I found the PLI course to be a good general overview, and their Patware CD to be a great way of taking practice exams, but the Bullseye course had a nice set of “frequently asked questions” and a solid outline.

In any case, the exam covers the entire Manual of Patent Examining Procedure (MPEP), a 3000-page monstrosity that includes all of the applicable patent laws, regulations, treaties, along with summaries of case, law, examples, and so on. The exam asks questions from just about everywhere in the MPEP. Although the exam is open book (you have a searchable PDF version of the MPEP on your exam computer), the tight time frame coupled with complicated fact patterns in questions makes it so that you don’t have enough time to look everything up.

Since questions are repeated from exam to exam, you need to memorize about 50 or so of the most commonly-asked questions in the hope that about 15-20 will show up on your exam. By having these questions down cold, you’ll buy yourself some time to look up answers to the rest.

Does the patent bar actually test your ability to be a good practitioner? Yes and no. Prosecuting patents is mostly about claim drafting, writing a good supporting description and arguing with the Patent Office. Knowing the matter tested by the MPEP is valuable, but is only part of the requisite knowledge. Perhaps these other topics are too hard to test.

I’m both surprised and relieved to have passed. The surprise is due to the fact that the exam was hard and at the end I wasn’t sure I had made it. And of course the relief is because I won’t have to sit through it again. Ever.

In the 1960’s, Paul Diederich and others conducted a study of how essays are graded. They managed to talk over 50 people, including teachers as well as professionals, into being graders. The essays were the product of high school students. Each grader had to anonymously grade a large number of essays, and categorize each on a scale of 1 (worst) to 9 (best).

Surprisingly (or perhaps not so surprisingly for those who have graded essays), every single essay received a wide range of disparate grades. One over third of the essays received every grade, from 1 to 9. All of the essays received at least five different grades.

But how could this be?

Diederich and team analyzed the results and found that each grader applied their own criteria when grading. While the experiment did not have graders try to explain their criteria, Diederich was able to find that most graders fell into one of five criteria categories. For example, some graders based grades largely on their perception of the quality of the essay writer’s ideas. Others based their grades on the writer’s mechanics, such as spelling and grammar. Some were mostly focused on organization. Other graders used even different criteria.

But what does it mean?

This is the hard question. Apparently, a student’s essay grades can vary widely based on who is doing the grading. This might be mitigated if the graders agree ahead of time on a “cut sheet” that assigns a number of points to various aspects of the essay. Either the cut sheet or a representation of it could be provided to the students so that they have a better idea how they will be graded and the weights the grader will assign to each criterion. Additionally, if the students and the grader work together for some time, the students will learn the grader’s “style” and adapt to it.

Perhaps this is the most important point. If the students are given enough examples of what the grader’s criteria are, their ability to adapt to that criteria will increase dramatically. If they are given fewer or no examples, the students will be more likely to view their grades as an arbitrary crap-shoot.

Source: Paul B. Diederich, Measuring Growth in English, National Council of Teachers of English, 1974 (this is the source of the facts, not my opinions).

For eight years, I taught part-time in the Electrical Engineering graduate program at Northwestern University. The MSIT program is for a professional degree, and focuses developing students’ technical and business skills. It is very selective, the students are quite good, and it was a pleasure for me to be part of it for so long.

Thinking about grades a lot lately, I’ve been data-mining eight years of raw scores to determine if there is anything to learn from them. I’m not sure that I’ll discover any new principle, but I’ll share the results as I go.

Each year my class consisted of anywhere from 20 to 33 students. Four out of the eight years I based the final grade on two exams, a midterm and a final, as well as four homeworks and class participation. The final grade was heavily based on the two exams. In 2000 I only gave two homeworks, in 2001 and 2002 I gave three homeworks, and in 2007 I assigned no graded homeworks.

Each year the class covered largely the same material. In latter years I moved faster and covered more topics in slightly less depth. Midterms and finals were similar year to year, often reusing the same questions, though most years I switched out at least 25% of each exam.

I gave model answers to all of the homeworks. I did not give model answers for the exam questions, but in latter years I did give “practice exams” that were fairly accurate views of real exams.

Most years, the exams consisted of 2-3 “mechanical” questions and 1-2 “essay” questions. Mechanical questions required solving a particular type of problem, such as IP subnetting and aggregation or determining the DNS servers involved in a query. Essay questions were open-ended, required some creativity and tended to make the student apply concepts and think “beyond the classroom” to analyze issues.

One of the first questions I had was how correlated was a student’s midterm exam grade to that student’s final exam grade? In other words, how accurate a predictor is a student’s performance on one exam of their performance on another exam? If the correlation is very high, then that would indicate that evaluating students with a single exam would be appropriate. If the correlation is low, that argues for using multiple exams to evaluate students.

Correlations per year are below:

• 2000: 0.18
• 2001: 0.77
• 2002: 0.81
• 2003: 0.32
• 2004: 0.61
• 2005: 0.65
• 2006: 0.62
• 2007: 0.72

Aside from 2000 and 2003, the correlations between midterm and final exam scores are quite high. This means that, for a significant number of students, looking at either the midterm or the final exam grade is a good predictor of how that student did on the other exam. However for a minority of students, midterm and final exam scores were quite skewed.

There could be any number of reasons for this discrepancy. The student had a bad day or didn’t get enough time to study. Or, they studied in general but missed a key topic on the exam. Or, they knew the material reasonably well, but wrote their answer poorly.

Of course there could be a degree of grader error as well, especially on the more subjective essay questions.

But what about 2000 and 2003? I may need to go back and look at the exams to see if I can shed any light on why the correlations are so low. 2000 was my first year teaching in the program and I remember giving a final that was much harder than the midterm. It is possible that the difficulty of the final threw off some of the students. But that doesn’t explain 2003. At this point I don’t have any hard facts as to why we see such small correlations in those years.

If there is anything to keep in mind as I continue this analysis is that it seems that it is more “fair” to given at least two major exams, as well as some other graded assignments. Putting all of my evaluation eggs in one basket would have changed the grades of a number of students.

For example, looking at 2002, the year with the highest correlation, if I had not given a midterm and based final letter grades just on the final exam, 15 of 30 students would have had a different final letter grade. However, the magnitude of this change would have been small - none would have changed more than half a letter grade (for example, a B to a B+ or a B+ to an A-).

This begs further analysis. Perhaps a job for tomorrow.

For years I’ve heard that the US schools are behind in teaching math, science and engineering, and here it is again.

U.S. grade school students continue to lag behind other developed countries in science and math, although fourth and eighth grade U.S. students showed steady gains in math since 1990. Only fourth graders showed gains in science compared to 1996.

But:

The U.S. is the largest, single, R&D-performing nation in the world supplying an estimated $340 billion for R&D in 2006, a record high.

One theory is that these surveys and tests report the progress of the average student, but it’s mostly the top students who go into math, science and engineering as a career. Thus, being average at teaching the average kids is ok as long as we’re doing well at teaching the kids who are interested in a science or engineering career path.

Or, there is so little motivation for top graduates of math, science and engineering universities to become teachers, that those who are best positioned to teach the material don’t. In other words, teaching salaries cannot compete with corporate and industry salaries.