August 18, 2005

Long Day

Posted by Scott at 10:00 PM

Debug - One of the things that makes a day seem long to me is when I spend the whole day debugging a problem and don't make significant headway. Usually once I start debugging, I get to the root within a couple of hours, but not today. I'm going to get a bit technical but hopefully not overly so. I won't describe the nature of the problem, only the things that obscure its resolution.

Almost every digital system consists of state machines. When designers code them up, they use decent labels for the various states, ie. idle, open, wait, close, synchronize, finish, cleanup, etc. In actual digital implementation these nice abstractions go away and are replaced with values. So when I look at the traces of the execution, it says the state went from 3 to 5 to 2 to 0. Oh, that's useful! With some effort I can have the tools decode these back into human readable labels, but it takes about 5 minutes to setup the translation, slowing down debug.

Later I wanted to watch the flow of data from the memory subsystem. "Back in the good ole days" memory systems were fairly straightforward. You made a request for data from Location A and almost immediately you got Result A. Looking at the traces of execution you might see: Request A, Answer A; Request B, Answer B; Request C, Answer C; etc. Nowadays for reasons of efficiency that parallel the automotive assembly line, memories can be excessively pipelined. I was looking at traces this afternoon and was seeing fetches that showed excessive amounts of what we call "split transactions" -- pipelining taken to the extreme. The requests and answers are decoupled so that you have a hard time tracking them. Effectively you see: Request A, Request B, Request C, ... and much later Answer A, Answer B, Answer C. The answers do come back in order but correlating them in your head makes your brain go pop. It's as if I asked you ten questions and you kept nodding your head saying "uh-huh". Then after the last one, you proceeded to answer them one by one in order, like Dustin Hoffman in Rainman.

Lastly I was double checking the reads and writes of some internal buffers. As I watched the sequence of addresses go by, I thought I spotted a bug. Usually when filling or reading a section of memory, you expect the addresses to increment one by one — ie. address 5, address 6, address 7, etc. For reasons which I won't go into, the addresses here need to use what are called "Gray Codes". Gray codes change only one digit at a time when sequencing. If decimal grey codes existed you might expect to see something like 17, 18, 19, 29, 20, 21. Freaky, eh? That's how I felt initially.

Without these three issues the debug would have been challenging enough to have taken all day. Obfuscation factors like these only drove me that much closer to mental meltdown. I've heard software guys complain about bad debug environments, but I've rarely seen anything that compares to what hardware guys deal with. I've grown accustomed to it, but today was particularly difficult.

Home - When I got home, I could tell it had been a long day for Michelle too. On the plus side we had a visit today from the Nix girls for most of the day. Michelle also barbecued dinner on the grill tonight. When I got home and ran upstairs to use the master bathroom, I noticed an empty mouthwash bottle on the countertop. I asked Michelle if she knew anything about it but she was as surprised as I. I saw a few drops of mouthwash on the bathroom floor and then lifted the toilet lid. The toilet water was "Scope" green! Timothy! So much for "child proof caps"! On the plus side, the toilet had a fresh minty smell, at least until I was done with it...

Thank goodness tomorrow is Friday.

Claire - As you can see, Claire also wrote an entry today. She actually scribbled it in a notebook on Monday, but left the notebook at the YMCA. She remembered to bring the notebook home Wednesday so that her thoughts could be put online.

Comments