Gregory's Serverso here's what i think happened:
1. my early version of the FPGA code had a typo that caused the BURST# line to be held low
2. this caused the DMA controller to get stuck and time out during the cache test, presumably a very early CPU test that checks for cache coherency.
3. this error is *sticky* and gets written to some nonvolatile memory (perhaps not CMOS since i couldn't clear it by pulling the battery.)
|
Top-level
 45 comments
it boots windows 3.1 now. it was trying to run a weird hdd power saving mode command I hadn't implemented. it also complains about the swap file because the filesystem is read only still. so about that write issue: it's an off-by-two error somewhere. two bytes being a single 16-bit word, so it's really an off-by-one error.  @tubetime of course it is. 🙂 Thinking it was an off by two error was off by one. 😂 figured it out and fixed it. i forget to set the "transfer request" flag to kick off DMA. in another routine, it sees that this flag is clear and assumes that a word has already been read using DMA, so it reads a crap value and then sets the transfer request flag again to start the next DMA transfer. that "crap value" pushes the valid data forward by one word. on to the next issues: randomly the ATN register mailbox flag gets set but the data in it is stale. also, the status interface register will randomly get read from by the host. I think these are two facets of the same problem: the mailbox flags sometimes respond when you access a register that they are not supposed to be monitoring! the mystery deepens. according to the logic analyzer, temp_atn_set never goes high. reg_atn_set (for crossing clock domains) is always 000. flag_atn is only set to 1 on this single line of code! and yet, somehow, it magically flips to a 1. looking at the generated logic, i see no explanation either. temp_atn_set (aka sd_cmd, my test point) never goes high. no glitches, no nothing. to set the flop, EN must be high and R must be low, and a clock edge must occur. there's a glitch! that's why I missed it before, it's only 2ns. this is the signal from the MCA bus clock domain, and it's getting picked up in my other clock domain's edge detector. and i believe this is the cause. this line right here. each signal, la_*, is an output from a flip flop latched by the micro channel bus cmd line. however, this line of code creates some combinational logic--there's a timing hazard here... the problem? the line (la_addr == REG_ATN) creates a bunch of gates that are slightly slower than the simple AND gates in the previous part of the line. so la_mca_op=1, ~la_s0_w_l=1, and (la_addr == REG_ATN) *is also a 1 for a very short time!!!* this is because the previous value of la_addr WAS a REG_ATN. what i need to do is take that entire wire and turn it into a latch (a reg) and clock it on cmd. so here's the solution: all the signals in the MCA bus domain go to a latch clocked in that domain (the first "always" block). then *without any combinational logic* the output of that latch goes *directly* to another latch (the second "always" block) located in the main clock domain. (i have another flip flop in main clock domain just for detecting the edge) next step is to optimize the interface speed. right now it takes 25us to read a sector from the SD card but ~5 milliseconds (ouch) to DMA it to the PC! it's mostly an issue with the Teensy-to-FPGA interface, which is async and simple: 4 address lines, 16 data lines, a read control line, and a write control line. everything else is done as a register in the 4-bit address space. flag register for status and mailbox sync bits. why is it 5ms per sector for DMA? well, it's about 20us per word. most of that time is wasted by the slow interface between the Teensy and the FPGA. I really should fix that. got rid of some delays and now we're down to 6us per word. but there are some unexpected wide gaps in between transfers between the FPGA and the Teensy. ahh that must be the issue, when i change from a read to a write, i have to set the port direction for 16 IO pins. pinMode is uhh not quick, so let's try writing to the port direction register directly. so how does this compare with the real drive? it transfers a word in about 1.6us (compared to 2us for mine), so it is slightly faster for sustained data transfer with data that fits in the internal buffer. however, seek times (in the ms range) more or less cancel this out. here's a DMA cycle for the real hard drive. it's using the bus more efficiently, requesting a new transfer as soon as the transfer completes. my design waits for the full DMA cycle to end before requesting a new one. maybe theoretical maximum transfer rates would make the comparison easier: if i put in a bunch of work and implemented burst mode DMA transfer, i might be able to hit 4.5MB/s. no more debugging wires. the DBA-ESDI drive is now running stand-alone!  I printed an angled carrier. the angle is because the socketed Teensy is really tall. time to lay out a version of this board for other PS/2 models, like the 50Z. this is the somewhat less uncommon 72-pin DBA-ESDI form factor. @tubetime Nice! (And LOLing because I JUST asked a client whether they really need the big teensy or could they go with the smaller one.)  @tubetime Reading the silk screen of your boards is always so much fun! 😂 looks like there's a rare problem when reading sectors, around 1 in 100,000 reads. ugh these can be hard to track down i thought it might be a timing issue caused by disabling USB debugging, but it still happens (with roughly the same statistics) with USB debugging turned on. looks like some individual data bits are getting flipped. what's really odd is that it is repeatable and it seems to be the same bits at the same addresses. also -- there seem to be two overlapping bugs. one causes a few bits to be corrupted for single sector reads, the other causes an entire sector's data to be shifted by several bytes.  huh, once I connect the logic analyzer, the problem takes much longer to reproduce; the probability of data corruption drops dramatically. interesting but it makes things much harder to debug. @tubetime hey @azonenberg weren’t you looking for examples of probes changing behavior of the DUT? looks like it is skipping a transfer here somehow. it transfers B66D, then it skips 6DDB and goes to DBB6. the preempt line seems to deassert right before arb/gnt goes high. that doesn't seem right.  @tubetime A coincidence that DBB6 == 6DDB << 1 ? Could it be slipping a bit? @jond no that is just the pattern in the disk image. presumably a disk area that was never written to after being formatted and still contains this bit pattern. the smoking gun: an invalid bus transfer. both rd and wr are asserted at the same time. before that we see the missing data on the bus: 6DDB. so the card thinks it is in a DMA cycle even though it is not, and it's putting data on the bus when it shouldn't be! i deleted 3 characters from my verilog and it works now. 😅 (i still need to test writes, and i need to test it without the logic analyzer card) @tubetime As the joke goes, its all about knowing where to tap...  @tubetime Ugh, that's NEVER good. I see lots of oscilloscope probing in your future. @tubetime Why is there a SD Card slot forseen on the baseboard as well? @1000millimeter in case i want to use a soft CPU core in the FPGA instead of a separate Teensy @tubetime This was really enjoyable to follow along with! Congrats on the progress |
this is all very good because i know the root cause and it's not something terrible like data bus contention, and it's thankfully not permanent damage.