Panels and testing
Fri 10 Mar 2017 by mskala Tags used: electronics, hardware, music, north-coastIt's been about a month since my last update on North Coast and my life in general, so here are some notes on how things have been going.
A few days ago, that is, just over five months from when I packed up and left my apartment in Copenhagen, I finally received the bulk of my possessions that I'd been having shipped from Sweden. The bill for transport by air freight ended up being about $4000, on goods with a declared value of $5800, and the shipping company after giving an estimate of three days delivery time sat on the shipment for a week in Sweden while they argued with my friend Steven who was organizing the shipment over whether the paperwork was in order. After assuring him that I wouldn't have to go to the port of entry clear the goods through Customs in person myself, guess what I had to spend a day travelling within Canada in order to do? And I needed my passport to identify myself for that, and it happened to be the last business day before my passport would expire, so it's good that A. they didn't delay it any later, and B. I had put off renewing my passport, because if I'd done that on time, it would still be in process and I wouldn't have had one at all.
There were little bonus issues, such as the shipping company insisting that they had to have a business name as the addressee (so Steven told them "North Coast Synthesis Ltd.") and then Canada Customs looking askance at a shipment of personal effects being addressed to a business. However, Customs accepted my explanation of how and why that had happened, and even though the paperwork looked different from what I'd been expecting, they said it was all in order. The only hard part of the clearance process was that I had to physically go to the port of entry (Hamilton Airport, which in the middle of nowhere not very near Hamilton, and does not have good bus service).
Did you know that Nordea Bank AB, which is is a Swedish company, cannot transfer money to recipients whose addresses contain the characters å or ö? For instance, those in and near the Swedish city of Malmö? Between that, and the requirement to verify transactions by receiving text messages to a Danish phone number, I found it easier to reimburse Steven for the shipping expenses by transfering money to Canada and back, instead of directly from my Danish account.
There was some damage to the goods. All boxes had clearly been handled roughly. There was external damage in the nature of crushing from things being stacked too high on top of each other, and internal damage indicating that a whole lot of shaking had gone on. Some boxes seemed to have been opened and repacked (by the shipping company, not Customs), not necessarily with due care. It may be unrealistic on this kind of shipment to expect that they won't open the boxes, but we'd been trying to avoid it, and that had factored into the decision to use this company instead of cheaper options.
My computer, which was the most important though not the most expensive item, seemed to have been unpacked, repacked upside down, and then to have been dropped on its corner in such a way that all the plastic bosses holding the front panel to the rest of the case were broken off. The plunger for the power switch was broken off inside the switch, in such a way that the computer could not boot up. The case damage doesn't seem to be reparable; what makes sense is to put that case out with the garbage, buy a new case, and transplant the entrails. I've already bought the new case and done the transplant. We're looking into whether the shipping company can be made to pay for it. Fortunately, the entrails of the computer appear to be unharmed and still usable once transplanted into the new case. The data was all on RAID and multiply backed up, but it's better not to have to make use of those measures. I'm still in the process of going through all the boxes and checking for, and photographing, the other damage.
I'm still waiting to get the paperwork on my HST and import/export account registration, which was supposed to be sent to my company's headquarters address about 15 to 20 working days after the incorporation date of January 9, then forwarded from there to me. I've checked with the tax people and they have actually already processed at least the HST registration, probably the import/export account as well. So either they didn't send the documents, or they did send the documents and the documents were not forwarded. I've been in touch with the paralegal firm who handled the filings and are supposed to be handling the mail forwarding, and they insist they have received no documents from the government. Maybe that's true. But since I have never yet received any forwarded mail from the headquarters address, I worry that maybe the mail forwarding isn't happening at all - which is a problem both because I'm paying a fair bit of money for that service, and because I don't know what other documents might be going to that address and not getting forwarded. These particular documents I can probably live without - the information is available online from the CRA. But it's a bad thing to have an official headquarters address that cannot actually be used for communication.
Soon I'm going to have to really find an accountant to go through my list of bookkeeping and tax questions. I should have done this already, and it's unfortunate that now we're in the personal income tax season, when accountants have lots of other stuff to deal with. But that's the way it worked out, partly because I wanted to wait until I had my main computer up and running.
I think I'm close to having a supplier lined up for aluminum front panels for my products. This has taken months. On the plus side I think I can say I've learned a lot about how to do business with companies in China, and that's going to be useful later, but on the other hand, I'm not happy about the time it has taken to get to this point, some of the compromises I've had to make, and the fact that it still isn't really resolved.
One thing I think I've learned is that these places run just like Dilbert's office. There are salespeople, who don't really have a clear understanding of the technology, and they will promise the customer whatever it takes to get an order before they take those promises to the engineers to find out whether keeping them is actually possible. With the Chinese companies it may also be the case that the salespeople are the only ones who understand English, so that my attempt to find out from the engineers whether they can or cannot make the product I want, has to be filtered through a translator who does not necessarily understand the information they're translating, and may skip over parts that they don't realize are critical.
My panels require basically three manufacturing processes: anodizing, machining, and printing. (Not necessarily in that order, though that is the order that's probably cheapest.) The "machining" could in principle be split into two steps (cut, and drill), but it's more likely that both would be done as a single operation on an NC mill. This is a sort of robotic Etch-A-Sketch machine with a spinning cutter on it. Now, the shape of the problem is that the companies I'm dealing with are primarily machine shops. They know exactly how to do the NC milling, that's their major business, and their salespeople can accurately quote prices on it and say what is and isn't possible in that process. But the NC milling is the easiest part of my specification. My tolerances are looser than what's the basic standard all the shops can do, the part being machined is not a complicated shape or anything, and so the salesperson can glance at the drawing and say, "Oh, yes, that'll be easy," if they only pay attention to the specifications that they understand well. Coming to an agreement on the other parts of the job is harder.
The anodizing is a little harder, but it's still well within the range of what the machine shops typically do. This is an electrochemical process that transforms a very thin layer of the surface of the metal into a hard ceramic, which looks nice and has some protective effect. Most places when you see a light grey aluminum surface that is not very shiny but still somewhat metallic-looking, it's anodized. Anodizing can also be done in colours, and you see that often on things like bicycle parts. I think most likely they would buy pre-anodized aluminum sheet (it's a standard off-the-shelf product) as input to their machining process, though that means having to be careful not to damage the surface during the machining. The other option would be to anodize the aluminum in their own shop after machining it, which is a reasonable way to do if they have the facilities. It is possible to do the anodizing last of all, after the printing, and that's ideal because it protects the printing from scratches, but doing that places extra constraints on the already-difficult printing job (special inks required and stuff) and I wouldn't demand that.
Then we come to the printing, and that's where it all gets complicated. I had my first prototypes done by Front Panel Express, in Seattle, who use a direct digital print process that can lay down variable thicknesses of ink. That allows the creation of smoothly-shaded colour gradients, and I used that capability to good effect on the prototypes. The results were perfect, and that provides a baseline: I know panels meeting my specifications can be manufactured. But FPE's price on them is too high for me to use in production. I hoped to use gradients on the production panels, even if it was somewhat expensive, as long as I could do it for significantly less than FPE charged, and I thought that ought to be possible given I was willing to order larger quantities and deal with the extra process of importing from China. If I really could not have that at an affordable price, then I'd come up with different art, but I was willing to pay more for the gradient. So what I really wanted to know was, can you do a gradient like FPE, and at what price?
The trouble is, first of all, that the direct digital print process with variable ink thickness is not commonly available. Most shops are set up to do silkscreening instead, and silkscreening cannot lay down variable thicknesses of ink. It is sort of possible to fake it using halftoning (a pattern of closely spaced dots of different sizes), just like a typical printing press on paper, but that doesn't work perfectly and it still involves pushing the silkscreen process to the limits. I knew that, and that's why I was trying to get a straight answer from each company, right at the start, on what their capabilities actually were. And instead they just wanted to tell me about their NC milling precision - which was not an interesting subject because that's the easy part.
Part of the reason they weren't eager to talk to me about the most important part of my specification turned out to be that few to none of the machine shops actually do printing themselves. When they say on their Web sites that they can offer printed products, what they really mean is that they can arrange to have subcontractors print the parts on which they themselves have done the milling. And the machine shop salespeople, who may not have a complete understanding of even their own company's capabilities for its core business of machining, have almost no clue what the printing subcontractors can or can't do. So the communication loop between me and the people who actually would be doing the printing and know what kind of printing is possible, becomes that much longer, and there's the constant temptation for the machine shop salespeople to short-circuit it and say "yes" to get an order without knowing whether the true answer is going to be "yes" or "no."
The one thing I most wanted if they couldn't meet the original spec was for them to offer suggestions on what they could do, but nobody I talked to was willing to follow such a process. I think that's another symptom of my negotiations being with a salesperson instead of an engineer: they didn't have the knowledge (or maybe not the confidence, or the authority) to make specific recommendations about the design of the product. It all had to be guessing games with proposals coming from my side: "Can you do this?" "Yes." "Really?" "Yes." "Really really?" "No." "Then what can you do?" "Something else." "Like what?" "What would you like?" I've noticed this from contractors in the English-speaking world, too; it seems making suggestions is just not how this kind of business operates, even though it seems to me to be a no-brainer that working with the customer instead of setting up oneself as an oracle, would be the way to close deals at all, and better deals.
In the last month I've often felt like I was trying to buy cheese.
That's mostly past. I now have samples from one manufacturer, and am waiting on samples from another. It seems reasonably likely that I'll be able to do business with one of them, and get a halftoned gradient print of acceptable quality at a price that can fit into my budget. And for future products, well, we'll see. I don't expect to be looking for gradient printing on most of my other products' panels - that was one reason for doing this one first, it represents a high-water mark of printing difficulty - so I'm hoping that if I can get good results on this deal, I'll also have a reliable supplier for future deals.
So now I have to think seriously about the electronics behind the panel. All this discussion is on the MSK 010 Fixed Sine Bank, the product discussed in my video streams. I'm planning to do more of those, by the way, but I have some computer upgrading to do first. My plan had been to launch the online store with the sine bank and also the MSK 008 octave switch, but now that I'm months behind my original timeline, I think I'm going to aim for having just the sine bank ready at launch time, see how that goes and hope to get some cash coming in, instead of trying to press the octave switch into production first thing as well.
I built a prototype of the sine bank and found that there were some serious issues with it, both on the mechanical construction and the electronics. I had planned to use 9mm standoffs between the circuit boards and the front panel, and after assembling the first prototype, I found that those weren't long enough. The jack sockets for connecting the cables are a little taller than I thought, and as a result, the standoffs need to be at least 10mm. That's not a huge issue - I just have to swap in some longer standoffs and make sure they fit - but it meant having to deal with suppliers again to order the longer standoffs. Just a couple days ago I got yet another installment of backordered parts from that order I placed with Newark on January 4, parts which had been in stock at that time and were only backordered because they failed to reserve the stock while they were messing around failing to verify my credit card. During that adventure I also discovered that if you mention "9mm standoffs" on Twitter, no matter what the context, your new-followers list instantly fills with gun nuts.
The first electrical issue was with the output voltage, which was too low. I'm pretty sure I know why: I had calculated that the circuit would require 2.4V Zener diodes to regulate the output voltage, on the assumption that they would turn on strongly at that voltage and pass very little current at any lower voltage. In fact, on closer reading of the data sheet for the diodes I ordered, as well as some benchtop testing, it's clear that these diodes are "weak-kneed": they start to turn on, just a little, at much lower voltage and current levels, and as a result, they limit the output voltage of the module to less than I had wanted. This is not a terribly bad thing; it just means I need higher-voltage diodes. The gentle turn-on effect at lower voltages may really be an advantage anyway, because it means the oscillators will produce less distortion. But it also means I have to test several real-life diodes on the bench to figure out which ones produce the voltage I want, because I don't have a sufficiently accurate model to calculate it. I have bought samples of higher-voltage diodes, and just need to find some time and clear some space to do the testing. Probably next week.
What seemed to be a more serious issue was crosstalk. There are eight oscillators in this module, all running at different frequencies; and we might expect them to exhibit some unusual behaviour because the frequencies are low, between one cycle per minute (17mHz, note that is not MHz) and 17Hz. The oscillators share a couple of TL074 op amp chips, four oscillators per chip, and I've had problems with crosstalk in those chips before. The data sheet advertises that they have 120dB of isolation between channels and that appears to be an outrageous lie.
So what happens is that there's a bicolour LED that flashes red-off-green-off-red for each oscillator, and on the slowest oscillators like the one-cycle-per-minute one, you can see the intensity of the light wavering, apparently picking up some of the much higher-frequency vibrations from the other oscillators. That could be a real problem if the voltage output of the oscillator is being used for synthesizer control purposes, because it could mean an audible high-frequency wavering in a tone or other parameter where you wanted something imperceptibly smooth. If severe and not easy to fix, it could be a dealbreaker for the entire module design. It was also worrying because I didn't know what would cause it.
However, it might not really be so bad. With the voltages lower than planned, it seemed possible that the LEDs themselves were much closer to the minimum voltage at which they could turn on, than was planned, and as a result they might be unusually sensitive to the crosstalk. If higher-frequency waves from the other oscillators were pushing the LED back and forth across the threshold at which it produced visible light, then it might show a lot of wavering for what was really a very small amount of crosstalk, and raising the voltage would make that less visible. In order to really evaluate how bad the problem was and have some hope of fixing it, I needed to be able to plug the module into my modular synthesizer and find out how it sounded; and I was waiting for the synthesizer to arrive from Sweden.
This week I finally got the synth and had the chance to test the sine bank module carefully, and I think it's going to be okay. When actually used to control synth parameters, the voltage outputs don't seem to have audible wavering problems; and they are as useful as I'd hoped in actual patching, so I think the basic module design is a winner. I expect the prototype will be even more fun once I deal with the voltage problem.
I also had the chance this week to put a spectrum analyser on the oscillator outputs and objectively measure just how much crosstalk there is. The answer is that crosstalk is about -42dB. Harmonic distortion is at about -22 or -25dB. These aren't great numbers, but I think they're good enough in the application.
I still don't know why there's so much crosstalk. I've gone through a number of hypotheses none of which seem to be right, and I'd feel a lot better about it if I could come up with an explanation.
I don't think it's a thermal side channel between the op amps units on a shared chip: these op amps are configured as plain noninverting amplifiers, a circuit in which the thermal sensitivity of the op amp is basically negligible. Stuff like insufficient decoupling on the chips' power inputs doesn't make sense because the frequencies are so low; I was suspecting a thermal mechanism because that's about the only thing slow enough for the observed effect. I suppose I could pursue this (and other per-chip mechanisms) further by checking whether there's much more crosstalk between oscillators on the same chip compared to oscillators not on the same chip; I haven't yet done that.
I thought it might possibly be related to the resistance of the 0V "ground" power connection, because (as is usual in modular synths, and the real and self-described experts have long arguments about this) that same net is used both for supplying power and as a voltage reference, and this module somewhat unavoidably puts all the current to and from the LEDs through that same connection. In principle, if a fast oscillator is alternately pushing and pulling its LED drive current from the 0V connection, and that connection is not well connected back to the power supply, then all the reference voltages of the other oscillators will see a bump at the frequency of the fast one, and that could cause some crosstalk.
That hypothesis gained some traction when I measured the old clip lead I'd been using for the 0V connection and found it was a little over 1ohm, which is a lot for a power supply connection. That would mean that 10mA, say, of LED drive current would be creating a 10mV bump on the 0V, which is the same order of magnitude as the crosstalk. But I tried swapping the bad clip lead with a good one, and in the other direction deliberately splicing a 10ohm resistor into the ground connection to make any high-resistance problem worse, and neither of those treatments changed the amount of crosstalk in a significant way. So that mechanism seems not to be the cause.
I think the crosstalk is at an acceptable level now, so it may not really be necessary to determine where it's coming from; but it remains that I'd prefer to know.
The harmonic distortion, as opposed to crosstalk, is from a well-understood source: the voltage-control diodes, which flatten out the peaks of the sine waves and push them more toward square waves. It's showing up primarily in the second and third harmonics, maybe just a little bit of the fifth before it drops below the general noise floor. We always hope for harmonic distortion to be as small as possible, but the present level is about what I was expecting and it doesn't need to be any better.
It's a little surprising that the second harmonic is almost as strong as the third, because this diode flattening would primarily create odd-numbered harmonics, but that can be explained by less-than-perfect matching among the individual diodes, and maybe their interactions with the LEDs (which have asymmetrical voltage response). It's not a problem. I'll be interested to see how the harmonic spectrum looks once I swap in the other diodes, at the proper voltage, instead of these first-try lower-voltage diodes.
I posted some pretty pictures from these debugging efforts in the last few days on my Twitter feed.
Further bulletins as events warrant...
4 comments
"At worst they'll think you're running a meth lab." You say that as if it wouldn't in fact be a HUGE problem. This isn't the States, but even in Canada, do you realize just how bad it *is* to give the police reasonable grounds to raid you for possibly running a meth lab? I've already got my hands full with other regulatory and insurance issues, making sure I'm not doing anything in my home office that would get me in trouble or attract the wrong kind of attention. Home anodizing doesn't fit well into my efforts to stay out of trouble; and it's not clear why I would want to do it, when it doesn't cost much to have someone else do it and then I don't have to think about whether I can achieve commercial quality reliably in my bathtub.
So, okay, I contract out the anodizing. But the anodizing isn't the hard part. Almost every place to whom I might contract out the machining, can do or arrange for the anodizing too. The real problem is the printing - and I can't do that at home to the quality level I want, even if there were no safety, regulatory, or start-up cost issues relevant to it.
Matt - 2017-03-13 12:35
Tony H. - 2017-03-13 11:58