Follow Us.
Become a Fan! 
Today wasn't looking good. But then neither was any day that had started with a phone call from the Shift Supervisor at 3:12 in the morning.
"I don't know what's happened," an all too familiar voice on the end of the line said. "We lost power about 45 minutes ago, and now that the power's back, we can't get Line 7 up."
Lucky line 7, Jim thought. This numbskull Supervisor was sure able to get him up.
"I'll be right in." Jim said. He knew that if a problem was important enough to warrant a call to the Plant Manager at this time in the morning, it wasn't going to go away anytime soon. Besides, it was only an hour and a half before he would have to wake up anyway.
That was only four hours ago, but it seemed a lot longer. Jim Davis had inherited this job just over 6 months ago, and its luster had certainly begun to wear off. Now, Jim sat staring at his desk. Or, rather, at what was on his desk. It seemed, according to these reports, that Line 12 hadn't been running quite right for the last month. It was almost as if the software change made to the HMI to handle the new thickness gauges was working on all of the lines but Line 12. A scary thought was slowly taking shape in the back of his mind - what if the changes made to the other 18 lines didn't actually make it to Line 12? The only way to find out was to walk out there and check the 'second' HMI screen. He knew that the new software included an extra temperature field on that screen, and it was the only way that he could really tell them apart. Before he could get up, Carlton Brady came through his door.
"One of the fork lift drivers just had an accident." Without another word, he turned and left. Jim got up and followed.
How ironic, thought Jim, as he looked at the PC that only moments before had been running line 12. There was no way now that he would be able to check that second HMI screen. Or any other screens, for that matter. Fortunately, the driver and the lift were OK, but the PC was definitely toast.
"Do we have a backup?" he asked, hopefully.
"We do, but not one with Windows NT on it. The backup PC's still have 3.11," Carlton said. "And Line 7's PC still won't boot. It looks like we may have to reinstall NT. That happens sometimes when you turn them off without shutting them down. I told you we needed back up power on all of our PC's."
Carlton seemed to enjoy it when there were problems. Jim didn't. Especially not today. Now he had two of his 19 lines down, with no assurance that they would be back up before he was supposed to have dinner with Mr. Earhart, the plant owner. Maybe there was still time to get some horrible stomach bug.
It was now a couple of hours past lunch. At least Rick Gilbert had been in town today. He was the best consultant that Simple Solutions had, and if anybody could fix his problems, Rick could. Yeah, it would cost a bunch, but maybe there was a chance the situation would look better tonight. Duo Plastics was Mr. Earhart's baby, and dinner would be a miserable experience if the plant wasn't running smoothly. While he was thinking of the best way to explain what had happened to almost eleven percent of the factory's output, Rick came in.
"I've got one of those good news, bad news stories," he said. "The good news is that I was able to get both lines up. It took awhile, but, fortunately, I had good backups."
Rick just stood there, as if he wanted some type of reward other than the $145 an hour that he was getting.
"And the bad news?"
"The bad news is that all of the PC's in the plant have a virus. I think it came from the game software that I found loaded over on Line 3. You really should install some type of locking enclosure on all of your PC's."
Line 3 was well hidden, and this kind of thing had happened before.
"What I am going to have to do," Rick continued, "is to take them offline one at a time and clean them up. It shouldn't take more than fifteen minutes per PC."
Fifteen minutes per line for the PC, Jim thought, and then thirty minutes more to get each line up and running. Jim started feeling that horrible stomach bug coming on. And, funny thing was, he didn't think that he was imagining it.
How much had this day cost? Well, there was the obvious cost of the nine hours for the consultant. Much worse, however, was the cost of lost production. Line 7 was down for eight and one half-hours, and Line 12 was down for five. And just how much had the insidious problem of the wrong HMI version cost in wasted product for the last month on Line 12? And, even more worrisome, what if some of the product that they thought was good had actually been out of spec? They wouldn't know about it for another week or so, and, by then, it may have cost them a customer.
Additionally, now there was the cost of adding a UPS and a locking enclosure to each line. That included a small cost for equipment, but it meant that all of the lines would have to go down again to put in the modifications. But, there was an even bigger cost - Jim was thinking seriously that accepting another job across town would be better for his health.
Jim liked getting to the plant early. His day wouldn't really start until 7:30, so the half-hour before then was usually all his.
Looking at the reports on his desk, it was obvious that the installation of the new thickness gauges had been a good idea. All 19 lines were now producing product that was consistently within spec, and rejects and start-up scrap had been reduced to basically nothing. Jim was just about to take a walk around the plant to make sure that all of the lines were running after that power outage that he heard they had last night when Carlton Brady came through his door.
"One of the fork lift drivers just had an accident." Without another word, he turned and left. Jim got up and followed.
What a mess, thought Jim, as he looked at the ThinTerm that only moments before had been running line 12. Well, he didn't know about the other lines, but at least it seemed that Line 12 was running OK, even though he couldn't see the HMI system's output. The line was still being controlled by the application running on the server somewhere inside the IT department. Fortunately, the driver and the lift were OK, but the ThinTerm was definitely toast.
"Do we have a backup client?" he asked, hopefully.
As he was asking the question, an operator came over with another ThinTerm box. Luckily, the monitor had escaped with little more than some aesthetic damage, and Jim watched as the operator plugged in the new ThinTerm and powered it on. Within a minute, it was back up running the same screen that its predecessor had been running before the accident. And now he could confirm his hunch - the line was still running within the proper control limits.
"What about the other lines - did they all come back OK after the power outage?"
"Yeah, sure," Carlton said. "They were all back up by about 3:00 this morning."
Jim was starting to look forward to dinner tonight with Mr. Earhart.
For more information on ACP Industrial Thin Client computers, please visit our web site at http://www.thinmanager.com
To sign up for the E-mail newsletter go here: ACP newsletter signup
For an archive of past newsletter articles go to: ACP Newsletter Archive
Copyright © Automation Control Products, Inc. All rights reserved.
