This blog is dedicated to open, interoperable manufacturing software and the coolest, latest and greatest things I see every day while conducting business under the banner of Inductive Automation.

Hello, my name is Steve Hechtman and I am president of Inductive Automation. During the span of one day there is more excitement, more discovery than I can possibly keep to myself. This blog is, therefore, my outlet. WARNING: This site is highly biased in favor of the most powerful, affordable manufacturing software in the world - Ignition by Inductive Automation!

Ignition Playback Templates

The other day I was wondering if I could create SCADA "playback" using standard Ignition components. I decided to try by using Ignition templates to create a "playback controller" and "playback container"  so that a controller could be dropped into a project and containers be bound to it.  Each container would contain a single SCADA object such as a gauge, tank, readout or whatever.

This turned out to be a really easy project (with a little help from tech support).  First, history needs to be turned on for each tag to be played back.  It just takes a check mark on the tag.  This can be done en mass with the multi-selection of tags.  Dropping the controller into a project gives a bindable property called Date_Time".  This property provides the date and time to display and increments forward from the selected start time at the playback speed selected.  That's all the controller does - specify time (selecting "RT" means realtime, gives the improbable year of 1900 which tells the containers to display realtime directly from the tag value.  I only wanted a single property to bind to).  The "playback container" uses the date and time from this property to query the tag historian for a value.

The "playback container" has some display objects on it which exist only for demo purposes.  Make a copy of this template, delete all the display objects, and add your own display object. Bind your display object's value to the container's "Value" property.  Now drop your new container object into the project as many times as you want.  For each container fill in the "Tag" property with the tagname. If the default database for your project is different from your History Tag Provider, add the name of your history provider in front of the tagname.  For example, [MyHist]N7:0.  Otherwise, just the tagname will work.  Then bind the container's "Controller" property to the "Date_Time" property of the controller.  That's it.

Selecting a date/ time out in the future will give the current values as they are logged because the query is configured to select the "Closest Value" (and the current value is the closest to any future value).

I exported these templates to a .proj and you can download them from here.  I built them in Ignition version 7.6.3.

DISCLAIMER:  These templates should not be considered "production ready" nor should they be considered a product of Inductive Automation.  They were just a "proof of concept" and I was just "an integrator having fun messing around."  It didn't take long.  Just dissect them and see how I did it. There are probably many other ways to accomplish the same thing.

When XP support ends next year, Who ya gonna call?

It was just brought to my attention by one of our customers that XP support will end next year.  That really pricked up my ears because tens of thousands of HMI, SCADA and MES systems (if not more) will require upgrading.

Now the shameless self-promotion part (sorry, I can't help it)...  When we say Ignition is cross-platform we mean it also runs on most versions of Windows right out of the box.  Usually people think we're just talking talking about Linux, Unix and OSX, but we also mean Windows.  Just saying.

Guerrilla Troubleshooting Tactics

I created a training video over ten years ago called Guerrilla Troubleshooting Tactics which simplified and codified successful troubleshooting patterns.  I recently realized it has broad applicability in other areas of our business here at Inductive Automation so I thought I would share the key points of it with you.

Troubleshooting procedure is very distinct from the technical knowledge of something.  Anyone can follow this procedure up until the last step, even if the're not technical, and troubleshoot successfully.  Sometimes it's just asking the right questions of someone who does have the technical knowledge.

The following simple questions are arranged in the order given so as to pick the low hanging fruit first.  A frequent mistake is to dive into too much detail with too little information too early. Here are the steps:

Diagnostic Sequence

1)  What is the problem?
2)  How is it supposed to work?
3)  Did it ever work?
4)  What changed?
5)  What else is affected?
6)  Has it ever happened before?
7)  Why do the affected outputs act this way?

A little explanation about these steps is in order.  Remember, the order of these steps is one of  trying the easy things first.  The problem could be, and many times is, resolved on step one.   The procedure shouldn't take very long - sometimes only minutes.

First Step
The first step is to get a clear description of the problem - obvious, right?  Yet, I have personally made the mistake of getting only a partial description to a problem and then taking off on a tangent (half cocked as they say), only to later have to back up and restart.  Take your time and get the whole description.  Another point is this... if they say there is a problem, there is a problem.  Sometimes troubleshooters "blow off" the user as "using it wrong" or being "nuts" or something.  But the troubleshooter should always assume full responsibility for the system and the person using it.  It's an end to end approach.  If the problem is how the user is using it, then this should become the target of the successful troubleshooter - education of the user.  Sometimes when doing this you discover something really is wrong with the system and not the user, which is a bit embarrassing, though productive.

Second Step
You'd better know how something is supposed to work before you attempt to "fix it."  It's really hard to fix something that's operating normally!  Sometimes the design is so poor you assume it would never work that way, but that would be a bad assumption on your part.  How would you get this information?  Ask the user and verify it with a manual.  You can Google a manual or other information on practically anything these days. You don't sit there and read the whole manual.  But get good at scanning quickly and picking out the relevant info.  Sometimes asking the user is sufficient because they most familiar with the thing (they are the user aren't they?)  If that leads you astray or into confusion then resort to the manual or use your own common sense.  This step is vital since troubleshooting is done by comparing how something is... to how it should be.

Third Step
When working with "one-off" systems the step, "did it ever work?", is vital or you'll waste massive amounts of time. Your approach will be totally different if it's never worked versus the case where it did work and then something changed and it stopped working.  The first case means the development-debugging process was never completed.  That's an engineering concern and the reason it doesn't work could be one or many reasons or even owe to a fundamentally flawed design.  In most cases that's where the troubleshooting ends and it goes back to the designers, or if not, at least you know what you're up against.  But if you can ascertain that it did work at some point then you've got something you can work with.  Now you've got certainty and a starting point.

Fourth Step
When you ask "what changed?" the response is often "Nothing."  But here is your stable datum - something changed - or else it would still be working.  So you might have to ask "when it was working fine?", and then ask "when did it stop working?"  Getting them to place it in time will often get them to realize what it was.  It will be something like "oh yeah, we changed such and such."  This is usually the point where the light bulb goes off and it will lead you right to the solution.  There is one caveat though.  Before you arrived on the scene someone else might have been troubleshooting it, and if they used a shotgun approach they could have caused new problems in addition to the original one.  They might have replaced parts with whatever was on hand which are the wrong ones.  They might tweaked and adjusted things out of desperation and thrown things out of alignment.  So you have to ask about what steps were taken exactly before you arrived, and in many cases you will need put every back to the original state before starting your troubleshooting.

Fifth Step
Usually the problem will resolve on step four.  But with particularly stubborn problems it's time to step back and look at the big picture.  Look for other things that depart from what they should be.  In step two you determined how it's supposed to work, so now try to find anything else that's affected adversely.  Things like, "oh yeah, memory use is also high", or "this component is also unusually hot."  Once you determine the other things that are affected, try to determine what the common denominator is.  That might lead you to the problem.  But be aware if there are multiple problems you could get really confused doing this step.  If that happens, factor into your thinking the multiple problem possibility and see if that helps.  If not, then go to the next step.

Sixth Step
Has it ever happened before?  Some people think this should be the first step and sometimes that works. But I have found that more often than not it can lead you astray if you haven't done the other steps first.  These steps are designed to lead toward greater understanding, whereas if you do step six first it's just rote procedure.  The point of the diagnostic sequence is to proceed methodically and with certainty toward problem resolution.  Doing step six first not only misses the root cause but also can lead to shotgun troubleshooting (a bad thing).  But if you've done the preceding steps already, you can now safely ask for previous similar problems and their resolution or look into the maintenance logs.  You can also call the manufacturer for help.  If they are any good they are going to ask you  steps one to five anyway.  Steps one to five put step six into the proper context.

Seventh Step
Why do the affected outputs act this way?  Malfunctions will usually show up as misbehaving outputs. Reports will have bad numbers.  Motors won't start.  Screens won't display.  Up until now any lay person with little or no technical knowledge could perform the diagnostic sequence and win in troubleshooting.  But on this step you need the technical background.  Fortunately, most problems resolve before this step.  But even without a technical background the lay person can direct the technician on this step to success.
The technician on this step traces logic, wiring, hydraulics, pneumatics, or  whatever the system consists of, back from a defective output to the exact thing that is causing the malfunction.  When there are multiple problem causes this step is usually the only one that works.  In this case it is an iterative process (trace one problem, trace the next, etc.).  As mentioned before, multiple problems can exist due to prior shotgun troubleshooting being done, but they always exist during the development-debug phase of any product or project. During this development cycle there are often hundreds or thousands of problems to rectify.  That's why you ask if it ever worked as in step three.  Sometimes projects are left unfinished or a few bugs slip by.

There are a few other factors to keep in mind.  If the system ever worked then odds are there is only single problem to find.  This procedure will help you find it quickly.  When things start getting complicated you should suspect multiple problem causes are present.  Start asking about prior troubleshooting attempts.  Another approach is to back up a step or two.  Maybe you missed something.

Another thing to know is that users will tell you all the things they have tried.  I recommend you verify everything for yourself and not trust their narratives.  I've been led astray by before I.e. "I tested all the fuses and they're all good" only to discover half an hour later that one of them is blown (they didn't know how to properly test fuses).  Only by moving forward methodically and gaining your own certainty of things can you conquer the problems.  Sometimes you need to be tactful and say "Please don't be offended if I recheck a few things. It's part of the procedure I use."  People are never offended, they're just glad you're there to help.

Realize that when you come onto the scene, people are really confused, or else they would have solved it already. Don't let their confusions become yours - rely on your procedure.  They will tell you all types of things. They will even tell you they have tried everything and try to tell you why it can't be fixed.  They are just trying to justify why they couldn't solve it  - they are usually embarrassed - so be kind.   Take their data (you need it) but don't take their recommendations - trust only your procedure and win!

Sequential Function Charts in Ignition

You say, "sequential function charts, who uses those?"  Turns out lots of people do.  Sequential Function Charts (SFCs) are part of the IEC61131-3 programming languages.  One usually thinks of SFCs as a PLC programming language, which is true, but SFCs are also used in industrial software applications quite extensively.

Personally, I seldom used SFCs in PLC programs.  I felt much more comfortable using ladder logic.  But customer feedback from a variety of industries shows the extensive use of SFCs in specialized application programs.  In fact, one customer calls them "everything a PLC isn't", referring to the fact that they excel in highly supervisory roles with long running processes requiring advanced logic handling capability.  

There is another aspect of SFCs which doesn't exist in the PLC version (at least not to my knowledge) and that is instantiation.  What this means is when you develop SFC diagrams you are actually developing templates, but when you run them you actually run a copy, or instance, of the template. Think of the uses.  SFCs could act like paper tracking and routing forms on the plant floor except with SFCs everything is kept in a database.  

Here's an example in the manufacture of widgets.  For every widget you plan to build you instantiate a new SFC and pass it parameters such as work order number, part number, etc.  The SFC instructions could handle every aspect of the widget's subsequent routing through the manufacturing process and would write all the information off to a database.  

Instantiated SFCs could track batches in a chemical process, discrete parts, parts assemblies, practically anything.  They could even handle the routing and tracking of front office paperwork.  Paper forms could be replaced with electronic versions which are automatically routed using instantiated SFCs. The instantiation is the form, so using actual paper forms is unnecessary.  Save a tree!

Think of SFCs as script supervisors.  Higher level SFCs determine which scripts run when. They also provide a bird's eye view of your program.  Everything you do with SFCs could be done with scripts alone, but you would quickly get lost in the details.  Additionally, SFCs can provide real-time status of scripts which are running and what conditions are required to run non-running scripts.

I'm excited about adding SFCs to Ignition because they add a whole new dimension to what you can do.  I won't give you a time frame for when they'll be done, but I can tell you the development team is well under way now.  

Why keep Reinventing the Wheel?

New software products hit the market every single week.  Most fulfill specialized purposes and it seems like most are targeted at the Oil & Gas industry.  But it makes me wonder, how many of these packages are developed from the ground up?

We talk about "the platform" all the time.  A platform is sort of like an operating system but higher up the software stack.  Why would you recreate deployment , development, security, licensing, connectivity , authentication, management and monitoring logic repeatedly when all of this could be part of a platform?  For example, with our product, all the non-specialized stuff is handled by the platform while "modules" handle the specialized part.  This allows module developers to concentrate on solving the problem and not the mundane stuff that's already been done before.

Let's look at what the Ignition platform actually does...  It is an OPU-UA Client.  It's really smart about databases in that it knows how to talk to a wide variety of them even with their subtle (or not so subtle) differences.  It optimizes database requests, handles database failover logic, handles store and forward logic, and far more.  It is also a web server and handles all the complicated details of client deployment, client communication, application synchronization and more.  It handles all the details of the unified development environment (yes, one environment for developing/ configuring any module installed in the platform).  It handles redundancy, system logging, auditing, alarming, security, authentication, licensing, the scripting engine, and if I listed everything else this post would be pages long.  The point is, 98% of your job is done when you leverage the platform.  And the point is, you know 98% will work as advertised so you can concentrate on the real problem you are trying to solve.

Most of the systems that are developed from the ground up are doomed to fail.  This is because they reinvent the wheel and with that comes a very long runway and an overwhelming amount of shaking out and fixing.  Most users aren't that patient so the undertaking fails.

SECS/GEM Driver now supports SECS-I

The newly released SECS/GEM driver for the semiconductor industry has been improved!  It now supports SECS-I (serial).  The first release only supported HSMS (Ethernet) messaging, but as it turns out, there is still a lot of legacy equipment out there and most of it only supports SECS-I communications.  You would think this to be a simple thing to implement but it wasn't that easy.  

TCP/IP has a lot of stuff built into it that gets leveraged for HSMS but when you get to SECS-I serial you lose that and you've got to build a lot of stuff from the ground up.  But that's been done now and the new SECS/GEM module can now be downloaded from our module marketplace.

We are in the process of adding simulator functionality to the SECS/GEM module as well, so that testing and development can be done in the absence of actual equipment and so that equipment definition files can be tested.  

The SECS/GEM module plugs right into the Ignition Gateway to provide seamless connectivity between tools, databases, GUI front-ends, ERP systems and more.

News Flash re OPC-UA

My recent post "Imagine OPC-UA Everywhere" has been one of the most active posts ever which indicates the high degree of interest in UA.  In that post I said once hardware providers start integrating OPC-UA into their products it will be a beautiful world, but I also said there is little motivation for them to do so.  Well the floodgates might be opening up sooner than I thought.  Invensys just made the following announcement...
"Invensys has also embedded OPC UA communications with its Triconex Trident and Triconex General Purpose safety instrumented systems. OPC UA maximizes interoperability between systems and streamlines connectivity through open platform architecture and future-proof design. The new communications interface module contains an embedded OPC UA server that supports up to 10 concurrent clients, delivering high performance and secure, reliable communication of real-time data, alarms and historical events."

The announcement just came out today.  The way I see it this is a major development in the forward progress of OPC-UA.  Who's next?

New SECS GEM Driver Availability

I've been working on a special project to develop a SECS driver for some time and it's almost here.  We'll be entering the beta test period soon.  If you aren't familiar with SECS it refers to Semiconductor Equipment Communications Standard.  Or from Wikipedia:

"The SECS/GEM is the semiconductor's equipment interface protocol for equipment-to-host data communications. In an automated fab, the interface can start and stop equipment processing, collect measurement data, change variables and select recipes for products. The SECS (SEMI Equipment Communications Standard)/GEM (Generic Equipment Model) standards do all this in a defined way."

SECS/ GEM Ignition Module
The GEM/SECS Module can stand alone in the Ignition Server for use with third-party applications or work seamlessly with other Ignition modules and projects to create rich applications. Third-party applications interface with equipment nicely by use of SQL database tables as the interface mechanism.

By use of a simple definition file any SECS message (including custom ones) can be supported.  Messages can be sent to equipment simply by  inserting a row into the NewMessage table.  Responses are stored in the Message table.  

Handling Complex Data

SECS messages can contain complex data structures in that they can have any number of items within lists and any number of lists within lists. For this reason SECS messages are serialized into JSON (JavaScript Object Notation, in this case it has nothing to do with Javascript) strings before they are inserted into the database. JSON is a human readable data-interchange format. JSON is well supported in most languages which is why it was selected to represent complex data.  Within Ignition JSON structures can be encoded/decoded to/from native structures like lists by using built-in scripting commands.

Handling Real-Time Data

Some SECS messages provide for streaming real-time data.  By configuring the message definition file, data can be marked for real-time usage so that equipment responses not only enter the message table but also the real-time table.  Rows only get inserted once and get updated when new data arrives.  So there is only one row for each monitored value (or complex data structure) and it always has the most recent data .  This way the real-time table never grows beyond the number of monitored values and keeps the table size manageable and query times low.  Eventually the real-time table will be reflected out into Ignition SQLTags for drag and drop ease of use.

This driver should be generally available the second quarter of 2013.

Controller with integrated OPC-UA

I said I hoped I was wrong about PLC vendors not supporting integrated OPC-UA. That was in my last post and as it turns out, to some extent I was wrong, which is good.  Matthias Damm pointed out that at least three PLC vendors are shipping with integrated OPC-UA interfaces, including Beckoff, Bosch Rexroth and B&R.

I did a little searching on Beckhoff, Bosch Rexroth and B&R to find out more about these controllers and found the Beckoff CX8091 controller right away but came away empty handed for the latter two.  I'm hoping someone can help point me in the right direction on those two.

According to their website the Beckhoff controller is estimated for market release in the 2nd quarter of 2013.  It is programmed using IEC 61131-3 languages and most importantly supports online programming changes.  As you can see, they have the OPC-UA logo in the lower middle of their product.

Another poster pointed out that SAP and iTAC (who is iTAC anyway?) have already implemented OPC-UA.  As I pointed out in my last post, it's not the software companies that are the problem.   Most pure software companies have already adopted.  It's the hardware companies that are lagging.

Kudos to the software companies who are now on-board, and kudos to Beckhoff for their new controller, but as I search the web for other hardware, particularly PLCs, with integrated OPC-UA, it is kind of spooky how empty the space is.  So if Bosch Rexroth and B&R have it then why can't I find it with a web search?  Are they just not promoting it?  Someone please shed some light on this.

In my last post I pointed out three firms that provide SDKs for embedded OPC-UA.  Matthias points out another provider is Unified Automation which provides a full range of OPC-UA SDKs and development tools for ANSI C, C++, Java and .NET.

All the tools are there, all that's missing is the motivation.

Imagine OPC-UA Everywhere

I did a little fact checking to find out if there are any PLC controllers available with OPC-UA embedded.  As far as I can tell, none are available yet.  I found a page for the OPC Technology Summit 2012 which has downloadable presentations from major automation vendors like Rockwell, Siemens, and ABB.  Each laid out their OPC-UA adoption roadmap.

How is it these presentations glow about the wonderful virtues of OPC-UA, yet you can't find a PLC with OPC-UA embedded?  (Please tell me if I'm wrong, I hope I am.) There's certainly no technological barrier preventing it. There are many OPC-UA development tools available from companies like Prosys OPC and OPC Labs. Embedded Labs even provides OPC-UA on a chip that costs less than five dollars.

Adoption Roadmap
But, there is a glimmer of hope. Each of these manufacturers has put embedded OPC-UA on their adoption roadmap. They say they are going to do it, but not one has said when. This is especially interesting when you consider Inductive Automation and most other pure software companies have had OPC-UA for years now. Works great.

So what's so special about OPC-UA? Secure, simple, cross platform, flexible, standardized.  Enough said?

Imagine OPC-UA Everywhere
Imagine a world of 100% OPC-UA.  Gets me all excited because it would take the pain out the subject of intercommunication, between devices, between enterprise applications, and between devices and enterprise applications. For example, from the Ignition server running right here on my desktop I can connect to any number of other Ignition servers running on other people's machines in just minutes. And I can connect to Kepware servers running on other machines with similar ease. Now if I only had a few OPC-UA-enabled PLCs to connect to, the world would be beautiful. I won't hold my breath.

Sadly, when I mention this beautiful vision people usually roll their eyes and sigh a wistful sigh. (Try it sometime, it's true.) They know it should be but there's a tacit reality that even though it would be right for end-users and integrators, there's little incentive for PLC vendors to be more open.

Security as an Incentive
However, there's a good reason for hardware vendors to adopt OPC-UA that has nothing to do with openness. It is the one thing they are being beaten up about and that's security. OPC-UA is very, very secure and that solves a lot of their problems. It has built-in encryption and can use certificates to ensure end-points are who they say they are. We wrote our OPC-UA client and server from the ground up (from the specification) which was no easy feat mainly because of all the security involved. And speaking of that, anyone who attempts to create proprietary security schemes based on secrecy is going to be the least secure of all. Encryption and security schemes that are exposed to general public scrutiny (like the OPC-UA specification) will be tested and vulnerabilities exposed so they can be fixed.

I wrote this post because I thought it would be fun to imagine what it would be like if the anarchy of endless communication protocols was gone and we could experience the simple, secure, and standardized beauty of OPC-UA to communicate between all devices and software applications.