General test info Mobile Info Links Embedded Links Twitter Other
How to break software: mobile and embedded device edition (finding the bugs that are already there)
site is provided based on Jon's book: Software Test Attacks to Break Mobile and
Embedded Devices (Sept 16, 2013 CRC press)
The industry is putting software everywhere and into everything. The really “hot” area of software these days is the mobile-smart phone world and embedding software in nearly every product made. It seems like every one has or will have an app for a smart phone. Soon smart phones will be in the hands of almost every human (all 7 billion of us). Further, there is software in our cars, TVs, factories, heating systems, light switches, and almost every electrically powered new device. Mobile and embedded software is the art of the continuum, which started 60 years ago with big main frame computers, moved to smaller computers, then along came desktop or personal computers PCs. And then, we moved to the internet or World Wide Web, and now, we’re into the Cloud. There are computers that fit into our pockets (smart phones) and there are devices that the user may not be aware that software is actually providing the major functionality of the device.
Because this is a continuum, many of the considerations that apply to IT software apply to the mobile and embedded domains, but the mobile and embedded software devices often introduce some new qualities, which traditional programmers may misunderstand. Software is about communication. The programmer communicates design to the computer. Hardware and software interact. The computer communicates to the user. The user wishes to communicate to or with other pieces of information in cyberspace. The misunderstanding in this communication stream results in what many people call a bug (i.e., a negative or undesired quality). This communication stream can also have many other qualities--positive and negative, which various interested parties (stakeholders) may wish to know about.
Software testing in its various dimensions such as verification, validation, checking, and evaluation, seeks to provide information about these qualities of software and the associated system. The different stakeholders have many views of the quality of software. As Drucker and Weinberg note, good quality is value someone is willing to pay for. Negative qualities imply negative value (something somebody does not want). Prime negative qualities most users of software want to avoid are bugs. This site and associated book are devoted to testing to find common bugs that occur over and over in the mobile and embedded software domain. We will note ways to provide other useful information on qualities, but since many users expect a device to work, we focus on ways to find bugs in the service of reducing software faults and failures.
At this point, I note that software will always have bugs and that much of the literature and information about software testing remains applicable to the domain of mobile and embedded. The concept of attack-based testing, trying to “break” the software, is about viewpoint, since by the time we are attacking the software, it is already broken and we are just trying to expose the bug. A person looking to test software should first be familiar with general testing concepts. This means you should be familiar with software testing principles (see Lee Copeland's A Practioner's Guide to Software Test Design and ISO 29119). My book, “Software Test Attacks to Break Mobile and Embedded Devices” and this site provide the next level of detail and information specific to “how to break software.” What these attacks and the test techniques attempt to do is provide information about the bug’s existence so that it can be fixed in the name of improving the software’s quality. It isn’t possible to test quality in to a software device. However, testers do provide valuable information about bugs and other qualities that decision makers and stakeholders (management, users, developers, testers, etc.) may find useful.
I take the view in my system testing, that a tester should attempt to demonstrate that the software does not work (i.e., the software fails to meet requirements, user needs, functional needs, and/or has non-functional characteristics). Many of the attacks in the book should be run at the system level where hardware, software, and operations are all at play. As part of a comprehensive set of test activities, we hope testers will use attacks and techniques to find bugs. We have found that all too often testers only cover “happy path” or "green light testing," providing a false sense of security to stakeholders. Since software can now kill people or has large impacts on companies and people, this viewpoint of attempting to break the software can be a good idea, and when we as testers “fail” to break the software or find a bug, we can provide a little more confidence that the users will find the software “working.” However, we don’t want to mislead anyone into thinking that complete testing is possible. There will always be bugs in software.
Short Bio: Jon Hagar is the principal (CEO/CTO) and senior software test engineer at Grand Software Testing. For over 30 years he has worked on systems and software engineering, specializing in testing/verification and validation. He is the lead editor on ISO/IEC/IEEE 29119 Software Test Standard, a member of the IEEE 1012 V&V Plan working group, and co-chair on the OMG UML testing profile standard. Jon holds a patent on web test technologies and has published numerous articles on software reliability, testing, test tools, formal methods, and embedded systems. He has an M.S. in computer science with a specialization in software engineering and testing from Colorado State University and a B.S. in mathematics with a specialization in civil engineering and software from Metropolitan State College in Denver, Colorado.
—30 + years developing, testing, V&V, manager, exploring, thinking, systems engineering, hardware, and support areas
—Specializing in: Mobile, Embedded, Automation
—Worked in: Agile, Confusion, Rockets, Critical Systems, Real-time—Standards: ISO 29119 Software Testing (project editor), IEEE 1012, OMG UML Test Profile (co-chair)
—Related web sites