Plan for Unexpected Complexity or Be Overwhelmed by It

complexity
The world is incredibly complex, and nowhere is this more evident than the art of writing software. Software is the result of intricate interactions between social structures, business desires, and limited knowledge. The complexity of those interactions inevitably shows up in code structure. Our job as engineers is to manage this, and prepare our abstractions to handle increasing complexity over time. The most important technique in this regard is to make space for pieces of logic to take on more complexity without drowning in it – with complexity, the dose makes the poison.
This is why one of the most common pieces of software writing advice that you’ll hear is to write lots of small, focused classes with logic split up into lots of small, focused methods. Software written in this style provides space for any individual piece to grow in complexity. On the Rescale development team, we’ve found that the time to split off logic from a growing method or class is much earlier than developers typically think. We prefer to start pulling out abstractions as we write first and second versions of classes and methods.
To illustrate with an example, we were recently writing some code to parse third party xml files that described workflows – files to transfer, analyses to run, and variable definitions. At first our parsing code was relatively simple, mostly because we didn’t know everything that we would need to parse. We started by just parsing input files and variables. Each of those are represented by an xml node, and each xml node indicates its type with an attribute named type, furthering indicating important values with attributes dependent on the type.
Some initial parsing code looked like:

Collection inputFileNames = getNodeStream()
  .filter(n -> n.getAttributes()
                .getNamedItem(“type”)
                .getNodeValue().equals(“inputFileType”))
  .map(n -> n.getAttributes()
             .getNamedItem(“filename”)
             .getNodeValue())
  .collect(Collectors.toList());
Collection inputVariableNames = getNodeStream()
  .filter(n -> n.getAttributes()
                .getNamedItem(“type”)
                .getNodeValue().equals(“inputVariableType”))
  .map(n -> n.getAttributes()
             .getNamedItem(“variableName”)
             .getNodeValue())
  .collect(Collectors.toList());

The helper method getNodeStream above converts a NodeList from the document into a stream for ease of manipulation.
There are two things to notice about this code – it uses magic strings instead of constants, and duplicates code to extract attribute values. After applying those simple refactorings, the parsing code is less cluttered with implementation details and reads more closely to its intent:

Collection inputFileNames = getNodeStream()
  .filter(n -> INPUT_FILE_TYPE.equals(getAttribute(n, TYPE)))
  .map(n -> getAttribute(n, FILE_NAME))
  .collect(Collectors.toList());
Collection inputVariableNames = getNodeStream()
  .filter(n -> INPUT_VARIABLE_TYPE.equals(getAttribute(n, TYPE)))
  .map(n -> getAttribute(n, VARIABLE_NAME))
  .collect(Collectors.toList());

This is our first example of how best practices can help prepare code for increasing complexity. On the surface, it doesn’t seem like we’ve done much, but by writing code that’s closer to what we mean, rather than what the computer does, we’ve made it easier to add more complexity to this code because we’ll have less context to keep in our heads as we write new additions.
If parsing out these collections of strings was all we ever did with this xml, it would be fine to leave this code as is. But, this being software, things got more complex. After writing out the third or fourth type/attribute parsing pair, we decided to extract some enums:

public enum NodeAttribute {
  TYPE(“type”),
  FILE_NAME(“filename”),
  VARIABLE_NAME(“variableName”),
  …
 private final String attrName;
 private NodeAttribute(String attrName) {
    this.attrName = attrName;
 }
 public String getValue(Node node) {
  return node.getAttributes()
             .getNamedItem(this.attrName)
             .getNodeValue();
}
public enum NodeType {
  INPUT_FILE(“inputFile”),
  INPUT_VARIABLE(“inputVariable”),
  OUTPUT_VARIABLE(“outputVariable”),
  …
  private final String value;
  private NodeType(String value) {
    this.value = value;
  }
  public boolean matches(Node node) {
    return NodeAttributes.TYPE.getValue(node).equals(this.value);
  }
}

Now the parsing code looks like:

Collection inputFileNames = getNodeStream()
  .filter(NodeType.INPUT_FILE::matches)
  .map(NodeAttribute.FILE_NAME::getValue)
  .collect(Collectors.toList());
Collection inputVariableNames = getNodeStream()
  .filter(NodeType.INPUT_VARIABLE::matches)
  .map(NodeAttribute.VARIABLE_NAME::getValue)
  .collect(Collectors.toList());
…

It seems like it might be overkill to extract logic here, but we were motivated to do so because the enums provide a central location to define these constants for reuse. Another benefit we soon learned of was that they provided space for the different pieces to become more complex.  Now you might be thinking, it’s just pulling out an attribute from an xml node, how could that become more complex? We certainly didn’t think it would or could.
But it turns out that in this third party xml, some nodes reference files with an attribute named filename and some referenced files with an attribute named fileName. This is the kind of thing that makes programmers curse, but luckily we were prepared to handle this with ease:

public enum NodeAttribute {
  TYPE(“type”),
  FILE_NAME(“filename”, “fileName”),
  VARIABLE_NAME(“variableName”),
  …
 private final String[] attrNames;
 private NodeAttribute(String... attrNames) {
    this.attrNames = attrNames;
 }
 public String getValue(Node node) {
  return Arrays.stream(this.AttrNames)
        .map(attr -> node.getAttributes().getNamedItem(attr))
        .filter(attr -> attr != null)
        .findFirst()
        .orElseThrow(() -> new NoSuchElementException(
            "Node: "
            + node.getNodeValue()
            + " didn't have any attributes named: "
            + Arrays.toString(mAttrNames)
       ));
}

None of our other parsing code had to change. If we had kept on using string constants, we would have had to make many updates across the parsing code to check for filename or fileName, or else write special methods for nodes referencing files. The code would have gotten more cluttered with if/else logic. Since we abstracted early, though, we had a place to put this logic. We want to reiterate that we didn’t expect this difference in attribute casing, but that’s exactly the point – you should expect code to become more complex in ways you don’t expect.
Why do some nodes use filename and others use fileName? We can guess that two different people worked on serializing the different nodes, and they didn’t know the capitalization scheme the other had chosen. Perhaps they communicated verbally and decided on “filename” as an attribute, but one used camel casing. Or perhaps they worked on one node after the other, and forgot what capitalization scheme had been used.
Whatever the case may be, the complexity of the social structure of the third party’s development team is manifesting in this difference of attribute names, and showing through to our codebase. It’s our job to be prepared for that kind of complexity, to be ready to handle it with appropriate structures.

Author

  • Adam McKenzie

    As CTO, Adam is responsible for managing the HPC and customer success teams. Adam began his career at Boeing, where he spent seven years working on the 787, managing structural and software engineering projects designing, analyzing, and optimizing the wing. Adam holds a B.S. in Mechanical Engineering cum laude from Oregon State University.

Similar Posts