Anthony Mastrean

Dashboarding

All I wanted was a huge TV hanging above our workspace displaying the build server status page. Too much to ask from the enormous IT market in 2015? Apparently.

We bought a 50” Panasonic LED TV (thankfully, with a generous corporate discount). The facilities folks eventually hung it on the wall and provided a quad power outlet and a network drop for whatever device we were going to tuck up there.

I thought we would just plug in a Chromecast and browse to our build status page. Haha, you have to be kidding me! You can’t just run a browser. Neither can the Amazon Fire TV Stick, Roku, or Apple TV. The Amazon Fire TV supports a browser, but only if you enable developer side-loading and want to run Mozilla Firefox unsupported.

Don’t forget, you might need to authenticate to your wifi network via captive portal. The Amazon Fire TV has that feature now and some of the other boxes will be getting support later.

You need a third-device “casting” to the dongle full-time!

  1. TV
  2. Chromecast (or similar)
  3. The device that’s actually doing the work :(

There are a variety of reasons not to do that…

  • Don’t want to remember to login to see the dashboard
  • Don’t want to leave a desktop session live and running all day (security)
  • Don’t want a reboot for Windows updates to take down the dashboard

The “set-top” box category of devices are letting us down. So, we’re looking for some kind of full-blown PC in a mini-box. Maybe the Asus Chromebox? Nope, it’s running ChromeOS, which can’t be authorized to connect to the corporate network.

The guest network doesn’t have permission to access the dashboard without a firewall rule exception request.

:‘(

So, now we have a $1000 black rectangle hanging on the wall. I’ll let you know what ends up working.

Caveats

  1. The corporate network requires domain authentication and/or some kind of device registration that requires meeting some arcane security measures.

A Project Management Revelation

The developer in this story had been attending status meetings. These were good meetings, they were attempts at transparency and cross-functional communication. They were snappy and there was always time for questions.

For weeks, the project management team had been presenting the general status of the many releases that were in various stages of the software development lifecycle. Yes, there were released versions in support, receiving patches and hotfixes, versions in active development, and versions being planned and experimented on for future release.

The statuses were like:

  • Bad, we’re going to miss a deadline or slip a feature. (-)
  • In trouble, we’re trending downwards. (~)
  • Good! Everything’s on track. (+)

And the presentation of these releases and status often looked like this, let’s say that 1.0 is the current released version, 2.0 and 2.1 are in various stages of development or testing, and 3.0 is in planning.

1
2
3
| 1.0 | 2.0 | 2.1 | 3.0 |
|-----|-----|-----|-----|
|  -  |  ~  |  ~  |  +  |

Of course, we’re “trying hard” to meet that deadline or to get that feature in. But, are we reorganizing the teams around the most important goals? Are we communicating hard messages to customers about that future service pack so that we can meet the 1.0 goals? It’s not really clear sometimes.

Let’s skip ahead and look at the reported progress as old releases rolled out and new ones popped in.

1
2
3
| 2.0 | 2.1 | 3.0 | 4.0 |
|-----|-----|-----|-----|
|  -  |  ~  |  ~  |  +  |

Wait, a minute… 2.0 isn’t improving and 2.1 is still in trouble. But, look, 4.0 is doing great! The key understanding was a quote(ish) from a project manager…

Of course, 4.0 is looking good because we haven’t started working on it yet! (chuckles)

If you understand that “working on it” means “starting the development phase”. The problems start when the damn developers get their hands on the perfect requirements. You’ll see, over time, that the statuses are fixed. They’ll always appear in that state in that order. The releases simply roll over them, like a wave.

1
2
3
4
5
| 1.0 | 2.0 | 2.1 | 3.0 | 4.0 | 4.1 |
|-----|-----|-----|-----|-----|-----|
|  -  |  ~  |  ~  |  +  |     |     |
|     |  -  |  ~  |  ~  |  +  |     |
|     |     |  -  |  ~  |  ~  |  +  |

I believe that the feedback loops between the teams are broken. The organization is not collaborating across the teams towards a shared goal. There is no real change, so the statuses will never change.

Collective Virtual Meetings

Though we don’t readily admit it, the team I’m on is part of a remote organization. We have offices in multiple time zones, work-from-home folks in different states, and offshore partners. We are trying to build a culture of remote/async work even when a lot of meetings happen in this office.

One part of that is encouraging collective ownership of meetings. For example, we started holding “lightning talks”. Mostly around technical topics, by developers for developers. But, everyone is invited and we’ve had QA folks and IT people come by and talk, too. Even managers got on board and started presenting softer topics!

I’ve been scheduling them, hosting the audio conference, starting the screen share, etc. But, that makes me a bottleneck. I want anyone to schedule these and for a quorum of participants to be able to start the meeting without waiting for some “official” meeting host.

Most enterprise email/calendering systems don’t make this easy because you can’t transfer ownership of a meeting or assign multiple owners. I’m looking at you, Outlook. So, I quit using recurring meetings. A recurrence makes it “mine” instead of “ours”. I developed a nice template that anyone can copy, edit, and deploy for their instance of a lightning talk.

Your audio conference line won’t start without the host. All the participants can be on the line, unable to chat, listening to muzak. Ugh. Not to mention, we have to “order” a conference line from our Corporate IT catalog! Virtual meeting rooms and screen sharing tools (like WebEx) have the same strong idea of single ownership. The host starts and controls the meeting. Without the host, you have no meeting, no sharing, no chat, no video, no recording.

So, here’s what we do to enable and encourage collective meeting ownership (some of these are specific to our tools, but you’ll get it):

  • Schedule the virtual meeting “room” for every day of the week for the longest possible period. WebEx doesn’t allow a 24-hour room, so we schedule two 12-hour meetings. This allows anyone anywhere at any time to start or join a virtual meeting. WebEx now has the concept of “personal” rooms that are always available. This is another option! But, a “permanent” room allows some kind of branding or titling to make the purpose more obvious.

  • Configure the virtual meeting room so that the first participant to sign in is the “presenter”. We don’t want to wait for a “host” to assign presenter permissions. This also lets the presenter get started and setup their rig early.

  • Provide the host key/code in the meeting agenda so that any participant can claim the host role and start a recording or configure this instance of the meeting. Again, the goal is not waiting on a single designated host!

  • Similarly, provide the host audio conference key/code to all participants so that the first participant actually starts the audio conference and they’re not stuck listening to hold music. It also helps to disable entry tones (the “bing” or required “say your name” part). We need to stop saying “Who’s on the line?” every time!

  • Allow the participants to use as many features as possible: chat, video, notes, file transfer, etc.

  • Be sure to record the meeting and provide a streaming and download link, a transcript (or minutes, etc.), and any relevant notes on a wiki and as a reply to the meeting invite, so that all the participants have access.

Mobile

If you must require a code to dial in to the audio line, most phones should support commas (soft-pause, around 2 secs) and semi-colons (hard-pause, requires you to tap a button) between the conference number and the code. For example:

1
1-800-123-4567,1234567#

Recordings

You should provide the link to the downloadable recording, because who wants to stream the recording when WebEx’s plugin asks for these permissions?

If you’re awesome, you’ll convert the recordings to a normal format using their recording editor.

Build Automation

I participated in an online panel on Build Automation: Quality and Velocity at Scale as part of Continuous Discussions (#c9d9), a series of community panels about Agile, Continuous Delivery and Devops. Automating the build pipeline has many challenges, including 3rd party dependencies, consistency and standardization, and testing.

Continuous Discussions is a community initiative by Electric Cloud, which powers Continuous Delivery at businesses like SpaceX, Cisco, GE and E*TRADE by automating their build, test and deployment processes.

Below are a few insights from my contribution to the panel:

What do build bottlenecks mean for your pipeline?

I work on a very large legacy application. We have a thick client run by retail pharmacies over dial up lines. There is an enormous centralized database in the backend. Recently I am hearing a lot about build automation. But the build is just one small piece of building value to the customers. I am thinking about my process throughout the pipeline and all the teams I am supporting; starting from when the pharmacy owner has a problem and we develop a solution, test it, and develop documentation and training materials. They need to get all their pharmacists together to learn the new solution. And then we finally can deploy a new version. This pipeline spans 18 months. And that’s not good. I wish it was a lot faster. But I am thinking, what good does it do if I speed up this one small part of the build?

If we can commit faster then there’s more in a branch that QA needs to test and that needs documentation and training. So the customers are less likely to take builds, because they have to get their work done. I am trying to think about making our build better. But honestly, I wish these guys would slow down. I wish that they were given some slack because development isn’t even our bottleneck, if you look at the full process. So that’s the conflict I’ve got professionally. There’s the duality that I really do want to have great tools, builds, pipelines, and good feedback. But it doesn’t matter if we can’t get stuff out for 18 months.

I’ll try not to be too much of a nay-sayer about build automation because I really am excited by it. But I am worried that we don’t focus enough on full system effectiveness.

I wonder if it’s a chicken and egg problem – I can’t make the people on the dev team more efficient if the people upstream of us can’t keep up. I do not control the entire organization, the trainers and others. But if I can make this team more efficient, maybe we can be a better partner in the overall solution. So I can still get excited about build automation for that reason.

What do you think about consistency and standardization in the process?

My biggest technical problem is with build tools that let you write a build step inside their GUI and all they’re doing is wrapping a shell exec - you write this complicated multi-step, post-condition, pre-condition build and only your build server knows about it.

Then if your build server goes down or if an AWS region goes down, nobody can run a build because it is not actually in source control. So as nice as those features are, they may be easy to set up but aren’t going to be maintainable. Let’s write all of our build steps in some sort of a DSL, let’s check it in, the build server should be some fancy central wrapper. When the teams are excited to do it, we build up a vagrant development machine.

I don’t care what my build server does or how the agents are configured - as long as they have a virtual box and Vagrant, they boot up a machine to do whatever. I look at those and offer advice. We share best practices. But honestly, whatever they do inside their Vagrant box is their own. They can fight with Ops later about how it deploys. At least I can take that box and build it on any machine anywhere.

Can integrated tests help speed up the pipeline?

I think integrated testing is a scam. Unit tests should be enormously fast. You should run thousands of them per second. So if you’re writing good unit tests that are running quickly, your product should not be large enough that your unit tests are going to take too long.

With the integrated tests cycle, when you do the mathematics and multiply out the combinations for even a simple screen, if you think you’re writing enough integrated tests to cover all the paths of your system, you probably aren’t. And the cost in time and effort to write and run enough integrated tests is just so mindboggling – I mean we’re talking about tens and hundreds of thousands of paths. So I think that integrated testers, if you write a couple of contract tests or do a single smoke test, for example: “Is the app up? Can I log in?” then I don’t see how that will ever slow you down. But then I haven’t gotten that far in my own application so it’s all theoretical.

Emphasis

Command-line tools should emphasize the feature that you most likely want to use given the situation (context-sensitive, so to speak). If you run the foo command in an empty directory, it should help you out.

1
2
$ foo
You ran `foo` in an empty directory. You might want to run `foo init` to get started.

Someone on a podcast I listened to recently called this “emphasis”.

Recovering Systems

I inherited a bad IT system: out-of-support servers, end-of-life applications, snowflake configurations, forgotten customizations, no monitoring, and no inventory. I wasn’t entirely surprised when I noticed that the directory service (OpenLDAP) was producing a zero-byte backup. It failed like this through the backup rotation window. We had no good backups on the machine, on the storage server, or offsite!

The scheduled backup job was not logging stderr and the service was not configured for logging, either. I hopped on the machine and tried

1
$ slapcat

The error appeared to be some corruption in the underlying database.

1
2
3
4
5
6
7
8
9
bdb_db_open: unclean shutdown detected; attempting recovery.
bdb_db_open: Warning - No DB_CONFIG file found in directory /var/lib/ldap: (2)
Expect poor performance for suffix dc=foo.
bdb_db_open: Recovery skipped in read-only mode. Run manual recovery if errors are encountered.
bdb_db_open: alock_recover failed
bdb(dc=erx): Unknown locker ID: 0
bdb_db_close: alock_close failed
backend_startup_one: bi_db_open failed! (-1)
slap_startup failed

An explicit recovery command existed that would revert to the last checkpoint.

1
$ slapd_db_recover

Unfortunately, checkpointing was not configured! We didn’t know what this command would do, how long it would take, if it would succeed or destroy data. With no backups, this was a very risky situation.

We tried querying every record out of the running system, and creating our own backup (LDIF), hoping the system didn’t fail under the stress.

1
2
$ ldapsearch -LLL
Size limit exceeded (4)

Oops, there are query limits hardcoded in the service configuration. I could have paged the results, but… c’mon.

1
2
3
4
-z sizelimit
       retrieve at most sizelimit entries for a search.  A sizelimit of 0 (zero) or  none  means
       no  limit.   A  sizelimit  of max means the maximum integer allowable by the protocol.  A
       server may impose a maximal sizelimit which only the root user may override.

We tried that special root service user, but no one had recorded the password.

1
$ ldapsearch -Wx -D "<rootdn>" -LLL

We could increase the query limit or reset the root service user’s password in the service configuration, using a root machine user, but we’d have to restart the service for the configuration to change. Again, that wasn’t safe without any backups.

It was time for an overblown technical response. We would archive the filesystem and load it in a virtual machine. We could try that recovery command, restart the service, or reconfigure to allow a full backup.

1
$ tar --create --verbose --gzip --file ldap.tar.gz /var/lib/ldap

Vagrant to the rescue! … (snipped all the mess getting a CentOS 5.5 i386 image running)… We designed a simple Puppet manifest to install OpenLDAP, using default configurations.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
package { [
  'openldap',
  'openldap-servers',
]:
  ensure => latest,
}

file { '/var/lib/ldap':
  ensure  => directory,
  owner   => ldap,
  group   => ldap,
  mode    => 'u=rwx',
  require => Package['openldap-servers'],
}

service { 'ldap':
  ensure    => running,
  enable    => true,
  require   => File['/var/lib/ldap'],
}

We would stop the service, extract the backed up filesystem, and restart the service.

1
2
3
$ service ldap stop
$ tar --extract --file ldap.tar.gz /var/lib/ldap
$ service ldap start

Wait a minute! The service restarted successfully despite the corrupted database. We now had some confidence that the corruption would be cleared up automatically.

We weren’t out of the woods yet. We wanted to understand exactly how to load an empty directory service from a backup, just in case. We extracted a backup from the first virtual machine.

1
$ slapcat > backup.ldif

We started another Vagrant machine using that simple Puppet manifest and reloaded the data.

1
2
3
$ service ldap stop
$ slapadd -l backup.ldif
$ service ldap start

You’re kidding…

1
2
str2entry: invalid value for attributeType objectClass #0 (syntax 1.3.6.1.4.1.1466.115.121.1.38)
slapadd: could not parse entry (line=12)

Of course, we had a custom schema with no changelog and, somehow, records in the directory service that didn’t comply with that schema. Gah! This option didn’t work as advertised.

1
2
3
-s     disable  schema checking.  This option is intended to be used when loading databases con-
       taining special objects, such as fractional objects on a partial replica.  Loading normal
       objects which do not conform to schema may result in unexpected and ill behavior.

We eventually figured out how to hand-edit the structure of the offending object to comply. With our hand-edited backup and recovery procedure ready, we set an off-hours maintenance period, notified the team, and restarted the directory service! Well, we were confident, so we let the machine do it.

1
2
> at midnight
service ldap restart

It worked and the backups started flowing! We also prioritized fixing some of those systemic problems…

  • Generated and securely stored a new password for the root service user (used slappasswd and the rootpw configuration directive).
  • Configured database checkpointing (the checkpoint configuration directive).
  • Configured service logging (the logfile configuration directive).

What Reviews Do You Practice?

A professor of mine asked about the kinds of “reviews” that I have practiced in the industry. They cover a wide range of practices in the verification & validation classes. Times, practices, and industry has changed and so some of these are more popular and others are practically unheard of anymore.

  • Ad Hoc Review
  • Peer Desk-Check
  • Passaround
  • Pair Programming
  • Walkthroughs
  • Team Reviews
  • Formal Inspections

I worked on contract teams, at a startup, and in the enterprise on enterprise software, mobile applications, and medical devices. So, I have some different perspectives, but it boils down to the following three practices.

Pair Programming

One of the teams I worked on was part of a rescue project. We rewrote the control software for a medical device. It was late, over-budget, and we had nothing salvagable except the theory of operations. We practiced pair programming on nearly all production and test code. This allowed us to come up to speed together and avoid a single point of failure or bottleneck (“We can’t complete this module because Chris is on vacation!”). We improved our “functional redundancy” or “resiliency” in this way. We were able to design and code at the same time (navigator/driver). We worked quickly and could iterate several times on a design or module in one pairing session.

Team Review

At most of my enteprise jobs, the only practice formally enforced was code review. There never was much informal review. Folks would occassionally ask for a “desk check” or the like. We often used separate repository browser tools (a popular one is Crucible) that support discussion over diffs, workflow (open, in-progress, completed), and integration with the issue tracker. In most of these places, the code was mediocre in quality and design. I don’t know that code reviews were the cause. An after-coding review may point out consistency, formatting, or glaring syntactical issues, but it cannot usually improve design.

Pull Request

In my open-source experience, we used pull requests from forks or branches. This method looks like discussion over diffs on steroids. You can view diffs and commit messages, carry on inline commentary, and continue to add commits to the request to address comments. GitHub simply has one of the best interfaces, which can make or break an energetic, deep review. A pull request also acts as a gate. The code is fully committed (safe) and reviewable (good), but it’s not in the “master” branch yet. The owners of the repository/branch get control of that decision.

Unemployment

I left a great team at a startup in digital pathology last year. I was there for a few years, changed roles a couple of times, and saw the product and team really grow. I left when the regulatory situation became unbearable (that’s the topic of another post, entirely).

I joined a small company that did contract work. I hoped to switch technology, business domain, and product lines more often. I was supposed to work on industrial imaging, device control, and interactive kiosks. But, the jobs didn’t come through and we still had to bill. So, I maintained legacy middleware, aging web technology, and spaghetti databases.

My new employer and I didn’t get along under those circumstances and I was let go quite suddenly…

I was surprised and upset, but not worried. I had deliberately cultivated a deep professional network, great communication skills, and an attractive public portfolio. Quick financial advice: pay off debt, save a few months pay, and invest in retirement!

I emailed or called contacts within about a dozen local companies. I was able to get an interview at about half (some of these never responded, even with open positions). I prepared for each interview. I didn’t want to get caught flat-footed. I talked to my contacts and grilled the HR person or manager phone screening.

  • What’s the product, customer, and the market?
  • What’s the business model and funding and/or revenue?
  • What’s the team size and structure?
  • What’s the technology stack and development process?

I wanted to know the biggest problem the team faced: social, process, or technical. I wanted to empathize and think about how I could help. Additionally, I scheduled a half or full day job shadow with each team. How the company responded told me a lot. What I saw during that day told me the rest!

The interviews were standard corporate fare. I talked to individuals and panels, developers and managers. We discussed development practices, technology, teams. Ok, and I waxed on about the future of automation, configuration and package management, and general devops topics. :)

One of the interviews included a technical challenge. They explained that I would create a new MVC project in such-and-such a framework, wire up a couple of views, and get at some data. I used their laptop, their development tools, and had no internet connection.

I never started a project with that framework from scratch, never bootstrapped that ORM, and didn’t know the routing or controller incantations. I maintained similar web projects. I worked with presentation patterns on mobile devices, desktop apps, and touch screen kiosks. I had experience with plenty of other libraries.

I didn’t complete much coding. So, I tried to engage them on the concepts of MVC and the implementation and philosophy of the relevant patterns, but they weren’t having it. I apparently failed that interview, greatly influenced by the outcome of that bogus challenge. I heard later that they changed the challenge part of the interview… oh, well!

Most dev interviews are like a long ass secret handshake to make sure you belong in the club. It’s ritual as much as it is evaluation.

Another company challenged me to answer questions about their specific configuration and deployment of Windows storage servers, without having heard or seen anything about it. I floundered, sounded like a dummy, and, when finally provided context, did not have time to explain what I thought. Another bogus challenge failed.

Most declined with canned HR messages or by never calling back. The rest were more descriptive, but it boiled down to a conflict of cultural fit.

Most companies hire as ‘come and do what you’ve always done with us’. Very few are ‘come and imagine and help build awesome with us’. #sad.

I ended up interviewing one last time. This company had an open configuration management position. I hadn’t strictly done that before, but it was an area of interest (can you say “devops”?). They saw it as a gateway for organizational change. My direct manager, the director, and the vice president of the department were all past colleagues.

I’m happy now, imagining and building awesome stuff. But, I’m worried that the effort I poured into my professional life is unbalanced in contrast to the ease of declining a candidate in an interview over cultural fit.

Interview With a Web Startup

Interview questions from a local web startup.

What is your ideal next career opportunity? Why?

I’m ready to get back into a product company! I did time on internal products and built one medical device before moving onto contracting. I thought I would like switching customers every few months, but it didn’t work out. So, I’m trying to get back into a product company, preferably, where I’ll get more web experience. I’ve got a lot of ideas about continuous integration/deploy and I work on several related open source projects. I’m excited to learn and apply new techniques and websites are much easier to deploy than medical software!

What is your career ambition?

My ambition is to become a real technical leader. I love learning, experimenting, and turning around and teaching people. I want to teach teams what I know about making software simple, quickly, and with happiness! I’m always inspired by what the guys like Dan North, James Shore, and Arlo Belshee are doing.

What do you hope to learn over the next year? Over the next 5 years? Why?

I want to practice continuous deployment in an agile environment and I believe that I can learn that by deploying websites. I’m convinced that continuous deployment is the next big step towards zero friction software development and will help drive down the risk of delivering software.

In the next five years, I’d like to teach and apply all of my hard-learned practices (TDD, SOLID principles, build scripting, deploy automation, etc) to new teams interested in experiencing an agile transformation. I think the Windows development market, especially in Pittsburgh, needs this!

Please describe your ideal relationship with a supervisor. If possible, give an example of a successful supervisor relationship from your prior career.

An ideal supervisor is invested in the people under her supervision. Of course, they care about the company and are passionate about the industry, but they know that happy people will work to delight the customer and make the best product/experience possible. A good supervisor will cultivate true leaders. You hired us for our experience and temperament, so empower us to lead!

What are your 2-3 greatest accomplishments? Why?

You could argue that my greatest accomplishments were at the medical device company, rescuing the device control project in just under 4 months (after 3 years of failure). But, while that seems like important work, I don’t think it was my best. Those accomplishments got stuck in regulatory process and helped no one for over a year (see more in the next question).

My greatest accomplishments have been my work in open source and community events. I was extremely proud to deliver a CQRS/Event-Sourcing talk to the .NET communities in Pittsburgh, Philadelphia, and Roanoke. I taught people a better way to build systems! I took over maintenance of an open-source library, Albacore, fixed bugs, accepted pull requests, and released new versions. This library is used by thousands of people in their build system every day, think about the effect that it’s had!

What is your greatest disappointment? Why?

My greatest disappointment was the inability to ship the medical device that we rescued. We executed technically well, in fact, at a level we never experienced before. The team was hand-selected and told to do whatever it took to get the product done. It was an exciting time. Until it came to actual ship the thing.

We had a very poor understanding of the regulatory environment and had to stop and return to the beginning to document, test, and “prove” all of the work and theory that went into the operation of the system. It was over a year of slow, arduous work to get the thing approved.

It was on this project that I got into continuous deployment and I wasn’t able to practice it completely because we couldn’t ship one!

What 3 books have had the greatest impact on your life? Please explain.

Oh, it’s so hard to pick just three! Luckily, I keep a reading list with a summary of what was exciting, awesome, or terrible about each book. So, here goes…

The Design of Everyday Things. This was the first book that I read in an effort to enhance my professional skills. At the time, I wrote, “My first design book and there’s not a single sentence about software. Norman really breaks down frustrating design. And what’s more frustrating than unusable software?” I’ll never look at doors the same way again!

The Cognitive Style of PowerPoint. This one changed the way I thought about information design and still affects the way I present information. I wrote, “My earliest foray into presentation zen. Tufte shows that PowerPoint reduces the analytical quality of presentations, corrupts statistical reasoning, and weakens verbal thinking.”

The Art of Agile Development. There’s not much to say here, Shore has influenced everything I do when it comes to creating software… “We refer to this book at work as “the XP book”. Yeah, there are others, but this is the one we read. The explanation of engineering principles are wonderful. I haven’t seen a section like “contraindications” in any other text. It’s advice for when you’re doing it wrong, because often we are.”

Describe one experience where you (individually or with your team) developed, delivered and supported a software product. Give specific examples of challenges that you personally experienced.

I’m going to pick on the medical device rescue project again and, while I mention not shipping for over a year, we did ship to alpha/beta sites, so I’m counting that as support. We had no end of challenges in this project (most of which, we overcame!) and even some advantages that became challenges later.

For example, the team was founded with the directive to just get it done, no matter what. We were exempted from the most frustrating scrum activities that the organization had built up over time. While that let us move quickly, experiment, and learn what worked best for us, we were resented, and even challenged, by the other teams for getting out of it.

We worked with the hardware and manufacturing folks in our NJ office, replacing their internal software team, with whom they had personal and professional relationships. The company said they failed and, as our software uncovered hardware defects, the relationship got worse. We worked very hard to gain the trust of that team and to prove that we were there to help us all succeed and not to make them look bad.

The regulatory process was an after-thought, tacked on at the “end” of development. We went through several heads of the regulatory department and, therefore, several tools, processes, and expectations. We were told to do “x” and don’t ask questions. We didn’t understand the reason and didn’t share a value system with the regulatory group. It was hard, as a technical and analytical group, not to have these answers. It took a long time to get in sync and to understand what was law, what was suggested guidance, and what was someone’s experience (often referred to as “mythology”).

I realize that I haven’t talked about any of the technical challenges, but I think it’s because they weren’t hard (we finished the majority of the features in 4 months). The technical problems are always surmountable, the people problems are much more nuanced and interesting!

In your opinion, what makes a software engineer great?

I think the best way to describe it is with the term “T-shaped.” I believe that you have to be deep in some areas (do you know your framework inside and out? are you a TDD fanatic? do you love studying XP practices?) and broad in many others (are you interested in the psychology of design? do you study other platforms for best practices? do you attend other communities group events?).

I think the engineering team has it very hard. They have to be technically savvy to create new software every day, as the technology, platforms, and standards change. They have to be well educated and love studying and practicing to stay on top trends. And they have to interface with most of the people in and outside of the organization (marketing, design, customers, support).

In your opinion, what makes a software engineering team great?

Trust is at the core of any great team. I’m a Patrick Lencioni fan (of The Five Dysfunctions of a Team and Three Signs of a Miserable Job). You must trust one another to be honest, to be aligned toward the same goal, and to avoid poisonous attitudes, behavior, and competition. Trust leads to authentic relationships inside and outside of work and professional interests, and that’s huge. I’m still very close friends with my best and most trusted teammates from previous positions.

What does your ideal development process look like?

This is a trick question, right? There isn’t one correct, objective process. The people, marketplace, and risks are different everywhere. The ideal “process” is a “meta” process of choosing the right process for right now (whoaaa… inception!). The goal is to become competent at shipping software. Or, as the agile leaders say: Work tiny. Prove it. Get done. Learn constantly. Work together. Tech first.

How does a software company ensure that users have the most satisfying possible experience with their product?

Another tough (trick) question, which I’ll answer with more indirection! I don’t think you (the company, director, or supervisor) should be actively managing “satisfaction”. There must be hundreds of identifiable experiences in your product (daily vs. one-time interactions, in and out-of-app (email) experiences) across many kinds of users (low-tech vs. power users). If you start managing/measuring one metric, you’ll get people gaming it (not out of spite, but out of human-laziness).

Optimize for the passion and happiness of your whole team. Let them get excited about user experience in their own way. Let your designers redesign the signup page until conversion is awesome! Let your engineers built a frictionless deployment system. Let your support team build the monitoring system that anticipates customer problems before they even call in.

The “company,” or more accurately, the business people, can measure the appropriate business aspects: signups, in-app purchases, value to the customer. But, they shouldn’t be micromanaging nitty-gritty technical measurements.

Building Internet-focused (“client”) software is different than developing traditional enterprise desktop or server software. Describe how.

I don’t accept the premise that it’s so different, except for the most trivial applications. The presentation patterns are similar, I’ve done some flavor of MVC/P/VM on mobile devices, desktops, and touch-screen kiosks. The pattern is the same, the details of the framework are different. WPF databinding brings a flavor of HTML templating to the desktop. The lifecycle of a web application view/controller interaction are only technically different than the lifecycle of an enterprise desktop application communicating with a remote service, but the principles are the same.

I see differences in the development process, particularly in release and branch management. Most websites only have one publicly released version (you can release features to slices of users, but usually through feature-toggles, not separate versions). You can more easily commit just to master and branch only long-running features. With a traditional enterprise application, you may maintain a long-lived branch per public release for regulatory or hotfix management.

I think websites get an easy break with continuous deployment, for these reasons! You’re pushing code/configuration to servers that you control and that your users only temporarily connect to. Try doing continuous deployment to desktop systems inside a firewall, inside a hospital, or to occasionally connected devices!

Describe any experience you have using the agile software engineering methodology.

At the medical device company, I was part of several multidisciplinary scrum teams operating within a regulated environment. We practiced a wide range of XP practices, vanilla-scrum activities, and experimented with some lean tools, like Kanban boards.

I can’t possibly do this question justice in text, please ask for more!

Describe how you handle dealing with conflicting priorities or objectives.

I like this question, because I know that “priority” is actually singular, there can be only one! And, luckily, humans are single-task creatures. I can time-slice, bringing up the dreaded context-switching costs and breaking flow.

I may not know which of the many priorities is actually the priority, but there are many interesting ways to find out. We can have a release or sprint planning game. We can lay things out into the “four quadrants” of activity management (important/not-important by urgent/not-urgent). Or I can just use my judgement.

In a different way, I believe that each team should be able to produce software competently, and by this, I mean, quickly and with low risk! We should be able to try all the conflicting priorities in as much time as a “normal” team could produce one.

Describe how you keep your skills and experience “up-to-date”.

(deep breath)… reading blogs, staying on top of trends by using twitter, listening to relevant podcasts, reading books on topics ranging from design to psychology to computation, participating in diverse user groups and professional events like conferences, hackathons, and code retreats, asking and, more importantly, answering questions on stackoverflow, reading and contributing to open source software, teaching and learning from coworkers, keeping in touch with professional colleagues in diverse positions, mentorship w/previous supervisors and/or colleagues.

What do you do when you are “stuck” when solving a problem?

Creating new software every single day in an environment that’s rapidly changing is really hard. Most of the metaphors that compare software to construction are wrong. The writing of the software is entirely a design activity. The computer does the constructing (compilers, hello?). The problems are non-linear! Sometimes you just have to disengage the problem solving part of your brain, turn away from the screen, take a walk, sleep on it, or just take a shower!

As well as do any of the things in the previous question… you never know where that bit of inspiration will come from!

Interview With a Ruby Shop

Interview questions for a small, local Ruby shop.

Do you have any experience programming in Ruby? If so, please describe.

I wrote and primarily maintain the Rake build for our .NET solution. We must be able to pull any released version of the software out of the repository and re-build it (for regulatory reasons). We were using TeamCity’s built-in runners, but were unable to store them with the code or audit it properly. Developing a Rake build allowed us to checkin the build script with every branch and release.

We used the Albacore rake tasks for .NET from Derick Bailey. We contributed an MSTest task (which was pulled in) and an MSBuild task extension (waiting in a pull request). I wrote some more .NET specific application metadata/configuration tasks (pull request coming later) and a simple deploy task. Both of these have been published to RubyGems.

My recent involvement in the Albacore project led me to request ownership from Derick Bailey. I organized the remaining active developers (Henrik Feldt and Jeff Schumacher) and created the Albacore organization. We’ve fixed some bugs, pulled in some code, and recently published a new version of the gem.

Describe your experience with object-oriented design.

I’m a SOLID guy at heart. I believe that several of these principles transcend static/dynamic functional/procedural paradigms, namely: SRP, OCP, and ISP. SRP is at the heart of what we do though and “extract method” is my hammer. This principle has led me down a path toward functional programming patterns. “Do one thing” is strict. An algorithm and how I execute that algorithm are two different things and can (should?) be separated.

OCP is a tough one and some languages give you just enough rope to hang yourself (mixins, multiple inheritance). I favor composition of components instead. It’s harder to mix state with composition. And easier to mock, test, and swap components this way. I think this has leaked into every level of my thinking. I think about the way I “close” namespaces, assemblies, and even higher, packages (think gems).

ISP has a sorry history in static language ecosystems. Interfaces should allow us to interact with objects based on what they do, not on how they do it. The canonical example makes me cringe

1
2
3
Cat : IAnimal {
  void Speak() { Console.WriteLine("meow"); }
}

The “correct” implementation is in the describing a behavior that I can perform, not a noun that I am!

1
2
3
Cat : ICanSpeak {
  void Speak() { Console.WriteLine("meow"); }
}

This principle has led me down the path of behavior driven development even in my unit tests. I try to ignore the “how” (usually, by extracting and then testing the new “what”) and focusing on the “what”.

Let’s not forget primitive obsession, which I’ve kicked! I won’t bore you here, but you can check out what I’ve already written.

Describe your professional passions, things that make you happy in your profession.

Great teams are passionate and expert at their craft. When empowered to make decisions, they will blow through barriers and accomplish amazing things. If you have a team like this, trust them, and unleash them on your business! Projects and companies succeed or fail not due to technical problems/expertise, but due to people. I’m a culture geek and this means a lot to me.

That said, I really like teaching and speaking to other developers and learning from them alike. I like “brown bag” meetings and encourage participation in local groups. Open spaces events are like rocket fuel to me. I think developers need to learn to speak in public or at least to the public. Blog or tweet!

This goes hand-in-hand with participating in open source, from submitting bugs or answering questions on StackOverflow to sending pull requests. Everyone can get involved in some meaningful way.

Describe for us areas you’d like to grow in professionally.

I’d like to get some experience in *nix systems. I’m an automation nut and have been using PowerShell and AutoHotkey on the Windows platform. But, I feel like it’s a generation or so behind. I also want to push the envelope on my devops skills. I’ve been configuring and deploying desktop applications on Windows in a regulated enviornment. You can only get so “continuous” with that process before there are gates to pass or manual pushes.

I want to continue growing in OSS contributions and, ultimately, leadership. I’ll be kind of conflicted leaving a .NET shop just as I help take over Albacore! But, I know this will reveal other opportunities. The OSS community on other platforms seems to be more mature.

Last, but not least, is my desire to really grok functional programming paradigms. That world has great ideas, but they don’t do themselves any favors with the way it’s presented (>>=, anyone?). It’s possible to learn and practice immutable objects and pure functions in any language.