Integrate Ansible Vault with 1Password Commandline

We are using Ansible to provision and deploy Tideways in development and production and the Ansible Vault feature to unlock secrets on production. Since we recently introduced 1Password I integrated them both and unlock the Ansible Vault using 1Password.

This way we can centrally change the Ansible Vault password regularly, without any of the developers with access to production/deployment needing to know the actual password.

To make this integration work, you can setup 1Password CLI to query your 1Password vault for secrets after logging in with password and two factor token.

Then you only need a bash script to act as an executable Ansible Vault password file.

First, download and install the 1Password CLI according to their documentation.

Next, you need to login with your 1Password account explicitly passing email, domain and secret key, so that the CLI can store this information in a configuration file.

$ op signin example.1password.com me@example.com
Enter the Secret Key for me@example.com at example.1password.com: A3-**********************************
Enter the password for me@example.com at example.1password.com:
Enter your six-digit authentication code: ******

After this one-time step, you can login more easily by just specifiying op signin example, so I create an alias for this in ~.bash_aliases (I am on Ubuntu).

alias op-signin='eval $(op signin example)'
alias op-logout='op signout && unset OP_SESSION_example'

The eval line makes sure that an environment variable OP_SESSION_example is set for this terminal/shell only with temporary access to your 1Password vault in subsequent calls to the op command. You can use op-logout alias to invalidate this session and logout.

Then I create the bash script in /usr/local/bin/op-vault that is used as Ansible Vault Password File. It needs to fetches the secret and print it to the screen.

#!/bin/bash
VAULT_ID="1234"
VAULT_ANSIBLE_NAME="Ansible Vault"
op get item --vault=$VAULT_ID "$VAULT_ANSIBLE_NAME" |jq '.details.fields[] | select(.designation=="password").value' | tr -d '"'

This one liner uses the command jq to slice the JSON output to print only the password. The tr command trims the double quotes around the password.

Make sure to configure the VAULT_ID and VAULT_ANSIBLE_NAME variables to point to the ID of your vault where the secret is stored in, and its name in the list. To get the UUIDs of all the vaults type op list vaults in your CLI.

Afterwards you can unlock your Ansible Vault with 1Password by calling:

ansible-playbook --vault-password-file=/usr/local/bin/op-vault -i inventory your_playbook.yml

This now only works in the current terminal/shell, when you called op-signin before to enter password and 2 factor token.

More about: Deployment / DevOps / Ansible / Automation

Unslacking Tideways Company

We have moved away from Slack at Tideways over the last three months, because I found Slack is already annoying, even with just a four person team (plus the occasional freelancer). For me, it disrupts deep work phases and knowledge lost in the depth of chat history.

As an engineer, I have learned to be productive when I have a quiet space and can tinker on a problem without getting interrupted. Slack makes this very difficult and at least for me, is a primary cause of anxiety and fear of missing out (FOMO).

While at first sight it seems chat is asynchronous it really is not.

  1. If you wait for a long time to reply to a message on chat, then the discussion thread is already spread in the history of the chat room and messages back and forth interleaved with many additional messages that are not related is not helpful.
  2. In addition chats online status indicators exist just so that you know if someone can answer any question directly, increasing communication anxiety.
  3. If you set yourself into some kind of do not disturb mode or mute a channel, then its easy to miss important conversations and get left out of discussions or decisions. Reading up on long conversations you have missed is hard with chat tools.
  4. By also sending Github commit messages, OpsGenie and Tideways alerts, excerpts of HelpScout ticket updates and other “events” of the business into Slack through various integrations we made Slack the primary tool to check if anything is going on. While it is important to see what is going on, this almost never has to happen in realtime and by connecting it to the chat, we excuberated the previous points and everyone is checking chat even more frequently.

Jason Fried summed it up much better than I could in this blog post.

We are now using Github (issues and pull requests) and Basecamp (Messages, Todolists) to replace Slack. Both tools allow us to have context sensitive, asynchronous discussions on specific topics.

Since work never happens in a vacuum, I would be happy to have only a single realtime notification tool (OpsGenie/PagerDuty) that sends notifications to poeple currently on-call, and only about problems that require realtime attention. Everything else, including chat, can be part of a daily summary e-mail or screen.

My ultimate goal is to get longer stretches of uninterrupted time to work on features, customer support or operational issues. Under time pressure the last 3 years I realized that productive and concentrated tinkering on projects is my number one driver of happiness at work. I consider this a primary value of my company, and it takes work and distance to the current status quo to make it happen.

If you want to follow up on this ideas on your own, I can recommend the books Deep Work by Cal Newport, The Entrepreneur’s Guide to Keeping Your Sh*t Together by Sherry Walling and It doesn’t have to be crazy at work by Jason Fried and David Heinemeier Hansson.

More about: Bootstrapping / DeepWork

How we paid the hidden costs of bootstrapping Tideways

Justin Jackson started an extremely interesting discussion about the Bootstrappers paradox and the hidden costs of bootstrapping with replies from many others.

In this series of posts (there are more) he is discussing how hard bootstrapping a SaaS business is in the first few years, when you are essentially investing a lot of personal time and money until you can finally get a return on this investment and pay yourself a decent salary.

All bootstrappers have to deal with the same tension: can I get this to
scale, while paying my bills, without burning out?

This is something I am thinking about a lot to understand the journey and history of Tideways and by writing about my thoughts I can share my unconventional route for “bootstrapping” a SaaS.

In his post Jason projects the growth to sustainable $ 20.000 Monthly Reccurring Revenue (MRR) to take 5 years, which is much too long for my taste. We projected to reach 20.000€ MRR (~ $23.000) after three years, and missing this target would probably have meant a re-evaluation, maybe even shutdown or sale of Tideways.

We reached this goal this summer, after almost exactly three years.

What went into funding Tideways instead of money?

  • I was able to invest 50+ hours/week and live on 50-75% of my previous salary for two years only because I don’t have credits to pay; my wife and I don’t have kids, no cars and no other large monthly expenses. We saved up a lot of money that we partially invested into the company and partially used for personal expenses during the first two years. Trading your own personal time for bootstrapping a business is probably the largest, most hidden cost and I for myself am pretty sure that I am never going to do this again.
  • My then employer Qafoo allowed me to work on Tideways a few days per month and invested a lot of their own time after we decided to found a company together and test product market fit. In return they now own shares of the business.
  • We struck a deal with our first business partner SysEleven to trade hosting for Tideways in return for highly discounted licenses for almost all their customers. This helped us with the large four-digit monthly hosting costs that we had from day one, because as a monitoring company you need more servers than other SaaS, In return for investing in us, they now have us as a happy paying customer and they still get a good deal on licenses for their customers.
  • With over 7000 Twitter followers from my Doctrine open source project days and my reach within the European and German PHP community I accidentily already built a large audience that would be potential customers for Tideways. We didn’t need to invest more time into building an audience and we luckily never had to pay for customer acquisition. However realistically I invested hundreds of unpaid hours into my open source work since 2009 that allowed me to build this audience.

I consider points 2 & 3 “unconventional” ways for bootstrapping Tideways and already having an audience (4) as a very long term investment that I didn’t plan and got lucky with.

Roughly summing up these investments now, they amount to costs of around 400.000-500.000€ that Tideways would have required funding for.

Maybe raising 500.000€ would be possible in hindsight, but without previous founding experience and no connections to investors, I don’t know if I would have succeeded.

I consider our approach similar to Fundstrapping but instead of money you raise freedom, time and resources from your investors.

Most specifically my advice is to find strategic early business parterns that either become your first large customer from day one, or make your product available to their large customer base.

More about: Bootstrapping

Proofing the Pythagorean theorem and what this means for Domain-Driven Design

According to Wikipedia the Pythagoraen theorem is the theorem with the highest number of known proofs, citing a book containing over 370 of them.

What does this mean for modelling under the Domain-Driven Design paradigm?

In discussions with other developers I often hear the opinion, that you should develop the model of a software entirely disconnected from technical decisions as long as possible and fully abstracted from the choices.

I held this opinion myself for a long time.

Discussing this with a colleague today I found a nice analogy to describe my growing pain with this approach, something I wasn’t able to formulate before.

Maybe the analogy is a bit of a stretch.

The Pythagoraen theorem shows that you can find an entirely different solution for the same problem by using a different toolbox. Finding one or more proofs for the theorem depends on your knowledge of different fields of mathematics. And depending on the toolbox, the solution is simple and elegant, or complex and theoretical.

The reverse conclusion could be, that if you develop a model (theorem) entirely abstracted from the future implementation and toolbox, then it must increase in generality and complexity to take into account the potential implementation with any kind of toolbox. Or the problem space is simple like the Pythagoraen theorem that its description allows many implementations.

Consequently, if you make technical decisions and restrict yourself to fewer approaches/toolboxes before building the model, then the resulting model can be simpler under the assumption that you only need it to be solved under the given technical constraints.

For example, take Redis as a choice of database. Its built in support for more complex data-types allows to implement extremely simple abstractions directly on top of Redis. If you know you use Redis up front for a problem that can be solved with lists, sets or maps, then the model and code describing this domain logic could be extremely simplified compared a solution that makes no assumptions about the data storage.

More about: DomainDrivenDesign

Hide Slack unread and highlight Tray Icon on Ubuntu/Linux

The Slack Ubuntu/Linux application doesn’t provide a way to disable the tray icon and that means the unread or highlight notification bubbles will distract you whenever somebody writes something in ANY channel that you are in, even when the messages are not even interesting for you.

I wrote to Slack support to make a feature request and they suggested to make clever use of an upstream Electron bug that makes the tray icon disappear in 17.04 as a workaround. This gave me the idea for a more stable workaround.

I poked around in the list of files of the slack-desktop package and found out that you can just exchange the files for the tray icons , so that the Slack app is fooled into thinking it is showing the unread or highlight icons.

sudo -i
cd /usr/lib/slack/resources/app.asar.unpacked/src/static
mv slack-taskbar-highlight.png slack-taskbar-highlight-backup.png
mv slack-taskbar-unread.png slack-taskbar-unread-backup.png
cp slack-taskbar-rest.png slack-taskbar-highlight.png
cp slack-taskbar-rest.png slack-taskbar-unread.png

Since I probably need to re-run this after every update to the Slack app, I packed this into a shell script slack-quiet.

More about: DeepWork