3 Different Ways to …Building Workflows That Last Forever
One of the more important mindsets we can have now which will pay dividends many times over down the road is to think a bit more long term than just the workflow we have open in front of us. Many times we Alteryx users are in the moment and building the world’s greatest workflow, we aren’t thinking about what that workflow, macro, or app is going to be doing in a month, or even a year from now, but here’s the kicker: We should be.
The more dynamic, thoughtful, and technically sound the workflows are the longer they can last. Anytime we don’t need to spend time on re-engineering or even re-building workflows the more we can focus on new tasks and projects that will move the needle. There are many ways we can address the ‘long term thinking’ within our workflows and in this post I’ll cover 3 very potent methods that will help us right out of the gate. Since there are quite a number of differences for users who are utilizing the Alteryx Server (aka Gallery) vs users who are just building on their own machines, I will actually provide 3 for building workflows on our desktop and 2 important concepts for building workflows for the Alteryx gallery.
Building Workflows for Desktop
If our organization doesn’t have the Alteryx Server Product then we will need to build workflows that are run from our own machine. If we are lucky enough to have the desktop scheduler then we have an immense power to be able to schedule our workflows. Automation should be a very high priority for us. Our ability to handle more and more work and get more and more done depends on it. All that aside, just because we might not have an Alteryx Server or even the desktop scheduler on our machine we need to think hard about how we are building our workflows. We don’t want to build 20, 30, or 50 workflows and then paint ourselves into the proverbial corner of having to manage them. We want to use them to make our days and weeks more productive and not have to keep updating or re-building our workflows.
Documentation
The first thing we can do that will have the biggest impact in our ability to navigate through the workflow, update if needed, and also troubleshoot is to properly document our workflows. This is nothing like where we have to document a system process that ends up being the size of a Stephen King novel. This is something we can actually do quite easily and complete it as we build. The key is to have something ready before we even start. What we want is to work off templates. Utilize a template that has documentation elements already in it and manage the workflow according to those elements. A modified version I’ve seen on the community (Original author: @balders93) looks like this:
Notice that this template has the following elements accounted for:
- Sections for each major area of a workflow
- Color to easily distinguish each section
- Documentation opportunities to ensure any user would be able to understand what’s going on
Folder Structure
The second thing we can do that will really help us to create workflows that last, that can be shared, and help others build off of them is to standardize the folder structure of our workflows assets. Many times we’ve been building a workflow and the input files are in one spot, the workflow is saved in a different spot, and the outputs are yet again somewhere else. This doesn’t help anyone. We need to build a simple folder structure and work from there each time we build a workflow. The point here is to create structure, if we need to send to someone else, simple zip up the directory and send. If we need to know what macros, input files, or output files are involved with our workflow, simply go to the folder. Here is a simple hierarchy that we can start with and if needed we can expand.
Top Level Folder – [Workflow/App/Macro]
[Folder 1] – Inputs
[Folder 2] – Outputs
[Folder 3] – Macros
[Folder 4] – Documentation (Optional)
Validation Checking
The third thing we can do is probably one of the most underrated capabilities we have at our fingertips. Nothing makes a workflow die a painful remorseful death like not having the right input. Correct inputs are kind of a big deal! Now if we all agree that our inputs to a workflow are critical then we should be checking them at runtime to ensure the inputs are exactly what are expected so that there are no mishaps, errors, failures, and anything else that would make us sad. We should be giving clear instructions to the user on what to use for input as well as validating that what they entered is correct. Inputs are not just file inputs, there are app inputs as well that we need to take into consideration if we are building apps or macros. The steps here are relatively simple:
- Document standard inputs (files, databases, etc)
- Document app, macro inputs
- Add usage of the ‘Error Message’ interface tool to perform validation on app inputs
- Add usage of the ‘Test’ Macro to ensure that the expectations we have are being met
Building Workflows for Alteryx Gallery
Now building workflows that will eventually be published to the internal Alteryx Gallery should take in all the practices described above but there are a couple more here that should be added as well. This is where Analytic Apps become our aha moment!
Turn Workflows Into Analytic Apps
Analytics Apps specifically allow us to enable a user of the workflow on the gallery to ‘choose’ the value or file that the workflow uses to run, therefore taking away the need for us to hardcode everything and in turn having to keep updating the workflow with new hardcoded values.
Let’s make this super practical. We have a workflow that has a excel file input. Each month we get an updated excel file which we then update our workflow and re-upload to the gallery….
Stop right there! We now know a much better way!
From now on we are going to make it more ‘dynamic’ by adding interface tools that will let the user pick the file to run the workflow with. We then upload one time and that’s it! We no longer have to upload that workflow just to keep changing the input.
The greatest part is that there is virtually nothing that can’t be updated at run time. To say another way, we can make every single piece of our workflow ‘dynamic’. Imagine if we built our workflows one time and uploaded them one time!
Use Events to Send Notifications on Failure
Even if we have inputs and use the error tool to validate or the test tool to validate inputs there is always going to be something that might come up. It might be right away or it might be months down the road but we don’t want our workflow to be sitting on the gallery and not being used or the scheduled job to stop because of some error. Here’s a way to ensure that doesn’t happen.
Another powerful tool we have at our disposal is the Events. We can take some action based on the outcome of our workflow for example if the workflow fails (and only when it fails). We can do this by setting our events to trigger to send us an email when a workflow fails like this:
Imagine if we have a 100 scheduled workflows on the Alteryx Gallery. We could go to the gallery and see the status of all our jobs but ain’t nobody got time for that! We could also flood our inbox by sending a notice every single time a workflow runs whether it errors or not and guess what…ain’t nobody got time for that either! We want the system to notify us if any one of them doesn’t run as planned. Now imagine if a month from now one of those 100 workflows fails (for any one of the million possible reasons) and guess what, we get notified, we fix it, and we get back on with our badassery.
In summary, by taking the little time from the beginning to build workflows in a purposeful manner we will actually save ourselves so much time in not having to go back and re-design, re-work, or spend unforeseen hours trying to figure out what we did 2 weeks, 6 months or 2 years ago on a workflow. We will have it well documented, a clear folder structure with validation checks throughout the workflow. We will also have made it more dynamic by turning our workflows into an analytic apps as well as using events to notify us if something doesn’t run as we expected. Sounds great doesn’t it?
