Production Level Learning Curve: From Local to Reality

Danny Bailey
11 minutes ago
7 min read

Diagram comparing localhost development with a fast direct link to one server and production deployment across a world map with slower response

In this blog, we will be looking at some major jumps in the learning journey for a piece of software/technology and the challenges at each stage, and more importantly, some techniques to help combat this and improve your professional development.

Firstly, some background on me; I recently joined RADAPEX as a DevOps Engineer at the start of June (shameless LinkedIn post plug!). After being out of the tech industry for a little over a year brought back the feelings of my first tech role, having good foundational knowledge of different tools but becoming lost and sometimes a little overwhelmed when working with production-level systems. This, I feel, is the biggest challenge when learning any tool, it turns the dial up to 11 and is far removed from local applications and personal projects that you may have built.

Learning a new technology is rarely a smooth path, even if the fundamentals can be easy to grasp. The journey involves distinct shifts in mindset and architecture, as the application or infrastructure develops. As an example, throughout this blog, I will be using Terraform as an IaC (Infrastructure as Code) tool to show snippets of differences in complexity in certain files (don't worry, I won't be adding a full production stack with multiple files and 100s of lines of code!).

The Different Learning Curves

The academic curve is usually the starting point, covering basic syntax, local execution and controlled environments. Everything here is fairly predictable in terms of variables and the overall environment; even if you are not fluent with the software, troubleshooting and error messages can be fairly intuitive to resolve. If not, call on the software documentation or (as a last resort) the help of your favourite AI Chatbot to help with the resolution of any issues.

The operational curve builds on the previous foundational knowledge but now involves systems at scale, security, and team collaboration. Most code bases at this level will be on a greater scale, with very specific environment configurations and able for multi-environment deployment. Yes, these systems would follow the same core principles, but going from a few .tf files (Terraform configuration file) to 10+, all linked together, can be jarring to get your head around at first.

Recognising that these two curves exist prevents the frustration that occurs when early success meets real-world friction.

Fundamental Application Knowledge and Syntax

The journey begins with literacy. At this stage, your goal is simply to understand how the tool thinks and communicates.

As with all learning journeys, you’ve got to start with the basics and the fundamentals. You are learning how the software actually works, the syntax, basic commands and the primary engine driving the technology.

Terraform uses HashiCorp Configuration Language (HCL) as its syntax. You learn about blocks: providers to connect to Cloud Providers, resources to declare infrastructure components and variables to make your code reusable.

# A basic Terraform configuration demonstrating HCL syntax

Dark code editor showing main.tf Terraform code for a random_password resource and generated_password output with special characters

# A basic Terraform configuration demonstrating HCL syntax

Not going into deeply what is actually happening in this file, the blog isn't designed to be a Terraform tutorial! To read more, the “random_password” module can be found in the Terraform Registry. Can probably guess what this module is doing!

Once you have a standard configuration, you then need to learn the core workflow of the application to deploy/run the software. The following to deploy Terraform infrastructure (at a very high level):

terraform init: To prepare your working directory.
terraform plan: To preview changes.
terraform apply: To execute the file and deploy.

This is the very basics for Terraform, from a blank document to an output into the terminal. Nothing too groundbreaking as of yet, but as mentioned more getting to grips with the syntax and how the application works at this stage.

Your First Application

After you’ve grasped the basic syntax and can produce something locally, usually something simple as having an output in the terminal. It will be quickly time to build your first application that has some real world components; usually deploying some Cloud Provider services (as this is the main use case of Terraform). In terms of Terraform files, the syntax won't be any more complicated, but you would need to know about the specific cloud service(s) that you are deploying (e.g. provisioning a EC2 Instance in AWS, information about parameters to pass can be found in the registry).

This can be common for different pieces of software, the software itself rarely runs as a standalone. Usually interconnected/relying on other services to host actual applications.

In our Terraform example, showing setting up an EC2 instance in AWS. Not too different to a local file, but now we will have an actual application that will deploy to a cloud platform (given AWS access etc has been granted/configured).

# Terraform config for AWS ec2 deployment

Dark code editor showing main.tf with Terraform AWS provider, Ubuntu AMI lookup, and aws_instance app_server config.

This is still a very basic example of an actual service that is actually running, and is a glimpse into how actual applications/web services are hosted, so they can be accessed by users. Still at a very high level, but you would encounter new issues vs using configurations locally.

You will then build on this as you move from a single component to a connected system. This forces you to think about how different elements interact, pass data and handle dependencies. The core of this will come from networking fundamentals and how the cloud provider services work and interact with each other. So regarding the Terraform examples that have been mentioned, we have moved swiftly on from the actual syntax of the software to overarching protocols that will make the software run as intended.

Real-World Application and Stack

So, you have learnt a new piece of software from the ground up; you know how the software operates and what commands perform which actions. You have experience using the software locally and deploying into a cloud environment with your own personal projects. Now, you are coming into a real world environment with production level systems (may have started a new position in tech or recently moved companies), and one of your first thoughts when looking at these repositories is … overwhelming!

The scale and level of detail in these files/repositories will be greater than anything you would have come across before, the file structure and syntax will make sense, but understanding the order of processing and what each file is doing may not be straight forward as it is at first glance.

With this environment, your technology stack will change greatly, through different factors that may not have been considered before in your personal projects:

Cost: Financial management (especially in a DevOps role) has to be adhered to. Just like any other non-technical position, your team/department will have a budget for work to be completed. With this in mind, this may change how you structure your architecture; you may have a solution that works perfectly in theory, but there may be a more cost effective solution to consider.
Compliance: More importantly than cost, the way you build your application/infrastructure must follow standards that have been laid out by your company. Again, this could change your plans in how you structure your architecture.
Security: Making sure, in addition to your infrastructure operating effectively, that your infrastructure is secure and resilient from potential threats. Making sure that, if your application fails, what failovers are in place to make sure that downtime is at an absolute minimum.
Multi-Environment Scale: Your infrastructure will likely be used across multiple different environments (development, testing, production, etc) and needs to work seamlessly across each, also making sure that they are developed in a modular way so dynamic values can be passed depending on the needs of each environment.
Older Software Version: Your organisation may be running the same software that you are familiar with, but operating on a much older version (this is to be expected; upgrading on a large scale can take years to plan and execute!). This could lead to quirks in the syntax or commands that you may not be familiar with.
Changing of Requirements at Operational Level: You may have learnt a software like Terraform for your IaC needs and your company is using this to deploy their infrastructure currently, but in the future your organisation may look to use a different software (AWS Cloud Development Kit, for example). This will come with all new syntax and commands to comprehend.

This is not an extensive list of challenges you will face with enterprise level systems, but they are some to be aware of. It is important to keep in mind the wider landscape of your team and the organisation. Why is the software running on an old version? Reasoning behind the change in software? How strict are the budget requirements?

How to Combat These Jumps

The gap between personal projects and production stacks can be massive (and rightly so!). I believe that the best way to combat this is to adopt the “fail fast, fail often” approach, and from these failures try to learn something new each time to build upon. Then, for your next project, you have a better understanding going forward.

To bridge these gaps effectively, you could look at utilising the following:

NB: Make sure you have permission from your organisation to perform some of these, this isn’t your system anymore and could have serious consequences if performed incorrectly!

Intentionally Break Your Environment: Only do this if in a testing or development environment, and it is not affecting anyone else that could be working in there! See what certain files do and what they are dependent upon; you will probably get more error messages before it actually works, but this is also part of the process to see how the entire architecture works.
Follow the Breadcrumbs: As mentioned, looking at a new repository can be overwhelming with the amount of files and the context in them. Try working through each to see where they connect and which module depends on which to get a better understanding of the architecture. The syntax will be familiar but more complex in connections.
Plan Ahead: Don’t just think of the service you are currently working on. Look at the entire architecture and how this will integrate with other services and what impact it will have.

Conclusion

I hope this has given some insights into the trials and tribulations of going from local systems to production level systems. As long as your mind is always open to learning new things, you can adapt to these potential challenges, there is always an opportunity to learn more!

As usual, don’t forget to check the RADAPEX blog page and RADAPEX on LinkedIn to read more of our blogs and success stories from other members of the RADAPEX team.