5 years as a software Developer

What I learned, what I regret doing, and not doing and the future might be

End of June, 2020, will mark the fifth year of my journey as a professional Software Engineer. It feels like yesterday, when I entered the long, air-conditioned, ubiquitous white corridors of a big software corporation as a wide-eyed Intern in early 2020, determined to make the world a better place through my Java Skills.

5 years on, a lot of things have changed.As a kid, 5 years is forever. As an adult, time just whizzes by.

It is hard to summarize this tumultuous period in one short article, nevertheless I will try my best to sum up my experiences and learnings in 5 points, to commemorate half a decade of professionalism.

A lot of these are merely observations, vague ideas and personal experience. Some of them are probably representative of the software industry in general, but most learns are personal, arising out of one’s luck, character and actions. Please do not be pedantic and ask for “citation required”. Consider this as a scientific experiment with a sample size of n=1.

When I started writing this article, my intention was to be a bit more structured and elaborate about my mistakes and learning. Along the process, I realized that there is a lot I would like to cover about those and they perhaps warrant longer articles of their own. I therefore have tried to just paint a broad picture of the most general observation and a very vague sketch of my next steps.

1.Real life is open-ended

One of the subjects in my 6th Semester course was OOAD - Object Oriented Analysis and Design. The subject consisted of memorizing “Design Patterns” , considered to be a standard engineering paradigm and applying the right pattern to the problem.

The problems were well defined - “What pattern would you apply for a Compiler” , “What pattern would you apply for a library of fonts” and the answers were precise. Problem a ? - Pattern X. Problem b ? - Patter Y.

Want to sort a bunch of numbers? What technique would you use ?

Ans: Quick sort. Why ? Complexity : O(n log n) approaching O(n) for some inputs

Unlike the blackboard and my exam sheets, Reality has a very basic problem : The problems are mostly not defined at all and the answers are almost never clear cut.

In fact, I would argue that finding the RIGHT problem to solve is the biggest challenge in itself. Startup culture calls this the “product/market” fit, the single biggest determiner of a startup’s ability to succeed. Many a startup pivots to a different service/product/offering mid-way once they realize that their original vision had no buyers and they will be running out of money soon.

Do you know that a “quicker” version of Quick sort exists, called Tim Sort (used by Python) ? . For small enough inputs, Time Sort simply performs an Insertion Sort and calls it a day, as the function call overhead/recursion overhead has a higher cost than the algorithmic inefficiency.

I worked on 4 months, pouring my heart and soul in a Service with the perfect set of tests, logging, API design and the whole shebang, only for my team to discard it after the demo, because there was no customer within my company for the service. Even my own team was not interested in running or maintaining this service.

People have been warning and prophesying of an AI takeover since the 1950s and especially after Deep Blue won a game against Garry Kasparov in 1997. It is 2020 and the M4 Competition findings have established that even sophisticated ML models are yet to thoroughly outperform statistical models. NN Taleb calls it the “Ludic Fallacy”, the incorrect assumption that real life resembles that of a game, with well-defined end states and goals that can be perfectly measured.

The open-ended-ness of real life leads to my next observation

2. Software Engineering is not Engineering

No Engineering is perfect from an “engineering” sense. Engineering conjures up a field of precise measurements, perfect laws and techniques wherein everything can be measured, a theory of cause and effect hypothesized, a solution drawn up mentally, an algorithm or design conjured up to solve the problem and the implementation solving the problem perfectly.

Real world engineering is messy. And this is not just in Software Engineering. But nowhere is it probably more apparent. Having worked in companies with billion dollars in revenue, I have seen how money making code is more often than not, a terrible patchwork of spit and glue clode, held together by an army of programmers .

Almost nobody sets out to write bad code. Time is limited, competition is cut-throat. The success or failure of a product often depends more on luck, timing and your (or your company’s) ability to make a sale, rather than purely on technical merit. Most software products probably developed as a horrible patchwork of hurriedly written code to get the product out of the door on time. Refactoring code costs a lot in terms of resources, time and money and management would rather engineers spend time adding more “features” than cleaning up existing ones. Even the almighty Google isn’t immune to this culture.

3. Environment matters far more than skills

“When a manager with a reputation for brilliance tackles a business with a reputation for bad economics, the reputation of the business remains intact.”

• Commonly attributed to Warren Buffett

Who do you think would “perform” better? A mediocre programmer in a good team, or a Rockstar developer in a bad team?

Side Note: I put perform within quotes, because the definition of performance, varies with perspective (more on this in the future)

My observations so far has been that engineering is no longer a solo venture, but a team game. Software is so complex that it is no longer possible for a single developer to maintain any moderately sized codebase in any company. You have front-end devs, SREs, back-end devs, DevOps guys, Managers and Leads all working and influencing the success of your work.

In such cases, how good you are probably doesn’t matter. It is far more important to be in the right team, among the right set of people, on projects that management and engineers care about.

If you manager or the VP doesn’t care about your team or the product, no matter how many long you burn the midnight oil, maintaining servers, writing tests and updating documentation, your career mobility and rewards will always be limited

It took me a long time to figure this out, and avoiding dead-end teams or companies, being able to negotiate higher salaries, will have a higher impact on your career than learning algorithms or technologies.

4. Playing the game you want to play

The Romans used the term “cursus honorum” or course of honor, to describe a series of public offices that aspiring politicians must hold to make it to the top (dictator, consul, praetor etc). To move up the cursus honorum of the modern corporate life, one must as my ex-boss and HR put it, it is not merely enough that you complete the work assigned to you on time, go beyond the expectations, to prove that one is hard-working and driven and committed to the firm, to proceed further up the hierarchy.

Like me, it is possible that you have a career as a programmer or would like to have a career in programming, because you loved playing with computers. You loved playing video games, making art, music or just hacking with a computer, putting it together or pulling it apart. Even if that weren’t the case, you are in because the pay is good and the hours are sane and the people, a little bit more easier to deal with than in the other industries.

The real politik of software engineering has more to do with internal politics, marketing and sales rather than pure engineering. Even within the company. There are rules, hierarchies, bosses and committees that you have to please to make a bit more than you were making last year, to get the ever elusive promotion to a higher title, to make it, in the industry.

While ideally progress in the long-term can roughly be defined as how good you were last year versus how good you are now, in reality , your rewards come from how well you performed compared to your team mates rather than how much you have improved in terms of skills and wisdom.

Maybe you decide to play the game. You work long hours, fix bugs that aren’t even described, write tests that the previous engineers ignored, yet find yourself languishing in a junior position, uninspired, unmotivated and feeling burnt out.

The good news is, you can changes the rules of the game, by playing a different game. Quit and start your own firm, change companies, change teams. Nowhere is jumping ship more easier than in software and infact, the industry rewards you more for doing so , in the form of joining bonuses, pay raises when you switch companies etc, than for sticking to a single team and slowly improving your team’s offering. These rewards however, will soon taper off and making the next jump from say, Senior to Principal , or Principal to Fellow, becomes exponentially harder.

To survive in the long term, it is important to understand what you want out of this career (or more robustly, what kind of jobs you DONT want to work on).

Do you want the security of a Fortune 500 firm ? A lifetime making 5-10% increments every year and retiring after 50 with a small fortune ? Or do you want to rough it out in the real world by taking the risk and starting your own firm ?

Do you like to work on hardcore engineering problems that are highly technical in nature ? Or do you like solving problems for your companies customers, no matter how simple or non-technical they are (Note: sometimes these distinctions converge, but it is best not to expect them to converge all the time).

The first step in having a satisfactory career is to play the game you want to play.

Choose your own cursus honorum.

5. There is a beautiful life outside of work

I have to admit that I am not a very diligent or a hardworking guy. Given a choice, I am more interested in programming as an exercise in pure knowledge, sort of like learning and doing math, than in solving real world problems that will bring in money.

I do not like working more than 8 hrs a day. Even 8 hrs is a drag and I get nervous and impatient and think of nothing more than jumping out of my desk and into the outdoors. I am a big outdoor aficionado. I love climbing, hiking, running, playing Table Tennis and listening to music, reading books and just about everything that doesn’t involve me sitting in a single position for a long time (except the book reading part). As such, it is hard for me to work day and night on any given problem and go beyond expectations at work.

I would rate myself as an average programmer. I can identify bugs, debug them, fix the problem and write good documentation about it. I would rate myself as barely OK for designing systems, given that so far in the past 5 years Iam yet run into a single problem where the main limitation of system was the inability of my system to scale.

I haven’t worked in any uber cool engineering teams so far, haven’t discovered kernel bugs, solved multi-threading problems or written ML systems in Rust and compiled them to run on the GPU in Web Assembly.

But honestly, I don’t miss or care for working on problems like that. The greatest joys of my life have so far been climing a 6000m mountain, running a 14 hr race without long breaks, hiking and sleeping the wild and playing a good game of Table Tennis and having a break with some cold drinks with my friends. My life is not defined by my work, my professional title or even my profession. My profession pays we well enough to afford a life I want to life and I am very thankful for it. But it is the non-work parts of my life that I enjoy more.

It is OK not to work 60+ hours on a dream project. It is OK not to make senior engineer in 5 years. It is OK not to have a house or a car after 5 years of making Engineer salaries (Indians, you will get this :)). Life outside is beautiful and personally, I’d rather live it and not regret it, say 30 years down the line, when my body is frail and my spirit is extinguished.

Next Steps

Honestly, I am not sure in which direction to proceed next. The first 5 years of my career have been spent on identifying what kind of a person I am and what kind of jobs I want to work on. What I realized is that it is far more fruitful for me to understand what sort of team, company or people I do not want to work with. This realization is important for 2 reasons

I realized that uninspiring work causes boredom for me and a bored me is a very self-destructive, disruptive me.

As a knowledge worker, the quality of the hours I work is more important than the quantity of hours I do. 2 hrs of motivated, in focus work is not the same as 40 hrs of distracted, barely in focus work. To be of value, I must be convinced of the work’s ambitions and goals.

The definition of what one deems “useful” or “motivational” differs with context and the person. As I highlighted in point 4, it is important to play the game that you want to play, in order to thrive in this industry in the long term.

So far, I can describe the first 5 years as a scouting phase, wherein I discovered how wide the software industry is and what are the options that are available, and what kind of work I find motivational.

I realized that I love developing, small, fast, focused tools that can help people organize their lives, work and their knowledge. Small because I like to work on them by myself. Fast because that is the entire practical purpose of personal computers. Focused because I love the UNIX philosophy of tools that do one thing and one thing very well. I am currently working on a small tool here as a tiny starting step towards writing software that will make my life better.

I am an autodidact and I honestly prefer learn things by myself, even if there is expert help available, due to a lifetime of being conditioned to learn so.

Over the next few years, I wan to ponder, and work on tools to help autodidacts like me learn and practice on just about anything, irrespective of where they are located. In an era of rising costs of education, rent seeking and regulatory capture, formal education is neither worth the cost nor is effective. COVID19, the rise of remote work and learning is so far just the tip of the iceberg and I see plenty of opportunity in this space. Hopefully in forthcoming posts, I’d be able to expand upon these vague ideas and hopefully build the tools that I would would love to use in the future.