The Downside Of Retiring Early: Beware Of Absolute Freedom!

There is a downside of retiring early that many people don't discuss. Instead, folks just focus on the positives of retiring early.

Not having to work a miserable job is a dream for most people. For 13 years, I toiled away in the stressful world of investment banking that gave me constant back pain. My bosses went through revolving doors and my clients were extremely demanding. Finally, at the age of 34, I negotiated a severance and never looked back.

It's been wonderful to be free since 2012. My wife and I have traveled all of the world for months each year. Now we're both stay at home parents to our wonderful little boy and daughter until they goes off to kindergarten.

We're even considering relocating from San Francisco to Hawaii as our home base because we have the passive income to allow us to live anywhere in the world. With the pandemic raging on, it would be nice to finally live near my parents in a tropical paradise.

But for all the glamour of living an early retirement lifestyle, there are several negatives I’ve come to discover since I permanently left my job. Let's discuss the downside of retiring early in more detail. Having absolute freedom is ironically one of the biggest downsides of retiring early!

The Downside Of Retiring Early

1) You will lose your identity. 

Your identity crisis may last as short as three weeks or it might last for years. It all depends on how wrapped up you were in your job, how long you spent getting educated after high school, and whether you have a clear plan post-retirement.

Job titles can be incredibly addictive. Why else do people try so hard to get promoted sooner and faster than everybody else? Do not underestimate the importance of being a manager, director, vice president, or even a C-level executive.

After all, the most common question people ask when they first meet each other is: What do you do for a living? And if you tell them you don’t do anything for a living, well then you might just feel like a loser. You’ll want to try to explain yourself, but by then, your first impression will no longer hold the other person’s attention.

What happened to me:

After working in banking for 13 years, it felt hollow to no longer have my Director title or be identified with my firm. I felt sad that I could no longer go to Asia for conferences or with clients. I felt important when clients would entrust me to show them around in a foreign land. 

For the first year after leaving my job, I wondered how the business was doing without me. Could they really survive without my expertise? After all, I was there for 11 consecutive years. Surely, they needed my relationships. But after months went by with no email or phone call from my old firm saying they wanted me back, I had to come to terms that I was no longer important to them.

I wanted to believe that my position meant something to the firm and to the people that I serviced. But at the end of the day, the person I trained to replace me as part of my severance agreement, was good enough. And because he was good enough, I concluded that I was no longer any good.

This ego hit took me a full year to get over.

The downside of retiring early

2) You'll feel completely stuck. 

Locked-in syndrome is a condition in which a patient is aware but cannot move or communicate verbally due to complete paralysis of nearly all voluntary muscles in the body except for vertical eye movements and blinking. You will get this way have you lose your identity.

When you suddenly have an extra 10 – 14 hours a day of free time, it’s very difficult to optimize your time wisely. Just like how kitchens naturally get messy, your mind naturally gets lazy the more time you have to do anything.

Your productivity will suffer in retirement. You will no longer feel motivated to achieve great wins. As a result, you may slowly start to get depressed. Only after some really deep soul-searching and some, what the hell am I doing with my life questioning will you begin to organize your time better and become more productive.

Your mind can be very dangerous because it can always second-guess your actions. Did I retire too soon? What if I run out of money? When you have a lot of time to think, your doubts go on and on.

What happened to me: 

Because I left work at the age of 34, I was worried for the first two years whether or not I had made the right choice. No rational person leaves a well-paying job to be unemployed in their mid-30s. Your late-30s is when you start to finally make good money. And by the time you reach your 40s, you should be at your maximum earnings power. 

During my first year of early retirement, to the outside world I proudly proclaimed I was retired from a career in finance. But on the inside, I was second-guessing my decision to leave. Because of my uncertainty, I decided to do some part-time consulting with some fintech companies like Personal Capital for ~25 hours a week. It was a great way to distract my mind from all my fears, earn some side income, and replug myself into society. I also kept in touch with multiple banks until my Series 7 and 63 licenses expired.

Finally, I dived deep into my writing on Financial Samurai. Writing has always been my most cathartic way to deal with any uncertainty or problems I might have. For example, now that I have a son, I've been worried about whether our passive income is enough to support a family of three.

As a result, I did a deep dive budget analysis for a family earning $300,000 a year, and it sure seems like this income is what's required if our boy can't get into a good public school. Alternatively, we can always move to a lower cost of living city.

3) People may treat you like an outcast.

Whether it's because retiring early is unconventional or because people are secretly jealous you aren't grinding away at a day job, people won't give you the same amount of respect as working class citizens. After all, if they can't describe what you do for a living, then they can't pigeonhole you into an archetype that is comfortable for them.

Having a job means you are a productive member of society. If you retire at a young age, people will assume you are simply slacking off and not paying any taxes. They'll sometimes look at you as a leech they want to flick off.

Further, if you are an outcast, then you won't be invited to social functions that other working people always get to attend. You're simply not top of mind to them. If you are an extrovert, early retirement will be much more difficult than if you are an introvert.

What happened to me: 

After the first year of early retirement, I no longer told anybody I retired early. Instead, I told anybody who asked that I was a writer, a tennis teacher, a fintech consultant, or simply in between jobs. Prior to that, I think a lot of people just assumed I was a trust fund baby who did not have to work. And the last thing this middle-class guy who went to public school wants to be known as is a trust fund baby. 

My favorite time of the year was during the winter holidays. I loved going to all the holiday parties and getting tipsy with fellow revelers. Now, I get invited to zero holiday parties because I don't work for anyone. Nor do I get invited to client holiday parties either, even though I have several partners who are based in the SF Bay Area.

It takes a lot of effort to build new social networks if you aren't part of a larger organization. There is no weekend BBQ party a colleague is hosting on Labor Day Weekend to attend. I've had to participate in various meet-up events in order to find new people to hang out with. So far, my social network only revolves around tennis and softball. But even then, it's not like I've found buddies who will come over and just chill in the hot tub over a beer or anything. 

4) You'll be disappointed that you aren't much happier.

So many people think that once they achieve financial freedom or leave a job they dislike, they'll suddenly be permanently happier. The truth of the matter is, your elevated happiness will only last at most six months. Eventually, you'll revert to your natural state of being.

Think back to your high school or college days when you didn't have any money compared to now. I'd venture to guess you were just as happy, if not happier when you were a broke college student.

Having the freedom to do what you want is priceless. But you will eventually take your freedom for granted like the air you breathe. On the days you feel angry or sad, you'll feel stupid for feeling unhappy when there are literally hundreds of millions of people in the world wondering whether they'll have enough to eat the next day.

You think, if I can't be happy when I'm financially independent, surely there must be something seriously wrong with me. And you could be right! Can you imagine being unhappy as a Norwegian? Norway is perpetually ranked as one of the top five happiest countries in the world.

What's going on with me: 

I thought I'd be much happier not having to report to a micromanager boss I did not respect. He was out of shape and uncharismatic. But my increased happiness was fleeting and only lasted for about a month before I was back to my regular self. Instead, my happiness was weighed down by months of uncertainty on whether I had made the right move to leave my job.  It was only after about two years did my doubt finally start to dissipate.

Although corporate politics no longer piss me off, other things end up filling the void. For example, drivers who decide to double park on a busy street in rush hour traffic really piss me off now. In the past, I could only allocate a small amount of annoyance to such incidences. 

I’m generally a happy-go-lucky type of guy. But I probably joke a little too much to my detriment because there are plenty of stiff people who easily take offense. With social media, it seems like getting outraged has become a national pass time!

For example, I had a bunch of people on Twitter get pissed off when I wrote, “Would you suffer through the indignity of going to public school for $1,000,000?” They basically told me I was an elitist a-hole for jokingly using the word “indignity” when if they read the post, they would realize I was all for going to public school and accepting a big fat check upon graduation.

Instead of being permanently at a happier level, I'm simply no longer as annoyed or as angry at things as frequently. Further, the volatility around my steady state of happiness is lower. In other words, I’ve mellowed out. That said, don’t offend me because I still enjoy a really good fight!

5) You constantly wonder what's next in retirement.

Retiring early is like finishing up your favorite longstanding TV show. You’re glad there’s a conclusion, but you’re also sad that it’s over. You hope to find a show that’s as good or better, but there are no guarantees. Oh how I miss Breaking Bad and The Sopranos!

Most of us spend 13 years going to grade school so we can spend four years in college in order to get a decent job. Then we spend decades trying to earn and save money in order to provide for our family and then one day retire by 65. With good luck, we'll live for another 20 years to enjoy all the fruits of our labor.

When you retire at a much earlier age, you are constantly left wondering what's next. You are mentally twiddling your thumbs waiting for the next big thing while your close friends are all at work. Early retirement can get extremely mundane and boring because you have nobody to spend time with.

As a result, you're repeatedly forced to will yourself into action. This constant self-starting attitude can become extremely trying to the point where you long to rejoin the workforce and be told what to do.

What's going on with me: 

Everybody gets bored at some point. When you're working, you don't have time to get bored because you're working. There's only so much tennis, golf, and softball I can play before my knees break apart. There are only so many churches to visit in Europe before they all start looking the same. 

My wife used to have vacations from me because I would be away traveling for work every month. Now she was seeing my silly face every single day! It's a good thing we had three bedrooms at the time. Otherwise, I'm pretty sure we'd both have gone crazy from seeing each other so often. 

It was only after our son was born in early 2017 that I felt a renewed sense of purpose. Before my boy, I felt my purpose was to help educate as many readers as possible about personal finance in order to one day be free.

After my boy was born, my purpose has expanded to keeping Financial Samurai running long enough to teach him about operating an online business out of fear he may have a tough time getting ahead due to his mixed race and his nystagmus, a visual condition where he likely won't have 20/20 vision, despite glasses or contacts.

Further, I now need to live long enough until he finds someone who loves him as much as I love my wife. I don't think I'd be able to die in peace if there's nobody to replace his mom or me. As a result, I'm exercising more, eating healthier, and meditating longer. 

The downside of retiring early

6) Health care costs are outrageous.

Only after you leave work will you truly appreciate all your work benefits. When I was working, I was paying about $280/month for healthcare. My wife didn't have to pay any premiums when she was working. Once we both left our jobs, we had to pay $1,300/month! Once we had our son in 2017, our healthcare monthly cost went up to $1,700.

In 2020, our healthcare cost is now $2,380 a month! At least it covers two adults and two young children who regularly see the doctors. But still, this amount of healthcare cost is too much. We should try and reduce our income to get some subsidies.

7) You're no longer paying into Social Security.

Not paying into Social Security could be a good thing or a bad thing. It depends on how big of a retirement passive income portfolio you have.

The amount of Social Security you get is based on how much you put in. If you stop working, or make less, then your future Social Security benefits will get negatively affected.

I personally am OK with not paying into Social Security given it is underfunded by ~25%. I'd rather build up my taxable invest portfolio and rely on myself. Who knows, by the time we are eligible for SS, we might be 70+!

Early Retirement Is Not A Cure-all

Early retirement savings rate chart

It might sound like after reading this article that I'm depressed. But I'm not. I'm simply highlighting some of the negatives I went through. You will probably go through some of these downsides if you decide to retire early too. The more extroverted you are and the higher power and pay your job was, the more you have difficulties making the early retirement adjustment.

Having the freedom to do what you want cannot be overstated. However, your mind will play games with your spirit during the first few years after leaving work. Some of you won't be able to handle early retirement life and will go back to work. Many of my peers have as well.

You must find your purpose in early retirement. Once you do, you will take the necessary steps to ensure you remain free forever.

Recommendation To Help You Retire Early

Manage Your Money In One Place. Sign up for Personal Capital, the web’s #1 free wealth management tool to get a better handle on your finances. In addition to better money oversight, run your investments through their award-winning Investment Checkup tool to see exactly how much you are paying in fees. I was paying $1,700 a year in fees I had no idea I was paying.

After you link all your accounts, use their Retirement Planning calculator. It pulls your real data to give you as pure an estimation of your financial future as possible . Definitely run your numbers to see how you’re doing. I’ve been using Personal Capital since 2012 and have seen my net worth skyrocket during this time. Better money management is huge.

Personal Capital Retirement Planner
Is your retirement plan on track? Find out for free after you link your accounts.

About the Author: Sam began investing his own money ever since he opened an online brokerage account in 1995. Sam loved investing so much that he decided to make a career out of investing by spending the next 13 years after college working at two of the leading financial service firms in the world. During this time, Sam received his MBA from UC Berkeley with a focus on finance and real estate. 

FinancialSamurai.com was started in 2009 and is one of the most trusted personal finance sites today with over 1.5 million organic pageviews a month. Financial Samurai has been featured in top publications such as the LA Times, The Chicago Tribune, Bloomberg and The Wall Street Journal.