It’s been six years and three months since they took my paycheck away. I sometimes stay up all night and sleep all day. At least I can do whatever I want and see professionally whomever I choose.

Gourmet sushi at fancy restaurants tastes so good. But sitting alone mixing wasabi can’t take away those occasional blues. Because nothing compares, nothing compares to the camaraderie of you dear ex-colleagues!

For all the glamour of living an early retirement lifestyle, there are plenty of negatives I’ve come to discover since I permanently left my job in 2012. I know why we revert back to our baseline state of happiness, no matter how much freedom and money you have.

Let’s go through some of the negatives of retiring early now that I’m a grizzled veteran.

When I first started Financial Samurai in 2009, I was proud of finally creating something. For too long, I had just followed orders. Yet, I couldn’t tell anybody about my hobby because I was still employed. I was afraid if my employer found out, they’d tell me to shut it down or at the very least, dock my bonus for being unfocused. After all, the site was started in the middle of the financial crisis and they were always looking for reasons to cut costs.

When I finally engineered my layoff three years later in 2012, I still felt it was unwise to tell anybody about my site even though I was internally beaming with pride about the site’s growth. I felt like a new parent who wanted to annoy the crap out of all his friends by cooing about every milestone their child hit.

But because I had five years of deferred compensation on the line, even if there was just a 1% chance my old employer would withhold payment, I didn’t want to risk losing years of living expenses. And so, I kept my mouth largely shut until I had a decision to make on 4/21/2017, when my last severance-related after-tax payment hit my bank account.

In a way, the five-year time period when I was collecting my deferred compensation was like a mental jail. Like Andy Dufresne in The Shawshank Redemption reaching for the stars as rain spatters his face, full emancipation only came years after leaving Corporate America.

With nothing holding me back in 2017, it would seem like finally, I could brag to everyone that I had created a website large enough to support a family in San Francisco. With some publicity would come growth. But alas, I felt it was much better to keep a low profile.

For those of you who are thirsty for attention, let me share some thoughts about being a nobody for so long and why it may be in your best interest to do the same. 

Financial loss creates stress. And given stress kills, your goal as a retiree should be to remove as much stress from your life to live as long and happy as possible.

Recently, I’ve been thinking about being OK with no longer making a lot of money from investments. With such huge returns in the S&P 500 and the SF real estate market since 2012, I kind of feel like Anthony Bourdain when he said,

“I should’ve died in my 20s. I became successful in my 40s. I became a dad in my 50s. I feel like I’ve stolen a car — a really nice car — and I keep looking in the rearview mirror for flashing lights. But there’s been nothing yet.”

I got in trouble the summer before starting college and could have easily given up on my future. But in college, I met my wife and landed a hard-to-get job for someone with my lack of pedigree.

Two months before likely being terminated at my first job as the internet bubble burst in 2000, I was able to find a new job in San Francisco for a raise and a promotion.

Instead of selling a home close to the bottom of the market in 2012, I was blessed that nobody took advantage of my uncertainty.

Instead of flaming out with Financial Samurai and having to go back to work a couple years later, this site has become a viable source of income to support my family.

Sure, there have been plenty of setbacks as well, but I’ve chosen to focus on the positives because life is better this way.

With low returns and a stable income, life becomes magnificent in retirement. All the worry about running out of money or having to go back to work full-time starts to fade away. 

Ever since landing my first job post college in 1999, I’ve been determined to build enough passive income in order to not have a job. A future that included getting into work by 5:30am and leaving after 7:30pm each day for decades seemed too brutal to endure.

In 2009, I decided that if I could earn about $80,000 in passive income, I would leave my job permanently and work on Financial Samurai while traveling instead. So I left work in 2012 at the age of 34. Once my wife decided to join me in 2015, also at the age of 34, I decided to shoot for $200,000 in passive income.

With $200,000 a year in passive income, I would have enough income to provide for a family of up to four in San Francisco given we bought a modest home in 2014. Now that we have a son, I’m happy to say that $200,000 indeed does seem like enough, especially if we can win the public school lottery to avoid paying $20,000 – $50,000 a year in private school tuition.

The Keys To Building Enough Passive Income

1) Save until it hurts each month. Most people think they are saving enough through their 401(k) or IRA, but they are not. Developing passive income requires an aggressive after-tax, after-401k/IRA savings amount each month because you can’t draw from your pre-tax retirement accounts before the age of 59.5 without a 10% penalty.

You must sacrifice the pleasures of today for the freedom you will earn tomorrow. In my 20s, I shared a studio with my best friend from high school and drove beater cars worth less than 10% of my annual gross income. I’d stay until after 7:30pm at work in order to eat the free cafeteria food. International vacations were replaced with staycations since work already sent me overseas 2-4X a year. Clothes were bought at thrift shops of course.

2) Focus on income producing assets. Internet growth stocks may be sexy, but they provide no income. To build a large enough passive income stream to survive, you must invest in dividend generating stocks, certificates of deposit, municipal bonds, government treasury bonds, corporate bonds, and real estate. You’re free to invest in non-income producing assets for capital appreciation too. You just want to earn reliable income when the day comes to leave your job.

My favorite type of semi-passive income was rental property because it was a tangible asset that provided reliable income. As I grew older, my interest in rental property waned because I no longer had the patience and time to deal with maintenance issues and tenants. Online real estate became more attractive, along with tax-free municipal bond income once rates started to rise.

3) Start as soon as possible. Building a livable passive income stream takes a tremendously long time largely due to declining interest rates since the late 1980s. Gone are the days of making a 5%+ return on a short-term CD or savings account. Today, the best 12-month CD is at 2.5%, and the best money market rate is around 1.85%, which is not bad considering such rates were below 0.5% just a couple years ago. Know that every $100 you save can generate at least $2.5 in passive income.

I knew I didn’t want to work 70 hours a week in finance forever. My body was breaking down and I was constantly stressed. As a result, I started saving every other paycheck and 100% of my bonus since my first year out of college in 1999. By the time 2012 rolled around, I was earning enough passive income (~$78,000) to negotiate a severance and be free.

4) Calculate how much passive income you need. It’s important to have a passive income goal, otherwise, it’s very easy to lose motivation. A good goal is to try and generate enough passive income to cover basic living expenses such as food, shelter, transportation, and clothing. If your annual expense number is $30,000, divide that figure by your expected rate of return to see how much capital you need to save. Unfortunately, you’ve got to then multiply the capital amount by 1.25 – 1.5 to account for taxes. For example, $30,000 / 3% = $1,000,000 in capital needed to generate $30,000 gross. But since you must pay tax on the $30,000 income, you really need closer to $1,250,000 to generate $30,000 in after-tax income at a 3% rate of return.

With the birth of our son in 2017, we’re aiming to generate $250,000 a year in passive income by the time he starts kindergarten in 2022.

5) Make sure you are properly diversified. Capital preservation is underrated. We saw a lost decade for tech stocks between 2000 – 2010 after the first dotcom burst. It actually took 13 years for NASDAQ investors to get back to even. Investors in the Borsa Istanbul Turkey stock market index just gave up 10 years worth of gains after they saw a plunge in their currency, partially due to increased tariffs by the US and no-confidence in the government. Your passive income needs to be properly diversified in order to take the hits.

I currently have 10 main sources of passive income as you’ll see in the chart below, with bonds as my largest source at 30.4% of total. I’ve worked through the 2000 and 2008 meltdowns, and don’t plan to ever lose so much money again. 

Congratulations everyone! According to this chart below, everybody is rich in America! Our assets are at the highest level compared to our income in all of history thanks to tremendous asset price appreciation.

Anybody who bought stocks and real estate between late 2009 and 2015 is sitting mighty pretty. When you’re sitting mighty pretty, you tend to spend more frivolously and take on debt to buy things you don’t really need. But at some point, a percentage of the population gets carried away, commits financial suicide, and brings the rest of us down with them.

I’m really hoping the good times will last for several more years. After all, it’s more fun making money hand over fist than losing money. Alas, I’m worried that with cash-out refinances reaching pre-crisis levels, we’re bound to repeat our mistakes from the previous financial crises.