Your X-Factor Is Key To Being Rich, Happy, And Free

If you want to be rich, happy, and free, you must cultivate your X-Factor.

Your X-Factor can be building something on the side that makes money beyond your day job. Your X-Factor can also be developing a top 1% talent in a sport, music, or art that provides you a tremendous amount of joy.

Without an X-Factor, life may not be as fulfilling. Let me share why everyone should develop an X-Factor even if they have the best job on earth. Because goodness knows, living to work will eventually lead to tremendous regret.

Build Your X-Factor Before Your Energy Fades

Out of the 500 or so college graduates I interviewed over a 13-year period in finance, every candidate was extremely enthusiastic. They were all for getting their butts kicked working 14-hour days so they could reach financial independence sooner than the average person.

When you could crack the six figure mark after your first full year, why not bear work torture to get ahead right? The situation is similar to an entrepreneur working 80 hours a week so she doesn't have to work 40 hours a week at a day job forever.

In the beginning, making outsized money requires outsized work hours. After all, almost all of us are cost centers when we first graduate from high school or college. Learn in your 20s, then earn in your 30s and beyond!

The unbridled enthusiasm of many 20-something year olds is quite common. Several of them used to tell me things like, “Sam, I love my job. Things are just so wonderful around here. I couldn't be happier!”

But to the 40+-year-olds hearing this type of joy, all we can do is think, you just wait and see.

You Must Forecast Your Misery

After these young bucks reach the 10-year mark, enthusiasm for work tends to drop off a cliff. Suddenly, doing the same old thing day in and day out isn't so exciting anymore. Further, the more money you have saved and invested, the less willing you are to put up with office politics.

Complaints about work start to rise. Those who ask me for advice about how to negotiate a severance package often tell me about how their micromanaging bosses are making them miserable.

They tell me that a co-worker is stabbing them in the back. Or that they got passed over for a raise and a promotion. Bitterness starts building up beyond the 10-year work milestone! All people want to do is get the hell out of there and do something more meaningful with their lives.

Without an X-Factor, it's hard to make a move. Not only have they not developed an X-Factor, their savings accounts are underfunded. As a result, they end up staying miserable at a job they hate for far longer than they should.

But as we finally exit this damn pandemic, many of us want to also get on with our lives. The YOLO Economy is just waiting to be lived.

Trying To Survive The Finance Industry

After my first six months of working on Wall Street, I felt I had very little chance of surviving the brutal work hours for longer than three years. Every day felt like we were in a pot of boiling water.

Doing well for three years is what was required to get promoted to Associate. But after a year and a half, it was evident I wasn't going to ascend. As a result, I saved all I could in order to provide myself a means to escape or survive if I was laid off.

Instead of renting a one-bedroom to split with a high school buddy, we decided to rent a studio that was walking distance from my job instead. We figured the studio was just a slightly more luxurious version of sharing a college dorm room.

Rather than constantly eat out at all the glorious food establishments in the city, I decided to chow down on the free cafeteria food at 85 Broad Street and doggy bag leftovers for breakfast. Part of the reason why I gained 15 pounds was because I felt like a bear who had to overeat in order to live off my fat during winter.

All told, I was able to save roughly 40% of my $40,000 a year income after taxes back in 1999. As my income thankfully grew over the years, I maintained a 50-70% after-tax savings rate because I continued to remind myself that saving money was the only way to escape.

My math was simple. Every year I worked and saved 50% of my after-tax income equated to one year of living expenses. Freedom and fear motivated me to no end.

My mini-X-Factor was to build enough passive income to buy me enough time to figure out what's next. The plan was working well until the financial crisis hit.

The Financial Crisis Derailed My Plans

Between 2008 and 2009, I lost anywhere from 35%-40% of my net worth in a matter of months. It felt like the past 10 years of diligent savings was for nothing.

I was depressed and angry that I continued to invest so much in stocks between 2006 – 2007. Back then, I still had ~15% of my net worth in CDs. But that wasn't enough to protect me from Armageddon. This clear memory of buying too much near the top of the market is why I've been de-risking my stock portfolio.

Losing so much of my net worth spurred me to finally act upon an idea my father had told me back in 2006. He realized that I enjoyed writing because he received all my investment newsletters that I had written for my clients. If you're a subscriber to my newsletters, know that I had 13-years of training before I began.

My father told me to go start a website and potentially earn a living as a writer. I brushed off his advice because I was too busy. I had just finished up my MBA part-time in 2006. All I wanted to do was relax a little.

For three years, I took Saturday classes for nine hours on top of regular 60-hour work weeks. I also had to spend at least five hours a week on group projects and homework. But when the crash happened, I finally hired some guy off Craigslist to help me launch Financial Samurai in July, 2009.

Financial Samurai became my new X-Factor.

Thoughts On Building Your X-Factor

My X-Factor set me free from a 13-year career I no longer enjoyed. But it took a damn financial crisis to get me to focus. My hope is that the global pandemic has jolted millions of people into cultivating their own X-Factor as well.

We know plenty of people plan to retire early once they've been fully vaccinated. They've gone through a pre-retirement checklist to make sure they don't mess things up.

But for those of you who are still relatively young and looking to change careers or try something completely new, your X-Factor will give you the courage to change.

Here are some thoughts on building your X-Factor today.

1) Set aside some time to think. 

My mother used to always encourage me to meditate for five minutes before going to bed when I was a kid because I was a naughty rascal. I found that when I did meditate, random epiphanies seemed to come to me like, “If you punch a kid in the solar plexus for hurtling racial slurs, you will get detention.” This was one of my first experiences with racism as an elementary school kid.

Life always gets in the way of calm. We have work to do, TV to watch, and families to feed. That said, I highly encourage everyone to set aside 15 minutes a week to think about ideas. You'll be surprised at what you'll come up with. Keep an open mind and let everything that creeps into your white room a chance.

Today, I like to go to my hot tub for an hour or two to come up with new ideas. Sometimes I will dictate a post. Other times, I'll just listen to a podcast. An idea always comes up while I'm soaking.

Figure out what helps fuel your creative side. If hot tubbing isn't your thing, perhaps taking a walk, drawing, or journaling could be your creative outlet for new ideas.

2) Spend money to get started.

Part of the reason why I didn't launch Financial Samurai sooner was because I didn't know how to launch a website. It wasn't until the great depression of 2009 that I finally forced myself to pay someone to help me. Now I’ve got a guide to make starting easy.

The first person I paid got around $350 and that site didn't last for more than 6 months. The second person I paid got $800, and the site lasted for four years until my redesign.

Don't let your lack of knowledge prevent you from starting. Spend some money to hire someone with expertise to get you going. Inexpensive resources are everywhere online today. There are plenty of freelancers ready to help you create.

Craigslist, Upwork, and a bunch of other sites are easy ways to find qualified people to help you fix problems. The freelance economy is booming during the pandemic! Perhaps your X-Factor is becoming a freelancer yourself.

3) Moonlight until you gain momentum. 

I wrote on Financial Samurai for almost three years before leaving my job. It was after the second year when I realized that maybe I could do this full-time. But I was still too scared to move because my job was all I knew for 12 years at the time. Further, my base salary was $250,000.

All I wanted to do was write and connect with people because it was so fun. I encourage everyone to spend several hours a week working on your ideas. The goal is to gain some traction. Once you have enough momentum, then you can better calculate when it's time to move on from your day job. Even if you fall flat on your face, you have many options to recover.

Let's say you have music talents. Why not start a website and offer music lessons online? Maybe you can start off with one music student a week and go from there. Let's say you are a great cook. Why not start a website and offer cooking lessons on the weekends? You will never know unless you try.

The one thing this pandemic has encouraged all of us entrepreneurially inclined folks to do is leverage the internet. Owning an online business that can't be shut down is almost like having a super power. No matter how much the government decides to crush offline business, so long as your website is legitimate, you can keep on going.

Nothing In Life Stays Static

After moving out to San Francisco in 2001, I thought I wanted to work in finance until I was 40 (2017). But by age 34, I was burned out. Pay at my firm also dropped, no matter how well we performed. I was working longer hours with more stress.

The industry I mostly enjoyed for 12 years stopped being fun anymore. I wanted out. Luckily, I figured out how to negotiate my way out with a severance. It gave me five years of breathing room to try out my main X-Factor.

Since 2009, Financial Samurai has provided me so much joy. The website has enabled me to regularly create something from nothing. During the darkest times in 2020, this X-Factor helped keep my mental health up. The revenue generated from the site is a nice bonus. But it's really the creative outlet that I cherish the most.

Don't wait until you are unsatisfied with your job to start working on your X-Factor. Give yourself at least an option to do something else once you leave your job. It might take years of cultivating before you decide to take a leap of faith. But believe me when I tell you that when the time comes, having a choice is priceless.

X-Factor And Maslov's Hierarchy Of Needs

Small Things To Keep You Motivated

If you still enjoy your job after so many years, you have hit the jackpot. Instead of working on an X-Factor that might one day extricate you from your job, work on an X-Factor that gives you joy and meaning outside of work.

Last year, I tried to become the best softball player possible. Every Saturday morning, I looked forward to batting practice and fielding drills. It was the perfect outdoor pandemic activity to keep me sane and motivated.

This year, I hope to finish a unique and useful book with Portfolio, a Penguin Random House imprint. If I do, it should be released in Fall 2022. How cool would it be if the book turned out to be a classic from some random guy who decided to change his path years ago.

As I come to the end of this post, I realize having an X-Factor is mainly about creating purpose and abundance in our lives.

Having financial independence is one thing. But it's not enough to feel fulfilled and happy. What we all need is to have something meaningful to do every day.

Please don’t wait until you are miserable to cultivate your X-Factor. Work on it today.

Related post:

How To Feel Rich Even If You Can't Technically Be Rich

Solving The Happiness Conundrum In Five Moves Or Less

Readers, have you been cultivating an X-Factor? What are some things you'd like to do, but never bothered trying due to the business of work? After how many years did you start feeling burnt out and wanting to do something new?

18 thoughts on “Your X-Factor Is Key To Being Rich, Happy, And Free”

  1. Great post!

    I think among the many, many important topics you hit here, was talking about taking time to think. I watch so many people want a better life for themselves, a life more fulfilling (because they finally realize the big paycheck just doesn’t do the job) but they don’t take time to sit in silence and actually think. Think about what they really want, what really brings them happiness. And then, think about the steps it would take to get there.

    I like to think we have the power to ‘bend the universe’– I truly believe with passion, determination, and a wicked work ethic anyone can achieve anything they want in life.

    But you have to stop to quiet your mind, quiet the noise and all the chatter. Allow your brain to make use of everything it has rolling around in there. Especially for creatives (like myself)–this is so crucial.

    If I didn’t take time to think about what I wanted in life I would never have made it to where I am now because I wouldn’t have been able to even come up with a plan. It doesn’t matter if the plan doesn’t pan out, you’ll come up with another one, but you’ve got to start with one in mind.

    Last thought, I love how you address the few that might really love their job and how this post still fits them too.
    I still love teaching and still do to this day– even though my investments bring in more money in a year than I could if I taught over several years. But some jobs will never allow you to retire YOUNG, with TIME left over to use as you want.
    And some won’t fulfill other aspects of who we are (I need constant creative outlets and intellectual stimulation and after over a decade in academics it’s not as stimulating as it was).

    So looking for other avenues of work to do, in addition, is great advice on your part. Plus multiple streams of income in one of the best ways to amass the wealth needed to retire and stay retired, right?

    Great read, great advice. Thanks Sam!

  2. There are some days I can’t believe my X-factor left. All I want to do these days is work 40 – 60 hours a week, make six figures, and wait a couple more years until I can start my own lifestyle business.

    It’s hard knowing that all the struggle I’ve been doing still hasn’t meant I reached my goals.

  3. Another interesting and thought provoking article. Setting aside time to think about the future is something we all need to do more often. This pandemic reminds me that your life can completely change overnight with no warning, and no fault of your own. If your aren’t thinking about what you want to do next in your life change can be extremely painful and depressing.

    My next X factor will be to have happy and productive retirement! I am enjoying my work, and retirement is a couple of year away, but if something happened tomorrow I could adjust.

  4. Love me some X-Factor! It helped me to sell one business, and then find new business partners to build another.

    You are spot on that there is a huge need to build in some time to think. I started morning page journaling shortly after I left my CEO gig, and it has been an immensely helpful way to hone my thoughts, vent, and focus on the now. I simply pull out my notebook and write while my kids watch TV when they first wake up in the morning. It is a great way to start my day and ensure that I have thinking time EVERY day.

    Also love the hot tub recommendation. No better way to relax than a hot tub or bath as well!

  5. This is such a great post. I feel like finding an X factor is so important. It helps you enjoy your life and feel like you are thriving vs surviving.

  6. Ms.Conviviality

    It was probably around year 9 of working as an internal auditor that I was almost desperately looking for a way out. Work was fine but I was bored. I also have a very creative side that I can’t hold back so I was often doing something artsy on the weekends. Since then, I started a side business doing wedding flowers, baked custom order cakes (think Cake Boss), and started an Etsy shop selling digital art. None of these creative endeavors have made enough to replace my work income though so I’m now 17 years into my career and still searching for a way out. It’s kind of difficult to replace my income when I’m in the top 10% of earners in my county. Currently, when I’m not at work, I’m putting all my efforts into real estate investing. If all goes according to plan, I can seriously consider leaving work within two years.

    1. I think things will turn out.

      Of course it would be nice to have your X-Factor completely replicate your day job income. But with the amount of money you’ve saved and invested over a 19-year period, having something meaningful to do that also provides supplementary retirement income is going to be awesome.


  7. What you say about anticipating your misery is true. It doesn’t mean you will definitely become miserable but building an x-factor is a smart way to have more options. Having more options and freedoms leads to more happiness for sure. Helpful tips and advice!

  8. Marie Jacobs

    All the time I spent thinking and building a side hustle / 1% talent on top of my day job and family just led me to burn out quicker until I had to give it all up after my health crashed. I’m so much happier now on the slow living train (low cost, low stress, zero time spent on any paid hustle). But I do tell the ambitious 20-somethings in my life to save and invest as soon as they can because life often takes a turn and it’s better to prepare for more options than to scramble with a frantic now what. I like the advice to add thinking time to imagine other possibilities as well.

    1. Yeah I am more focused on just saving and investing so that I can go without any paid job within a few years. It’s not that I dont want to do something or be of service, but frankly, as soon as you do something for money it starts to ruin it IMO

    2. That is true. Right now is tough for many to do given so much has gone on in the past 15 months.

      The key is to work on your X-Factor BEFORE one is burned out with so many responsibilities. It is hard to forecast one’s misery.

      So perhaps the best rule is to work on your X-Factor when you have the most energy, the most amount of enthusiasm, and the most amount of fun.

  9. This is great advice for people starting out in life. Myself, I’m going in the opposite direction. With taxes and regulations at the state, Washington State” and federal level going up I’m gonna do everything I can to eliminate my X factor. For me it’s YOLO.

    1. The missing item or “X-Factor” is so damn true. You can have millions and still feel unfulfilled and unsatisfied. One of the ways I have minimalistic filled a small caveat is by studying for
      professional Licenses as current one is for Mortgage Loan Originator. God knows I might even be able to use to seek solid employment whether full or part time hustle.
      Good work my man Sam!

  10. Great advice on finding an X factor and on how to do it. Setting aside time regularly (ideally each day) will help turn it into a habit so it sticks. Based on my experience, the earlier in the day you set aside this time the better so that you don’t get too busy with other things. Kind of like booking the first appointment for the day before things get backed up.

Leave a Comment

Your email address will not be published. Required fields are marked *