A few days ago, a friend asked me whether we were gearing up for early retirement. My answer was pretty boring and lengthy. Yes, we are preparing for early retirement. Unfortunately, we’re not there yet.

At this point, I think we are on schedule to reach financial independence in around 11 years – when we’re 46. Around that time, our retirement accounts should be fully stocked. Meanwhile, our primary residence and rental homes will all be paid off the same year.

I have to be honest – retiring completely at 46 sounds like a dream. On the other hand, I can think of a million things that could go wrong. Here are five issues that seriously concern me when it comes to our early retirement dreams:

I worry about the cost of college.

When I decided to have children, I felt like it was my duty to pay for at least part of their college education. To achieve that goal, I started saving in a 529 for each of them as soon as they got a social security number.

Here’s the problem: When we turn 46, our kids will only be 17 and 15. In other words, we may have no idea how much we’ll need to pay for their college – or even where they want to go. This is part of the reason I doubt we’ll retire until they are through college. As we all know, there is a huge price difference between the cost of community college and an Ivy-League school!

I am concerned about the cost of health care.

As I mentioned earlier this year, we opted out of Obamacare and joined a healthcare sharing ministry instead. This was mostly because of our high incomes; since subsidies drop off for families of four who make over $95,000, premiums in our state and county were crazy-high.

When we retire, however, I might be willing to get a regular health insurance policy. Provided that the new healthcare law still stands, we might be able to get a much cheaper policy when we are living off of our investments. Still, that isn’t all that concerns me. I’ve also thought a lot about the cost of nursing home care and whether we should get long-term care insurance. I’m still undecided on that front, but the potential costs do scare me.

I worry about our investment timeline.

Since I plan to live a really long time, I also worry about our investment timeline. Even if we retire at 50, we could feasibly need to live off of our money for another forty years. This begs the question, will it be enough?

This is also part of the reason I think it would be wise if we had some side hustles in retirement, or even just pursued partial retirement. It would be nice to make it through to 60 without spending any of our nest egg, if possible!

I don’t want to be a burden on my children.

I’m extremely conservative when it comes to money, but that doesn’t mean I’m perfect. One thing that always keeps me up at night is whether we’ll make some kind of mistake and wind up being a burden on our children in old age.

As we all know, there are no “do-overs” when it comes to retirement. And if we underestimate how much we’ll need, we could end up regretting it. While I believe in a variation of the “4 percent rule” and formulate our plans around it, I still worry that I will mess things up somehow – and that it will hurt my children.

I worry I won’t be able to live the life I want.

Although there are a ton of reasons I want to retire early, the main one is simply being able to live the life I want to live. Who knows what that will mean ten or fifteen years from now, but right now it means plenty of travel, plenty of relaxation, and work without deadlines or a sense of urgency.

If we don’t save enough or set enough aside, I worry that we’ll end up poor in retirement. And that’s honestly the last thing I want. If I quit working early, I want it to be so I can have more freedom, not less.

How I’m Overcoming My Worries About Early Retirement

Although all of those concerns are real and legitimate, there are some things I don’t worry about at all when it comes to our early retirement dreams. For example, our primary home and rental properties will be paid off within 11 years, meaning we will be 100% debt-free. That means that, even if our plan doesn’t go perfectly, we will at least have a cheap place to live (we’ll still have to pay property taxes and homeowner’s insurance) and at least $2,000 per month in somewhat-passive income.

Another thing I’m not worried about is our current savings rate. Even though we are paying a boatload in taxes right now, we’re saving around 50% of our incomes. That’s one good thing about self-employment – even though we don’t get a 401(k) match or benefits, we are able to save 25% of our incomes – up to $52,000 per year) in a Vanguard SEP IRA. And for the last few years, we have been maxing out our Vanguard Roth IRAs as well.

The rest stresses me out, but there is little I can do aside from planning the best I can. Our goals right now include a) saving as much as possible in retirement and investment accounts, b) paying off our home and rental properties, and c) keeping our spending as low as possible.

We’re also working on a few side hustles that would be easy to do on a part-time basis. My hope is, even if we can’t retire completely, we can scale down to location independent jobs that require less than 20 hours per week when we’re ready. Since we’re location independent already, I think that should be fairly easy to achieve.

So I guess I’ll just keep planning. When it comes down to it, I suppose that’s all anyone can do to prepare for early retirement. I just hope I’m able to pull the trigger when I finally get there.

What worries you most about your retirement plans? What steps are you taking now to make your transition into retirement as seamless as possible?