There are many programs that will format your book for you. This is not an advertisement for that. In fact this is mostly how it is done in Word. Open office will be very similar. Smashwords has a more indepth guide if you want to try that. Except it is only good for ebooks and will give you a clean manuscript but it won't be very imaginative.
Get rid of all the hidden formatting you have no idea is there. You do this by copying your manuscript (Control A and then Control C) then open a new document and right click and paste text only.
There is even more formatting you didn't know was there and you have to get rid of it. If you click on the Paragraph button, It looks like a fancy backwards P, then you will see all the formatting marks. You will also see hidden spaces that you didn't see before. Get rid of spaces at the end of lines etc. Just tidy it all up. This button is a life saver in this project so remember where it is.
Set styles for the text. This is how you can do it in Word and this is how it is done in Open Office.
I have at least two but everything that looks different should have a new style made for it. Label them and set them as default as you will need them again. Go through and apply those styles to the different sections of your text.
I recommend an indented first line and Garamond at 12 for the paragraph as that is my favourite. If you want you can take the indent out of the first line. If you want one of those fancy letters at the start it gets trickier and I suggest InDesign if you want to get that fancy.
Page breaks. On the page layout tab you will see you can put in Page Breaks. Do this and a lot. end of a chapter yes, end of a section yes, after the front matter yes. It will make the difference for the next bit. One thing to pay attention to is that they are on the same line as your text as it will sometimes add in odd pages and make your text stretch across the page to fit. Better to drop it just below and you have less trouble.
The dreaded page numbers. I always have trouble getting the page number on the right side of the page once I upload it. No one really notices so don't worry about it too much but if you really want to get that right you will need something better than Word but like I said no one notices. Your Page breaks make this so much easier. Also, click on different first page. Start where the chapters start and click on the link to previous this will detach it from the front matter bit and then put in the page number. Format it by saying that you don't want to carry on from previous section. I like the numbers at the bottom and you can do any font you like as long as it is readable. This is a great place to give your book some character.
Header. I like to put my author name and the title of the book at the top but the name of the chapter is also a good idea if you have witty titles. This part can be skipped but I like it. To get two different things in the header just click on the tab and uncheck the odd and even box.
That is the bulk of it. You can add in chapter headings and section dividers and if you do make sure you use the insert button. Be careful to also make a style for this as the indent will put it off center if you just leave the paragraph style in place. Make sure the image is the right size when you put it in. Some converters will muck it around if it isn't and will revert it back to the original size. So size outside of Word not in Word.
Front matter and back matter are just the things at the front and back. The front matter is only important in print books as the ebooks usually start at chapter 1 anyway. I like to have a half page with the copyright and a list of other books. The back matter should have a Call to Action to either sign up to your website or to review the book. Snippets from books are good or sneak peeks.