One problem I've found with the recommendations of professionals about collecting is their expectations are too high about what people can afford and how much depth they want to get into the hobby. A young child will not have the means (or attention span) to maintain a professional-quality collection; teenagers may want to be more involved in the hobby but can't afford the more expensive pieces and equipment. Or, someone could have the means but not want to go into the hobby as deeply as the professionals. These suggestions try to take into account different levels of interest and different amounts of money available to pursue the hobby.
All you really need is an interest in coins, a sharp eye, and a box to put them in. You can look in your pocket change, in penny dishes at department stores, at flea markets, or in the cheap bins at coin shops. Don't worry about if it's worth anything- just start with whatever strikes your fancy. If you want to take care of your coins so they don't get scratched or dented in the box, you can put each one in a small paper envelope before you put it in a box.
At this point, you should consider something to protect each individual coin, such as the little paper envelopes or albums (cardboard albums designed for a series of coins, like pennies, can be very cheap, as well as some of the plastic albums). You may also want to invest in a book for identifying and/or pricing coins. A magnifying glass would be helpful too but it's optional. If your collection starts to get large enough, you may want to start some sort of inventory of your collection. It can be as simple as a notebook or index cards, or you could look around for a catalog program (they have some cheap Shareware ones). You can learn more about the history of the coins themselves from the public library or the internet. You'll probably be able to afford some of the more common coins in the coins shops.
Your interest level and your budget are the only limits for you! Look at the FAQ sheet to get an idea of what's out there and how in-depth you want to get. In any case, start slowly and simply at first- this way, if you find you're losing interest in the hobby at some point you won't have lost too much money buying materials and expensive coins. If you find your interest increasing over time, gradually upgrade your supplies (a better storage setup, a more thorough book, etc) and read up on the more expensive coins before adding them to your collection. If you feel you've progressed beyond the basics but don't know how to make the next step, look through the Now What? section to pick up some ideas.