Some sort of storage is a must. For more on this subject, see How should I store my coins?
A magnifying glass is optional but it helps to make out the details on dirty or damaged or small coins. You can start out with a very inexpensive magnifying glass, and eventually work your way up a high-power jeweler's loupe. Many coins supply stores sell these.
A small desk lamp is useful for seeing the fine details on coins and for seeing any flaws.
A price guide for identifying and/or grading them is optional but useful. There are many books available for coin collectors at all levels. There are hardcover and paperback guides for identifying and pricing coins, as well as how to grade them. I would suggest a fairly good quality book if it's just identifying coins since the coins themselves don't change very much, but get an inexpensive, up-to-date paperback for pricing them. Prices vary greatly and new editions come out almost every year so no one book will stay definitive for long. A favorite among collectors for American coins is A Guide to United States Coins, commonly called the Redbook, by R. S. Yeoman. It's published every year. The definitive guide for world coins is the Krause Standard Catalog of World Coins by Krause Publishers.
An inventory list of some kind is optional, too, but handy. Since it's not always easy to write on the envelope or baggie or album what the coin is, it might be easier to keep track of your collection with a notebook or index cards or a database. There are even software programs available that help you record and organize your collection. Having an inventory of your collection helps you avoid repeat items and talk with other collectors about what you have and what you need or want.
Now that you know some of the things you'll need to manage your collection, here are the places to find them:
Most coin shops sell some sort of storage supplies, although they may not carry the particular kind you want. Their prices may or may not be good; that varies a lot depending on the size of the store and the type of supplies. You can find local coins shops through the phone book under "coin dealers" or "collectibles", or online dealer directories (see the links main page).
A general hobby store might sell coin collecting albums, and baseball card stores usually sell the plastic envelopes for sports cards that work well as temporary storage for coins. You can find these in the phone book under "collectibles" or "hobby shops".
These stores are good for finding small paper envelopes. The larger stores often have a section for archival supplies. This is even better since archival supplies are acid-free (some office supplies don't use acid-free paper; the acid in the paper can damage a coin if it's stored in one for a long time). Sometimes office supply stores will sell photo and other albums or 3 ring binders. They often have boxes of different shapes and sizes, as well as locked boxes occasionally. You can find these stores in the phone book.
I've come across several suppliers online. I have a list of them in the links section.