Self Guided Walking Tours
You could start the day with an early lunch at Cafe Degas on Esplanade, open 11am-3pm), afterward walk up Esplanade Ave (lined with beautiful, traditional New Orleans mansions) and stop by St. Louis Cemetery No. 3, a traditional New Orleans-style cemetery - supposedly they also arrange tours though I think just wandering around for a bit may be sufficient).
From St. Louis, walk down beautiful Bayou St. John for a few minutes to get to the Pitot House, open for guided tours (10am-3 pm), a traditional Creole Colonial country home with guided tours that are supposed to be highly informative.
From Pitot House make the short walk up across Carrollton Ave, into City Park and towards the New Orleans Museum of Art. Depending on the weather you'll have a number of options. If it's nice outside you can check out the sculpture garden, located right next to NOMA - it's beautiful and free to access. In case of rain or heat, head into NOMA for "Where Y'Art," a weekly events series held each Friday starting at 5:30pm - you can get a cocktail and wander around the galleries while listening to live music, talks, etc.
For the evening you could head down towards the French Quarter (still a must-see, especially if there are folks in the group who haven't been to New Orleans before) via streetcar (it leaves right outside City Park and heads down Canal towards the quarter). The options here for food and music are obviously endless... for great food (high-end but still a bit off the beaten path) I recommend Bayona - Susan Spicer is one of New Orleans' most well-known chefs and this restaurant is fantastic. It's small so I would definitely suggest reservations. Bayona is known for its roasted garlic soup... Another higher-end/famous New Orleans chef option for food may be John Besch's Domenica, located inside the beautiful Roosevelt hotel. Even if you're not up for another drink, take a look inside the Sazerac bar where the famous cocktail was invented.
(If you wanted to stay in Mid-City for dinner there are obviously more off the beaten path options here than in the quarter... Redemption restaurant, located inside an old church, Cafe Minh for Asian-Cajun fusion, Mandina's for true New Orleans tradition, etc.... Angelo Brocato's gelato for dessert....)
Finally, for some music... for traditional New Orleans jazz Preservation Hall right in the heart of the quarter comes highly recommended. For a more casual/nightlife experience you can't go wrong with Frenchmen Street - lots of bars, clubs and great music (DBA, Cafe Negril, Snug Harbor, Three Muses...). You can just walk around and listen to great music everywhere. For those who have some energy left even after that... Cafe Du Monde is open (almost) around the clock and walking distance from Frenchmen - late-night beignets and cafe au lait are a quintessential New Orleans experience.
You could spend a day in Lower 9th ward, Holy Cross, Marigny/Bywater - however, this area is somewhat less manageable walking so maybe a tour would be a better option. I am sure there are tours of the Lower 9th available... not sure about Bywater/Marigny but I would be happy to look into it. My very favorite restaurant is located in the Bywater - Elizabeth's can compete with the most famous of all NOLA restaurants but is definitely off the beaten path... a nice option for dinner. Cake Cafe in the Marigny is one of my favorite brunch/lunch spots. Frenchmen Street and the quarter are an easy walk from the Marigny so you could end up there for the evening as suggested above.
The warehouse district/CBD could be another great option to spend the day - especially in case of rain. The World War II museum is fantastic and you can spend lots of time there, possibly with a guided tour as well. The Ogden Museum is also close. There are a number of very well-known restaurants around there - Cochon stands out because it is in walking distance from the WWII museum and its chef (Donald Link) just won the James Beard award for best Southern chef. My meat-eating friends have been there and loved it! There are a number of art galleries in the warehouse district which may be of interest to some as well. The Arts District has its own website with more information. From the CBD you could take a streetcar ride either up St. Charles into the Garden District (many people do a round trip just to get a tour of the beautiful mansions) or the short ride back towards Canal and into the quarter.
Finally, if there are a number of folks who haven't been to New Orleans and want to spend more time in the French Quarter, this walking tour may be a fun way to see it from a different angle:http://www.mondecreole.com/.
New Orleans does not stop with the French Quarter, although that is most famous part, the one you hear about. So, let's start there. Cafe Du Monde is where you go for cafe au lait(strong chicory coffee with a large splash of milk) and beignets(the square, airy powdered doughnuts New Orleans is so famous for). I prefer Croissant D'or Patisserie, on Ursulines Avenue, not as touristy, more of a local haunt. There are other breakfast places, and Court of Two Sisters lays a mean spread for brunch daily, full of local fare. One of my favorite dishes is Turtle Soup, with a touch of Sherry. Locals argue who makes the best, mine being a restaurant in Metarie that is no longer in business.
The Cabildo Museum is adjacent to the St. Louis Cathedral. There are a few Irish pubs for refreshment. The one I like is on 625 St. Philip, Flanagans Pub. An unpretentious place, it's open 24 hours and serves food. There's a mixture of people that go there, locals and tourists alike. It's a good spot to wind down after a night on the town. They also have a ghost tour. Pat O's is another place to sit on the patio and have their signature drink - a Hurricane. Preservation Hall has live Jazz music nightly.
For dinner, don't miss out on eating at one of the Brennans restaurants. My favorite is Mr. B's Bistro. Both Brennans and Mr. B's are located on Royal Street. I couldn't get into all the restaurants there are in New Orleans, in the French Quarter. There are websites dedicated to just that topic. I like urbanspoon.com and tripadvisor.com as they rely on consumer reviews. I will say that some do not allow shorts or t-shirts, and require jackets for Dinner. Reservations are usually needed.
In the Central Business District (CBD), down near the Morial Convention Center, there are other restaurants, museums (The Nationa WWII Museum, The Louisiana Childrens Museum, The American Italian Museum), the Aquarium of the Americas, riverboats(Steamboat Natchez), and Harrah's Casino. Herbsaint Restaurant and Bar, a Chef Donald Link restaurant, is close to Lafayette Square and Gallier Hall, popular in movie scenes. At the end of Poydras Street, on the Mississippi River, is the Riverwalk - mainly mall type shopping and small food kiosks.
Walking away from the Mississippi River eight blocks, you come to St. Charles Avenue. Catch a street car down the Avenue and see all the beautiful homes. You also pass Loyola and Tulane University. At the end of St. Charles Avenue, uptown New Orleans, where it intersects Carrolton, there are a number of restaurants and bars. This is a hot spot for the University crowd. Camellia Grill is known for their chili cheese omlet, but they also have the best coconut cream pie I have ever tasted, and is open in to the wee hours of the morning. Cooter Browns is a bar across the street from Camelia Grill. It also serves food. There are also some very good sushi restaurants (Hana Japanese Restaurant, Ninja Sushi) around the corner from there. I like to go to Carrollton Station for their open mike night on Sunday nights. Walk towards the river about four blocks, and you will hit Magazine Street. At the end of Magazine, is the Audubon Zoo. Across from there is Audubon park. All along Magazine Street, there are ecclectic restaurants and antique and specialty shops. My favorite places to imbibe/eat on Magazine include The Bulldog, Mona's Cafe, Theo's Pizza, Taqueria Corona, and Whole Foods Market. The Bulldog has fifty beers on tap, as well as bottled beers, and more bottled. They also serve food. Mona's is Middle Eastern food, Theo's is great Pizza, and Taqueria Corona is authentic Mexican food. If you want more high-end fare, there's Upperline, Lilette, Commanders Palace, Emeril's Delmonico, to name a few. Coquette is a farly new and happening place. Roasted, or grilled oysters are a big thing these days, although most restaurants have been serving Oysters Rockefeller and Oysters Bienville for years. Be sure to get you some oysters, cher! As I stated earlier, some restaurants do not allow shorts or t-shirts, and require jackets for Dinner. The Columns Hotel Bar on St Charles Avenue (Uptown), has a mixture of tourists, locals, and college students, and is a good spot to begin or end an evening.